JP2020199883A - Display control device, head-up display device, method and computer program - Google Patents

Abstract

To suppress the reduction in the visibility of an image while preventing the oversight of the image.SOLUTION: An image display unit displays an image formed of one or a plurality of image elements visually recognized so as to be along the cross direction of the own vehicle in a region overlapped on a foreground when viewed from a driver of the own vehicle, in a case where a first condition consisting of at least one of states that a driver does not visually recognizes a tip end region 240 of an image longer than a prescribed time, the driver gazes at a rear end region 220 of the image and the driver gazes at a region on the upper side with respect to the tip end region 240 is satisfied, changes the angle of the tip end region 240 so as to be raised from the angle along a road surface 330 on which the own vehicle travels and visually recognized.SELECTED DRAWING: Figure 5

Description

The present disclosure relates to a display control device, a head-up display device, a method, and a computer program used in a vehicle to superimpose and visually recognize an image on the foreground of the vehicle.

In Patent Document 1, the driver's attention is directed to the information image that is not seen by the driver (viewer) by increasing the brightness of the information image or thickening the line (increasing the visibility). The image display device to be used is described.

In Patent Document 2, a virtual object as if it actually exists in the foreground (actual view) of the own vehicle is expressed by an image with a sense of perspective, augmented reality (AR) is generated, and the virtual object is virtualized in the foreground. A head that recognizes information while improving the connection between the information shown by the virtual object (image) and the actual scene by blending the object (image) and reducing the visual load on the viewer (driver of the own vehicle). An up-display device is described.

Japanese Unexamined Patent Publication No. 2017-138350 Japanese Unexamined Patent Publication No. 2016-21238

However, when a virtual object (image) is displayed so as to blend into the actual scene, the visibility of the displayed image as in Patent Document 1 is improved, and more specifically, the visibility of the entire image is increased to improve the visibility of the actual scene. The familiarity of the virtual object (image) with respect to is reduced. It is also assumed that this will reduce the ease of grasping the spatial information possessed by the virtual object (image).

A summary of the particular embodiments disclosed herein is presented below. It should be understood that these aspects are presented solely to provide the reader with an overview of these particular embodiments and do not limit the scope of this disclosure. In fact, the present disclosure may include various aspects not described below.

The outline of the present disclosure suppresses a decrease in the visibility of an image while preventing an image from being overlooked, and more specifically, suppresses a decrease in the ease of grasping spatial information possessed by a virtual object (image). Also related to that.

Therefore, in the display control device described in the present specification, the driver does not visually recognize the front end region of the image for longer than a predetermined time, is gazing at the rear end region of the image, and is from the tip region. If the first condition consisting of at least one of the fact that the vehicle is gazing at the upper region is satisfied, the angle of the tip region is adjusted so that the vehicle can be seen from an angle along the road surface on which the vehicle is traveling. change.

It is a figure which shows the application example of the display system for a vehicle which concerns on some embodiments. It is a block diagram of the display system for a vehicle which concerns on some embodiments. It is a figure explaining the difference angle between the line-of-sight direction of a driver, and a specific angle region, and is the figure seen from the top of own vehicle. It is a figure explaining the difference angle between a driver's line-of-sight direction and a specific angle region, and is the figure seen from the left of the own vehicle. It is a perspective image displayed by the image display unit which concerns on some embodiments, and is the figure which shows the display example before changing a part angle. It is a perspective image displayed by the image display unit which concerns on some embodiments, and is the figure which shows the display example after changing a part angle. It is a graph which shows the angle with the road surface for each area of the perspective image shown in FIG. 5, the horizontal axis shows the area of a perspective image, and the vertical axis is the angle of a perspective image virtually set with respect to the road surface [degree]. Is shown. Another example of a graph showing the angle of each area of the perspective image with respect to the road surface according to some embodiments, the horizontal axis indicates the area of the perspective image, and the vertical axis is virtually set with respect to the road surface. The angle [degree] of the perspective image is shown. It is a flow chart of the process which raises the visibility of an image. It is another example of the perspective image displayed by the image display part which concerns on some embodiments, and is the figure which shows the display example before changing a part angle. It is another example of the perspective image displayed by the image display part which concerns on some embodiments, and is the figure which shows the display example after changing a part angle. It is another example of the perspective image displayed by the image display part which concerns on some embodiments, and is the figure which shows the display example after changing a part angle. It is a graph which shows the angle with the road surface for each area of the perspective image shown in FIG. 9, the horizontal axis shows the area of a perspective image, and the vertical axis is the angle of a perspective image virtually set with respect to the road surface [degree]. Is shown. Another example of a graph showing the angle of each area of the perspective image with respect to the road surface according to some embodiments, the horizontal axis indicates the area of the perspective image, and the vertical axis is virtually set with respect to the road surface. The angle [degree] of the perspective image is shown. It is a timing chart from changing a part of the angle of the perspective image to returning to the original angle, the horizontal axis shows the time, and the vertical axis is a part of the perspective image virtually set for the road surface. Indicates the angle [image]. It is a figure which shows the example of the perspective image displayed by the display system for a vehicle which concerns on some embodiments. It is a figure which shows the example of the perspective image displayed by the display system for a vehicle which concerns on some embodiments. It is a figure which shows the example of the perspective image displayed by the display system for a vehicle which concerns on some embodiments. It is a flow chart of the process which changes the angle of a part of a perspective image which concerns on some embodiments.

Hereinafter, FIGS. 1 and 2 provide a description of the configuration of an exemplary vehicle display system. 3A and 3B provide an explanation of an example of gaze determination. 4 to 12C provide a description of a display example. FIG. 13 describes a flow of an exemplary display control process. The present invention is not limited to the following embodiments (including the contents of the drawings). Of course, changes (including deletion of components) can be made to the following embodiments. Further, in the following description, in order to facilitate understanding of the present invention, description of known technical matters will be omitted as appropriate.

See FIG. The vehicle display system 10 includes an image display unit 20 and a display control device 30 that controls the image display unit 20. In the description of the present embodiment, the left-right direction when the driver 4 seated in the driver's seat of the own vehicle 1 faces the front of the own vehicle 1 is the X-axis (the left direction is the X-axis positive direction), and the vertical direction is The Y-axis (upward direction is the Y-axis positive direction) and the front-back direction is the Z-axis (front direction is the Z-axis positive direction).

The image display unit 20 in the vehicle display system 10 is a head-up display (HUD: Head-Up Display) device provided in the dashboard 5 of the own vehicle 1. The HUD device emits the display light 20a toward the front windshield 2 (an example of the projected member) of the own vehicle 1 and emits the display light 20a toward the front side (Z) of the front windshield 2 (an example of the projected member). The image is visually recognized in the display area 100 (in the positive direction of the axis). As a result, the driver 4 can visually recognize an image that overlaps with the foreground 300, which is a real space that is visually recognized through the front windshield 2, in the depth direction (front-back direction).

The display area 100 is an area of a plane, a curved surface, or a partially curved surface in which an image generated inside the HUD device is imaged as a virtual image (visible as an image), and is also called an imaging surface. The display area 100 itself is so low in visibility that it is not actually visible to the driver 4 or is difficult to see. In the display area 100, an angle (tilt angle θt in FIG. 1) formed with the horizontal direction (XZ plane) about the left-right direction (X-axis direction) of the own vehicle 1 is set. In the present embodiment, the display area 100 is described as a surface area (two-dimensional surface or curved two-dimensional surface), but if the image display unit 20 is a 3D display using a known technique, the display area 100 can be a three-dimensional region.

The tilt angle θt is, for example, 90 [degree], and is provided so as to be perpendicular to the horizontal plane (for example, the road surface 330 of the traveling lane in which the own vehicle 1 travels) (display area 110 in FIG. 1). However, the upper part (the part in the positive direction of the Y axis) of the display area 100 seen from the driver 4 is the lower part (Y) so as to be along the horizontal plane (the road surface 330 of the traveling lane on which the own vehicle 1 travels). It may be set at an angle that is farther from the driver 4 than the portion in the negative axis direction (display area 120 in FIG. 1). The tilt angle θt is not limited to these, and is preferably set to 0 to 90 [degree]. Further, the tilt angle θt may be arbitrarily changed by the driver 4, for example, by rotating the relay optical system 24 described later with an actuator (not shown) or the like. The tilt angle θt may be set to 91 to 180 [degree], that is, the upper portion (the portion in the positive direction of the Y axis) of the display area 100 as seen from the driver 4 is the lower portion (Y). The angle may be set closer to the driver 4 than the portion in the negative axis direction).

The image display unit 20 includes a display 22 having a display surface for displaying an image, and a relay optical system 24. The display 22 may be a transmissive display that transmits light from a backlight such as an LCD, or may be a projection type display that projects an image on a screen. In these cases, the display surface is the display surface in the transmissive display and the screen of the projection display.

The relay optical system 24 is arranged on the optical path of the light of the image from the display 22 between the display 22 and the front windshield 2, and the light of the image from the display 22 is transmitted to the front outside the image display unit 20. It is composed of one or a plurality of optical members projected on the windshield 2. The relay optical system 24 includes one concave mirror (an example of a reflection optical member), and in addition to this, for example, one or more refractive optical members such as a lens, a diffraction optical member such as a hologram, a reflection optical member, and the like. Alternatively, a combination thereof may be included.

Further, the image display unit 20 may be a head-mounted display (hereinafter, HMD) device. The driver 4 attaches the HMD device to his / her head and sits on the seat of the own vehicle 1, so that the displayed image is superimposed on the foreground 300 via the front windshield 2 of the own vehicle 1 and visually recognized. .. The display area 100 on which the vehicle display system 10 displays a predetermined image is fixed (adjustably arranged) at a specific position with respect to the coordinate system of the own vehicle 1, and when the driver 4 faces that direction, The image displayed in the display area 100 fixed at the specific position can be visually recognized.

The image display unit 20 exists in the foreground 300, which is a real space (actual view) that is visually recognized through the front windshield 2 of the own vehicle 1 based on the control of the display control device 30, and is an obstacle (pedestrian) described later. , Bicycles, motorcycles, other vehicles, etc.), the road surface 330 of the driving lane described later, road signs, and the vicinity of the real object 310 such as features (buildings, bridges, etc.) (the specific positional relationship between the image and the real object). (Example), at a position overlapping the real object 310 (an example of a specific positional relationship between the image and the real object), or at a position set based on the real object 310 (an example of a specific positional relationship between the image and the real object). By displaying the image, the visual augmented reality (AR) can be perceived by the viewer (typically, the driver 4 seated in the driver's seat of the own vehicle 1).

FIG. 2 is a block diagram of the vehicle display system 10 according to some embodiments. The display control device 30 includes one or more I / O interfaces 31, one or more processors 33, one or more image processing circuits 35, and one or more memories 37. The various functional blocks described in FIG. 2 may consist of hardware, software, or a combination of both. FIG. 2 is only one embodiment, and the illustrated components may be combined with a smaller number of components, or there may be additional components. For example, the image processing circuit 35 (eg, graphic processing unit) may be included in one or more processors 33.

As shown, the processor 33 and the image processing circuit 35 are operably connected to the memory 37. More specifically, the processor 33 and the image processing circuit 35 execute a program stored in the memory 37 to generate and / or transmit image data, for example, and display the vehicle display system 10 (image display). The operation of unit 20) can be performed. The processor 33 and / or the image processing circuit 35 includes at least one general purpose microprocessor (eg, central processing unit (CPU)), at least one application specific integrated circuit (ASIC), and at least one field programmable gate array (FPGA). , Or any combination thereof. The memory 37 includes any type of magnetic medium such as a hard disk, any type of optical medium such as a CD and DVD, any type of semiconductor memory such as a volatile memory, and a non-volatile memory. The volatile memory may include DRAM and SRAM, and the non-volatile memory may include ROM and NVROM.

As shown, the processor 33 and the image processing circuit 35 are operably connected to the memory 37. More specifically, the processor 33 and the image processing circuit 35 execute the program stored in the memory 37 to operate the vehicle display system 10, such as generating and / or transmitting image data. It can be carried out. The processor 33 and / or the image processing circuit 35 includes at least one general purpose microprocessor (eg, central processing unit (CPU)), at least one application specific integrated circuit (ASIC), and at least one field programmable gate array (FPGA). , Or any combination thereof. The memory 37 includes any type of magnetic medium such as a hard disk, any type of optical medium such as a CD and DVD, any type of semiconductor memory such as a volatile memory, and a non-volatile memory. The volatile memory may include DRAM and SRAM, and the non-volatile memory may include ROM and NVROM.

As shown, the processor 33 is operably linked to the I / O interface 31. The I / O interface 31 communicates with, for example, the vehicle ECU 401 described later or other electronic device (reference numerals 403 to 415 described later) provided in the vehicle according to the standard of CAN (Controller Area Network) (also CAN communication). ). The communication standard adopted by the I / O interface 31 is not limited to CAN, and is, for example, CANFD (CAN with Flexible Data Rate), LIN (Local Interconnect Network), Ethernet (registered trademark), MOST (Media Oriented Systems Transport). : MOST is a registered trademark), a wired communication interface such as UART, or USB, or a local such as a personal area network (PAN) such as a Bluetooth (registered trademark) network, or an 802.1x Wi-Fi (registered trademark) network. In-vehicle communication (internal communication) interface, which is a short-range wireless communication interface within several tens of meters such as an area network (LAN), is included. In addition, the I / O interface 31 is a wireless wide area network (WWAN0, IEEE 802.16-2004 (WiMAX: Worldwide Interoperability for Microwave Access)), IEEE 802.16e base (Mobile WiMAX), 4G, 4G-LTE, LTE Advanced, An external communication (external communication) interface such as a wide area communication network (for example, an Internet communication network) may be included according to a cellular communication standard such as 5G.

As shown in the figure, the processor 33 is interoperably connected to the I / O interface 31 to provide information with various other electronic devices and the like connected to the vehicle display system 10 (I / O interface 31). Can be exchanged. The I / O interface 31 includes, for example, a vehicle ECU 401, a road information database 403, a vehicle position detection unit 405, an external sensor 407, a line-of-sight direction detection unit 409, an eye position detection unit 411, a mobile information terminal 413, and an external communication device. 415 and the like are operably connected.

The display 22 is operably connected to the processor 33 and the image processing circuit 35. Therefore, the image displayed by the image display unit 20 may be based on the image data received from the processor 33 and / or the image processing circuit 35. The processor 33 and the image processing circuit 35 control the image displayed by the image display unit 20 based on the information acquired from the I / O interface 31. The I / O interface 31 may include a function of processing (converting, calculating, analyzing) information received from another electronic device or the like connected to the vehicle display system 10.

The vehicle ECU 401 uses sensors and switches provided in the own vehicle 1 to determine the state of the own vehicle 1 (for example, mileage, vehicle speed, accelerator pedal opening, brake pedal opening, engine throttle opening, injector fuel injection amount, engine). Rotation speed, motor rotation speed, steering angle, shift position, drive mode, various warning states, posture (including roll angle and / or pitching angle), vehicle vibration (magnitude, frequency, and / or frequency of vibration) (Including)), etc., and collects and manages (may also include control) the state of the own vehicle 1. As a part of the function, the numerical value of the state of the own vehicle 1 (for example,). A signal indicating the vehicle speed of the own vehicle 1) can be output to the processor 33 of the display control device 30. The vehicle ECU 401 simply transmits the numerical value detected by the sensor or the like (for example, the steering steering angle is 10 [degree] to the right) to the processor 33, or instead, includes the numerical value detected by the sensor. Judgment result based on one or more states of own vehicle 1 (for example, own vehicle 1 is trying to change course to the right) and / and analysis result (for example, driver 4 follows the guidance route). In addition, the intention to turn right at the branch road) may be transmitted to the processor 33. For example, the vehicle ECU 401 may output a signal indicating a determination result indicating that the own vehicle 1 satisfies a predetermined condition stored in advance in a memory (not shown) of the vehicle ECU 401 to the display control device 30.

Further, the vehicle ECU 401 may output an instruction signal indicating an image to be displayed by the vehicle display system 10 to the display control device 30, and at this time, it is necessary to notify the coordinates, size, type, display mode, and image of the image. The degree and / or the necessity-related information that is the basis for determining the notification necessity may be added to the instruction signal and transmitted.

The road information database 403 is included in a navigation device (not shown) provided in the own vehicle 1 or an external server connected to the own vehicle 1 via an external communication interface (I / O interface 31), and the own vehicle will be described later. Based on the position of the own vehicle 1 acquired from the position detection unit 405, the road information (lane, white line, white line,) on which the own vehicle 1 travels, which is the information around the own vehicle 1 (information related to the actual object around the own vehicle 1). Stop line, crosswalk, width of road, number of lanes, intersection, curve, branch road, traffic regulation, etc.), presence / absence of feature information (buildings, bridges, rivers, etc.), position (including distance to own vehicle 1) ), Direction, shape, type, detailed information, etc. may be read and transmitted to the processor 33. Further, the road information database 403 may calculate an appropriate route (navigation information) from the departure point to the destination and output a signal indicating the navigation information to the processor 33.

The own vehicle position detection unit 405 is a GNSS (Global Navigation Satellite System) or the like provided in the own vehicle 1, detects the current position and orientation of the own vehicle 1, and sends a signal indicating the detection result to the processor 33. It is output to the road information database 403, the portable information terminal 413 described later, and / or the external communication device 415 via or directly. The road information database 403, the mobile information terminal 413 described later, and / or the external communication device 415 acquires the position information of the own vehicle 1 from the own vehicle position detection unit 405 continuously, intermittently, or at a predetermined event. Then, the information around the own vehicle 1 may be selected and generated and output to the processor 33.

The vehicle exterior sensor 407 detects a real object 310 existing around the own vehicle 1 (front, side, and rear). The real object 310 detected by the external sensor 407 is, for example, an obstacle (pedestrian, bicycle, motorcycle, other vehicle, etc.), a road surface 330 of a traveling lane described later, a lane marking, a roadside object, and / or a feature (building). Etc.) may be included. Examples of the vehicle exterior sensor include a radar sensor such as a millimeter wave radar, an ultrasonic radar, and a laser radar, and a camera sensor composed of a camera and an image processing device, which may be composed of a combination of both a radar sensor and a camera sensor. It may be composed of only one of them. Conventional well-known methods are applied to object detection by these radar sensors and camera sensors. By detecting an object with these sensors, the presence or absence of a real object in the three-dimensional space, and if the real object exists, the position of the real object (relative distance from the own vehicle 1 and the traveling direction of the own vehicle 1). Horizontal direction (horizontal position, vertical position, etc.), size (horizontal direction (horizontal direction), height direction (vertical direction), etc.), movement direction (horizontal direction (horizontal direction)) , Depth direction (front-back direction)), movement speed (horizontal direction (horizontal direction), depth direction (front-back direction)), and / or type may be detected. One or more external sensors 407 detect a real object in front of the own vehicle 1 for each detection cycle of each sensor, and the real object information (presence or absence of the real object, the real object is an example of the real object information). If it exists, information such as the position, size, and / or type of each real object) can be output to the processor 33. The actual object information may be transmitted to the processor 33 via another device (for example, the vehicle ECU 401). Further, when a camera is used as a sensor, an infrared camera or a near-infrared camera is desirable so that a real object can be detected even when the surroundings are dark such as at night. Further, when a camera is used as a sensor, a stereo camera capable of acquiring a distance or the like by parallax is desirable.

The line-of-sight direction detection unit 409 captures the driver 4's face with, for example, an infrared camera or a visible light camera, analyzes the captured image, identifies the line-of-sight direction GZ of the driver 4, and analyzes the analysis result (line of sight). A signal indicating a gaze position where the direction GZ is stagnant for a while) can be output to the processor 33. The processor 33 may specify the line-of-sight direction GZ of the driver 4 by acquiring an captured image (information capable of estimating the line-of-sight direction) from the line-of-sight direction detection unit 409 and analyzing the captured image. Further, the method of acquiring information capable of estimating the line-of-sight direction of the driver 4 of the own vehicle 1 or the line-of-sight direction of the driver 4 is not limited to these, and is not limited to these, and is limited to the EOG (Electro-occlera) method and the corneal reflex. It may be obtained using other known line-of-sight detection (estimation) techniques such as the method, scleral reflex method, Purkinje image detection method, search coil method, infrared fundus camera method.

The line-of-sight direction detection unit 409 may include an infrared camera or a visible light camera that captures the face of the driver 4 seated in the driver's seat of the own vehicle 1, and may output the captured image to the processor 33. The processor 33 acquires an captured image (an example of information capable of estimating the line-of-sight direction) from the line-of-sight direction detection unit 409, and analyzes the captured image to obtain the line-of-sight direction GZ (and / or the gaze position) of the driver 4. Can be identified. The line-of-sight direction detection unit 409 may analyze the captured image from the camera and output a signal indicating the line-of-sight direction GZ (and / or the gaze position) of the driver 4 as the analysis result to the processor 33. The method for acquiring information capable of estimating the line-of-sight direction GZ of the driver 4 of the own vehicle 1 is not limited to these, and the EOG (Electro-oculogram) method, the corneal reflex method, the scleral reflex method, and the like. It may be obtained using other known line-of-sight detection (estimation) techniques such as Purkinje image detection, search coil, and infrared fundus camera. The processor 33 adjusts the angle of a part of the image displayed by the image display unit 20 with respect to the road surface 330 (ZX horizontal plane) according to the gaze direction (furthermore, the gaze position indicating where the gaze is). This will be described later.

The eye position detection unit 411 may include a camera such as an infrared camera that detects the eye position of the driver 4 seated in the driver's seat of the own vehicle 1, and may output the captured image to the processor 33. The processor 33 acquires an captured image (an example of information capable of estimating the eye position) from the eye position detection unit 411, and can identify the eye position of the driver 4 by analyzing the captured image. The eye position detection unit 411 may analyze the image captured by the camera and output a signal indicating the position of the driver 4's eyes, which is the analysis result, to the processor 33. The method of acquiring the information capable of estimating the eye position of the driver 4 of the own vehicle 1 or the eye position of the driver 4 is not limited to these, and the known eye position detection (estimation) is used. It may be obtained using technology. The processor 33 adjusts at least the position of the image based on the position of the eyes of the driver 4, so that the image superimposed on the desired position of the foreground 300 (the position having a specific positional relationship with the real object) is displayed on the eyes. The visual operator (driver 4) who has detected the position may visually recognize the position.

The personal digital assistant 413 is a smartphone, a laptop computer, a smart watch, or other information device that can be carried by the driver 4 (or another occupant of the own vehicle 1). The I / O interface 31 can communicate with the mobile information terminal 413 by pairing with the mobile information terminal 413, and the data recorded in the mobile information terminal 413 (or the server through the mobile information terminal). To get. The mobile information terminal 413 has, for example, the same functions as the above-mentioned road information database 403 and own vehicle position detection unit 405, acquires the road information (an example of real object-related information), and transmits it to the processor 33. May be good. Further, the mobile information terminal 413 may acquire commercial information (an example of information related to a real object) related to a commercial facility in the vicinity of the own vehicle 1 and transmit it to the processor 33. The mobile information terminal 413 transmits the schedule information of the owner (for example, the driver 4) of the mobile information terminal 413, the incoming information on the mobile information terminal 413, the received information of the mail, etc. to the processor 33, and the processor 33 and The image processor 35 may generate and / or transmit image data relating to these.

The external communication device 415 is a communication device that exchanges information with the own vehicle 1, for example, another vehicle connected to the own vehicle 1 by vehicle-to-vehicle communication (V2V: Vehicle To Vehicle), and pedestrian-to-vehicle communication (V2P: Vehicle). It is a network communication device connected by pedestrians (portable information terminals carried by pedestrians) connected by To Pedestrian) and by road-to-vehicle communication (V2I: Vehicle To Roadside Infrastructure). In a broad sense, it is a network communication device with the own vehicle 1. Includes everything connected by communication (V2X: Vehicle To Everything). The external communication device 415 includes, for example, a pedestrian, a bicycle, a motorcycle, another vehicle (preceding vehicle, etc.), a road surface (reference numeral 330 in FIG. 4), a lane marking (reference numeral 331, 332 in FIG. 4), a roadside object, and /. Alternatively, the position of a feature (building, etc.) may be acquired and output to the processor 33. Further, the external communication device 415 has the same function as the above-mentioned own vehicle position detection unit 405, and may acquire the position information of the own vehicle 1 and transmit it to the processor 33, and further, the above-mentioned road information database 403. The road information (an example of real object-related information) may be acquired and transmitted to the processor 33. The information acquired from the external communication device 415 is not limited to the above.

The software components stored in the memory 37 are the actual object information detection module 502, the actual object position identification module 504, the notification necessity determination module 506, the eye position detection module 508, the image type setting module 510, the image arrangement setting module 512, and the like. It includes an image size setting module 514, a line-of-sight direction determination module 516, a graphic module 518, and an action determination module 520.

The real object information detection module 502 acquires information (also referred to as real object information) including at least the position of the real object 310 existing in front of the own vehicle 1. The real object information detection module 502 is, for example, from the vehicle exterior sensor 407, the position of the real object 310 existing in the foreground 300 of the own vehicle 1 (from the driver 4 in the driver's seat of the own vehicle 1 to the traveling direction of the own vehicle 1 (forward). ) Is the position in the height direction (vertical direction), the horizontal direction (horizontal direction), and the position in the depth direction (forward direction) may be added to these positions), and the position of the real object 310. Information (an example of real object information) including the size (size in the height direction and the size in the lateral direction) and the relative speed with respect to the own vehicle 1 (including the relative moving direction) may be acquired. Further, the real object information detection module 502 includes the position, relative speed, type of the real object (for example, another vehicle), the lighting state of the direction indicator of the other vehicle, the steering angle operation state, and the steering angle operation state via the external communication device 415. / Or, information (an example of information related to a real object) indicating a planned progress route and a progress schedule by the driving support system may be acquired.

Further, the real object information detection module 502 is determined by the position of the lane marking 331 (see FIG. 4) on the left side of the road surface 330 (see FIG. 4) of the traveling lane of the own vehicle 1 and the lane marking 332 (see FIG. The position of (see FIG. 4) may be acquired, and the region (road surface 330 of the traveling lane) between the left and right lane markings 331 and 332 may be recognized.

Further, the real object information detection module 502 is information about a real object existing in the foreground 300 of the own vehicle 1 which is a source for determining the content of the perspective image 200 (hereinafter, also appropriately referred to as “image type”) described later. (Real object related information) may be detected. The real object-related information includes, for example, type information indicating the type of the real object such as the real object being a pedestrian or another vehicle, moving direction information indicating the moving direction of the real object, distance to the real object, and the like. Distance time information indicating the arrival time, or individual detailed information of the real object such as the charge of the parking lot (real object) (but not limited to these). For example, the real object information detection module 502 acquires type information, distance / time information, and / or individual detailed information from the road information database 403 or the mobile information terminal 413, and the type information, movement direction information, and / or from the vehicle exterior sensor 407. Alternatively, the distance / time information may be acquired and / or the type information, the moving direction information, the distance / time information, and / or the individual detailed information may be detected from the external communication device 415.

The real object position identification module 504 acquires the observation position indicating the current position of the real object 310 from the vehicle exterior sensor 407 or the external communication device 415 via the I / O interface 31, or obtains two or more observation positions thereof. The observation position of the fused real object is acquired, and the position (also referred to as a specific position) of the real object 310 is set based on the acquired observation position. The image arrangement setting module 512, which will be described later, determines the position of the image based on the specific position of the real object 310 set by the real object position specifying module 504.

The real object position specifying module 504 may specify the position of the real object 310 based on the observed position of the real object 310 acquired immediately before, but is not limited to this, and at least observes the real object 310 acquired immediately before. The position of the real object 310 may be specified (estimated) based on the predicted position of the real object at a predetermined time predicted based on the observed position of one or more real objects 310 acquired in the past including the position. .. That is, by executing the real object position specifying module 504 and the image arrangement setting module 512 described later, the processor 33 can specify the position of the perspective image 200 based on the observed position of the real object 310 acquired immediately before. Based on the predicted position of the real object 310 at the display update timing of the perspective image 200 predicted based on the observed position of one or more real objects 310 acquired in the past including the observed position of the real object 310 acquired at least immediately before. The position of the perspective image 200 can be set. There are no particular restrictions on the method of calculating the predicted position by the real object position specifying module 504, and the prediction is performed based on the observation position acquired in the past from the display update timing targeted by the real object position specifying module 504. Any method may be used as long as it is used. The real object positioning module 504 uses, for example, a least squares method, a prediction algorithm such as a Kalman filter, an α-β filter, or a particle filter, and uses one or more observation positions in the past to determine the next value. You may try to predict. The vehicle display system 10 only needs to be able to acquire the observed position and / or the predicted position of the real object, and does not have to have the function of setting (calculating) the predicted position of the real object. A part or all of the function of setting (calculating) the predicted position may be provided separately from the display control device 30 of the vehicle display system 10 (for example, the vehicle ECU 401).

The notification necessity determination module 506 determines whether each perspective image 200 displayed by the vehicle display system 10 is the content to be notified to the driver 4. The notification necessity determination module 506 may acquire information from various other electronic devices connected to the I / O interface 31 and calculate the notification necessity. Further, the electronic device connected to the I / O interface 31 in FIG. 2 transmits information to the vehicle ECU 401, and the notification necessity determination module 506 detects (acquires) the notification necessity determined by the vehicle ECU 401 based on the received information. ) May. The "necessity of notification" is, for example, the degree of danger derived from the degree of seriousness that can occur, the degree of urgency derived from the length of reaction time required to take a reaction action, the own vehicle 1 or the driver 4 ( Alternatively, it can be determined by the effectiveness derived from the situation of the own vehicle 1 (other occupants of the own vehicle 1), or a combination thereof (the index of the necessity of notification is not limited to these). The notification necessity determination module 506 may detect the necessity-related information that is the basis for estimating the notification necessity of the image 200, and may estimate the notification necessity from this. The necessity-related information that is the basis for estimating the notification necessity of an image may be estimated by, for example, the position and type of a real object or traffic regulation (an example of road information), and is connected to the I / O interface 31. It may be estimated based on other information input from various other electronic devices or in addition to other information. That is, the notification necessity determination module 506 may determine whether to notify the driver 4 and may choose not to display the image described later. The vehicle display system 10 only needs to be able to acquire the notification necessity, and does not have to have a function of estimating (calculating) the notification necessity, and some or all of the functions for estimating the notification necessity are , It may be provided separately from the display control device 30 of the vehicle display system 10 (for example, the vehicle ECU 401).

The eye position detection module 508 detects the position of the eyes of the driver 4 of the own vehicle 1. The eye position detection module 508 determines where the eye height of the driver 4 is located in the height region provided in a plurality of stages, detects the eye height of the driver 4 (position in the Y-axis direction), and Detection of driver 4 eye height (Y-axis position) and depth position (Z-axis position) and / or driver 4 eye position (X, Y, Z-axis position) ) Includes various software components for performing various actions related to detection. The eye position detection module 508 can acquire, for example, the eye position of the driver 4 from the eye position detection unit 411, or can estimate the eye position including the eye height of the driver 4 from the eye position detection unit 411. Information is received and the eye position including the eye height of the driver 4 is estimated. The information that can estimate the position of the eyes is, for example, the position of the driver's seat of the own vehicle 1, the position of the face of the driver 4, the height of the sitting height, the input value by the driver 4 at the operation unit (not shown), and the like. May be good.

The image type setting module 510 needs to notify, for example, the type and position of the real object detected by the real object information detection module 502, and the type, number, and / or notification of the real object-related information detected by the real object information detection module 502. The type of image to be displayed for the real object can be determined based on the magnitude of the (estimated) notification necessity detected by the degree determination module 506. Further, the image type setting module 510 may increase or decrease the types of images to be displayed depending on the determination result by the line-of-sight direction determination module 516 described later. Specifically, when the real object 310 is difficult to be visually recognized by the driver 4, the number of types of images visually recognized by the driver 4 may be increased in the vicinity of the real object.

The image arrangement setting module 512 is a position (observation position) of the real object 310 specified by the real object position specifying module 504 so that the perspective image 200 is visually recognized in a specific positional relationship with the real object 310 (road surface 330). Or, based on the predicted position), the coordinates of the perspective image 200 (horizontal direction (X-axis direction) and vertical direction (Y-axis direction) when the driver 4 sees the direction of the display area 100 from the driver's seat of the own vehicle 1). Includes at least). In addition to this, the image arrangement setting module 512 is when the driver 4 looks at the direction of the display area 100 from the driver's seat of the own vehicle 1 based on the determined position of the real object 310 set by the real object position specifying module 504. The front-back direction (Z-axis direction) may be determined. The image arrangement setting module 512 adjusts the position of the perspective image 200 based on the position of the driver 4's eyes detected by the eye position detection unit 411. Specifically, the image arrangement setting module 512 determines the positions of the perspective image 200 in the horizontal direction and the vertical direction so that the perspective image 200 can be visually recognized in the area (road surface 330) between the division lines 331 and 332. To do.

Further, the image arrangement setting module 512 can set the angle of the perspective image 200 (pitching angle about the X direction, yaw rate angle about the Y direction, rolling angle about the Z direction). The angle of the perspective image 200 is a preset angle, and can be set to be parallel to the front-rear and left-right directions (XY planes) of the own vehicle 1.

The image size setting module 514 may change the size of the perspective image 200 according to the position, shape, and / or size of the corresponding real object 310. For example, the image size setting module 514 can reduce the size of the perspective image 200 if the position of the associated real object 310 is far away. Further, the image size setting module 514 can increase the size of the perspective image 200 if the size of the associated real object 310 is large.

Further, the image size setting module 514 can determine the size of the perspective image 200 based on the magnitude of the (estimated) notification necessity detected by the notification necessity determination module 506.

The image size setting module 514 may have a function of predicting and calculating the size of the perspective image 200 to be displayed in the current display update cycle based on the size of the real object 310 a predetermined number of times in the past. As a first method, the image sizing module 514 tracks the pixels of the real object 310 between two past images captured by a camera (an example of an outside sensor 407), using, for example, the Lucas-Kanade method. Therefore, the size of the real object 310 in the current display update cycle may be predicted, and the size of the perspective image 200 may be determined according to the predicted size of the real object 310. As a second method, the rate of change in the size of the real object 310 is obtained based on the change in the size of the real object 310 between the two past captured images, and the perspective image 200 is obtained according to the rate of change in the size of the real object 310. You may decide the size of. The method of estimating the size change of the real object 310 from the viewpoint that changes in time series is not limited to the above, and includes, for example, an optical flow estimation algorithm such as the Horn-Schunkk method, Buxton-Buxton, and Black-Jepson method. A known method may be used.

The line-of-sight direction determination module 516 is used to perform various actions related to the determination that the driver 4 of the own vehicle 1 is looking at a specific area and / or is not looking at a specific area. Includes software components. Specifically, the line-of-sight direction determination module 516 determines whether or not the driver 4 of the own vehicle 1 is looking at the tip region 240 (see FIG. 4) of the perspective image 200 described later (seeing a specific region). (Example of determination of that), determination of whether or not the line-of-sight direction is far from a predetermined line-of-sight threshold from the tip region 240 of the perspective image 200 described later (an example of determination of not seeing a specific region), self. Determining whether the driver 4 of the vehicle 1 is looking at the front end region 240 or the rear end region 220 (see FIG. 4) of the perspective image 200 described later (an example of determining that the driver 4 is looking at a specific region). , And / or determination of whether or not the driver 4 of the own vehicle 1 is looking at a side farther from the tip region 240 of the perspective image 200 described later (an example of determination of determining that the driver 4 is looking at a specific region). Includes various software components for performing various operations.

An example of a method of determining whether the driver 4 is looking at (or gazing at) a specific direction (also referred to as a specific angle region E) will be described. The line-of-sight direction determination module 516 determines whether or not the specific angle region E is being viewed based on the line-of-sight direction GZ detected by the line-of-sight direction detection unit 409. The line-of-sight direction determination module 516 calculates the difference angle Δθ between the line-of-sight direction GZ and the specific angle region E (for example, the direction in which the perspective image 200 described later is displayed), and the difference angle Δθ is less than a predetermined difference angle threshold value. Is determined. When the difference angle Δθ is not less than the difference angle threshold value, it is determined that the person is not gazing in a specific direction, and the gaze determination counter is cleared to an initial value (for example, 0). The gaze determination counter is a counter for measuring a period in which it is determined that the line-of-sight direction is directed to a specific direction, and is stored in the memory 37 during the operation of the processor 33.

On the other hand, when the difference angle Δθ is less than the difference angle threshold value, the line-of-sight direction determination module 516 increments the gaze determination counter. That is, the line-of-sight direction determination module 516 estimates that the driver 4 has started the gaze action, and starts incrementing the gaze determination counter. The line-of-sight direction determination module 516 determines whether or not the value of the gaze determination counter exceeds a predetermined gaze determination threshold value, and as a result, when the gaze determination counter exceeds the gaze determination threshold value, gaze determination, That is, it is determined that the driver 4 is gazing at the specific angle region E (for example, the direction in which the perspective image 200 described later is displayed).

3A and 3B are diagrams for explaining the difference angle Δθ between the line-of-sight direction GZ of the driver 4 and the specific angle region E on which the perspective image 200 is displayed. In FIG. 3A, the difference between the horizontal line-of-sight direction GZx of the driver 4 and the specific angle region Ex of the perspective image 200 displayed in the display area 110 along the vertical and horizontal directions (XY planes) seen from the driver 4. The angle is expressed as Δθx. In FIG. 3B, the difference between the vertical line-of-sight direction GZy of the driver 4 and the specific angle region Ey of the perspective image 200 displayed in the display area 110 along the vertical and horizontal directions (XY planes) seen from the driver 4. The angle is expressed as Δθy.

The line-of-sight direction determination module 516 includes the position information of the image 200 input from the image arrangement setting module 512, the size information of the image 200 input from the image size setting module 514, and the driver 4 input from the eye position detection module 508. From the eye position information, a specific angle region E (Ex, Ey) from the current eye position of the driver 4 toward the perspective image 200 is set. Next, the line-of-sight direction determination module 516 determines the difference angle Δθ (Δθx, Δθy) from the set specific angle region E (Ex, Ey) and the line-of-sight direction GZ (GZx, GZy) input from the line-of-sight direction detection unit 409. Is calculated, and it is determined whether the difference angles Δθx and Δθy are less than the difference angle thresholds provided for comparison. If both the difference angles Δθx and Δθy are less than the difference angle thresholds, a specific angle is specified. It is determined that the line of sight is directed to the area E. The difference angle threshold that is the comparison target of the difference angle Δθx and / or the difference angle threshold that is the comparison target of the difference angle Δθy may be zero, that is, the line-of-sight direction GZ is in the specific angle region E. Therefore, it is not necessary to determine that the specific angle region E is being watched. The line-of-sight direction determination module 516 transmits to the processor 33 a determination result of whether or not the driver 4 is gazing at the specific angle region E. The gaze determination method is not limited to these means. Further, the display control device 30 (processor 33) only needs to be able to acquire the area seen by the driver 4 of the own vehicle 1 (the area being watched), and the area seen from the line-of-sight direction GZ (the area being watched). ) Does not have to be determined (calculated), and some or all of these functions are separate from the display control device 30 (processor 33) of the vehicle display system 10 (for example, the vehicle ECU 401). ) May be provided. Furthermore, in the display control device 30 (processor 33), the driver 4 of the own vehicle 1 is looking at a specific area, is not looking at a specific area for a longer time than a predetermined time, and the like. It suffices if the determination result can be obtained, and it is not necessary to have the function of making the above determination from the gaze direction of the driver 4, and the driver 4 of the own vehicle 1 is looking at a specific area or a specific area. It is not necessary to have a function of determining that the vehicle is not being viewed, and some or all of these functions are separate from the display control device 30 (processor 33) of the vehicle display system 10 (for example, It may be provided in the vehicle ECU 401).

Further, the line-of-sight direction determination module 516 may detect what the driver 4 is viewing other than the perspective image 200. For example, the line-of-sight direction determination module 516 includes the position of the real object 310 existing in the foreground 300 of the own vehicle 1 detected by the real object information detection module 502, and the line-of-sight direction GZ of the driver 4 acquired from the line-of-sight direction detection unit 409. , The real object 310 to be watched may be specified, and the information for identifying the real object 310 to be visually recognized may be transmitted to the processor 33.

The graphic module 518 includes various known software components for performing image processing such as rendering to generate image data and driving the display 22. The graphic module 518 also provides the type, arrangement (position coordinates, angle), size, display distance (in the case of 3D), visual effect (eg, brightness, transparency, saturation, contrast, or contrast) of the displayed image. Other visual characteristics), may include various known software components for modification. The graphic module 518 has the type set by the image type setting module 510 and the position coordinates set by the image arrangement setting module 512 (the left-right direction when the driver 4 sees the direction of the display area 100 from the driver's seat of the own vehicle 1 (left-right direction). Position coordinates including at least the position coordinates in the vertical direction (Y-axis direction) and the vertical direction (Y-axis direction).) Angles set by the image arrangement setting module 512 (pitching angle centered on the X direction, yaw rate angle centered on the Y direction) , Rolling angle about the Z direction), and image data is generated so that the driver 4 can see the size set by the image size setting module 514, and the image data is displayed on the image display unit 20.

The graphic module 518 displays a perspective image 200 composed of one or a plurality of image elements visually recognized along the front-rear direction of the own vehicle 1. The perspective image 200 is arranged so as to have a predetermined positional relationship with the real object 310 (analog) of the foreground 300, and in this case, it is also called a (Kontaktanalogue) image. In the description of the present embodiment, the perspective image 200 will be described as a (Kontakatanalog) image, but the perspective image 200 may be drawn using perspective and does not necessarily have a predetermined positional relationship with the real object 310 in the foreground 300. It may be a non-(Kontaktanalogue) image in which the display position is fixed (or adjustable semi-fixed) regardless of the real object 310.

FIG. 4 is a diagram showing an example in which the perspective image 200 is displayed. The perspective image 200 of FIG. 4 is composed of six (plural) arrow-shaped images (also referred to as route images) showing the guide route, and is located at a position overlapping the road surface 330 in the foreground 300 of the own vehicle 1 among the guide routes. The reference numerals 201, 202, 203, 204, 205, and 206 are arranged in this order from the side closest to the own vehicle 1. The guidance route shows a right turn at an intersection after going straight, that is, perspective images 201, 202, and 203 showing the guidance route to the intersection overlap the road surface 330 of the traveling lane of the own vehicle 1 as seen from the driver 4. Perspective images 204, 205, and 206, which indicate the straight direction (Z-axis positive direction) toward the intersection ahead and show the guidance route from the intersection, are the road surfaces of the branch road in the right turn direction when viewed from the driver 4. Instruct the right direction (X-axis negative direction) so that it overlaps 330. The perspective images 201 to 206 are arranged so as to be visually recognized along the road surface 330, that is, the image arrangement setting module 512 has an angle between the virtual object shown by the perspective images 201 to 206 and the road surface 330. The arrangement (angle) is set so as to be 0 [image] (in other words, parallel to the road surface 330).

The graphic module 518 of the present embodiment changes the angle of the perspective image 200 with respect to a part of the road surface 330 according to the determination result by the line-of-sight direction determination module 516. FIG. 5 shows a display example after changing the angle of a part of the perspective image 200 of FIG. 4 (tip region 240 (204, 205, 206)) with respect to the road surface 330. When it is determined by the line-of-sight direction determination module 516 that the driver 4 has not seen the tip region 240 (an example of the specific angle region E) of the perspective image 200 for a while, the graphic module 518 determines the tip region 240 (204) of the perspective image 200. , 205, 206) to get up toward the driver 4 (in other words, the pitching angle of the tip region 240 (204, 205, 206) about the X direction with respect to the road surface 330 is set from 0 [degree] to 90. Change to [degree]).

Further, in some embodiments, when the line-of-sight direction determination module 516 determines that the driver 4 is gazing at the rear end region 220 (an example of the specific angle region E) of the perspective image 200, the graphic module 518 , The tip region 240 (204, 205, 206) of the perspective image 200 may be raised toward the driver 4 (in other words, the pitching angle of the tip region 240 about the X direction with respect to the road surface 330). It may be changed from 0 [degree] to 90 [degree]).

Further, in some embodiments, the driver 4 is gazing at a region (an example of a specific angle region E) above the tip region 240 of the perspective image 200 (in the positive direction of the Y axis) by the line-of-sight direction determination module 516. If it is determined that, the graphic module 518 may raise the tip region 240 (204, 205, 206) of the perspective image 200 toward the driver 4 (in other words, the X direction with respect to the road surface 330 as a reference). The pitching angle of the tip region 240 as the axis may be changed from 0 [degree] to 90 [degree]).

FIG. 6A is a diagram showing arrangements (angles) set in each region of the perspective image of FIG. 5, where the horizontal axis represents each region 201 to 206 of the perspective image and the vertical axis is virtual with respect to the road surface 330. Indicates the pitching angle [image] set in. 0 [degree] is set in the perspective images 201, 202, 203 of the rear end region 220, and 90 [degree] is set in the perspective images 204, 205, 206 of the front end region 240.

FIG. 6B is a diagram showing another example of the arrangement (angle) set in each area of the perspective image 200. The graphic module 518 may be angled so that when the tip region 240 (204, 205, 206) of the perspective image 200 is raised toward the driver 4, the tip region 240 (204, 205, 206) of the perspective image 200 is gradually raised toward the tip of the perspective image 200. (In other words, the pitching angle of the tip region 240 about the X direction with respect to the road surface 330 may be changed so as to gradually approach 90 [graphic] toward the 0 [degree] tip.) Specifically, 0 [degree] is set in the perspective images 201, 202, 203 of the rear end region 220, 30 [degree] is set in the perspective image 204 of the front end region 240, and 60 [degree] is set in the perspective image 205. The perspective image 206 may have 90 [degree], that is, the angle between the state-of-the-art image element (reference numeral 206) and a plurality of image elements (reference numerals 205, 204) connected to the state-of-the-art image element may be different. The degree of change in the angle may be set larger toward the cutting edge.

In some embodiments, the perspective image 200 may be composed of one image element. FIG. 7 shows a display example before the perspective image 200 composed of one image element is deformed according to the gaze determination. Further, FIG. 8 shows a display example after the perspective image 200 of FIG. 7 is deformed according to the gaze determination. In the example of FIG. 8, the angle of the tip region 240 of the perspective image 200 is set so as to rise suddenly, but the angle is not limited to this. Therefore, the graphic module 518 may set the angle so that when the tip region 240 of the perspective image 200 is raised toward the driver 4, the tip region 240 is gradually raised toward the leading edge of the perspective image 200 (in other words,). The pitching angle of the tip region 240 about the X direction with respect to the road surface 330 may be changed so as to gradually approach 90 [graphic] from 0 [degree] toward the cutting edge.)

Further, in some embodiments, the graphic module 518 may change the angle of the rear end region 220 of the perspective image 200 with respect to the road surface 330 depending on the determination result by the line-of-sight direction determination module 516. FIG. 9 shows a display example after changing the angle (rolling angle) of a part of the perspective image 200 of FIG. 4 (rear end region 220 (201, 202, 203)) with respect to the road surface 330. When the line-of-sight direction determination module 516 determines that the driver 4 has not seen the rear end region 220 (an example of the specific angle region E) of the perspective image 200 for a while, or the line-of-sight direction determination module 516 determines that the driver 4 has not seen the perspective image. When it is determined that the tip region 240 (an example of the specific angle region E) of the 200 is being watched, the graphic module 518 refers to the rear end region 220 (201, 202, 203) of the perspective image 200 with reference to the road surface 330. The roll angle of the rear end region 220 about the Z direction may be changed from 0 [image] to 90 [graphic]. It should be noted that the virtual object represented by the perspective image 200, which is visually recognized as being lined up (extending) in the depth direction (Z direction) when viewed from the driver 4, is raised at a large angle with respect to the road surface 330. When visually recognizing, the position of the raised perspective image 200 may be shifted to the left or right from the center of the traveling lane of the own vehicle 1.

10A is a diagram showing arrangements (angles) set in each region of the perspective image of FIG. 9, the horizontal axis represents each region 201 to 206 of the perspective image, and the vertical axis is virtual with respect to the road surface 330. Indicates the rolling angle [image] set to. 90 [degree] is set in the perspective images 201, 202, 203 of the rear end region 220, and 0 [degree] is set in the perspective images 204, 205, 206 of the front end region 240.

FIG. 10B is a diagram showing another example of the arrangement (angle) set in each area of the perspective image 200. The image arrangement setting module 512 sets the rolling angle so that when the rear end region 220 (201, 202, 203) of the perspective image 200 is raised from the road surface 330, it gradually rises toward the rearmost end of the perspective image 200. (In other words, the rolling angle of the rear end 230 about the Z direction with respect to the road surface 330 may be changed so as to gradually approach 90 [image] from 0 [image] toward the rearmost end. ). Specifically, 0 [degree] is set in the perspective images 204, 205, 206 of the front end region 240, 30 [degree] is set in the perspective image 203 of the rear end region 220, and 60 [degree] is set in the perspective image 202. 90 [degree] is set for the perspective image 201 at the rearmost end.

FIG. 11 is a timing chart from changing a part of the angle of the perspective image 200 to returning to the original angle, the horizontal axis shows the time, and the vertical axis is virtually set with respect to the road surface 330. The rolling angle [degree] of a part of the perspective image 200 is shown. Based on the determination result of the line-of-sight direction determination module 516, at the time t11 when a specific condition is satisfied, the graphic module 518 is a part of the perspective image 200 virtually set for the road surface 330 (rear end region 220, Alternatively, the angle of the tip region 240) is increased, the vehicle is raised from the road surface 330, and the perspective image 200 is changed gradually (for example, over 30 seconds) from the time t12 after a predetermined time (for example, 10 seconds) has elapsed. The angle of the portion (rear end region 220 or tip region 240) is restored (time t13). By raising a part of the perspective image 200 that is not being watched from an angle along the road surface 330, the visual attention of the driver 4 can be quickly directed to the perspective image 200.

Further, according to the above embodiment, since the other part of the perspective image 200 connected to a part of the perspective image 200 whose angle has changed continues to be displayed at an angle along the road surface 330, the image (other than the perspective image 200). Part) can easily blend into the foreground 300, and as a result, the discomfort of the image in the virtual reality can be reduced and the visual attention can be made excessively difficult to direct.

Further, according to the above embodiment, the time taken to return a part of the perspective image 200 not being watched to the angle along the road surface 330 (t12 to t13 in FIG. 11, also referred to as a change holding time) is defined. The visual attention of the driver 4 by making a part of the non-gaze perspective image 200 longer than the time it takes to wake up from an angle along the road surface 330, i.e. slowly returning it to an angle along the road surface 330. Can be returned to the original state in which the perspective image 200 is easily blended into the foreground 300 without being excessively taken.

The specific type of the perspective image 200 is not limited, and in addition to the above-mentioned image showing the guide route, which consists of one or a plurality of image elements visually recognized along the front-rear direction of the own vehicle 1, for example, Lane guide image displayed along the lane markings 331 and 332 (reference numeral 200 in FIG. 12A), lane departure warning (LDW, Lane) displayed along the lane marking when the vehicle is about to deviate from the lane markings 331 and 332. The image relating to Departure Warning) (similar to or similar to reference numeral 200 in FIG. 12A) and the inter-vehicle distance set when following another vehicle 350 traveling in front of the same traveling lane as the own vehicle 1 are superimposed on the road surface 330. It is predicted that the image related to the constant distance control (ACC, Adaptive Cruise Control) (reference numeral 200 in FIG. 12B) or another vehicle 370 traveling in the adjacent traveling lane will invade the traveling lane of the own vehicle 1. In this case, it may be an image (reference numeral 200 in FIG. 12C) showing the predicted intrusion route of the other vehicle 370.

See FIG. 2 again. The action determination module 520 detects whether or not the driver 4 has taken an appropriate action with respect to the information shown by the perspective image 200. Each of the perspective images 200 is associated with the accurate action of the driver 4 and stored in the memory 37. In particular, after changing the angle of the perspective image 200 with respect to a part of the road surface 330, the action determination module 520 determines whether or not an accurate action associated with the perspective image 200 is detected. For example
When the perspective image 200 includes the guide route information, the driver 4 gazes at the same direction as the guide route indicates, or the steering is performed to move in the same direction as the guide route indicates. As a condition, it may be determined that an accurate action associated with the perspective image 200 has been detected. Further, when the perspective image 200 is an image related to the lane deviation warning, it is determined that an accurate action associated with the perspective image 200 is detected on condition that the steering is performed so as to avoid the lane deviation. You may. The information collected by the action determination module 520 for determining the action is the state of the own vehicle 1 (for example, mileage, vehicle speed, accelerator pedal opening, engine throttle opening, injector fuel injection amount, etc., which is input from the vehicle ECU 401. The engine speed, motor speed, steering angle, shift position, drive mode, various warning states), the state of the driver 4 input from the line-of-sight direction detection unit 409 (for example, the line-of-sight direction), etc., but are limited to these. Not done. After changing the angle of the perspective image 200 with respect to a part of the road surface 330, the graphic module 518 determines that an appropriate action has been taken by the action determination module 520, even if the change holding time does not elapse. A part of the angle of 200 may be changed back to the angle along the road surface 330. In this case, the speed at which the angle is returned is faster than the speed at which the angle of the perspective image 200 is returned after the change holding time has elapsed, provided that the action determination module 520 determines that an accurate action has been taken. May be good. As a result, when the driver 4 makes an accurate operation, by returning a part of the perspective image 200 to an angle along the road surface 330, the entire perspective image 200 can be easily blended into the foreground 300 again, and as a result. , The discomfort of the image in the virtual reality is reduced, and the visual attention can be made excessively difficult to direct.

FIG. 13 is a flow chart of a process for changing the angle of a part (front end region 240 or rear end region 220) of the perspective image 200 according to some embodiments.
First, one or more processors 33 acquire the line-of-sight direction GZ of the driver 4 of the own vehicle 1 from the line-of-sight direction detection unit 409 (step S11), and acquire the position of the currently displayed image from the memory 37. (Step S12).

Next, the processor 33 compares the line-of-sight direction GZ (or the gaze direction estimated from the line-of-sight direction GZ) acquired in step S11 with the position of the image acquired in step S12, and changes the angle of the tip region 240. It is determined whether or not the condition to be performed is satisfied (step S13). Specifically, the processor 33 executes the line-of-sight direction determination module 516, and the driver 4 of the own vehicle 1 has not seen the front end region 240 for a while, is gazing at the rear end region 220, or When it is determined that the driver is gazing at the upper side (Y-axis positive direction) of the tip region 240 (Yes in step S13), the angle of the tip region 240 is raised from the angle along the road surface 330 and visually recognized. (Step S14), and the process proceeds to step S17.

On the other hand, when the condition for changing the angle of the tip region 240 is not satisfied (No in step S13), the processor 33 sets the line-of-sight direction GZ (or the gaze direction estimated from the line-of-sight direction GZ) acquired in step S11 and the step. By comparing with the position of the image acquired in S12, it is determined whether or not the condition for changing the angle of the rear end region 220 is satisfied (step S15). Specifically, the processor 33 executes the line-of-sight direction determination module 516, and the driver 4 of the own vehicle 1 has not visually recognized the rear end region 220 for a while, or is gazing at the front end region 240. When is determined (Yes in step S15), the angle of the rear end region 220 is changed so that it can be visually recognized by rising from an angle along the road surface 330 (step S16), and the process proceeds to step S17. If the condition for changing the angle of the rear end region 220 is not satisfied (No in step S15), the process is terminated and the process is started again from step S11.

In step S17, the processor 33 determines whether or not the condition for restoring the angle of the front end region 240 or the rear end region 220 is satisfied. Specifically, the processor 33 timers the elapsed time after changing the angle of the front end region 240 in step S14 or the elapsed time after changing the angle of the rear end region 220 in step S16. When the measurement is performed in (not shown) and a predetermined time has elapsed (Yes in step S17), the angle of the front end region 240 or the rear end region 220 is returned to the angle along the road surface 330 (step S18), and the process is completed. , The process is started again from step S11. Further, when the processor 33 executes the action determination module 520 and detects that the driver 4 has taken an appropriate action with respect to the information indicated by the perspective image 200 (Yes in step S17), the tip region 240 or later. The angle of the end region 220 may be returned to an angle along the road surface 330 (step S18), the process may be terminated, and the process may be started again from step S11.

The operation of the processing process described above can be carried out by executing one or more functional modules of an information processing device such as a general-purpose processor or a chip for a specific purpose. These modules, combinations of these modules, and / or combinations with common hardware that can replace their functionality are all within the scope of the protection of the present invention.

The functional blocks of the vehicle display system 10 are optionally performed by hardware, software, or a combination of hardware and software to implement the principles of the various embodiments described. The functional blocks described in FIG. 2 may be optionally combined or one functional block separated into two or more subblocks in order to implement the principles of the embodiments described. It will be understood by those skilled in the art. Accordingly, the description herein optionally supports any possible combination or division of functional blocks described herein.

1: Own vehicle 2: Front windshield 4: Driver 5: Dashboard 10: Vehicle display system 20: Image display unit 20a: Display light 22: Display 24: Relay optical system 30: Display control device 31: I / O interface 33: Processor 35: Image processing circuit 37: Memory 100: Display area 200: Perspective image 220: Rear end area 240: Tip area 300: Foreground 310: Real object 330: Road surface 331: Compartment line 332: Compartment line 350: Other vehicle 370: Other vehicle 401: Vehicle ECU
403: Road information database 405: Own vehicle position detection unit 407: Vehicle outside sensor 409: Line-of-sight direction detection unit 411: Eye position detection unit 413: Mobile information terminal 415: External communication device 502: Real object information detection module 504: Real object position Specific module 506: Notification necessity determination module 508: Eye position detection module 510: Image type setting module 512: Image arrangement setting module 514: Image size setting module 516: Line-of-sight direction determination module 518: Graphic module 520: Action determination module E: Specific angle area

Claims (19)

An image display unit (20) for displaying an image (200) composed of one or a plurality of image elements visually recognized along the front-rear direction of the own vehicle in an area overlapping the foreground when viewed from the driver of the own vehicle. In the display control device (30) to be controlled
One or more I / O interfaces (31) that can acquire information from the outside,
With one or more processors (33),
Memory (37) and
It comprises one or more computer programs stored in the memory (37) and configured to be executed by the one or more processors (33).
The one or more processors (33)
The driver has not visually recognized the tip region (240) of the image (200) for longer than a predetermined time, is gazing at the rear end region (220) of the image (200), and the tip. When the first condition consisting of at least one of watching the region above the region (240) is satisfied.
Execute a command to change the angle of the tip region (240) so that the vehicle can be seen from an angle along the road surface (330) on which the vehicle travels.
Display control device. The one or more processors (33)
The driver has not seen the rear end region (220) of the image (200) for longer than a predetermined time, and is gazing at the tip region (240) of the image (200). If the second condition consisting of at least one of
Execute a command to change the angle of the rear end region (220) so that it can be seen from an angle along the road surface (330).
The display control device according to claim 1. The image (200) is composed of a plurality of image elements.
The one or more processors (33)
When the first condition is satisfied,
The state-of-the-art image element, which is not all of the plurality of image elements, and one or a plurality of the state-of-the-art image elements connected to the state-of-the-art image element so as to be visually recognized by rising from an angle along the road surface (330). Change the angle of the image element, execute the command,
The display control device according to claim 1. The image (200) is composed of one image element.
The one or more processors (33)
When the first condition is satisfied,
A command is given to bend the tip region (240) of the image (200) to change the angle of the tip region (240) so that the image (200) can be seen from an angle along the road surface (330). Execute,
The display control device according to claim 1. After changing the angle of the tip region (240),
The one or more processors (33)
When it can be estimated that the driver has taken an appropriate action with respect to the information shown in the image (200).
Return the angle of the modified tip region (240), execute further instructions,
The display control device according to claim 1. The one or more processors (33)
The speed at which the modified tip region (240) is returned is made faster than the speed at which the angle at the tip region (240) is changed so that it can be seen from an angle along the road surface (330).
The display control device according to claim 5. A head-up display device (20) that displays an image (200) composed of one or a plurality of image elements visually recognized along the front-rear direction of the own vehicle in an area overlapping the foreground when viewed from the driver of the own vehicle. In
One or more I / O interfaces (31) that can acquire information from the outside,
With one or more processors (33),
Memory (37) and
It comprises one or more computer programs stored in the memory (37) and configured to be executed by the one or more processors (33).
The one or more processors (33)
The driver has not visually recognized the tip region (240) of the image (200) for longer than a predetermined time, is gazing at the rear end region (220) of the image (200), and the tip. When the first condition consisting of at least one of watching the region above the region (240) is satisfied.
Execute a command to change the angle of the tip region (240) so that the vehicle can be seen from an angle along the road surface (330) on which the vehicle travels.
Head-up display device. The one or more processors (33)
The driver has not seen the rear end region (220) of the image (200) for longer than a predetermined time, and is gazing at the tip region (240) of the image (200). If the second condition consisting of at least one of
Execute a command to change the angle of the rear end region (220) so that it can be seen from an angle along the road surface (330).
The head-up display device according to claim 7. The image (200) is composed of a plurality of image elements.
The one or more processors (33)
When the first condition is satisfied,
The state-of-the-art image element, which is not all of the plurality of image elements, and one or a plurality of the state-of-the-art image elements connected to the state-of-the-art image element so as to be visually recognized by rising from an angle along the road surface (330). Change the angle of the image element, execute the command,
The head-up display device according to claim 7. The image (200) is composed of one image element.
The one or more processors (33)
When the first condition is satisfied,
A command is given to bend the tip region (240) of the image (200) to change the angle of the tip region (240) so that the image (200) can be seen from an angle along the road surface (330). Execute,
The head-up display device according to claim 7. After changing the angle of the tip region (240),
The one or more processors (33)
When it can be estimated that the driver has taken an appropriate action with respect to the information shown in the image (200).
Return the angle of the modified tip region (240), execute further instructions,
The head-up display device according to claim 7. The one or more processors (33)
The speed at which the modified tip region (240) is returned is made faster than the speed at which the angle at the tip region (240) is changed so that it can be seen from an angle along the road surface (330).
The head-up display device according to claim 11. An image display unit (20) for displaying an image (200) composed of one or a plurality of image elements visually recognized along the front-rear direction of the own vehicle in an area overlapping the foreground when viewed from the driver of the own vehicle. It ’s a way to control
Acquiring the line-of-sight direction of the driver
Based on at least the position of the image (200) displayed by the image display unit (20) and the line-of-sight direction, the driver sets the tip region (240) of the image (200) for a predetermined time. At least one of not seeing for a longer time, gazing at the rear end region (220) of the image (200), and gazing at the region above the tip region (240). Determining whether or not the first condition consisting of
When the first condition is satisfied,
Including changing the angle of the tip region (240) so that the vehicle can be seen from an angle along the road surface (330) on which the own vehicle travels.
Method. Based on at least the position of the image (200) displayed by the image display unit (20) and the line-of-sight direction, the driver determines the rear end region (220) of the image (200). It is determined whether or not the second condition consisting of at least one of not visually recognizing for longer than the time of and watching the tip region (240) of the image (200) is satisfied. ,
When the second condition is satisfied,
Further including changing the angle of the rear end region (220) so that it can be seen from an angle along the road surface (330).
13. The method of claim 13. The image (200) is composed of a plurality of image elements.
When the first condition is satisfied,
The state-of-the-art image element, which is not all of the plurality of image elements, and one or a plurality of the state-of-the-art image elements connected to the state-of-the-art image element so as to be visually recognized by rising from an angle along the road surface (330). Change the angle of the image element,
13. The method of claim 13. The image (200) is composed of one image element.
When the first condition is satisfied,
The tip region (240) of the image (200) is bent to change the angle of the tip region (240) so that the image (200) can be seen from an angle along the road surface (330).
13. The method of claim 13. After changing the angle of the tip region (240),
Acquiring the state of the own vehicle and / or the state of the driver,
Estimating from the state of the own vehicle and / or the state of the driver whether the driver has taken an appropriate action with respect to the information shown in the image (200).
When it is estimated that the driver has taken an appropriate action with respect to the information shown in the image (200).
Further including returning the angle of the modified tip region (240).
13. The method of claim 13. The speed at which the modified tip region (240) is returned is made faster than the speed at which the angle at the tip region (240) is changed so that it can be seen from an angle along the road surface (330).
17. The method of claim 17. A computer program comprising instructions for performing the method according to any one of claims 13-18.

Images