WO2020158601A1

WO2020158601A1 - Display control device, method, and computer program

Info

Publication number: WO2020158601A1
Application number: PCT/JP2020/002504
Authority: WO
Inventors: 誠秦
Original assignee: 日本精機株式会社
Priority date: 2019-01-29
Filing date: 2020-01-24
Publication date: 2020-08-06

Abstract

The present invention smooths the transition of displaying images on the basis of a predicted position that is a predicted position of a real object, and quickly directs visual attention.　A display control device is capable of executing a first position setting process for setting a position of an image 210 on the basis of the most recently acquired position of a real object 310, and a second position setting process for setting a position of an image 220 on the basis of a predicted position of the real object 310 in a display update cycle of the image 210 that is predicted on the basis of one or more previously acquired positions of the real object 310, including at least the most recently acquired position of the real object 310. The display control device executes the first position setting process in a first display update cycle Fα immediately after the acquisition of the position of the real object 310 existing in front of an own vehicle, and executes the second position setting process in a second display update cycle Fβ that is not immediately after the acquisition of the position of the real object 310 existing in front of the own vehicle.

Description

Display control device, method, and computer program

The present disclosure relates to a display control device, a method, and a computer program that are used in a vehicle and that superimpose an image on the foreground of the vehicle for visual recognition.

Patent Document 1 discloses data in which the position of an object (other vehicle) around the vehicle is detected, and the size and position of the object after a predetermined time is predicted from the relative speed between the vehicle and the object. There is disclosed a display device that performs display according to the position of an object by generating the image and performing display based on the generated image.

JP, 2014-177275, A

However, when the position of an image or the like is adjusted only by the data of predicting the size and position of an object after a predetermined time as in the display device of Patent Document 1, there is a large difference between the predicted position and the actual observed position. However, it is assumed that it is difficult for the actual observation position to be reflected in the image quickly, and it becomes slower for the viewer to pay attention to the real object to be noted.

A summary of the specific embodiments disclosed herein is provided below. It should be understood that these aspects are presented only to provide an overview of the reader to these particular embodiments and are not intended to limit the scope of this disclosure. Indeed, the present disclosure may include various aspects not described below.

The outline of this disclosure relates to prompt visual attention to the target. More specifically, the present invention also relates to prompting visual attention while smoothing the display transition of an image based on the predicted position where the position of a real object is predicted.

Therefore, the display control device described in the present specification sets the first position setting process for setting the position of the image 200 based on the position of the real object acquired immediately before, and at least the position of the real object acquired immediately before. Second position setting process for setting the position of the image 200 based on the predicted position of the real object in the display update cycle of the image 200 predicted based on the position of one or more real objects acquired in the past And a second display that is not immediately after the position of the real object is acquired in the first display update cycle Fα immediately after the position of the real object is acquired. In the update cycle Fβ, the second position setting process is executed. Further, in some embodiments, in the first image 220, the first position setting process is executed in the first display update period Fα, and the second position setting process is executed in the second display update period Fβ. In the second image 230, the second position setting process is executed in the first display update cycle Fα and the second display update cycle Fβ.

It is a figure which shows the example of application to the vehicle of the display system for vehicles which concerns on some embodiment. FIG. 3 is a block diagram of a vehicular display system according to some embodiments. FIG. 6 is a diagram showing a first AR image displayed in association with a first real object and a second AR image displayed in association with a second real object according to some embodiments. It is a figure explaining how the specific position of the 1st real object and the specific position of the 2nd real object are set for every display update cycle of AR image concerning some embodiments.

1 and 2, below, a description of the configuration of an exemplary vehicle display system is provided. FIG. 3 provides a description of the image displayed by the vehicular display system. Also, FIG. 4 provides a description of the processing method. The present invention is not limited to the following embodiments (including the contents of the drawings). Of course, changes (including deletion of components) can be added to the following embodiments. In addition, in the following description, in order to facilitate understanding of the present invention, description of known technical matters is omitted as appropriate.

Refer to Figure 1. The image display unit 11 in the vehicle display system 10 is a head-up display (HUD: Head-Up Display) device provided in the dashboard 5 of the vehicle 1. The HUD device emits the display light 11 a toward the front windshield 2 (which is an example of a member to be projected) and displays the image 200 in the virtual display area 100, so that the display light 11 a is transmitted through the front windshield 2. The image 200 is overlaid and visually recognized on the foreground 300 which is the visually recognized real space. In the description of the present embodiment, the horizontal direction when the driver 4 seated in the driver's seat of the host vehicle 1 faces the front of the host vehicle 1 is the X axis (the left direction is the positive direction of the X axis), and the vertical direction is the vertical direction. The Y-axis (the upward direction is the positive Y-axis direction) and the front-back direction is the Z-axis (the forward direction is the positive Z-axis direction).

Further, the image display unit 11 may be a head mounted display (hereinafter, HMD) device. The driver 4 mounts the HMD device on his/her head and sits on the seat of the own vehicle 1 to visually recognize the displayed image 200 by superimposing it on the foreground 300 through the front windshield 2 of the own vehicle 1. To do. The display area 100 in which the vehicle display system 10 displays the predetermined image 200 is fixed at a specific position based on the coordinate system of the host vehicle 1, and when the driver 4 turns in that direction, the display area 100 is displayed at the specific position. The image 200 displayed in the fixed display area 100 can be visually recognized.

The image display unit 11 is an obstacle (pedestrian) existing in a foreground 300 which is a real space (real scene) visually recognized through the front windshield 2 of the vehicle 1 under the control of a display control device 13 described later. , Bicycles, motorcycles, other vehicles, etc.), road surfaces, road signs, and the vicinity of real objects 310 such as features (buildings, bridges, etc.) (an example of a specific positional relationship between images and real objects), real objects 310 To display the image 200 at a position overlapping with (an example of a specific positional relationship between the image and the real object) or at a position set with reference to the real object 310 (an example of a specific positional relationship between the image and the real object). Thus, it is possible to form a visual augmented reality (AR). The image display unit 11 changes the display position according to the position of the real object 310 (described in detail later) and a non-AR image (not shown) that does not change the display position according to the position of the real object 310. And can be displayed.

FIG. 2 is a block diagram of a vehicle display system 10 according to some embodiments. The vehicle display system 10 includes an image display unit 11 and a display control device 13 that controls the image display unit 11. The display controller 13 comprises one or more I/O interfaces 14, one or more processors 16, one or more memories 18, and one or more image processing circuits 20. The various functional blocks depicted in FIG. 2 may be implemented in hardware, software, or a combination of both. FIG. 2 is only one embodiment of an implementation and the illustrated components may be combined into fewer components or there may be additional components. For example, image processing circuitry 20 (eg, a graphics processing unit) may be included in one or more processors 16.

As illustrated, the processor 16 and the image processing circuit 20 are operably connected to the memory 18. More specifically, the processor 16 and the image processing circuit 20 execute a program stored in the memory 18 to operate the vehicular display system 10, for example, to generate or/and transmit image data. be able to. The processor 16 or/and the image processing circuit 20 includes at least one general purpose microprocessor (eg, central processing unit (CPU)), at least one application specific integrated circuit (ASIC), at least one field programmable gate array (FPGA). , Or any combination thereof. The memory 18 includes any type of magnetic media such as a hard disk, any type of optical media such as CDs and DVDs, any type of semiconductor memory such as volatile memory, and non-volatile memory. Volatile memory may include DRAM and SRAM, and non-volatile memory may include ROM and NVROM.

As shown, the processor 16 is operably connected to the I/O interface 14. For example, the I/O interface 14 uses the vehicle display system 10 as a personal area network (PAN) such as a Bluetooth (registered trademark) network or a local area network (LAN) such as an 802.11x Wi-Fi (registered trademark) network. It may include a wireless communication interface for connecting to a wide area network (WAN) such as a 4G or LTE cellular network. Also, the I/O interface 14 may include a wired communication interface such as, for example, a USB port, a serial port, a parallel port, an OBDII, and/or any other suitable wired communication port.

As shown in the figure, the processor 16 is operably connected to the I/O interface 14 so as to communicate with various other electronic devices connected to the vehicle display system 10 (I/O interface 14). Can be given and received. The I/O interface 14 includes, for example, a vehicle ECU 401 provided in the host vehicle 1, a road information database 403, a host vehicle position detection unit 405, a vehicle exterior sensor 407, a line-of-sight direction detection unit 409, an eye position detection unit 411, and portable information. The terminal 413, the vehicle exterior communication connection device 420, and the like are operably connected. The image display unit 11 is operably connected to the processor 16 and the image processing circuit 20. Therefore, the image displayed by the image display unit 11 may be based on the image data received from the processor 16 and/or the image processing circuit 20. The processor 16 and the image processing circuit 20 control the image displayed by the image display unit 11 based on the information obtained from the I/O interface 14. The I/O interface 14 may include a function of processing (converting, calculating, analyzing) information received from another electronic device or the like connected to the vehicle display system 10.

The host vehicle 1 is in the state of the host vehicle 1 (for example, mileage, vehicle speed, accelerator pedal opening, engine throttle opening, injector fuel injection amount, engine speed, motor speed, steering angle, shift position, drive mode). , A vehicle ECU 401 for detecting various warning states). The vehicle ECU 401 controls each unit of the host vehicle 1, and can transmit vehicle speed information indicating the current vehicle speed of the host vehicle 1 to the processor 16, for example. It should be noted that vehicle ECU 401 may transmit the determination result of the data detected by the sensor and/or the analysis result to processor 16 in addition to or instead of simply transmitting the data detected by the sensor to processor 16. For example, information indicating whether the host vehicle 1 is traveling at a low speed or stopped may be transmitted to the processor 16.

Further, the vehicle ECU 401 may transmit an instruction signal for instructing the image 200 displayed by the vehicle display system 10 to the I/O interface 14, in which case the coordinates of the image 200, the notification necessity degree of the image 200, or / And the necessity degree related information that is a basis for determining the notification necessity degree may be added to the instruction signal and transmitted.

The host vehicle 1 may include a road information database 403 including a navigation system and the like. The road information database 403 is based on the position of the vehicle 1 acquired from the vehicle position detection unit 405 described later, and is road information (lane, white line, stop line, Crosswalk, width of road, number of lanes, intersection, curve, branch road, traffic regulation, etc.), presence/absence of feature information (buildings, bridges, rivers, etc.), position (including distance to own vehicle 1), direction The direction (based on the host vehicle 1), shape, type, detailed information, etc. may be read and transmitted to the processor 16. Further, the road information database 403 may calculate an appropriate route from the starting point to the destination and send it to the processor 16 as navigation information.

The host vehicle 1 may include a host vehicle position detection unit 405 such as a GNSS (Global Navigation Satellite System). The road information database 403, the portable information terminal 413, which will be described later, and/or the external communication connecting device 420 acquires the position information of the own vehicle 1 from the own vehicle position detection unit 405 continuously, intermittently, or at every predetermined event. Thus, the information around the vehicle 1 can be selected and/or generated and transmitted to the processor 16.

The host vehicle 1 may include one or more vehicle exterior sensors 407 that detect the real objects 310 existing around the host vehicle 1 (front, side, and rear). The real object 310 detected by the vehicle exterior sensor 407 is, for example, a pedestrian, a bicycle, a motorcycle, another vehicle (the preceding vehicle 320 or the like), a road surface (a traveling lane 330), a marking line, a roadside object, and/or a feature (building. Etc.) etc. may be included. As the vehicle exterior sensor, for example, a millimeter wave radar, an ultrasonic radar, a radar sensor such as a laser radar, there is a camera sensor consisting of a camera and an image processing device, may be configured by a combination of both the radar sensor, the camera sensor, It may be configured with only one of them. A conventionally known method is applied to the object detection by the radar sensor and the camera sensor. By detecting the objects by these sensors, the presence or absence of a real object in the three-dimensional space, and if the real object exists, the position of the real object (relative distance from the own vehicle 1, the traveling direction of the own vehicle 1 Position in the left-right direction in the front-back direction, vertical position, etc.), size (size in the horizontal direction (left-right direction), height direction (up-down direction), etc.), movement direction (lateral direction (left-right direction)) , Depth direction (front-back direction), movement speed (lateral direction (left-right direction), depth direction (front-back direction)), and/or type may be detected. One or more vehicle exterior sensors 407 detect a real object in front of the own vehicle 1 for each detection cycle of each sensor, and detect real object related information (presence or absence of real object, which is an example of real object related information). When a real object exists, information such as the position, size, and/or type of each real object) can be transmitted to the processor 16. The real object related information may be transmitted to the processor 16 via another device (for example, the vehicle ECU 401). An infrared camera or a near-infrared camera is desirable when using a camera as a sensor so that a real object can be detected even when the surroundings are dark such as at night. Further, when using a camera as a sensor, a stereo camera capable of acquiring a distance and the like by parallax is desirable.

The host vehicle 1 may include a line-of-sight direction detection unit 409 including an infrared camera that detects the gaze direction of the driver 4 (hereinafter, also referred to as “line-of-sight direction”) and that captures an image of the face of the driver 4. The processor 16 can specify the line-of-sight direction of the driver 4 by acquiring an image captured by the infrared camera (an example of information that can estimate the line-of-sight direction) and analyzing the captured image. Note that the processor 16 may acquire the line-of-sight direction of the driver 4 specified by the line-of-sight direction detection unit 409 (or another analysis unit) from the image captured by the infrared camera from the I/O interface 14. The method of acquiring the driver's 4 line-of-sight direction of the vehicle 1 or the information capable of estimating the driver's 4 line-of-sight direction is not limited to these, and the EOG (Electro-oculogram) method, the corneal reflex Method, scleral reflection method, Purkinje image detection method, search coil method, infrared fundus camera method, and other known gaze direction detection (estimation) techniques may be used.

The host vehicle 1 may include an eye position detection unit 411 including an infrared camera that detects the position of the eyes of the driver 4. The processor 16 can specify the eye position of the driver 4 by acquiring an image (an example of information that can estimate the eye position) captured by the infrared camera and analyzing the captured image. The processor 16 may acquire the information on the position of the eyes of the driver 4 identified from the image captured by the infrared camera from the I/O interface 14. The method for acquiring the position of the eyes of the driver 4 of the vehicle 1 or the information capable of estimating the position of the eyes of the driver 4 is not limited to these, and known eye position detection (estimation) It may be acquired using a technology. The processor 16 adjusts at least the position of the image 200 based on the position of the eyes of the driver 4 so that the image 200 in which the image 200 superimposed on a desired position of the foreground 300 is detected is the viewer (driver 4). May be visually confirmed.

The mobile information terminal 413 is a smartphone, a laptop computer, a smart watch, or other information device that can be carried by the driver 4 (or another occupant of the vehicle 1). The I/O interface 14 can communicate with the mobile information terminal 413, and acquires the data recorded in the mobile information terminal 413 (or the server via the mobile information terminal). The mobile information terminal 413 has, for example, the same function as the road information database 403 and the vehicle position detection unit 405 described above, acquires the road information (an example of real object-related information), and transmits it to the processor 16. Good. The mobile information terminal 413 may also acquire commercial information (an example of real object-related information) related to a commercial facility in the vicinity of the host vehicle 1 and send it to the processor 16. The portable information terminal 413 transmits schedule information of the owner (for example, the driver 4) of the portable information terminal 413, incoming information at the portable information terminal 413, mail reception information, etc. to the processor 16, and the processor 16 and The image processing circuit 20 may generate or/and transmit image data regarding these.

The outside-vehicle communication connection device 420 is a communication device for exchanging information with the own vehicle 1, and for example, another vehicle connected to the own vehicle 1 through vehicle-to-vehicle communication (V2V: Vehicle To Vehicle), pedestrian-to-vehicle communication (V2P: It is a network communication device connected by a pedestrian (a portable information terminal carried by a pedestrian) and a road-to-vehicle communication (V2I: Vehicle To road Infrastructure) connected by a Vehicle To Pedestrian. Includes everything connected by V2X (Vehicle To Everything). The extra-vehicle communication connection device 420 acquires the position of, for example, a pedestrian, a bicycle, a motorcycle, another vehicle (such as a preceding vehicle), a road surface, a marking line, a roadside object, and/or a feature (such as a building), and the processor 16 May be sent to. In addition, the vehicle exterior communication connection device 420 may have the same function as the own vehicle position detection unit 405 described above, may acquire the position information of the own vehicle 1 and may transmit the position information to the processor 16, and further, the above road information database. It also has the function of 403, and may acquire the road information (an example of real object related information) and send it to the processor 16. The information acquired from the vehicle exterior communication connection device 420 is not limited to the above.

The software components stored in the memory 18 are the real object related information detection module 502, the real object position setting module 504, the difference determination module 506, the distance determination module 508, the speed determination module 510, the notification necessity determination module 512, and the image position. A decision module 514, an image size decision module 516, and a graphics module 518 are included.

The real object related information detection module 502 acquires information (also called real object related information) including at least the position of the real object 310 existing in front of the host vehicle 1. The real object-related information detection module 502 uses, for example, the vehicle exterior sensor 407 to detect the position of the real object 310 existing in the foreground 300 of the host vehicle 1 (the traveling direction of the host vehicle 1 from the driver 4 in the driver's seat of the host vehicle 1 ( The position in the height direction (vertical direction) and the lateral direction (horizontal direction) when visually recognizing the front), and the position in the depth direction (front direction) may be added to these), and of the real object 310. You may acquire the information (an example of real object relevant information) containing size (height direction, horizontal size) and relative speed with respect to the own vehicle 1 (a relative moving direction is also included). In addition, the real object related information detection module 502 uses the external communication connection device 420 to detect the position, relative speed, and type of the real object (other vehicle), the lighting state of the direction indicator of the real object (other vehicle), and the steering angle operation. (Or an example of the actual object and related information) indicating the state, or/and the planned traveling route and the traveling schedule by the driving support system.

In addition, the real object related information detection module 502 detects the position of the left lane marking 331 (see FIG. 3) of the vehicle driving lane 330 (see FIG. 3) and the right lane marking 332 (see FIG. 3)), and the area (running lane 330) between the left and

right partition lines

331 and 332 may be recognized.

The real object position setting module 504 acquires an observation position indicating the current position of the real object 310 from the road information database 403, the vehicle exterior sensor 407, the portable information terminal 413, or the vehicle exterior communication connection device 420 via the I/O interface 14. Alternatively, the observation position of the real object obtained by fusing these two or more observation positions is acquired, and the position of the real object 310 (also referred to as a specific position) is set based on the acquired observation position. An image position determination module 514, which will be described later, determines the position of the AR image 210 based on the specific position of the real object 310 set by the real object position setting module 504.

The real object position setting module 504 sets the specific position of the real object 310 based on the observation position of the real object 310 acquired immediately before, and the one acquired in the past including at least the observation position of the real object 310 acquired immediately before. Alternatively, it is possible to set the specific position of the real object 310 based on the predicted position of the real object at a predetermined time predicted based on the observed position of the real object 310 or more. That is, by executing the real object position setting module 504 and the image position determination module 514 described later, the processor 16 sets the specific position of the AR image 210 based on the observation position of the real object 310 acquired immediately before. Position setting processing and a real object in the display update cycle of the AR image 210 predicted based on the observation positions of one or more real objects 310 acquired in the past including at least the observation position of the real object 310 acquired immediately before A second position setting process of setting the position of the AR image 210 based on the predicted position of 310 can be executed.

For example, in the first AR image 220, the real object position setting module 504 sets the specific position of the real object 310, which is the reference of the position to be displayed, in the time series of the first position setting process and the second position setting process. In the second AR image 230, the specific position of the real object 310 that serves as the reference of the display position is calculated by the second position setting process. FIG. 3 is a diagram showing a first AR image 220 displayed in association with the first real object 320 and a second AR image 230 displayed in association with the second real object 330. Here, the first AR image 220 is an arcuate shape that draws attention to the preceding vehicle (first real object) 320 preceding the traveling lane 330 of the own vehicle 1 and is visually recognized so as to surround the preceding vehicle 320 behind. This is a warning image. In addition, the second AR image 230 is a route image having a single arrow shape that shows the planned route of the host vehicle 1 and is visually recognized in an overlapping manner on the traveling lane (second real object) 330 of the host vehicle 1.

The processor 16 uses the real object-related information detection module 502 to observe the position Im of the preceding vehicle (first real object) 320, which serves as a reference for the position where the attention image (first AR image) 220 is displayed, and the route image. The observation position In indicating the position of the traveling lane (second real object) 330 that serves as a reference for the position where the (second AR image) 230 is displayed is acquired. Next, the processor 16 uses the observation positions Im to Im-5 of the first real object 320 acquired in the past, including the observation position Im acquired immediately before, from the observation positions Im to Im-5 of the first time for each display update cycle k+2 to k−2 of the AR image 210. From the specific positions Pk+2 to Qk-2 of the real object 320 and the observation positions In to In-5 of the second real object 330 including those acquired in the past, the second at every display update cycle k+2 to k-2 of the AR image 210. The specific positions Qk+2 to Qk-2 of the real object 330 are set.

FIG. 4 is a diagram for explaining how the specific position of the first real object and the specific position of the second real object are set for each display update cycle of the AR image. In the figure, the display update cycle k+2 is the newest display update cycle, and becomes older in the order of k+1, k, k-1, k-2. The same applies to the specific position of the first real object 320 and the specific position of the second real object 330, and Pk+2 (Qk+2) is the newest in the figure. Im(In) is the newest observation position of the first real object 320 (second real object 330) in the figure, and Im-1(In-1), Im-2(In-2)... It becomes old in order. In the figure, the cycle in which the observation position Im of the first real object 320 is acquired and the cycle in which the observation position In of the second real object 330 is acquired are described as different, but they are acquired. The periods may be aligned.

First, the setting of the specific position Pk of the first real object 320 will be described. Immediately after the processor 16 acquires the specific position Pk of the first real object 320 existing in front of the host vehicle 1 from the one or more I/O interfaces 14 in the setting of the specific position Pk of the first real object 320. In the first display update cycle Fα, the first position setting processing is executed, and in the second display update cycle Fβ which is not immediately after the observation position Im of the first real object 320 is acquired, the second position setting processing is executed. To do. In the figure, the display update cycle k-1, k+1 is the first display update cycle Fα immediately after the observation positions Im-1, Im are acquired, and the display update cycle k-2, k, k+2 is the observation position Im-1. , Im is the second display update cycle Fβ that is not the first display update cycle Fα immediately after the acquisition. Since the display update cycle k is the second display update cycle Fβ, the specific position Pk in the display update cycle k is determined by the second position setting process, and specifically, the four observation positions Im− 1, Im-2, Im-3, Im-4. Since the next display update cycle k+1 is the first display update cycle Fα, the specific position Pk+1 in the display update cycle k+1 is determined by the first position setting process, and specifically, the first actual acquired immediately before. The observation position Im of the object 320 is set. Since the next display update cycle k+2 is the second display update cycle Fβ, the specific position Pk+2 in the display update cycle k+2 is determined by the second position setting process, and specifically, the four observation positions before that. It is predicted based on Im, Im-1, Im-2 and Im-3. There is no particular limitation on the method of calculating the predicted position by the real object position setting module 504, and the real object position setting module 504 obtains a display update cycle (eg, the display update cycle k in FIG. 4) that is the target of processing in the past. Any method may be used as long as the prediction is performed based on the observed positions (observed positions Im-1, Im-2,... In FIG. 4). The real object position setting module 504 uses the least squares method or a prediction algorithm such as a Kalman filter, an α-β filter, or a particle filter, and uses one or more past observed positions to calculate the next value. May be predicted.

Next, the setting of the specific position Qk of the second real object 330 will be described. The processor 16 executes the second position setting process in the first display update cycle Fα and the second display update cycle Fβ in setting the specific position Qk of the second real object 330. In the figure, the display update cycle k-2, k, k+2 is the first display update cycle Fα immediately after the observation positions In-2, In-1, In are acquired, and the display update cycle k-1, k+1 is observed. The second display update cycle Fβ is not the first display update cycle Fα immediately after the positions In-2, In-1, and In are acquired. That is, the display update cycle k is the first display update cycle Fα, but the specific position Qk in the display update cycle k is determined by the second position setting process. Predicted based on In-1, In-2, In-3, In-4. The specific position Qk+1 in the next display update cycle k+1 is also determined by the second position setting process, and specifically, the four observation positions In-1, In-2, In-3, In-4 before that are determined. Predicted based on. The next display update cycle k+2 is the second display update cycle Fβ, but the specific position Qk+2 in the display update cycle k+2 is determined by the second position setting process, and specifically, the four observation positions before that. Predicted based on In, In-1, In-2, In-3.

The difference determination module 506 of FIG. 2 predicts the predicted position in the first display update cycle Fα based on the observation position acquired immediately before and one or more observation positions including at least the observation position acquired immediately before. And are compared, and it is determined whether the difference between them is larger than a predetermined difference threshold value stored in the memory 18. When these differences are larger than the predetermined threshold value, the processor 16 executes the first position setting process in the first display update cycle Fα, and executes the second position setting process in the second display update cycle Fβ. After continuing this for a predetermined number of display update cycles, the second position setting process may be executed in the first display update cycle Fα and the second display update cycle Fβ. Further, when these differences are not larger than the predetermined threshold value, the processor 16 executes the second position setting process in the first display update cycle Fα and the second display update cycle Fβ. The memory 18 may store two or more difference thresholds, and the distance determination module 508 may determine the degree of difference between the observed position acquired immediately before and the predicted position in three or more stages. Further, these difference thresholds may be variable. For example, the difference determination module 506 may be changed according to the relative speed between the real object 310 and the host vehicle 1, and in this case, the higher the relative speed, the longer the difference threshold may be set.

Further, in some embodiments, the difference determination module 506 may determine whether the observed position is closer to the host vehicle 1 than the predicted position. When the observation position acquired immediately before the real object by the difference determination module 506 is closer to the host vehicle 1 than the predicted position, the processor 16 executes the first position setting process in the first display update cycle Fα. In the second display update cycle Fβ, when the second position setting process is executed and the observed position acquired immediately before is not nearer to the vehicle 1 than the predicted position, the first display update cycle Fα and the second display update In the period Fβ, the second position setting process may be executed. According to this, when the real object is assumed to be closer than the predicted position, the AR image 210 can be quickly displayed based on the observation position where the real object is highly likely to exist.

The distance determination module 508 determines the degree of distance between the real object 310 and the host vehicle 1. For example, the distance determination module 508 determines that the distance between the real object 310 and the host vehicle 1 that can be acquired by executing the real object related information detection module 502 is greater than the predetermined distance threshold stored in the memory 18. You may judge whether it is long. The memory 18 may store two or more distance thresholds, and the distance determination module 508 may determine the degree of the distance between the real object 310 and the vehicle 1 in three or more steps. Further, these distance thresholds may be variable. For example, the distance determination module 508 may change the distance according to the relative speed between the real object 310 and the host vehicle 1. In this case, the faster the relative speed, the longer the distance threshold may be set.

The speed determination module 510 determines the degree of relative speed between the real object 310 and the host vehicle 1. For example, the speed determination module 510 calculates the real object 310 and the own vehicle based on the time change of the distance between the real object 310 and the own vehicle 1 that can be acquired by the real object related information detection module 502 being executed. It may be determined whether the relative speed with respect to 1 is faster than a predetermined relative speed threshold value stored in the memory 18. The memory 18 may store two or more relative speed thresholds, and the speed determination module 510 may determine the degree of relative speed between the real object 310 and the vehicle 1 in three or more stages. Further, these relative speed thresholds may be variable. For example, the speed determination module 510 may change the speed according to the distance between the real object 310 and the host vehicle 1. In this case, the longer the distance between the real object 310 and the host vehicle 1, the relative speed. You may set so that a threshold value may become fast.

The notification necessity degree determination module 512 determines whether or not each image 200 displayed by the vehicle display system 10 is the content to be notified to the driver 4. The notification necessity degree determination module 512 may obtain information from various other electronic devices connected to the I/O interface 14 and calculate the notification necessity degree. In addition, the electronic device connected to the I/O interface 14 in FIG. 2 transmits information to the vehicle ECU 401, and the notification necessity degree determination module 512 detects (acquires) the notification necessity degree determined by the vehicle ECU 401 based on the received information. ) May be. The "information need level" is, for example, a risk level derived from the degree of seriousness of the possibility itself, an urgency level derived from the length of the reaction time required to take a reaction action, the own vehicle 1 or the driver 4 ( Alternatively, it can be determined based on the effectiveness derived from the situation of other occupants of the vehicle 1 or a combination thereof (the indicator of the notification necessity degree is not limited to these). That is, the notification necessity degree determination module 512 may determine whether to notify the driver 4 and may select not to display the route image 220, the warning image 230 described below, or both of them. The vehicle display system 10 may not have a function of estimating (calculating) the notification necessity degree, and a part or all of the function of estimating the notification necessity degree may be displayed by the vehicle display system 10. It may be provided separately from the control device 13.

The image position determination module 514 determines the determined position (observation position or predicted position) of the real object 310 set by the real object position setting module 504 so that the image 200 is visually recognized in a specific positional relationship with the real object 310. Based on the coordinates of the image 200 (including at least the horizontal direction (X-axis direction) and the vertical direction (Y-axis direction) when the driver 4 views the direction of the display area 100 from the driver's seat of the vehicle 1) decide. In addition to this, the image position determination module 514 determines when the driver 4 views the direction of the display area 100 from the driver's seat of the own vehicle 1 based on the determined position of the real object 310 set by the real object position setting module 504. The front-back direction (Z-axis direction) may be determined. The image position determination module 514 adjusts the position of the image 200 based on the position of the eyes of the driver 4 detected by the eye position detection unit 411. For example, the image position determination module 514 determines the horizontal position and the vertical position of the image 200 so that the center of the image 200 is visually recognized so as to overlap with the center of the real object. The “specific positional relationship” can be adjusted depending on the situation of the real object or the host vehicle 1, the type of the real object, the type of the displayed image, or the like.

The image size determination module 516 may change the size of the AR image 210 in accordance with the position and/or size of the real object 310 to be associated. For example, the image size determination module 516 can reduce the size of the AR image 210 if the position of the real object 310 to be associated is distant. Further, the image size determination module 516 can increase the size of the AR image 210 if the size of the real object 310 to be associated is large.

In addition, the image size determination module 516 detects the type, number, and/or notification necessity determination module 512 of the real object that displays the image 200 detected by the real object related information detection module 502 in association with each other ( The size of the image 200 may be determined based on the (estimated) notification need.

The image size determination module 516 may have a function of predicting and calculating the size of the AR image 210 to be displayed in the display update cycle of this time, based on the size of the real object a predetermined number of times in the past. As a first method, the image size determination module 516 tracks pixels of the real object 310 between two past captured images captured by the camera (an example of the vehicle exterior sensor 407) using, for example, the Lucas-Kanade method. Thus, the size of the real object in the current display update cycle may be predicted, and the size of the AR image may be determined according to the predicted size of the real object. As a second method, the rate of change of the size of the real object is obtained based on the change of the size of the real object between the past two captured images, and the size of the AR image is determined according to the rate of change of the size of the real object. You may. Note that the method of estimating the size change of the real object from the viewpoint that changes in time series is not limited to the above, and known methods including optical flow estimation algorithms such as the Horn-Schunkk method, the Buxton-Buxton method, and the Black-Jepson method, for example. You may use the method of.

The graphics module 518 provides visual effects (eg, brightness, transparency, saturation, contrast, or other visual characteristic), size, display position, distance (distance from the driver 4 to the image 200) of the displayed image 200. ) Includes various known software components for modifying. The graphic module 518 displays the coordinates set by the image position determination module 514 (the left-right direction (X-axis direction) and the up-down direction (Y-axis when the driver 4 views the direction of the display area 100 from the driver's seat of the vehicle 1 ). The image 200 is displayed so as to be visually recognized by the driver 4 with the image size set by the image size determination module 516.

As described above, the display control device according to the present embodiment includes the image display unit 11 that superimposes and displays the image 200 on the position associated with the real object existing in the foreground viewed from the driver 4 of the vehicle 1. In the display controller 13 to control, one or more I/O interfaces 14, one or more processors 16, a memory 18, and one or more processors 16 stored in the memory 18. One or more computer programs configured to be executed, the one or more processors 16 set the position of the image 200 based on the position of the most recently acquired real object. The first position setting process and the predicted position of the real object in the display update cycle of the image predicted based on the positions of one or more real objects acquired in the past including at least the position of the real object acquired immediately before. And a second position setting process for setting the position of the image 200 based on the position of the real object existing in front of the vehicle 1 from one or more I/O interfaces 14. In the first display update cycle Fα, the first position setting process is not executed and the position of the real object existing in front of the own vehicle 1 is acquired from the one or more I/O interfaces 14. In the display update cycle Fβ, the second position setting process is executed. According to this, since the display position of the AR image is determined based on the predicted position of the real object from the past observation position, the observation position of the real object is acquired while preventing the display position of the image from changing rapidly. In the display update cycle immediately after, the display position of the AR image is determined based on the observation position of the real object, so that the viewer can quickly recognize the accurate position of the real object.

Also, in some embodiments, the image 200 includes a first AR image 220 and a second AR image 230 that is of a different type than the first AR image 220, and one or more processors 16 may cause the first AR image 220 to include the first AR image 220. In the first display update period Fα, the first position setting process is executed, in the second display update period Fβ, the second position setting process is executed, and in the second AR image 230, the first display update period Fα and In the second display update cycle Fβ, the second position setting process may be executed. According to this, in one of the types of images to be displayed, the position of the image can be updated only at the predicted position of the real object, and a smoothly changing image in which abrupt changes are suppressed can be displayed.

Further, in some embodiments, the first AR image 220 may be a warning image that calls attention to the real object, and the second AR image 230 may be a route image that shows the route of the host vehicle 1.

Note that the first AR image 220 is an image that requires a relatively high degree of notification, and includes signs such as "stop" on the road surface, road surface conditions (wet, freezing) that may cause slippage, and automatic manual operation of the host vehicle 1. It may be an image showing a driving switching point. In addition, the second AR image 230 is an image having a lower notification necessity than the first AR image 220, and shows road signs, signboards, POI information, information about the direction of the final destination, and the like, which are not so important as "stop". It may be an image.

Further, in some embodiments, the one or more processors 16 include the position of the real object acquired immediately before and the position of the real object acquired at least immediately before in the first display update cycle Fα. The predicted position of the real object in the display update cycle of the image predicted based on the acquired position of one or more real objects is compared, and if the difference between these is greater than a predetermined threshold value, the first display update In the period Fα, the first position setting process is executed, in the second display update period Fβ, the second position setting process is executed, and when the difference is not larger than the predetermined threshold, the first display update period Fα and the second display The second position setting process may be executed in the update cycle Fβ. According to this, when a difference occurs between the observed position and the predicted position of the real object, the image is quickly displayed at the position based on the latest observed position, and the AR image and the actual image in which the AR image is associated with each other are associated with each other. The visual attention of the driver 4 can be promptly directed to the object.

Further, in some embodiments, the one or more processors 16 include the position of the real object acquired immediately before and the position of the real object acquired at least immediately before in the first display update cycle Fα. The predicted position of the real object in the display update cycle of the image predicted based on the acquired position of one or more real objects is compared, and the position of the real object acquired immediately before is the predicted position of the real object. When it is closer to the host vehicle 1, the first position setting process is executed in the first display update cycle Fα, the second position setting process is executed in the second display update cycle Fβ, and the actual position acquired immediately before is executed. When the position of the object is not nearer to the host vehicle 1 than the predicted position of the real object, the second position setting process may be executed in the first display update cycle Fα and the second display update cycle Fβ. According to this, when the latest observed position of the real object is closer to the host vehicle 1 than the predicted position, and the rapid approach of the real object can be assumed, the image is displayed at the position based on the latest observed position, so that the AR image is displayed. Also, the visual attention of the driver 4 can be promptly directed to the real object associated with this AR image.

Further, in some embodiments, one or more processors 16 may use the one or more I/O interfaces 14 to set the position of the image 200 relative to a reference real object and the host vehicle 1. When the information capable of estimating the speed is acquired and the relative speed is faster than the predetermined threshold value, the first position setting process is executed in the first display update cycle Fα, and the second position setting process is executed in the second display update cycle Fβ. If the relative speed is not faster than the predetermined threshold value, the second position setting process may be executed in the first display update cycle Fα and the second display update cycle Fβ. According to this, for a real object with a high relative speed, the latest observation position is quickly reflected in the image to draw visual attention, and for a real object with a low relative speed, the observation position is predicted. Even if there is an instantaneous difference between the position and the position, it is possible to prevent the visual attention from being excessively directed without promptly reflecting it on the image.

Also, in some embodiments, one or more of the processors 16 may determine, in the first display update period Fα, if the real object serving as the reference for setting the position of the image 200 is determined to be approaching. When the first position setting process is executed, the second position setting process is executed in the second display update cycle Fβ, and it is not determined that the real object serving as the reference for setting the position of the image 200 is approaching, the first position setting process is performed. In the display update cycle Fα and the second display update cycle Fβ, the second position setting process may be executed. According to this, when it can be assumed that the real object is approaching, the driver displays the AR image and the real object associated with the AR image by displaying the image at the position based on the latest observation position. The visual attention of 4 can be directed quickly.

Further, as shown in FIG. 1, the display area 100 is not limited to an arrangement that is substantially along a plane (XY plane) consisting of up, down, left and right as seen from the driver 4. For example, the display region 100 may be rotated about the left-right direction (X-axis direction) viewed from the driver 4 and arranged substantially along the traveling lane 330 (ZX plane). The display area 100 may be a curved surface instead of a flat surface. Further, a stereoscopic display may be adopted for the image display unit 11 and the image 200 may be displayed in the display area 100 which is a three-dimensional area.

The second AR image 230 may be a route image with two or more illustrations or the like that are visually recognized to be superimposed on the traveling lane 330 of the host vehicle 1.

DESCRIPTION OF SYMBOLS 1... Own vehicle, 2... Front windshield, 4... Driver, 5... Dashboard, 10... Vehicle display system, 11... Image display part, 11a... Display light, 13... Display control device, 14... I/O Interface, 16... Processor, 18... Memory, 20... Image processing circuit, 100... Display area, 200... Image, 210... AR image, 220... First AR image (path image), 230... Second AR image (attention image) , 300... Foreground, 310... Real object, 320... First real object (preceding vehicle), 330... Second real object (running lane), 401... Vehicle ECU, 403... Road information database, 405... Own vehicle position detection unit , 407... Out-of-vehicle sensor, 409... Gaze direction detecting section, 411... Eye position detecting section, 413... Portable information terminal, 420... Out-of-vehicle communication connecting device, 502... Real object related information detecting module, 504... Real object position setting module, 506... Difference determination module, 508... Distance determination module, 510... Speed determination module, 512... Notification necessity determination module, 514... Image position determination module, 516... Image size determination module, 518... Graphic module, Fα... First display Update cycle, Fβ... Second display update cycle

Claims

In a display control device that controls an image display unit that displays an image in an overlapping manner at a position associated with a real object existing in the foreground seen from the driver of the own vehicle,
One or more I/O interfaces,
One or more processors,
Memory and
One or more computer programs stored in the memory and configured to be executed by the one or more processors;
The one or more processors are
A first position setting process for setting the position of the image based on the position of the real object acquired immediately before;
Based on the predicted position of the real object in the display update cycle of the image predicted based on the position of one or more real objects acquired in the past including the position of the real object acquired at least immediately before A second position setting process for setting the position of the image, and
In the first display update cycle immediately after acquiring the position of the real object existing in front of the own vehicle from the one or more I/O interfaces, the first position setting process is executed,
The second position setting process is executed in the second display update cycle that is not immediately after the position of the real object existing in front of the own vehicle is acquired from the one or more I/O interfaces.
Display controller.
The image includes a first image and a second image of a different type from the first image,
The one or more processors are
In the first image,
In the first display update cycle, the first position setting process is executed,
In the second display update cycle, the second position setting process is executed,
In the second image,
In the first display update cycle and the second display update cycle, the second position setting process is executed.
The display control device according to claim 1.
The first image is a warning image that calls attention to the real object,
The second image is a route image showing a route of the own vehicle,
The display control device according to claim 2.
The one or more processors are
In the first display update cycle, prediction is performed based on the position of the real object acquired immediately before and the position of one or more real objects acquired at least including the position of the real object acquired immediately before. Comparing the predicted position of the real object in the display update cycle of the image,
If these differences are greater than a given threshold,
In the first display update cycle, the first position setting process is executed,
In the second display update cycle, the second position setting process is executed,
If the difference is not greater than a predetermined threshold,
In the first display update cycle and the second display update cycle, the second position setting process is executed.
The display control device according to claim 1.
The one or more processors are
In the first display update cycle, prediction is made based on the position of the real object acquired immediately before and the position of one or more real objects acquired at least including the position of the real object acquired immediately before. Comparing the predicted position of the real object in the display update cycle of the image,
If the position of the real object acquired immediately before is closer to the host vehicle than the predicted position of the real object,
In the first display update cycle, the first position setting process is executed,
In the second display update cycle, the second position setting process is executed,
If the position of the real object acquired immediately before is not near the own vehicle than the predicted position of the real object,
In the first display update cycle and the second display update cycle, the second position setting process is executed.
The display control device according to claim 1.
The one or more processors are
From the one or more I/O interfaces, obtain information capable of estimating the relative speed between the real object and the own vehicle, which serves as a reference for setting the position of the image,
If the relative speed is faster than a predetermined threshold,
In the first display update cycle, the first position setting process is executed,
In the second display update cycle, the second position setting process is executed,
If the relative speed is not faster than a predetermined threshold,
In the first display update cycle and the second display update cycle, the second position setting process is executed.
The display control device according to claim 1.
The one or more processors are
When it is determined that the real object that is the reference for setting the position of the image is approaching,
In the first display update cycle, the first position setting process is executed,
In the second display update cycle, the second position setting process is executed,
If it is not determined that the real object that is the reference for setting the position of the image is approaching,
In the first display update cycle and the second display update cycle, the second position setting process is executed.
The display control device according to claim 1.
In a method of controlling an image display unit for displaying an image in an overlapping manner at a position associated with a real object existing in the foreground seen by the driver of the own vehicle,
A first position setting process for setting the position of the image based on the position of the real object acquired immediately before;
Based on the predicted position of the real object in the display update cycle of the image predicted based on the position of one or more real objects acquired in the past including the position of the real object acquired at least immediately before A second position setting process for setting the position of the image, and
Executing the first position setting process in a first display update cycle immediately after acquiring the position of the real object existing in front of the own vehicle;
Executing the second position setting process in a second display update cycle that is not immediately after acquiring the position of the real object existing in front of the own vehicle.
Method.
The image includes a first image and a second image of a different type from the first image,
In the first image,
In the first display update cycle, the first position setting process is executed,
Performing the second position setting process in the second display update cycle;
In the second image,
Performing the second position setting process in the first display update cycle and the second display update cycle.
The method of claim 8.
The first image is a warning image that calls attention to the real object,
The second image is a route image showing a route of the own vehicle,
The method according to claim 9.
In the first display update cycle, prediction is performed based on the position of the real object acquired immediately before and the position of one or more real objects acquired at least including the position of the real object acquired immediately before. Comparing the predicted position of the real object in the display update cycle of the image,
If these differences are greater than a given threshold,
In the first display update cycle, the first position setting process is executed,
Performing the second position setting process in the second display update cycle;
If the difference is not greater than a predetermined threshold,
Performing the second position setting process in the first display update cycle and the second display update cycle.
The method of claim 8.
In the first display update cycle, prediction is performed based on the position of the real object acquired immediately before and the position of one or more real objects acquired at least including the position of the real object acquired immediately before. Comparing the predicted position of the real object in the display update cycle of the image,
If the position of the real object acquired immediately before is closer to the host vehicle than the predicted position of the real object,
In the first display update cycle, the first position setting process is executed,
Performing the second position setting process in the second display update cycle;
If the position of the real object acquired immediately before is not near the own vehicle than the predicted position of the real object,
Performing the second position setting process in the first display update cycle and the second display update cycle.
The method of claim 8.
Acquiring information capable of estimating a relative speed between the real object and the own vehicle, which serves as a reference for setting the position of the image,
If the relative speed is faster than a predetermined threshold,
In the first display update cycle, the first position setting process is executed,
Performing the second position setting process in the second display update cycle;
If the relative speed is not faster than a predetermined threshold,
Performing the second position setting process in the first display update cycle and the second display update cycle.
The method of claim 8.
When it is determined that the real object that is the reference for setting the position of the image is approaching,
In the first display update cycle, the first position setting process is executed,
Performing the second position setting process in the second display update cycle;
If it is not determined that the real object that is the reference for setting the position of the image is approaching,
Performing the second position setting process in the first display update cycle and the second display update cycle.
The method of claim 8.
A computer program comprising instructions for performing the method according to any of claims 8-14.