WO2020121486A1 - Display control device and display control method - Google Patents

Abstract

A host vehicle information acquisition unit (11) acquires, as host vehicle information, information including the position and orientation of a host vehicle. An object information acquisition unit (12) acquires, as object information, information including the position of an object that is located within a predetermined distance from the host vehicle. A road shape acquisition unit (13) acquires the road shape of the road on which the object is located. An inclination acquisition unit (14) acquires the inclination of the object with respect to the host vehicle on the basis of the host vehicle information, the object information, and the road shape. An AR image generation unit (15) generates an AR image to be overlaid on the object. An AR image control unit (16) changes the display inclination of the AR image with respect to the object on the basis of the inclination of the object acquired by the inclination acquisition unit (14).

Description

Display control device and display control method

The present invention relates to a display control device and a display control method for superimposing and displaying an AR (Augmented Reality) image on a real environment.

Some AR display systems that use AR technology that adds virtual objects to the real environment include ARs for objects determined based on information obtained from sensors, radars, GPS (Global Positioning System), and cameras. There is a system that displays images in a superimposed manner. An AR display system mounted on a vehicle uses an object as a preceding vehicle, detects a vehicle-to-vehicle distance between a host vehicle and a preceding vehicle by using a camera, etc. AR images representing the existence are overlaid (for example, see Patent Document 1).

JP 2004-58919 A

The conventional AR display system did not consider the inclination of the preceding vehicle when generating the AR image to be displayed according to the distance to the preceding vehicle. Therefore, when the preceding vehicle is obliquely facing the own vehicle, if the AR image generated by the AR display system is displayed as it is, the displayed AR image and the preceding vehicle do not have the same inclination, and There is a problem that the AR image is tilted. Therefore, in the conventional AR display system, the AR display feels strange.

The present invention has been made to solve the above problems, and an object thereof is to display an AR image in consideration of the inclination of an object.

The display control device according to the present invention includes a vehicle information acquisition unit that acquires information including the position and orientation of the vehicle as vehicle information, and a position of an object within a predetermined distance of the vehicle. An object information acquisition unit that acquires information as object information, a road shape acquisition unit that acquires the road shape of the road on which the object is located, and own vehicle information, object information, and a vehicle shape based on the road shape. A tilt acquisition unit that acquires the tilt of the target object, an AR image generation unit that generates an AR image that is displayed superimposed on the target object, and a target of the AR image based on the tilt of the target object acquired by the tilt acquisition unit. And an AR image control unit for changing the tilt of the display.

According to the present invention, since the AR image in which the inclination of the object is taken into consideration can be displayed, it is possible to reduce the discomfort caused by the deviation of the inclination between the object and the AR image.

3 is a block diagram showing a configuration example of a display control device according to the first embodiment. FIG. 2A and 2B are diagrams for explaining an example when the display control device according to the first embodiment changes the inclination of the AR image in accordance with the inclination of the target object, and is ahead of the orientation of the host vehicle. The case where the directions of vehicles are the same is shown. FIG. 3A and FIG. 3B are diagrams illustrating an example in which the display control device changes the inclination of the AR image according to the inclination of the target object, and when the orientation of the preceding vehicle is different from the orientation of the own vehicle. Indicates. FIG. 4A and FIG. 4B are diagrams for explaining an example when the display control device changes the inclination of the AR image according to the inclination of the object. Since the direction of the preceding vehicle is different from the direction of the own vehicle, The case where the inclination of the AR image is changed is shown. 5 is a flowchart showing an operation example of the display control device according to the first embodiment. FIG. 6A and FIG. 6B are diagrams for explaining a method of reducing the deviation of the inclination between the object and the AR image by the AR image control unit, in which the direction of the subject vehicle is the same as the direction of the own vehicle. This is an example of a case. FIG. 7A and FIG. 7B are diagrams for explaining a method for reducing the deviation of the inclination between the object and the AR image by the AR image control unit, and an example in the case where the direction of the preceding vehicle is different from the direction of the own vehicle. .. 7 is a flowchart showing an operation example of the display control device according to the second embodiment. 9A and 9B are diagrams illustrating a method in which the AR image control unit changes the inclination of the AR image according to the visibility of the AR image. This is an example when the directions are the same. 10A and 10B are diagrams illustrating a method in which the AR image control unit changes the inclination of the AR image according to the visibility of the AR image, and when the direction of the preceding vehicle is different from the direction of the own vehicle. An example is shown. 11A and 11B are diagrams for explaining a method in which the AR image control unit changes the inclination of the AR image according to the visibility of the AR image, and when the direction of the preceding vehicle is different from the direction of the own vehicle. Another example of FIG. 12A and FIG. 12B are diagrams showing the result of transforming the AR image in the case shown in FIG. 11. 13A and 13B describe an example in which the display control device according to the third embodiment changes the inclination of the AR image in accordance with the inclination of the moving body when the object is the moving body. FIG. 9 is a block diagram showing a configuration example of a display control device according to a third embodiment. 9 is a flowchart showing an operation example of the display control device according to the third embodiment. 16 is a flowchart showing details of operations in steps ST8 and ST4 of FIG. 15 when the object is a moving body. It is a figure explaining the determination method of a straight ahead area by a course prediction part, and is an example in which roads are connected at right angles. It is a figure explaining the determination method of a straight ahead area by a course prediction part, and is an example where roads are connected diagonally. It is a figure explaining the course prediction method of a course prediction part in case a preceding vehicle which is an object goes straight ahead at an intersection. It is a figure explaining the direction estimation method of the preceding vehicle by the inclination acquisition part in case the preceding vehicle which is an object goes straight ahead at an intersection. It is a figure explaining the course prediction method of a course prediction part in case a preceding vehicle which is an object turns right at an intersection. It is a figure explaining the direction estimation method of the preceding vehicle by the inclination acquisition part, when the preceding vehicle which is an object turns right at an intersection. It is a figure which shows an example of the hardware constitutions of the display control apparatus which concerns on each embodiment. It is a figure which shows another example of the hardware constitutions of the display control apparatus which concerns on each embodiment.

Hereinafter, in order to explain the present invention in more detail, modes for carrying out the present invention will be described with reference to the accompanying drawings.
Embodiment 1.
FIG. 1 is a block diagram showing a configuration example of the display control device 1 according to the first embodiment. The display control device 1 according to the first embodiment includes a vehicle information acquisition unit 11, an object information acquisition unit 12, a road shape acquisition unit 13, an inclination acquisition unit 14, an AR image generation unit 15, and an AR image control unit 16. I have it. A vehicle direction detection sensor 2, a navigation device 3, a driving support device 4, and a display device 5 are connected to the display control device 1. The own vehicle direction detection sensor 2 is a gyro sensor or the like, and detects the direction of the own vehicle and outputs it to the display control device 1. The navigation device 3 outputs the position information and map information of the own vehicle to the display control device 1. The driving support device 4 executes an ACC (Adaptive Cruise Control) function and the like, and outputs information regarding the function being executed to the display control device 1.

The display control device 1 is, for example, a device mounted in a vehicle or a portable terminal carried by a passenger and brought into the vehicle. The mobile terminal is, for example, a portable navigation device, a tablet PC (Personal Computer), or a smartphone.

The display control device 1 controls the display device 5 that superimposes and displays the AR image on the actual landscape (hereinafter, referred to as “background”), and displays the AR image in the background as an object to be superimposed and displayed (hereinafter, This is for allowing the user to visually recognize a video image that is displayed overlaid on the "target object").

Note that in Embodiment 1, the user is assumed to be a passenger of the vehicle. Further, the AR image is assumed to be a mark that points the target object to the user. Further, the vehicle on which the user is boarded is referred to as “own vehicle”. The object is a moving body such as another vehicle or a motorcycle that is visually recognized by the user, or a facility such as a building.

The display control device 1 changes the tilt of the AR image so that the AR image that matches the tilt of the object is displayed. The inclination of the object is the difference between the orientation of the host vehicle and the orientation of the object. The display control device 1 acquires the orientation of the vehicle from the vehicle direction detection sensor 2. Further, the display control device 1 determines the orientation of the target object based on the road shape at the position of the target object.

The display device 5 is, for example, a head-up display. The head-up display includes a liquid crystal display (not shown) and the like, and a "display surface 5a" such as a windshield or a combiner. This head-up display projects an AR image displayed on a liquid crystal display or the like onto a windshield or a combiner of the host vehicle to reflect the AR image, and allows a passenger to recognize the virtual image of the AR image. It is displayed superimposed on the background.

The display device 5 may be a liquid crystal display, an organic EL (Electroluminescence) display, or the like. In this case, the display control device 1 causes the display device 5 to display an image of a real landscape captured by a camera (not shown) and superimposes an AR image on the image of the real landscape. In this case, the image of the actual landscape corresponds to the “background”. A display unit such as a liquid crystal display or an organic EL display corresponds to the "display surface 5a".

Here, an example in which the display control device 1 changes the inclination of the AR image in accordance with the inclination of the object will be described with reference to FIGS. 2 to 4. The display control device 1 superimposes and displays the mark as an AR image 40 below the preceding vehicle 30 on the display surface 5a of the display device 5 so as to indicate the preceding vehicle 30 that is the object.

2 shows a case where the direction of the preceding vehicle 30 is the same as the direction 21 of the own vehicle 20, FIG. 2A is an overhead view, and FIG. 2B is a display example of the display surface 5a. In this case, the preceding vehicle 30 is not inclined with respect to the host vehicle 20. Further, the direction 21 of the host vehicle 20 and the background displayed on the display surface 5a are interlocked. Therefore, even if the display control device 1 displays the AR image 40 on the display surface 5a without changing the tilt 41, it does not change the tilt 41 of the AR image 40 because it matches the tilt 31 of the preceding vehicle 30.

FIG. 3 shows a case where the direction of the preceding vehicle 30 is different from the direction 21 of the own vehicle 20, FIG. 3A is an overhead view, and FIG. 3B is a display example of the display surface 5a. In this case, the preceding vehicle 30 is inclined with respect to the host vehicle 20. Therefore, when the display control device 1 displays the tilt 41 of the AR image 40 without changing the tilt 41, it does not match the tilt 31 of the preceding vehicle 30, and thus the tilt 41 of the AR image 40 needs to be changed as shown in FIG.

FIG. 4 shows a case where the inclination 41 of the AR image 40 is changed because the direction of the preceding vehicle 30 is different from the direction 21 of the own vehicle 20, FIG. 4A is an overhead view, and FIG. 4B is a table of the display surface 5a. It is an example. In this case, the display control device 1 changes the inclination 41 of the AR image 40 in the yaw direction of the preceding vehicle 30 so as to match the inclination 31 of the preceding vehicle 30, and displays the AR image 40a with the changed inclination 41a.

Next, the operation of the display control device 1 will be described.
FIG. 5 is a flowchart showing an operation example of the display control device 1 according to the first embodiment. The display control device 1 starts the operation shown in the flowchart of FIG. 5 when, for example, the ignition switch of the host vehicle is turned on, and repeats this operation until the ignition switch is turned off. Alternatively, the display control device 1 performs the operation shown in the flowchart of FIG. 5 when the navigation device 3 or the driving support device 4 gives an instruction to display an AR image.

In step ST1, the vehicle information acquisition unit 11 acquires vehicle information from the vehicle direction detection sensor 2 and the navigation device 3. The own vehicle information includes the position coordinates of the own vehicle and the direction of the own vehicle. The position coordinates of the host vehicle are represented by a three-dimensional coordinate system (X axis, Y axis, and Z axis) set in the real space. The orientation of the host vehicle includes at least one of pitch, roll, or yaw of the host vehicle in this three-dimensional coordinate system. Then, the vehicle information acquisition unit 11 outputs the acquired vehicle information to the object information acquisition unit 12 and the inclination acquisition unit 14.

In step ST2, the target object information acquisition unit 12 acquires information from the navigation device 3 and the driving support device 4, and displays an AR image within a predetermined distance of the vehicle on the basis of the acquired information in an overlapping manner. The relative position of the target object and the type of the target object are acquired. The relative position of the object is the position of the object based on the position of the host vehicle.

The target object information acquisition unit 12 acquires, from the navigation device 3, for example, information about a facility that a passenger of the vehicle can visually recognize while traveling or a facility that guides the passenger. In addition, the object information acquisition unit 12 acquires, from the driving support device 4, information on, for example, a preceding vehicle preceding in the same lane as the own vehicle. Which of the preceding vehicles and facilities and the like acquired from the navigation device 3 and the driving support device 4 is to be an object is determined according to the function provided to the passenger of the own vehicle, the object information acquisition unit 12, the navigation device 3 Alternatively, it may be determined by the driving support device 4. For example, when the navigation device 3 is guiding and guiding the planned traveling route of the own vehicle, the target object information acquisition unit 12 sets the facility to be guided to the passenger as a target object. Further, for example, when the driving support device 4 is executing the ACC function, the target object information acquisition unit 12 sets the preceding vehicle as a target object.

Subsequently, the object information acquisition unit 12 acquires the own vehicle information from the own vehicle information acquisition unit 11, and based on the position coordinates and the direction of the own vehicle included in the acquired own vehicle information and the relative position of the object. , Get the absolute position of the object. The absolute position of the object is the position coordinate on the map of the object, that is, the position coordinate in the three-dimensional coordinate system set in the real space. Then, the target object information acquisition unit 12 uses the acquired target object information including the position coordinates and the type of the target object as the road shape acquisition unit 13, the inclination acquisition unit 14, the AR image generation unit 15, and the AR image control. Output to the unit 16.

In step ST3, the road shape acquisition unit 13 acquires the object information from the object information acquisition unit 12 and the map information from the navigation device 3. The road shape acquisition unit 13 uses the map information acquired from the navigation device 3 to acquire the shape of the road corresponding to the position coordinates of the target object included in the target object information acquired from the target object information acquisition unit 12. The shape of the road corresponding to the position coordinates of the target object is information such as the curvature of the curve, the altitude, and the inclination degree in a predetermined range when the position coordinates of the target object and the map information are collated. is there. The predetermined range is, for example, a range of several meters around the object including the object. Then, the road shape acquisition unit 13 outputs the acquired road shape to the inclination acquisition unit 14.
When the object is a facility or the like and is on the side of the road instead of on the road, the road shape acquisition unit 13 uses the map information to calculate the curvature, altitude, and slope of the curve of the road close to the position coordinates of the facility or the like. The topographical information such as the degree may be acquired.

In step ST4, the inclination acquisition unit 14 acquires the object information from the object information acquisition unit 12 and the road shape from the road shape acquisition unit 13. The inclination acquisition unit 14 acquires the orientation of the target object based on the target object information acquired from the target object information acquisition unit 12 and the road shape acquired from the road shape acquisition unit 13. The orientation of the object is the traveling direction and posture of the object when the object travels along the shape of the road. The orientation of the object includes at least one of pitch, roll, or yaw. For example, when the road shape is a straight line, the inclination acquisition unit 14 sets the yaw direction of the target object to be the same as the direction of the road included in the map information. In addition, for example, when the road shape is a curve, the inclination acquisition unit 14 calculates the yaw direction of the target object from the curvature of the curve of the road included in the map information, and the tangent line at the position coordinate of the target object. The same as the direction of.

Subsequently, the inclination acquisition unit 14 acquires the own vehicle information from the own vehicle information acquisition unit 11, and determines the difference from the orientation of the object when the orientation of the own vehicle included in the obtained own vehicle information is used as a reference. The inclination of the object. Then, the tilt acquisition unit 14 outputs the acquired tilt of the object to the AR image control unit 16.

In step ST5, the AR image generation unit 15 acquires the target object information from the target object information acquisition unit 12 and generates an AR image based on the type of the target object included in the acquired target object information. The AR image is a graphic that indicates a part or the whole of the preceding vehicle when the preceding vehicle is being followed while the ACC function is being executed, for example. Hereinafter, the AR image will be described as being three-dimensional image data including three-dimensional information. For example, the AR image generation unit 15 holds a plurality of types of AR images in advance and selects an AR image according to the type of target object. Then, the AR image generation unit 15 outputs the generated AR image to the AR image control unit 16.

In step ST6, the AR image control unit 16 acquires the inclination of the object from the inclination acquisition unit 14 and acquires the AR image from the AR image generation unit 15. The AR image control unit 16 changes the tilt of the AR image, which is the three-dimensional image data acquired from the AR image generation unit 15, according to the direction and amount of the tilt of the target object acquired from the tilt acquisition unit 14. The tilt of the AR image is the pitch, roll, or yaw tilt of the AR image that is three-dimensional image data.

For example, assume that the vehicle width direction of the vehicle is equal to the X axis of the three-dimensional coordinate system set in the real space, the vertical direction is equal to the Y axis direction, and the longitudinal direction is equal to the Z axis. Then, it is assumed that the preceding vehicle, which is an object, is turning on a curve of the road. In this case, the object is tilted in the yaw direction with respect to the Y axis. Therefore, the AR image control unit 16 tilts the AR image, which is the three-dimensional image data, in the yaw direction with the Y axis as a reference.

Subsequently, the AR image control unit 16 acquires the target object information from the target object information acquisition unit 12, and based on the position coordinates of the target object included in the acquired target object information, the AR image after the tilt change is set as the target object. The position, size, etc. of the AR image, which is the two-dimensional image data, which is visually recognized through the display surface 5a so as to be displayed in a superimposed manner are changed. Then, the AR image control unit 16 outputs the AR image with the tilt, the position, the size, and the like changed to the display device 5. The display device 5 displays the AR image acquired from the AR image control unit 16 on the display surface 5a.

Here, with reference to FIGS. 6 and 7, a method of reducing the deviation of the inclination between the target object and the AR image by the AR image control unit 16 will be described.
6 shows a case where the direction 33 of the preceding vehicle 30, which is the object, is the same as the direction 21 of the host vehicle 20, FIG. 6A is a bird's-eye view, and FIG. 6B is a display example of the display surface 5a. In this case, the inclination acquisition unit 14 estimates the direction 33 of the preceding vehicle 30 from the direction of the road at the position 32 of the preceding vehicle 30. Then, the inclination acquisition unit 14 acquires the difference between the orientation 21 of the vehicle 20 and the orientation 33 of the preceding vehicle 30 included in the vehicle information as the inclination of the preceding vehicle 30. In the example of FIG. 6, since the orientation 21 of the host vehicle 20 and the orientation 33 of the preceding vehicle 30 are the same, the inclination of the preceding vehicle 30 is 0 degree. Therefore, the AR image control unit 16 does not change the inclination of the AR image 40.

FIG. 7 shows a case where the direction 33 of the preceding vehicle 30 is different from the direction 21 of the own vehicle 20, FIG. 7A is a bird's-eye view, and FIG. 7B is a display example of the display surface 5a. Here, the case where the shape of the road at the position 32 of the preceding vehicle 30 is a curve is shown. In this case, the inclination acquisition unit 14 determines that the preceding vehicle 30 is inclined along the shape of the curve. Then, the inclination acquisition unit 14 estimates the traveling direction of the preceding vehicle 30 traveling along the curve based on the curvature of the curve and the position 32 of the preceding vehicle 30, and sets it as the direction 33 of the preceding vehicle 30. Then, the inclination acquisition unit 14 acquires, as the inclination θ of the preceding vehicle 30, the difference between the direction 21 of the own vehicle 20 and the direction 33 of the preceding vehicle 30 included in the own vehicle information. The AR image control unit 16 tilts the AR image 40, which is three-dimensional image data, in the same direction as the tilt θ of the preceding vehicle 30 by θ degrees.

In the example of FIG. 7, the preceding vehicle 30 is tilted in the yaw direction with respect to the Y axis. The reference of the inclination θ of the preceding vehicle 30 is not limited to the Y axis, but may be the X axis (that is, when tilting in the pitch direction) or the Z axis (that is, when tilting in the roll direction). In that case, the inclination acquisition unit 14 may acquire the inclination θ of the preceding vehicle 30 based on information such as the altitude and inclination of the road included in the road shape.

As described above, the display control device 1 according to the first embodiment includes the vehicle information acquisition unit 11, the object information acquisition unit 12, the road shape acquisition unit 13, the inclination acquisition unit 14, the AR image generation unit 15, and the AR. The image control unit 16 is provided. The own vehicle information acquisition unit 11 acquires information including the position and direction of the own vehicle as the own vehicle information. The target object information acquisition unit 12 acquires, as the target object information, information including the position of the target object within a predetermined distance of the host vehicle. The road shape acquisition unit 13 acquires the road shape of the road on which the object is located. The inclination acquisition unit 14 acquires the inclination of the object with respect to the own vehicle based on the own vehicle information, the object information, and the road shape. The AR image generation unit 15 generates an AR image to be displayed over the target object. The AR image control unit 16 changes the display inclination of the AR image with respect to the object based on the inclination of the object acquired by the inclination acquisition unit 14. With this configuration, the display control device 1 can display the AR image in consideration of the inclination of the target object on the display device 5. As a result, it is possible to reduce the discomfort caused by the deviation of the inclination between the target object and the AR image. In the case of a configuration in which the inclination of the target object is estimated from the captured image and the AR image is changed according to the inclination of the target object, the processing load for image processing increases, but the display control device according to the first embodiment In No. 1, the processing load does not increase because the inclination of the object is not estimated from the captured image.

Embodiment 2.
In the display control device 1 according to the second embodiment, when changing the inclination of the AR image, the ratio of the display area of the AR image after the change to the display area of the AR image before the change is equal to or larger than a predetermined threshold value. To be Accordingly, the display control device 1 prevents the AR image from becoming hard to see by changing the inclination of the AR image.

Since the configuration of the display control device 1 according to the second embodiment is the same as the configuration shown in FIG. 1 of the first embodiment in the drawing, FIG. 1 is referred to below.
The AR image control unit 16 of the second embodiment changes the tilt of the AR image based on the tilt of the target object acquired from the tilt acquisition unit 14 as in the first embodiment. At this time, the AR image control unit 16 of the second embodiment sets the AR image so that the ratio of the display area of the AR image after the change to the display area of the AR image before the change becomes equal to or more than a predetermined threshold value. Change the inclination of. Hereinafter, the ratio of the display area of the AR image after the change to the display area of the AR image before the change will be referred to as a “visibility rate”.

FIG. 8 is a flowchart showing an operation example of the display control device 1 according to the second embodiment. After the operation in step ST5 in FIG. 5, the display control device 1 performs the operation in steps ST11 to ST18 in FIG. 8 instead of the operation in step ST6 in FIG.

In step ST11, when the AR image control unit 16 changes the tilt of the AR image, which is the three-dimensional image data acquired from the AR image generation unit 15, according to the tilt of the target object acquired from the tilt acquisition unit 14, The visibility of the AR image is calculated. As described above, the visibility is the ratio of the display area of the AR image after the change to the display area of the AR image before the change. The display area of the AR image is the area when the AR image, which is the three-dimensional image data, is displayed on the display surface 5a of the display device 5, that is, the area of the two-dimensional image data. The AR image control unit 16 calculates the display area of the AR image after the change by using the direction and amount of tilting the AR image.

In step ST12, the AR image control unit 16 determines whether or not the visibility of the AR image calculated in step ST11 is equal to or higher than a predetermined threshold value. The threshold value is set in advance based on the type of the target object and the like. Further, the threshold value may be different depending on whether the AR image is composed only of figures and the case where characters are included. Since it becomes difficult to see the character when the inclination is large, it is preferable that the threshold value when the AR image includes the character is larger than the threshold value when the AR image does not include the character.

When the visibility of the AR image is equal to or more than the threshold value (step ST12 “YES”), in step ST13, the AR image control unit 16 is three-dimensional image data corresponding to the inclination of the target object acquired by the inclination acquisition unit 14. Tilt the AR image. Further, as in step ST6 of FIG. 5, the AR image control unit 16 uses the display surface 5a so that the AR image after the tilt change is superimposed and displayed on the target object based on the position coordinates of the target object. The position and size of the visually recognized AR image are changed. Then, the AR image control unit 16 outputs the AR image with the tilt, the position, the size, and the like changed to the display device 5.

When the visibility of the AR image is less than the threshold value (step ST12 “NO”), in step ST14, the AR image control unit 16 adds the inclination of the target object acquired by the inclination acquisition unit 14 to a constant value in the same direction. The visibility of the AR image in the case of tilting by the amount of tilt is calculated.

In step ST15, the AR image control unit 16 determines whether the visibility rate of the AR image calculated in step ST14 is equal to or more than a threshold value. The threshold value of step ST15 is the same as the threshold value of step ST12.

When the visibility of the AR image is equal to or higher than the threshold value (step ST15 “YES”), the AR image control unit 16 sets the tilt of the target object acquired by the tilt acquisition unit 14 and the predetermined amount of tilt to 3 in step ST16. The AR image, which is three-dimensional image data, is tilted. Further, as in step ST6 of FIG. 5, the AR image control unit 16 uses the display surface 5a so that the AR image after the tilt change is superimposed and displayed on the target object based on the position coordinates of the target object. The position and size of the visually recognized AR image are changed. Then, the AR image control unit 16 outputs the AR image with the tilt, the position, the size, and the like changed to the display device 5.

When the visibility of the AR image is less than the threshold value (step ST15 “NO”), the AR image control unit 16 calculates the limit amount of inclination at which the visibility ratio is the same as the threshold value in step ST17. Specifically, the AR image control unit 16 uses the display area of the AR image before the change and the threshold value to calculate the display area of the AR image when the visibility is equal to the threshold value, and the display area is calculated. Use to calculate the limit of slope backwards.

In step ST18, the AR image control unit 16 tilts the AR image, which is the three-dimensional image data, by the limit amount of tilt calculated in step ST17. Further, as in step ST6 of FIG. 5, the AR image control unit 16 uses the display surface 5a so that the AR image after the tilt change is superimposed and displayed on the target object based on the position coordinates of the target object. The position and size of the visually recognized AR image are changed. Then, the AR image control unit 16 outputs the AR image with the tilt, the position, the size, and the like changed to the display device 5.

Here, a method in which the AR image control unit 16 changes the tilt of the AR image according to the visibility of the AR image will be described with reference to FIGS. 9 to 12. 9 to 12, it is assumed that the threshold value of the visibility rate is set to "50".
FIG. 9 shows a case where the direction 33 of the preceding vehicle 30, which is the object, is the same as the direction 21 of the host vehicle 20, FIG. 9A is an overhead view, and FIG. 9B is a display example of the display surface 5a. When the orientation 21 of the host vehicle 20 and the orientation 33 of the preceding vehicle 30 are the same, the AR image control unit 16 does not change the inclination of the AR image 40. Therefore, the display area of the AR image 40 before the change is the same value as the display area of the AR image after the change, and the visibility is “100”.

10 shows an example in which the direction 33 of the preceding vehicle 30 is different from the direction 21 of the own vehicle 20, FIG. 10A is a bird's eye view, and FIG. 10B is a display example of the display surface 5a. Here, it is assumed that the inclination of the preceding vehicle 30 is θ degrees, the display area of the AR image 40 before the change is “500”, and the display area of the AR image 40a after the inclination of θ degrees in the three-dimensional coordinate system is “400”. In this case, the visibility rate is 400/500×100=80, which is equal to or greater than the threshold value “50”. Therefore, the AR image control unit 16 tilts the AR image 40, which is the three-dimensional image data, by θ degrees to generate the AR image 40a.

FIG. 11 shows another example in which the direction 33 of the preceding vehicle 30 is different from the direction 21 of the host vehicle 20, and FIG. 12 shows the deformation result of the AR image in the case shown in FIG. 11A and 12A are overhead views, and FIGS. 11B and 12B are display examples of the display surface 5a. In FIG. 11, the inclination of the preceding vehicle 30 is θ degrees, the display area of the AR image 40 before the change is “500”, and the display area of the AR image 40a after the inclination of θ degrees in the three-dimensional coordinate system is “100”. .. In this case, the visibility rate is 100/500×100=20, which is less than the threshold value “50”. Assuming that a predetermined amount of tilting the AR image is predetermined to be 90 degrees, the AR image control unit 16 adds the tilt angle θ of the preceding vehicle 30 and the constant amount 90 degrees to the three-dimensional image by (θ+90) degrees. An AR image 40b is generated by inclining the AR image 40, which is data, and the display area of the AR image 40b is calculated. When the display area of the changed AR image 40b shown in FIG. 12 is “450”, the visibility is 450/500×100=90, which is equal to or more than the threshold “50”. Therefore, the AR image control unit 16 tilts the AR image 40, which is the three-dimensional image data, by (θ+90) degrees to generate the AR image 40b.

As described above, the AR image control unit 16 according to the second embodiment sets the ratio of the display area of the AR image after the change to the display area of the AR image before the change to a predetermined threshold value or more, The inclination of the display of the AR image with respect to the object is changed. With this configuration, the display control device 1 can reduce the sense of discomfort due to the deviation of the tilt between the target object and the AR image while maintaining the visibility rate equal to or higher than the threshold value.

Embodiment 3.
When the type of the object is a moving body, the display control device 1 according to the third embodiment predicts the path of the object and acquires the inclination of the object based on the path. Accordingly, the display control device 1 can change the inclination of the AR image according to the inclination of the target even when the inclination of the target cannot be acquired only by the road shape, such as when the target is at an intersection.

Here, with reference to FIG. 13, an example in which the display control device 1 changes the inclination of the AR image in accordance with the inclination of the moving body when the object is the moving body will be described. FIG. 13 shows a case where the preceding vehicle 30 as the object turns right at the intersection, FIG. 13A is a bird's-eye view, and FIG. 13B is a display example of the display surface 5a. The preceding vehicle 30 is turning to enter the road on the right turn destination, and the direction 33 of the preceding vehicle 30 is different from the direction 21 of the host vehicle 20. Therefore, the inclination acquisition unit 14 acquires the difference between the orientation 21 of the host vehicle 20 and the orientation 33 of the preceding vehicle 30 as the inclination θ of the preceding vehicle 30. The AR image control unit 16 tilts the AR image 40 by θ degrees in the same direction as the tilt of the preceding vehicle 30 to generate the AR image 40a.

FIG. 14 is a block diagram showing a configuration example of the display control device 1 according to the third embodiment. The display control device 1 according to the third embodiment has a configuration in which a route prediction unit 17 is added to the display control device 1 of the first embodiment shown in FIG. 14, parts that are the same as or correspond to those in FIG. 1 are assigned the same reference numerals and explanations thereof are omitted.

FIG. 15 is a flowchart showing an operation example of the display control device 1 according to the third embodiment. The display control device 1 starts the operation shown in the flowchart of FIG. 15 when the ignition switch of the own vehicle is turned on, for example, and repeats this operation until the ignition switch is turned off. Alternatively, the display control device 1 performs the operation illustrated in the flowchart of FIG. 15 when the navigation device 3 or the driving support device 4 issues an AR image display instruction.

The operation in steps ST1 to ST3 in FIG. 15 is the same as the operation in steps ST1 to ST3 in FIG. However, the target object information acquisition unit 12 outputs the target object information to the route prediction unit 17 (step ST2). The road shape acquisition unit 13 outputs the road shape to the route prediction unit 17 (step ST3).

In step ST7, the route prediction unit 17 acquires the target object information from the target object information acquisition unit 12 and determines whether the type of the target object included in the target object information is a moving body. When the object is a moving body (step ST7 “YES”), in step ST8, the route prediction unit 17 acquires the road shape from the road shape acquisition unit 13 and acquires the map information from the navigation device 3. Then, the route prediction unit 17 predicts the route of the target object based on the acquired target object information, road shape, and map information. Then, the route prediction unit 17 estimates the trajectory of the target object based on the predicted route of the target object, and outputs the estimated trajectory to the tilt acquisition unit 14.

When the object is a moving body (step ST7 “YES”), in step ST4, the inclination acquisition unit 14 acquires the trajectory of the object from the route prediction unit 17 and compares the acquired trajectory with the map information. , Get the tilt of the object. In step ST5, the AR image generation unit 15 generates an AR image. In step ST6, the AR image control unit 16 changes the AR image acquired from the AR image generation unit 15 based on the inclination of the target object acquired from the tilt acquisition unit 14, and the changed AR image is superimposed on the target object. After adjusting the position and size so as to be displayed, it is output to the display device 5.

If the object is not a moving body (step ST7 “NO”), the route prediction unit 17 does not perform the route prediction. In that case, the operation in steps ST4 to ST6 is the same as the operation in steps ST4 to ST6 in FIG.

FIG. 16 is a flowchart showing details of the operation in steps ST8 and ST4 of FIG. 15 when the object is a moving body.
In step ST21, the route prediction unit 17 determines whether or not the target object is at the intersection based on the target object information acquired from the target object information acquisition unit 12. Specifically, the route prediction unit 17 compares the position coordinates of the target object included in the target object information with the map information acquired from the navigation device 3, and determines the target object within the range of the intersection included in the map information. Whether or not the object is at the intersection is determined based on whether or not the position coordinates are present.

If the object is not at the intersection (step ST21 “NO”), the route prediction unit 17 outputs a notification that the object is not at the intersection to the inclination acquisition unit 14.

When the target object is at the intersection (step ST21 “YES”), in step ST22, the route prediction unit 17 determines the straight traveling area. The straight-ahead region is a range of a predetermined width (for example, 1/3 of the width of the lane) within the intersection centered on a straight line extending along the direction of the object just before the object enters the intersection. is there. The direction of the object immediately before entering the intersection is acquired by the route prediction unit 17 from the inclination acquisition unit 14 and stored, or estimated from the road shape acquired by the route prediction unit 17 from the navigation device 3.

17 and 18 are diagrams for explaining a method of determining a straight ahead region by the route predicting unit 17. At the intersection shown in FIG. 17, the roads 51 to 54 are connected at right angles to the other roads 51 to 54 adjacent to each other. At the intersection shown in FIG. 18, the road 53 is obliquely connected to the roads 52 and 54. Further, FIGS. 17 and 18 show a straight-ahead region 55 when the preceding vehicle 30 as an object enters the intersection from the road 53. At this time, the direction 33 of the preceding vehicle 30 immediately before entering the intersection is acquired by the route prediction unit 17 from the inclination acquisition unit 14 and stored, or is estimated from the shape of the road 53 by the route prediction unit 17. Then, the route prediction unit 17 determines, as the straight traveling region 55, a range of a predetermined width within the intersection centering on the straight line extending along the estimated direction 33 of the preceding vehicle 30.

In step ST23, the route prediction unit 17 determines whether or not the object is in the straight traveling region based on the straight traveling region determined in step ST22 and the target object information acquired from the target object information acquisition unit 12. Specifically, the route prediction unit 17 determines whether or not the target object is in the straight-ahead region based on whether or not the position coordinates of the target object included in the target object information are in the straight-ahead region.

If the target object is in the straight ahead area (step ST23 “YES”), in step ST31, the route prediction unit 17 determines that the target object goes straight at the intersection. Then, the route prediction unit 17 outputs to the tilt acquisition unit 14 a notification that the target object is going straight.

If the object is not in the straight traveling area (step ST23 “NO”), the route prediction unit 17 determines in step ST24 that the object turns right or left at the intersection. Then, the route prediction unit 17 determines whether the object is on the right side or the left side of the straight-ahead region.

When the target object is on the right side of the straight ahead area (step ST24 “YES”), the route prediction unit 17 determines in step ST25 that the target object turns right at the intersection.
In subsequent step ST26, the course predicting unit 17 determines the right turn road having the position coordinates closest to the position coordinates of the object among the right turn roads to the right of the straight ahead region as the course of the object. In addition, the route prediction unit 17 acquires the position coordinates and the direction of the right-turn road determined as the route of the object.

When the object is on the left side of the straight-ahead area (step ST24 “NO”), the route prediction unit 17 determines in step ST29 that the object turns left at the intersection.
In subsequent step ST30, the route prediction unit 17 determines the left-turn road having the position coordinates closest to the position coordinates of the target object among the left-turn roads on the left of the straight-ahead region as the route of the target object. In addition, the route prediction unit 17 acquires the position coordinates and the direction of the left turn road determined as the route of the object.

In step ST27, the route prediction unit 17 estimates the curve connecting the position of the object immediately before entering the intersection and the road determined as the route of the object as the trajectory of the object. More specifically, the route prediction unit 17 makes contact with a straight line extending with respect to the direction of the object immediately before the object enters the intersection and a straight line in the traveling direction of the road determined as the path of the object. The curvature of is calculated, and the curve having the calculated curvature is set as the trajectory of the object. The trajectory when the object turns right or left is not limited to a curved line, and may be a straight line depending on the road shape. Then, the route prediction unit 17 outputs the estimated trajectory of the target object to the inclination acquisition unit 14.

When acquiring the trajectory of the target object from the path prediction unit 17, the tilt acquisition unit 14 determines the orientation of the target object at the position of the target object based on the trajectory of the target object and the position coordinates of the target object in step ST28. presume. The orientation of the object is the traveling direction at the position of the object when the object travels along the predicted trajectory. When the trajectory of the target object is a straight line, the inclination acquisition unit 14 estimates the direction of the target object based on the straight line. When the trajectory of the object is a curve, the inclination acquisition unit 14 estimates the direction of the object based on the tangent line of the curve at the object position. Then, the inclination acquisition unit 14 acquires the inclination of the target object based on the orientation of the target object.

When the inclination acquisition unit 14 acquires from the route prediction unit 17 the notification that the target object is going straight, in step ST32, the direction of the target object immediately before the target object enters the intersection is the current target object direction. .. Then, the inclination acquisition unit 14 acquires the inclination of the target object based on the orientation of the target object.

When the inclination acquisition unit 14 acquires the notification that the target object is not at the intersection from the route prediction unit 17, the target object when the target object travels along the road shape acquired from the road shape acquisition unit 13 in step ST33. The traveling direction at the position is the direction of the object. Then, the inclination acquisition unit 14 acquires the inclination of the target object based on the orientation of the target object.

Note that the route prediction unit 17 and the inclination acquisition unit 14 may acquire the inclination that changes from moment to moment of the object traveling at the intersection by repeating the operations in steps ST21 to ST33. In that case, the route prediction unit 17 determines the straight-ahead region only when it first determines that the object is at the intersection (step ST21 “YES”) (step ST22), and the first-determined straight-ahead region after the second time. Should be used. In addition, the course prediction unit 17 may estimate the trajectory of the target object only when it first determines that the target object is outside the straight traveling region (step ST23 “NO”) (steps ST24 to ST27, ST29, ST30). .. Then, the tilt acquisition unit 14 may estimate the orientation of the target object by using the trajectory of the target object estimated first after the second time (step ST28).

FIG. 19 is a diagram illustrating a route prediction method of the route prediction unit 17 in the case where the preceding vehicle 30 as the target object goes straight through the intersection. FIG. 20 is a diagram illustrating a method of estimating the direction of the preceding vehicle 30 by the inclination acquisition unit 14 when the preceding vehicle 30 that is an object travels straight at an intersection. In FIG. 19 and FIG. 20, the preceding vehicle 30, which is the object, goes straight from the road 53 to the road 51 at the intersection.

First, the route prediction unit 17 compares the position 32 of the preceding vehicle 30 included in the object information with the intersection range of the map information acquired from the navigation device 3, and determines whether the preceding vehicle 30 is at the intersection. To do. In the example of FIG. 19, since the preceding vehicle 30 is at the intersection, the route prediction unit 17 determines the straight traveling area 55 at the intersection.

Subsequently, the route prediction unit 17 determines whether the preceding vehicle 30 is in the straight traveling area 55 of the intersection. In the example of FIG. 19, since the preceding vehicle 30 is in the straight-ahead region 55, the route prediction unit 17 determines that the preceding vehicle 30 is going straight at the intersection.

As shown in FIG. 20, the inclination acquisition unit 14 estimates the direction 33a of the preceding vehicle 30 immediately before entering the intersection as the current direction 33 of the preceding vehicle 30.

FIG. 21 is a diagram for explaining the route prediction method of the route prediction unit 17 when the preceding vehicle 30 as the object turns right at the intersection. FIG. 22 is a diagram illustrating a method of estimating the direction of the preceding vehicle 30 by the inclination acquisition unit 14 when the preceding vehicle 30 that is the object turns right at the intersection. 21 and 22, the preceding vehicle 30, which is the object, turns right at the intersection from the road 53 to the road 52.

First, the route prediction unit 17 compares the position 32 of the preceding vehicle 30 included in the object information with the intersection range of the map information acquired from the navigation device 3, and determines whether the preceding vehicle 30 is at the intersection. To do. In the example of FIG. 21, since the preceding vehicle 30 is at the intersection, the route prediction unit 17 determines the straight traveling area 55 at the intersection.

Subsequently, the route prediction unit 17 determines whether the preceding vehicle 30 is in the straight traveling area 55 of the intersection. In the example of FIG. 21, since the preceding vehicle 30 is not in the straight traveling region 55, the route prediction unit 17 determines that the preceding vehicle 30 turns right or left at the intersection. Then, the route prediction unit 17 determines whether the preceding vehicle 30 is on the right side or the left side of the straight traveling region 55. In the example of FIG. 21, since the preceding vehicle 30 is on the right side of the straight-ahead region 55, the route prediction unit 17 determines that the preceding vehicle 30 turns right at the intersection.

Subsequently, the route prediction unit 17 determines the right-turning road 52 closest to the preceding vehicle 30 as the route of the preceding vehicle 30. Then, the route prediction unit 17 calculates and calculates the curvature based on the straight line representing the direction of the preceding vehicle 30 immediately before entering the intersection and the straight line in the traveling direction of the road 52 determined as the route of the preceding vehicle 30. The curve which becomes the trajectory of the preceding vehicle 30 is estimated based on the curvature.

As shown in FIG. 22, the inclination acquisition unit 14 acquires a tangent line at the position 32 of the preceding vehicle 30 of the curve 33b representing the trajectory of the preceding vehicle 30 acquired from the route prediction unit 17. Then, the traveling direction when the preceding vehicle 30 travels along the tangent is estimated as the direction 33 of the preceding vehicle 30.

As described above, the display control device 1 according to the third embodiment includes the route prediction unit 17 that predicts the route of the object when the type of the object is a moving body. The inclination acquisition unit 14 acquires the inclination of the object with respect to the own vehicle based on the own vehicle information, the object information, and the course predicted by the course prediction unit 17. With this configuration, the display control device 1 makes the AR according to the inclination of the object even when the inclination of the object cannot be acquired from only the road shape, such as when the moving body that is the object is traveling at an intersection. You can change the image. This makes it possible to reduce the discomfort caused by the deviation of the tilt between the target object and the AR image regardless of the type and location of the target object.

Finally, it is a figure explaining the hardware constitutions of the display control apparatus 1 which concerns on each embodiment.
23 and 24 are diagrams showing a hardware configuration example of the display control device 1 according to each embodiment. The functions of the vehicle information acquisition unit 11, the object information acquisition unit 12, the road shape acquisition unit 13, the inclination acquisition unit 14, the AR image generation unit 15, the AR image control unit 16, and the route prediction unit 17 in the display control device 1 are: Is realized by the processing circuit. That is, the display control device 1 includes a processing circuit for realizing the above functions. The processing circuit may be the processing circuit 100 as dedicated hardware, or may be the processor 101 that executes a program stored in the memory 102.

As shown in FIG. 23, when the processing circuit is dedicated hardware, the processing circuit 100 includes, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, and an ASIC (Application Specific Integrated Circuit). ), FPGA (Field Programmable Gate Array), or a combination thereof. The functions of the own vehicle information acquisition unit 11, the target object information acquisition unit 12, the road shape acquisition unit 13, the inclination acquisition unit 14, the AR image generation unit 15, the AR image control unit 16, and the route prediction unit 17 have a plurality of processing circuits 100. May be realized by, or the functions of each unit may be collectively realized by one processing circuit 100.

As shown in FIG. 24, when the processing circuit is the processor 101, the vehicle information acquisition unit 11, the object information acquisition unit 12, the road shape acquisition unit 13, the inclination acquisition unit 14, the AR image generation unit 15, the AR image. The functions of the control unit 16 and the route prediction unit 17 are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 102. The processor 101 realizes the function of each unit by reading and executing the program stored in the memory 102. That is, the display control device 1 includes a memory 102 for storing a program that, when executed by the processor 101, results in the steps shown in the flowchart of FIG. 5 or the like being executed. In addition, this program is a procedure of the vehicle information acquisition unit 11, the object information acquisition unit 12, the road shape acquisition unit 13, the inclination acquisition unit 14, the AR image generation unit 15, the AR image control unit 16, and the course prediction unit 17. Alternatively, it can be said that the method causes a computer to execute the method.

Here, the processor 101 is a CPU (Central Processing Unit), a processing device, a computing device, a microprocessor, or the like.
The memory 102 may be a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), or a non-volatile or volatile semiconductor memory such as a flash memory, a hard disk, a flexible disk, or the like. Magnetic disk, or an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc).

The functions of the vehicle information acquisition unit 11, the object information acquisition unit 12, the road shape acquisition unit 13, the inclination acquisition unit 14, the AR image generation unit 15, the AR image control unit 16, and the route prediction unit 17 are partially described. May be realized by dedicated hardware, and a part thereof may be realized by software or firmware. As described above, the processing circuit in the display control device 1 can realize the above-described functions by hardware, software, firmware, or a combination thereof.

Note that, in the present invention, within the scope of the invention, it is possible to freely combine each embodiment, modify any constituent element of each embodiment, or omit any constituent element of each embodiment.

INDUSTRIAL APPLICABILITY The display control device according to the present invention is configured to reduce a sense of discomfort due to a deviation of the tilt between the target object and the AR image, and thus is suitable for use in a display control device for controlling the display of an AR display system mounted on a vehicle. Are suitable.

1 display control device, 2 vehicle direction detection sensor, 3 navigation device, 4 driving support device, 5 display device, 5a display surface, 11 vehicle information acquisition unit, 12 object information acquisition unit, 13 road shape acquisition unit, 14 Inclination acquisition unit, 15 AR image generation unit, 16 AR image control unit, 17 course prediction unit, 20 own vehicle, 21 orientation, 30 preceding vehicle (object), 31 inclination, 32 position, 33, 33a orientation, 33b curve, 40, 40a, 40b AR image, 41, 41a tilt, θ tilt.

Claims (5)

1. A vehicle information acquisition unit that acquires information including the position and orientation of the vehicle as vehicle information;
   An object information acquisition unit that acquires information including the position of an object within a predetermined distance of the vehicle as object information,
   A road shape acquisition unit that acquires the road shape of the road on which the object is located;
   A tilt acquisition unit that acquires the tilt of the object with respect to the vehicle based on the vehicle information, the object information, and the road shape;
   An AR image generation unit that generates an AR image to be displayed over the target object;
   An AR image control unit that changes an inclination of a display of the AR image with respect to the object based on the inclination of the object acquired by the inclination acquisition unit.
2. The display control device according to claim 1, wherein the object information includes a type of the object.
3. The AR image control unit with respect to the object of the AR image such that the ratio of the display area of the AR image after the change to the display area of the AR image before the change becomes equal to or more than a predetermined threshold value. The display control device according to claim 1, wherein a display inclination is changed.
4. A path prediction unit that predicts a path of the object when the type of the object is a moving body,
   The said inclination acquisition part acquires the inclination of the said object with respect to the said own vehicle based on the said vehicle information, the said object information, and the course estimated by the said course estimation part, The said 2nd characterized by the above-mentioned. Display controller.
5. The own vehicle information acquisition unit acquires information including the position and direction of the own vehicle as the own vehicle information,
   The object information acquisition unit acquires information including the position of the object within a predetermined distance of the vehicle as the object information,
   The road shape acquisition unit acquires the road shape of the road on which the object is located,
   An inclination acquisition unit acquires the inclination of the object with respect to the own vehicle based on the own vehicle information, the object information, and the road shape,
   An AR image generation unit generates an AR image to be displayed on the target object in an overlapping manner,
   A display control method in which an AR image control unit changes a display inclination of the AR image with respect to the object based on the inclination of the object acquired by the inclination acquisition unit.

Images