JP7484132B2 - Superimposed image display device and computer program - Google Patents

Description

The present invention relates to a superimposed image display device and a computer program that provide vehicle driving support.

Conventionally, various means have been used as information providing means for providing vehicle occupants with various information for driving support such as route guidance and warnings of obstacles. For example, this can be a display on a liquid crystal display installed in the vehicle, or a sound output from a speaker. In recent years, one such information providing means is a device that provides information by displaying an image superimposed on the driver's surroundings (landscape, real scene). For example, this includes head-up displays, windshield displays, and methods of displaying an image superimposed on a captured image of the vehicle's surroundings displayed on a liquid crystal display.

For example, Japanese Patent Application Laid-Open No. 2005-69800 discloses a technology that allows vehicle occupants to accurately grasp the location of an intersection where a right or left turn is to be made by superimposing a virtual image of a circle or a person on the location of the intersection in a head-up display device for a vehicle and making it visible.

JP 2005-69800 A (FIGS. 4 and 7)

Here, in the technology described in Patent Document 1, a virtual image of a circle or a person is displayed so as to be superimposed on the position of the guidance intersection where a right or left turn is to be made. However, there are often objects (e.g., artificial structures, other vehicles, pedestrians, etc.) around the guidance intersection where the image is to be superimposed, and in such cases the virtual image may be superimposed on the object. For example, as shown in FIG. 13, when a virtual image 101 of an arrow indicating a right turn at a guidance intersection ahead of the vehicle in the travel direction is displayed, if another vehicle 102 is present around the guidance intersection where the right turn is to be made, the virtual image 101 of the arrow will be superimposed on the other vehicle 102.

In that case, the virtual image of the arrow 101 is visually perceived as being in front of the other vehicle 102, so even though the exit road 103 of the guidance intersection indicated by the virtual image of the arrow 101 is behind the other vehicle 102, it is mistakenly recognized as a right turn in front of the other vehicle 102, or the position to turn right cannot be identified. The same problem occurs when the image of the arrow is superimposed on a captured image of the vehicle's surroundings displayed on a liquid crystal display.

The present invention has been made to solve the above-mentioned problems in the conventional technology, and aims to provide a superimposed image display device and computer program that enables accurate recognition of the positional relationship between the guide image and the object when the guide image is superimposed on the scenery in front of the vehicle, and enables more appropriate guidance using the guide image.

In order to achieve the above object, a first superimposed image display device according to the present invention comprises: guide image arranging means for arranging a guide image in a three-dimensional space corresponding to the periphery of a vehicle; virtual body arranging means for arranging a virtual body simulating an object existing in the periphery of the vehicle at a position in the three-dimensional space where the object exists; viewpoint setting means for setting a viewpoint at a position in the three-dimensional space corresponding to the current position of the vehicle; shape acquisition means for acquiring shapes of the guide image and the virtual body as viewed from the viewpoint with the guide image and the virtual body arranged in the three-dimensional space; and an image display means for displaying the guide image having the shape acquired by the shape acquisition means and an object image which is a black image showing the shape of the virtual body acquired by the shape acquisition means as a single image on an image display screen provided in the vehicle, and for reflecting the guide image and the object image which are single images displayed on the image display screen onto the windshield of the vehicle so that they are visible to an occupant of the vehicle, thereby superimposing a virtual image of the guide image on the scenery in front of the vehicle and allowing the occupant of the vehicle to view it, and for superimposing a virtual image of the object image at the position of the object in the scenery in front of the vehicle in a manner that cannot be viewed by the occupant of the vehicle.
In addition, a second superimposed image display device according to the present invention has a guide image placement means for placing a guide image in a three-dimensional space corresponding to the periphery of the vehicle, a virtual body placement means for placing a virtual body simulating an object present in the periphery of the vehicle at a position in the three-dimensional space where the object is present, a viewpoint setting means for setting a viewpoint at a position in the three-dimensional space corresponding to the current position of the vehicle, a shape acquisition means for acquiring the shapes of the guide image and the virtual body viewed from the viewpoint with the guide image and the virtual body placed in the three-dimensional space, and an image display means for displaying the guide image having the shape acquired by the shape acquisition means and an object image which is a transparent color image showing the shape of the virtual body acquired by the shape acquisition means as an integrated image on an image display screen provided in the vehicle and displaying a scenery in front of the vehicle, thereby superimposing the guide image on the scenery in front of the vehicle so as to be visible to an occupant of the vehicle, and superimposing the object image at the position of the object in the scenery in front of the vehicle in a manner that cannot be seen by the occupant of the vehicle.
The term "target object" includes stationary objects and moving objects around the vehicle. Stationary objects include, for example, artificial structures (buildings, road signs, billboards, etc.) and natural objects (trees, etc.), while moving objects include, for example, other vehicles, people, bicycles, etc.
Further, the "scenery" includes not only the scenery actually viewed from the vehicle (actual scenery), but also an image of the scenery captured, an image of the scenery reproduced, and the like.

A first computer program according to the present invention is a program for providing vehicle driving support. Specifically, the computer functions as: a guide image arrangement means for arranging a guide image in a three-dimensional space corresponding to the periphery of the vehicle; a virtual body arrangement means for arranging a virtual body simulating an object present in the periphery of the vehicle at a position in the three-dimensional space where the object exists; a viewpoint setting means for setting a viewpoint at a position in the three-dimensional space corresponding to the current position of the vehicle; a shape acquisition means for acquiring the shapes of the guide image and the virtual body viewed from the viewpoint in a state in which the guide image and the virtual body are arranged in the three-dimensional space ; and an image display means for displaying the guide image having the shape acquired by the shape acquisition means and an object image which is a black image showing the shape of the virtual body acquired by the shape acquisition means as an integrated image on an image display screen provided in the vehicle, and reflecting the guide image and the object image which are integrated images displayed on the image display screen onto the windshield of the vehicle so that the occupants of the vehicle can view them, thereby superimposing a virtual image of the guide image on the scenery ahead of the vehicle and allowing them to be viewed, and superimposing a virtual image of the object image on the position of the object in the scenery ahead of the vehicle in a manner that cannot be viewed by the occupants of the vehicle.
A second computer program according to the present invention is a program for providing vehicle driving support. Specifically, the computer is caused to function as a guide image arrangement means for arranging a guide image in a three-dimensional space corresponding to the periphery of the vehicle, a virtual body arrangement means for arranging a virtual body simulating an object existing in the periphery of the vehicle at a position in the three-dimensional space where the object exists, a viewpoint setting means for setting a viewpoint at a position in the three-dimensional space corresponding to the current position of the vehicle, a shape acquisition means for acquiring shapes of the guide image and the virtual body viewed from the viewpoint in a state in which the guide image and the virtual body are arranged in the three-dimensional space , and an image display means for displaying the guide image having the shape acquired by the shape acquisition means and an object image which is a transparent color image showing the shape of the virtual body acquired by the shape acquisition means as an integrated image on an image display surface provided in the vehicle and displaying a scene ahead of the vehicle, thereby superimposing the guide image on the scene ahead of the vehicle so as to be viewed by a vehicle occupant, and superimposing the object image on the position of the object in the scene ahead of the vehicle in a manner that the vehicle occupant cannot view it.

According to the first and second superimposed image display devices and computer programs of the present invention having the above configurations, when a guide image is superimposed on the scenery ahead of the vehicle and is visually recognized, the guide image is prevented from being visually recognized by the vehicle occupants in the area where it overlaps with an object located closer to the vehicle than the guide image, so that the vehicle occupants can be made to visually recognize a guide image that accurately shows the positional relationship between the guide image and the object. As a result, guidance using a virtual image of the guide image can be more appropriately performed.

1 is a schematic configuration diagram of a superimposed image display device according to a first embodiment. 1 is a block diagram showing a navigation device according to a first embodiment. 5 is a flowchart of a driving support processing program according to the first embodiment. FIG. 2 is a diagram showing a three-dimensional relief map generated based on three-dimensional map information. FIG. 11 is a diagram showing an example of a guide image arranged in three-dimensional map information. FIG. 10 is a diagram showing an example of a target virtual object arranged in three-dimensional map information. 11 is a diagram showing the shape of a guide image visually recognized from a vehicle position in three-dimensional map information and the shape of an overlapping object virtual body. FIG. 11A and 11B are diagrams showing examples of displaying a guide image and a black image. 11A and 11B are diagrams showing examples of virtual images of a guide image visually recognized by a vehicle occupant. 13 is a flowchart of a sub-processing program of the display range determination process. 13 is a diagram showing an example of a guide image displayed on a liquid crystal display in the superimposed image display device according to the second embodiment. FIG. 11A and 11B are diagrams showing examples of display of a guide image and a transparent image. FIG. 1 is a diagram illustrating problems with the conventional technology.

Hereinafter, a first embodiment and a second embodiment of the superimposed image display device according to the present invention will be described in detail with reference to the drawings.

[First embodiment]
First, a schematic configuration of a superimposed image display device 1 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic configuration diagram of the superimposed image display device 1 according to the first embodiment.

As shown in FIG. 1, the superimposed image display device 1 basically comprises a navigation device 3 mounted on a vehicle 2, and a front display 4 also mounted on the vehicle 2 and connected to the navigation device 3. The front display 4 functions as a head-up display together with a windshield 5 of the vehicle 2 as described below, and serves as an information providing means for providing various information to an occupant 6 of the vehicle 2.

The navigation device 3 has functions to search for a recommended route to the destination, display a map image of the area around the current position of the vehicle 2 based on map data acquired from a server or stored in memory, and provide driving guidance along a set guide route and warnings of obstacles together with the front display 4. Note that the navigation device 3 does not need to have all of the above functions, and the present invention can be constituted as long as it has at least the function of providing driving guidance along a guide route and warnings of obstacles. Note that the structure of the navigation device 3 will be described in detail later.

On the other hand, the front display 4 is a liquid crystal display that is installed inside the dashboard 7 of the vehicle 2 and has the function of displaying images on an image display surface provided on the front. For example, a CCFL (cold cathode fluorescent lamp) or a white LED is used as the backlight. Note that, in addition to a liquid crystal display, the front display 4 may also be an organic electroluminescence display or a combination of a liquid crystal projector and a screen.

The front display 4 functions as a head-up display together with the windshield 5 of the vehicle 2, and is configured to reflect images output from the front display 4 on the windshield 5 in front of the driver's seat so that the images can be viewed by the occupant 6 of the vehicle 2. The images displayed on the front display 4 include information about the vehicle 2 and various information used to assist the occupant 6 in driving. For example, the images include warnings for objects that should be warned to the occupant 6 (other vehicles, pedestrians, guide signs), guide routes set by the navigation device 3 and guide information based on the guide routes (arrows indicating the direction to turn right or left, etc.), warnings displayed on the road surface (collision warning, speed limit, etc.), lane markings on the lane the vehicle is traveling in, current vehicle speed, shift position, remaining energy, advertising images, facility information, map images, traffic information, news, weather forecasts, time, the screen of a connected smartphone, television programs, etc.

When the occupant 6 sees the image displayed on the front display 4 reflected by the windshield 5, the image displayed on the front display 4 is perceived by the occupant 6 not at the position of the windshield 5, but at a distant position beyond the windshield 5, as a virtual image 10. The virtual image 10 is displayed superimposed on the surrounding environment (landscape, real scene) in front of the vehicle, and can be displayed superimposed on any object (road surface, building, object that is the subject of a warning, etc.) located in front of the vehicle.

The position where the virtual image 10 is generated, more specifically the distance L from the occupant 6 to the virtual image 10 (hereinafter referred to as the imaging distance), is determined by the position of the front display 4. For example, the imaging distance L is determined by the distance (optical path length) along the optical path from the position where the image is displayed on the front display 4 to the windshield 5. For example, the optical path length is set so that the imaging distance L is 1.5 m.

In the first embodiment, the front display 4 installed inside the dashboard 7 is used as a means for displaying an image to be superimposed on the scenery around the vehicle, but other means may be used. For example, a head-up display (HUD) device may be provided as an in-vehicle device, or a windshield display (WSD) that displays an image on the windshield 5 may be used. In the WSD, the windshield 5 may be used as a screen to display an image from a projector, or the windshield 5 may be a translucent liquid crystal display. The image displayed on the windshield 5 by the WSD is an image that is superimposed on the scenery around the vehicle, similar to a HUD.

In addition, it is possible to display the scenery around the vehicle captured by the front camera 11 (described below) on an LCD display inside the vehicle, and to display an image that is superimposed on the scenery displayed on the same LCD display. In this case, the image displayed on the LCD display will be an image that is superimposed on the scenery around the vehicle, just like the HUD.

A front camera 11 is also installed above the vehicle's front bumper or behind the rearview mirror. The front camera 11 is an imaging device having a camera using a solid-state imaging element such as a CCD, and is installed with its optical axis facing forward in the direction of travel of the vehicle. Image processing is then performed on the image captured by the front camera 11 to detect the condition of the forward environment (i.e. the environment on which the virtual image 10 is superimposed) as viewed by the occupant 6 through the windshield. Note that a sensor such as a millimeter wave radar may be used instead of the front camera 11.

An in-vehicle camera 12 is also installed on the top surface of the vehicle's instrument panel. The in-vehicle camera 12 is an imaging device having a camera using a solid-state imaging element such as a CCD, and is installed with its optical axis direction facing the driver's seat. The range in the vehicle where the face of an occupant is generally expected to be located is set as the detection range (imaging range of the in-vehicle camera 12), and an image of the face of the occupant 6 sitting in the driver's seat is captured. Then, image processing is performed on the captured image captured by the in-vehicle camera 12 to detect the positions of the left and right eyes of the occupant 6 (starting point of gaze) and the gaze direction.

Next, the schematic configuration of the navigation device 3 constituting the superimposed image display device 1 will be described with reference to FIG. 2. FIG. 2 is a block diagram showing the navigation device 3 according to the first embodiment.

As shown in FIG. 4, the navigation device 3 according to the first embodiment includes a current position detection unit 13 for detecting the current position of the vehicle 2 on which the navigation device 3 is mounted, a data recording unit 14 for recording various data, a navigation ECU 15 for performing various calculation processes based on input information, an operation unit 16 for accepting operations from the user, a liquid crystal display 17 for displaying to the user a map of the area around the vehicle and facility information related to facilities, a speaker 18 for outputting voice guidance related to route guidance, a DVD drive 19 for reading a DVD, which is a storage medium, and a communication module 20 for communicating with an information center such as a VICS (registered trademark: Vehicle Information and Communication System) center. The navigation device 3 is also connected to the front display 4, front camera 11, and in-vehicle camera 12, etc., described above, via an in-vehicle network such as a CAN.

Each of the components of the navigation device 3 will be described below in order.
The current position detection unit 13 is composed of a GPS 21, a vehicle speed sensor 22, a steering sensor 23, a gyro sensor 24, etc., and is capable of detecting the current vehicle position, direction, vehicle running speed, current time, etc. Here, the vehicle speed sensor 22 in particular is a sensor for detecting the travel distance and vehicle speed of the vehicle, and generates pulses in response to the rotation of the drive wheels of the vehicle and outputs the pulse signal to the navigation ECU 15. The navigation ECU 15 then calculates the rotation speed and travel distance of the drive wheels by counting the generated pulses. It is not necessary for the navigation device 3 to be equipped with all of the above four types of sensors, and the navigation device 3 may be configured to be equipped with only one or a plurality of these types of sensors.

The data recording unit 14 also includes a hard disk (not shown) as an external storage device and recording medium, and a recording head (not shown) which is a driver for reading the map information DB 31 and predetermined programs recorded on the hard disk and writing predetermined data to the hard disk. The data recording unit 14 may have a flash memory, a memory card, or an optical disk such as a CD or DVD instead of a hard disk. The map information DB 31 may also be stored on an external server, and the navigation device 3 may acquire it through communication.

Here, the map information DB 31 stores two-dimensional map information 33 and three-dimensional map information 34. The two-dimensional map information 33 is map information used in general navigation devices 3, and includes, for example, link data related to roads (links), node data related to node points, facility data related to facilities, search data used in route search processing, map display data for displaying the map, intersection data related to each intersection, search data for searching for locations, etc.

On the other hand, the three-dimensional map information 34 has not only planar information but also height information, and is map information for expressing a map in three dimensions. In particular, in the first embodiment, the map information is for expressing road contours, building shapes, road division lines, traffic lights, road signs, signboards, etc. in three dimensions. The three-dimensional map information 34 may also include information other than the road contours, building shapes, road division lines, traffic lights, road signs, and signboards. For example, it may also include information for expressing road trees and road markings in three dimensions. In addition, the three-dimensional map information 34 may store the map itself in which each object such as the road contours, building shapes, road division lines, traffic lights, road signs, and signboards is arranged in a three-dimensional space, or it may store information necessary for expressing a map in three dimensions (such as three-dimensional coordinate data of road contours, building shapes, road division lines, traffic lights, road signs, signboards, etc.). When the information necessary to represent a map in three dimensions is stored, the navigation device 3 uses the information stored as three-dimensional map information 34 at the required timing to generate a map that represents the target area in three dimensions.

The navigation device 3 uses the two-dimensional map information 33 for general functions such as displaying a map image on the liquid crystal display 17 and searching for a guide route. As described below, the navigation device 3 also uses the three-dimensional map information 34 in addition to the two-dimensional map information 33 for processing related to displaying a guide image.

On the other hand, the navigation ECU (electronic control unit) 15 is an electronic control unit that controls the entire navigation device 3, and includes a CPU 41 as a calculation device and a control device, a RAM 42 that is used as a working memory when the CPU 41 performs various calculation processes and stores route data when a route is searched, a ROM 43 that stores a driving support processing program (FIG. 3) described later as well as a control program, and a flash memory 44 that stores a program read from the ROM 43. The navigation ECU 15 has various means as processing algorithms. For example, the guide image placement means places a guide image in a three-dimensional space corresponding to the periphery of the vehicle. The virtual body placement means places a virtual body that imitates an object existing around the vehicle at the position where the object exists in the three-dimensional space. The viewpoint setting means sets a viewpoint at a position in the three-dimensional space corresponding to the current position of the vehicle. The shape acquisition means acquires the shape of the guide image and the virtual body that are visually recognized from the viewpoint. The guidance image display means displays a guidance image of the shape acquired by the shape acquisition means superimposed on the scenery ahead of the vehicle. The object image display means displays an object image, which is an image showing the shape of the virtual object acquired by the shape acquisition means, superimposed on the scenery ahead of the vehicle in a manner that is not visible to the vehicle occupants.

The operation unit 16 is operated when inputting a departure point as the starting point of the journey and a destination point as the end point of the journey, and has a number of operation switches (not shown) such as various keys and buttons. The navigation ECU 15 controls the execution of various corresponding operations based on switch signals output by pressing each switch. The operation unit 16 may have a touch panel provided on the front of the liquid crystal display 17. It may also have a microphone and a voice recognition device.

The LCD display 17 also displays map images including roads, traffic information, operation guidance, operation menus, key guidance, guided route from the departure point to the destination, guidance information along the guided route, news, weather forecasts, time, emails, television programs, etc. In the first embodiment, the front display 4 is provided as a means for displaying information, so the LCD display 17 may be omitted if the map images and the like are displayed on the front display 4.

The speaker 18 also outputs voice guidance to guide the driver along the guide route based on instructions from the navigation ECU 15, as well as traffic information.

The DVD drive 19 is a drive capable of reading data recorded on recording media such as DVDs and CDs. Based on the read data, music and video are played, and the map information DB 31 is updated. Instead of the DVD drive 19, a card slot for reading and writing to a memory card may be provided.

The communication module 20 is a communication device for receiving traffic information, such as congestion information, regulation information, and traffic accident information, transmitted from a traffic information center, such as a VICS center or a probe center, and is, for example, a mobile phone or DCM.

Next, the driving support processing program executed in the navigation device 3, particularly in the superimposed image display device 1 having the above configuration, will be described with reference to FIG. 3. FIG. 3 is a flowchart of the driving support processing program according to the first embodiment. Here, the driving support processing program is executed after the vehicle's ACC power (accessory power supply) is turned on, and is a program that provides various information to the vehicle occupants by superimposing an image displayed on the front display 4 on the surrounding environment around the vehicle and making it visible. The programs shown in the flowcharts in FIG. 3 and FIG. 10 below are stored in the RAM 42 and ROM 43 of the navigation device 3, and are executed by the CPU 41.

The following description will be directed to an example in which the front display 4 is used to provide a guide arrow that indicates the direction of travel of the vehicle along the guide route set by the navigation device 3. However, it is also possible to provide guidance and information other than the above. For example, it is possible to display warnings to the occupants 6 of the vehicle 2 about objects that are the subject of a warning (e.g., other vehicles, pedestrians, guide signs), warnings displayed on the road surface (collision warning, speed limits, etc.), lane markings on which the vehicle is traveling, advertising images, facility information, map images, traffic information, news, weather forecasts, time, the screen of a connected smartphone, television programs, etc.

First, in the driving assistance processing program, in step (hereinafter abbreviated as S) 1, the CPU 41 acquires an image of the scenery in front of the vehicle captured by the front camera 11. Furthermore, the CPU 41 reads out three-dimensional map information 34 of the surroundings of the vehicle's current position from the map information DB 31. The three-dimensional map information 34 stores information necessary to express a map in three dimensions (such as three-dimensional coordinate data for road contours, building shapes, road dividing lines, traffic lights, road signs, billboards, etc.).

Next, in S2, the CPU 41 generates a three-dimensional relief map (a map that shows buildings, roads, etc. in three dimensions) showing the area around the current position of the vehicle based on the three-dimensional map information 34 read in S1. Specifically, the three-dimensional relief map is generated by executing the following process.

First, the CPU 41 acquires objects (shape data) that model roads, buildings, road signs, signs, etc. in a three-dimensional space from the three-dimensional map information 34. The CPU 41 may be configured to acquire the relevant object from among objects that have been modeled in advance and stored in the DB, or may be configured to perform modeling in S2 and create a new object. When performing modeling, information that specifies the shape and position of the road and structures around the road (buildings, road signs, signs, etc.) is acquired from the three-dimensional map information 34, and modeling processing is performed based on the acquired information. However, it is not necessary to model all roads, buildings, road signs, and signs; for example, only the roads, or only the roads and buildings may be modeled.

Here, modeling refers to the process of creating the shape of a model (object) in three-dimensional space, and more specifically, involves determining the coordinates of each vertex, and the parameters of equations that represent boundaries and surfaces. Note that modeling is a well-known technique, so details will be omitted. Depending on the application, the modeled object (shape data) is then expressed in the form of a "wireframe model" that displays only the edges, or a "surface model" that displays the surfaces, etc. The three-dimensional space in which each object is formed is then considered a three-dimensional relief map.

In addition, in S2, the CPU 41 may generate a three-dimensional space corresponding to the surroundings of the current position of the vehicle (particularly forward in the direction of vehicle travel) based on general two-dimensional map information 33 and the current position of the vehicle, without using the three-dimensional map information 34, to create a three-dimensional solid map. Also, the three-dimensional space may be generated based on an image captured by the front camera 11. For example, by performing point cloud matching on the image captured by the front camera 11, it is possible to detect structures on the road and around the road and generate a three-dimensional space.

In addition, in S2, the CPU 41 also identifies the current position and orientation of the vehicle on the generated 3D map based on the parameters detected by the current position detection unit 13. In order to facilitate the process of matching the captured image with the 3D map described below, it is preferable that the current position of the vehicle is the installation position of the front camera 11 installed on the vehicle, and the orientation of the vehicle is the optical axis direction of the front camera 11. In addition, the current position and orientation of the vehicle in the 3D space are updated as appropriate as the vehicle moves.

The CPU 41 also detects the positions and shapes of stationary and moving objects in the vicinity in front of the vehicle by performing image processing on the captured image of the scenery in front of the vehicle acquired in S1. Stationary objects include, for example, artificial structures (buildings, road signs, billboards, etc.) and natural objects (trees, etc.), while moving objects include, for example, other vehicles, people, bicycles, etc. If a stationary or moving object not included in the three-dimensional map information 34 is detected, information specifying the position and shape of the detected stationary or moving object is included in the three-dimensional map. In S2, it is not necessary to model the detected stationary or moving object on the three-dimensional map, but modeling may be performed.

Here, FIG. 4 is a diagram showing an example of the three-dimensional relief map 51 generated in S2. As shown in FIG. 4, on the three-dimensional relief map 51, objects 52 indicating roads and structures (buildings, road signs, billboards, etc.) are arranged in three-dimensional space. In particular, objects 52 indicating roads and structures located around the current position of the vehicle are arranged. In addition, a vehicle position mark 53 indicating the current position and direction of the vehicle is also arranged on the three-dimensional relief map 51.

Next, in S3, the CPU 41 compares the captured image acquired in S1 with the 3D map generated in S2 to determine whether there is any misalignment between the two. Specifically, the CPU 41 sets the current position of the vehicle (more specifically, the installation position of the front camera 11, taking into account the height) set on the 3D map as the viewpoint, sets the orientation of the vehicle as the line of sight, and compares the image when viewing the 3D map from the set viewpoint and line of sight with the captured image. Note that the 3D map does not include moving objects such as pedestrians and other vehicles, and some fixed objects such as trees, so the determination is basically made while ignoring any misalignment caused by these. In addition, the absence of misalignment is not limited to the case where the two completely match, but should desirably be considered to be absent even when there is a misalignment within a certain tolerance range.

If it is determined that there is no misalignment between the captured image acquired in S1 and the 3D relief map generated in S2 (S3: YES), the process proceeds to S5. On the other hand, if it is determined that there is a misalignment between the captured image acquired in S1 and the 3D relief map generated in S2 (S3: NO), the process proceeds to S4.

In S4, the CPU 41 corrects the current position and orientation of the vehicle set in the 3D map generated in S2 so as to reduce the discrepancy between the captured image acquired in S1. The current position and orientation of the vehicle in the 3D map may be fixed and the object side may be corrected. This makes it possible to identify the accurate current position and orientation of the vehicle (more specifically, the installation position and optical axis direction of the front camera 11) in the 3D map. Then, the process returns to S3.

On the other hand, in S5, the CPU 41 acquires the guide route set in the navigation device 3, and generates a guide arrow indicating the future travel direction of the vehicle based on the 3D map generated in S2 as a guide image to be displayed on the front display 4. In the first embodiment, the guide image is an arrow image placed on the road surface along the guide route of the vehicle (for example, on the road surface of the road up to 100 m ahead from the current position of the vehicle). For example, as shown in FIG. 5, if the future travel direction of the vehicle is to turn right at the guide intersection 54 ahead in the travel direction, an arrow indicating that the vehicle will proceed straight to the guide intersection 54, turn right at the guide intersection 54, and proceed to the exit road is generated as the guide image 55. The shape of the guide image can be changed as appropriate, and for example, it may be a line segment placed along the guide route instead of an arrow.

Next, in S6, the CPU 41 places the guidance image generated in S5 on the three-dimensional map generated in S2. The position at which the guidance image is placed is a position at which the guidance image is superimposed and visible, for example, on the road surface up to 100 m ahead from the current position of the vehicle along the guidance route (including the guidance intersection, if any). As a result, the guidance image 55 is placed on the three-dimensional map 51 as shown in FIG. 5.

Next, in S7, the CPU 41 extracts, from among the objects (buildings, road signs, billboards, guide images, pedestrians, other vehicles, guide images, etc.) included in the 3D map, objects other than the guide images as objects present around the vehicle. Note that objects include stationary objects such as artificial structures (buildings, road signs, billboards, etc.) and natural objects (trees, etc.), as well as moving objects such as other vehicles, people, and bicycles. However, roads (road surfaces) are excluded from the objects to be extracted.

Next, in S8, the CPU 41 places the model of the object extracted in S7 (a virtual object of the object, hereinafter referred to as the virtual object object) on the 3D map generated in S2 at the position where the object is located. Information specifying the position of the object is stored in the 3D map, including cases where the object is a moving object. The virtual object object placed on the 3D map has a shape that imitates the shape of the actual object. For example, if it is a pedestrian, the virtual object object will have a human shape, and if it is a vehicle, the virtual object object will have a vehicle shape. The size also corresponds to the size of the actual object.

However, taking into consideration that the position of the object visible to the occupants may be distorted due to vibrations occurring in the vehicle, etc., it is desirable to make the size of the virtual object body slightly larger than the actual size of the object (for example, 110%). Also, rather than expanding the size uniformly overall, it is desirable to expand the size significantly in the relative direction of movement (the direction in which the object moves as seen by the vehicle occupants). Note that, as an exception, if the object is shapeless (its shape and type cannot be identified), the virtual object body is made the smallest rectangular parallelepiped that contains the entire object.

Next, in S9, the CPU 41 first acquires an image (hereinafter referred to as a visual image) of the 3D map on which the guide image and virtual object are arranged, visually recognized in the traveling direction of the vehicle from the viewpoint of the vehicle (passenger). In particular, the viewpoint of the vehicle is the position of the eyes of the vehicle occupant. The position of the eyes of the occupant can be detected by the in-vehicle camera 12. That is, the acquired visual image is an image that can be visually recognized when each object (guide image, virtual object, etc.) arranged on the 3D map is visually recognized in the traveling direction of the vehicle from the viewpoint of the vehicle (passenger), and corresponds to the field of view of the vehicle occupant. The visual image does not necessarily have to be an image visually recognized in the traveling direction of the vehicle as long as it is an image visually recognized from the viewpoint of the vehicle (passenger). However, the visual image must at least include the guide image and the virtual object around the guide image.

Then, in S9, the CPU 41 determines whether or not there is an object virtual object that overlaps with the guide image in the visual image among the object virtual objects arranged on the three-dimensional map in S8, i.e., whether or not there is an object virtual object that overlaps with the guide image and corresponds to an object that is visually recognized by the vehicle occupant while overlapping with the guide image (hereinafter, referred to as an overlapping object virtual object). If it is determined that there is an overlapping object virtual object, the overlapping object virtual object is displayed in black. Note that an object virtual object that at least partially overlaps with the guide image in the visual image is regarded as an overlapping object virtual object, regardless of whether it is located in front of the guide image (toward the vehicle) or behind the guide image.

Here, FIG. 6 is a diagram in which a guide image 55 and virtual object bodies 56-59 are arranged on a three-dimensional map 51. For example, when a virtual object body 56 of a building, a virtual object body 57 of another vehicle, a virtual object body 58 of a bicycle, and a virtual object body 59 of a road sign are arranged on the three-dimensional map 51 as shown in FIG. 6, the virtual object body 57 of the other vehicle will be visually recognized as overlapping with the guide image 55. Therefore, the virtual object body 57 of the other vehicle corresponds to an overlapping virtual object body, and will be displayed in black.

Next, in S10, the CPU 41 stores the shape of the guide image and the shape of the overlapping object virtual body included in the visual image as the shape of the image to be displayed by the front display 4. Here, the shape of the guide image stored in S10 is the shape of the guide image that can be seen when each object (guide image, object virtual body, etc.) arranged on the three-dimensional relief map is seen from the viewpoint of the vehicle (passenger). On the other hand, the shape of the overlapping object virtual body included in the visual image is the shape of the overlapping object virtual body that can be seen when each object (guide image, object virtual body, etc.) arranged on the three-dimensional relief map is seen from the viewpoint of the vehicle (passenger). Therefore, for example, when the shape of the guide image stored in S10 is seen overlapping with the overlapping object virtual body 57 located in front of the guide image 55 as shown in FIG. 7, the overlapping portion is removed. On the other hand, when the shape of the overlapping object virtual body stored in S10 is seen overlapping with the guide image located in front of the overlapping object virtual body, the overlapping portion is removed.

Furthermore, in S11, the CPU 41 acquires the position of the guide image arranged on the three-dimensional relief map. For example, this is on the road surface up to 100 m ahead from the current position of the vehicle. Similarly, the CPU 41 acquires the position of the overlapping object virtual body arranged on the three-dimensional relief map. Specifically, this is the position where there is an object that is visually recognized as overlapping with the guide image.

Then, in S12, the CPU 41 performs a display range determination process (FIG. 10) described below. In the display range determination process, the range in which the guide image and black image are displayed on the front display 4 (the range in which the guide image and black image are projected onto the windshield 5, or the range in which the guide image and black image are displayed on the front display 4) is determined.

Next, in S13, the CPU 41 transmits a control signal to the front display 4, and causes the front display 4 to display a guide image of the shape stored in S10 in the display range determined in S12. Similarly, the CPU 41 displays a black image (object image) having the shape of the overlapping object virtual body stored in S10 in the display range determined in S12. The guide image and the black image are basically displayed as a single image. Also, the overlapping object virtual body placed on the 3D stereoscopic map does not necessarily have to be black, but even in this case, the image displayed on the front display 4 in accordance with the overlapping object virtual body is a black image.

Here, when a vehicle occupant views an image displayed on the front display 4, if the image displayed on the front display 4 is a black image, the virtual image of the black image cannot be seen (it is in a transparent state). Therefore, even if the black image is displayed at a position where it overlaps with an object, the virtual image of the black image is not seen by the vehicle occupant (more accurately, it is seen in a transparent state), and only the object is visible.

As a result, when a vehicle occupant views the guide image 55 and black image 66 displayed on the front display 4 as shown in Figure 8, the virtual image is not visible (transparent) for the portion corresponding to the black image 66 (i.e., the portion overlapping with the object in the real scene). Also, for the virtual image 67 of the guide image 55, the portion hidden by the object 68 that cannot be seen (the portion indicated by the dashed line in the right diagram of Figure 8) is not visible due to the black image 66, but the other portion (the portion indicated by the solid line in the right diagram of Figure 8) is visible.

Therefore, when a vehicle occupant visually recognizes the virtual image 67 of the guidance image 55, the shape of the virtual image 67 enables the occupant of the vehicle to recognize the front-to-back relationship between the virtual image 67 and the object 68 that is visually perceived as overlapping with the virtual image 67. For example, FIG. 9 shows an example of an image displayed on the front display 4 and the virtual image 67 of the guidance image 55 visually perceived by the vehicle occupant through the windshield 5. In the example shown in FIG. 9, it is clear that the virtual image 67 of the guidance image 55 is superimposed on the guidance intersection 69 located further back (farther away) than the object 68 (another vehicle in the example shown in FIG. 9). Therefore, the position of the guidance intersection 69 to be turned right or left, which is indicated by the virtual image 67 of the guidance image 55, and the exit road at the guidance intersection 69 can be accurately grasped.

When drawing the guide image and black image on the front display 4, they may be drawn on different layers. In terms of order, the one located closer to the vehicle is drawn on a higher (upper) layer. For example, in the example shown in FIG. 9, the guide image 55 located further back (farther from the vehicle) than the overlapping object virtual body is drawn on the lowest layer. Next, the black image 66 corresponding to the overlapping object virtual body is drawn on a higher layer. In that case, even if the guide image 55 and black image 66 are drawn in their entirety, it is possible for the vehicle occupants to view the virtual image 67 of the guide image 55 shown in FIG. 9.

Next, the sub-processing of the display range determination process executed in S12 will be described with reference to FIG. 10. FIG. 10 is a flowchart of the sub-processing program of the display range determination process.

First, in S21, the CPU 41 detects the eye positions of the vehicle occupants based on the captured image captured by the in-vehicle camera 12. The detected eye positions are specified by three-dimensional position coordinates.

Next, in S22, the CPU 41 determines whether the display on the front display 4 is ON. The display on the front display 4 can be switched ON or OFF by an occupant of the vehicle. It may also be automatically switched ON or OFF based on the surrounding conditions or the state of the vehicle.

If it is determined that the display on the front display 4 is ON (S22: YES), the process proceeds to S23. On the other hand, if it is determined that the display on the front display 4 is OFF (S22: NO), the process ends without displaying a guide image on the front display 4.

In S23, the CPU 41 acquires the position coordinates of the windshield 5 onto which the image is to be projected by the front display 4. The position coordinates of the windshield 5 are specified as three-dimensional position coordinates.

Next, in S24, the CPU 41 acquires coordinates that specify the positions of the guide image and the overlapping object virtual object arranged on the three-dimensional map acquired in S11 as the position coordinates of the guide image and the overlapping object virtual object, respectively. Note that the position coordinates of the guide image and the overlapping object virtual object are also specified by three-dimensional position coordinates.

Next, in S25, the CPU 41 determines the projection range of the guide image and the black image on the windshield 5 based on the position coordinates acquired in S21, S23, and S24. Furthermore, the display range of the guide image and the black image on the front display 4 is also determined from the determined projection range. The display range of the guide image is the range in which the guide image having the shape stored in S10 is superimposed on the guide image arranged on the three-dimensional map by the vehicle occupant. The display range of the black image is the range in which the black image having the shape of the overlapping object virtual object stored in S10 is superimposed on the overlapping object virtual object (i.e. the object in the real scene) arranged on the three-dimensional map by the vehicle occupant. Then, the process proceeds to S13, where the virtual image is displayed using the front display 4 based on the determined projection range and display range. The processes of S6 to S13 are repeated until the display on the front display 4 is turned off.

As described above in detail, according to the superimposed image display device 1 and the computer program executed by the superimposed image display device 1 according to the first embodiment, when a virtual image of a guidance image is superimposed on the scenery ahead of the vehicle and viewed, a virtual body that imitates an object present around the vehicle is placed together with the guidance image in three-dimensional space (S8), the shapes of the guidance image and the virtual body when viewed from the position of the vehicle in three-dimensional space are respectively acquired (S10), and a black image showing the shape of the acquired virtual body is displayed on the front display 4 together with the guidance image of the acquired shape (S13), so that it is possible for the vehicle occupants to view a virtual image of the guidance image that accurately shows the positional relationship between the virtual image of the guidance image and the object. As a result, it is possible to provide more appropriate guidance using the virtual image of the guidance image.

[Second embodiment]
Next, a superimposed image display device according to a second embodiment will be described with reference to Fig. 11 and Fig. 12. In the following description, the same reference numerals as those in the configuration of the superimposed image display device 1 according to the first embodiment shown in Figs. 1 to 10 indicate the same or corresponding parts as those in the configuration of the superimposed image display device 1 according to the first embodiment.

The schematic configuration of the superimposed image display device according to the second embodiment is almost the same as that of the superimposed image display device 1 according to the first embodiment. In addition, various control processes are also almost the same as those of the superimposed image display device 1 according to the first embodiment.
However, the superimposed image display device 1 according to the first embodiment uses a head-up display system using the front display 4 as a means for displaying an image to be superimposed on the scenery around the vehicle, whereas the superimposed image display device according to the second embodiment displays an image captured by the front camera 11 on the liquid crystal display 17 of the navigation device 3, and displays a guide image by superimposing it on the displayed image. Note that the display for displaying the captured image may be a display other than the liquid crystal display 17, so long as it is a display arranged inside the vehicle.

Figure 11 is a diagram showing an example of a driving guidance screen 91 displayed on the liquid crystal display 17 in the superimposed image display device according to the second embodiment. As shown in Figure 11, the liquid crystal display 17 displays the current view in front of the vehicle captured by the front camera 11. Then, a guide image 92 having the shape stored in S10 is displayed with respect to the view in front of the vehicle. The display range of the guide image 92 is the range in which the guide image arranged in the three-dimensional map is visible from the position of the vehicle occupant. In addition, a transparent image (object image) 93 having the shape of the overlapping object virtual object stored in S10 with respect to the view in front of the vehicle is also displayed. The display range of the transparent image 93 is the range in which the overlapping object virtual object arranged in the three-dimensional map is visible from the position of the vehicle occupant, that is, the range in which it overlaps with the object 94 displayed on the liquid crystal display 17. The transparent image 93 is an image with a transmittance of 100%, and is an image that cannot be seen by the vehicle occupant.

Here, when the image displayed on the liquid crystal display 17 is viewed by a vehicle occupant, the transparent image 93 displayed on the liquid crystal display 17 cannot be viewed (is in a transparent state). Therefore, even if the transparent image 93 is displayed at a position where it overlaps with the object 94, the transparent image 93 is not viewed by the vehicle occupant (more accurately, it is viewed in a transparent state), and only the object 94 is substantially viewed.

As a result, when a vehicle occupant views the guide image 92 and the transparent image 93 displayed on the liquid crystal display 17 as shown in Figure 12, the portion of the image that corresponds to the transparent image 93 (i.e., the portion that overlaps with the object in the real scene) is not visible (is in a transparent state). Also, for the guide image 92, the portion that is hidden by the object and cannot be seen (the portion indicated by the dashed line in the right diagram of Figure 12) is not visible (is made transparent) by the transparent image 93, while the other portion (the portion indicated by the solid line in the right diagram of Figure 12) is visible.

As a result, when a vehicle occupant visually recognizes the guide image 92, the shape of the guide image 92 enables the occupant to recognize the front-to-back relationship between the guide image 92 and the object 94 that is visually overlapping the guide image 92. For example, in the example shown in FIG. 11, it is clear that the guide image 92 is superimposed on a guide intersection 95 that is located behind (farther away from) the object 94 (another vehicle in the example shown in FIG. 11). Therefore, the position of the guide intersection 95 to turn right or left, which is indicated by the guide image 92, and the exit road at the guide intersection 95 can be accurately grasped.

When drawing a guide image and a transparent image on the liquid crystal display 17, they may be drawn on different layers. In terms of order, the one located closer to the viewer (closer to the viewer's vehicle) is drawn on a higher (upper) layer. For example, in the example shown in FIG. 11, a guide image 92 located further back (farther from the viewer's vehicle) than the overlapping object virtual body is drawn on the lowest layer. Next, a transparent image 93 corresponding to the overlapping object virtual body is drawn on a higher layer. In that case, even if the guide image 92 and the transparent image 93 are drawn in their entirety, the guide image 92 shown in FIG. 11 can be visually recognized by the vehicle occupants.

In addition, in the superimposed image display device according to the second embodiment, the display range determination process of S12 is not necessary, and the display range of the guide image on the liquid crystal display 17 is the range in which the guide image arranged on the three-dimensional map is visible from the position of the vehicle occupant as described above. Also, the display range of the transparent image on the liquid crystal display 17 is the range in which the overlapping object virtual object arranged on the three-dimensional map is visible from the position of the vehicle occupant.

Incidentally, the present invention is not limited to the above-described embodiment, and it is needless to say that various improvements and modifications are possible without departing from the spirit and scope of the present invention.
For example, in the first embodiment, the front display 4 is configured to generate a virtual image in front of the windshield 5 of the vehicle 2, but the virtual image may be generated in front of a window other than the windshield 5. In addition, the target onto which the image is reflected by the front display 4 may be a visor (combiner) installed around the windshield 5, rather than the windshield 5 itself.

In the first embodiment, a head-up display system is used as a means for displaying an image to be superimposed on the scenery around the vehicle, but a windshield display (WSD) that displays an image on the windshield 5 may also be used. In that case, it is preferable that the object image displayed corresponding to the overlapping object virtual body is a transparent image as in the second embodiment, rather than a black image.

In the first and second embodiments, the guidance image is an image of an arrow indicating the direction of travel of the vehicle along the guidance route set by the navigation device 3, but other images may be used. For example, a warning image of an object to be warned to the vehicle occupants (e.g., another vehicle, a pedestrian, or a guide sign), an image of the lane markings on which the vehicle is traveling, etc. may be used.

In addition, in the first and second embodiments, the guide image is an image that needs to be kept superimposed on a specific object (e.g., the road surface) included in the scenery visible from the vehicle, but it may be an image that does not need to be kept superimposed on a specific object (e.g., an image from a smartphone, a map image, etc.).

In addition, in the first and second embodiments, only black images or transparent images corresponding to objects present around the vehicle that overlap with the guide image and are visible to the vehicle occupants are displayed on the front display 4 or liquid crystal display 17, but black images or transparent images corresponding to all objects present around the vehicle may be displayed on the front display 4 or liquid crystal display 17.

In the first embodiment, a black image with 0% transmittance is displayed as the object image on the front display 4, but a semi-transparent black image (for example, 50% transmittance) may be displayed. In that case, the guide image (the dotted line part in the right diagram of Figure 8) that is hidden by the black image is also displayed semi-transparently. As a result, it becomes possible for the vehicle occupants to view the virtual image of the guide image that is hidden by the object. Furthermore, the above-mentioned semi-transparent image display may be performed only when a guide image of particular importance (for example, an arrow indicating a right or left turn) is to be displayed.

On the other hand, in the second embodiment, the object image displayed on the liquid crystal display 17 is an image with a transmittance of 100%, but a semi-transparent (e.g., transmittance of 80%) or opaque (transmittance of 0%) image may be displayed. If the object image is an opaque image, the object included in the scenery displayed on the liquid crystal display 17 will be hidden by the object image and will not be visible, but this is not a major problem because the vehicle occupants can see the object through the windshield.

In the first and second embodiments, the navigation ECU 15 of the navigation device 3 executes the driving assistance processing program (FIG. 3), but the execution entity can be changed as appropriate. For example, the execution entity can be a control unit of the front display 4, a vehicle control ECU, or other in-vehicle device. Note that when the execution is performed by a control unit of the front display 4, the superimposed image display device according to the present invention can be configured with only the front display 4.

Although the above describes specific examples of the superimposed image display device according to the present invention, the superimposed image display device can also have the following configuration, which provides the following effects:

For example, the first configuration is as follows.
A guide image arrangement means (41) for arranging a guide image (55, 92) in a three-dimensional space (51) corresponding to the periphery of a vehicle (2), a virtual body arrangement means (41) for arranging a virtual body (57) simulating an object (68, 94) existing around the vehicle at a position in the three-dimensional space where the object exists, a viewpoint setting means (41) for setting a viewpoint at a position in the three-dimensional space corresponding to the current position of the vehicle, a shape acquisition means (41) for acquiring the shapes of the guide image and the virtual body as viewed from the viewpoint in a state where the guide image and the virtual body are arranged in the three-dimensional space, and and an image display means (41) for displaying the guide image having the shape acquired by the shape acquisition means and an object image (66, 93), which is a black image showing the shape of the virtual body acquired by the shape acquisition means, as a single image on an image display surface provided in the vehicle, and for reflecting the guide image and the object image, which are single images displayed on the image display surface, onto the windshield of the vehicle so that they can be viewed by an occupant of the vehicle, thereby superimposing a virtual image of the guide image on the scenery in front of the vehicle and allowing the occupant of the vehicle to view it, and for superimposing a virtual image of the object image at the position of the object in the scenery in front of the vehicle in a manner that cannot be viewed by the occupant of the vehicle.
According to the superimposed image display device having the above configuration, when the guide image is superimposed on the scenery ahead of the vehicle and is visually recognized, the guide image is prevented from being visually recognized by the vehicle occupants in the area where it overlaps with an object located closer to the vehicle than the guide image, so that the vehicle occupants can visually recognize the guide image that accurately shows the positional relationship between the guide image and the object. As a result, guidance using a virtual image of the guide image can be performed more appropriately.

Furthermore, with the superimposed image display device having the above configuration, when a virtual image of a guidance image is superimposed on the scenery ahead of the vehicle and viewed, a black image is displayed in an area corresponding to the real image of the object, so that the vehicle occupants can view the virtual image of the guidance image that accurately shows the positional relationship between the virtual image of the guidance image and the object. As a result, guidance using the virtual image of the guidance image can be performed more appropriately.

The second configuration is as follows.
A guide image arrangement means (41) for arranging a guide image (55, 92) in a three-dimensional space (51) corresponding to the periphery of a vehicle (2), a virtual body arrangement means (41) for arranging a virtual body (57) simulating an object (68, 94) existing in the periphery of the vehicle at a position in the three-dimensional space where the object exists, a viewpoint setting means (41) for setting a viewpoint at a position in the three-dimensional space corresponding to the current position of the vehicle, and a visual display means (41) for displaying the guide image and the virtual body as viewed from the viewpoint in a state where the guide image and the virtual body are arranged in the three-dimensional space. The vehicle has a shape acquisition means (41) for acquiring a shape , and an image display means (41) for displaying the guide image of the shape acquired by the shape acquisition means and an object image (66, 93), which is a transparent color image showing the shape of the virtual body acquired by the shape acquisition means, as a single image on an image display surface provided in the vehicle and displaying the scenery in front of the vehicle, thereby superimposing the guide image on the scenery in front of the vehicle so that it can be viewed by an occupant of the vehicle, and superimposing the object image at the position of the object in the scenery in front of the vehicle in a manner that cannot be viewed by the occupant of the vehicle.
According to the superimposed image display device having the above configuration, when the guide image is superimposed on the scenery ahead of the vehicle and viewed, the guide image that accurately shows the positional relationship between the guide image and the object can be made transparent to the vehicle occupants by making the area of the guide image that overlaps with the object on the vehicle side transparent. As a result, guidance using the guide image can be performed more appropriately.

The third configuration is as follows.
The image display means (41) displays the guide image at a display position on the image display surface for visual recognition by superimposing the guide image at a predetermined superimposition position.
According to the superimposed image display device having the above configuration, it is possible to visually superimpose a guide image at a desired position on the scenery ahead of the vehicle.

The fourth configuration is as follows.
The image display means (41) displays the object image (66, 93) at a display position on the image display surface (4) for visual recognition by superimposing the object image (66, 93) on the object (68, 94).
According to the superimposed image display device having the above configuration, the visibility of the object is not hindered by the object image, and the vehicle occupant can visually recognize the object. In addition, the guide image makes it possible to visually recognize the other parts of the guide image, while the part of the guide image that is hidden by the object cannot be visually recognized by the object image. As a result, the vehicle occupant can visually recognize the guide image that accurately shows the positional relationship between the guide image and the object.

The fifth configuration is as follows.
The virtual body placement means (41) places the virtual body (57) having a size larger than the target object (68).
With the superimposed image display device having the above configuration, even if the position of the object visible to the occupants is blurred due to vibrations or the like occurring in the vehicle, it is possible for the occupants of the vehicle to view a guide image that accurately shows the positional relationship between the guide image and the object.

The sixth configuration is as follows.
The image display means (41) displays the object image corresponding to the object that is visible to the vehicle occupant, overlapping with the guide image, among objects present around the vehicle.
According to the superimposed image display device having the above configuration, the display processing of the object image is performed only on the object related to the guide image, so that it is possible to reduce the processing load associated with the display processing of the object image.

REFERENCE SIGNS LIST 1 Superimposed image display device 2 Vehicle 3 Navigation device 4 Front display 5 Windshield 6 Passenger 7 Dashboard 17 Liquid crystal display 41 CPU
42 RAM
43 ROM
55 Guidance image 56-59 Virtual object 66 Black image 68 Object 91 Driving guidance screen 92 Guidance image 93 Transparent image 94 Object

Claims (8)

a guide image arrangement means for arranging a guide image in a three-dimensional space corresponding to the periphery of the vehicle;
a virtual body placement means for placing a virtual body simulating an object existing around a vehicle at a position where the object exists within the three-dimensional space;
a viewpoint setting means for setting a viewpoint at a position in the three-dimensional space corresponding to a current vehicle position;
a shape acquisition means for acquiring shapes of the guide image and the virtual body viewed from the viewpoint in a state in which the guide image and the virtual body are arranged in the three-dimensional space ;
and an image display means for displaying the guide image having the shape acquired by the shape acquisition means and an object image which is a black image showing the shape of the virtual body acquired by the shape acquisition means as a single image on an image display screen provided in the vehicle, and for reflecting the guide image and the object image which are single images displayed on the image display screen onto the windshield of the vehicle so that they are visible to occupants of the vehicle, thereby superimposing a virtual image of the guide image on the scenery in front of the vehicle and allowing the occupants of the vehicle to view it, and for superimposing a virtual image of the object image at the position of the object in the scenery in front of the vehicle in a manner that cannot be seen by the occupants of the vehicle. a guide image arrangement means for arranging a guide image in a three-dimensional space corresponding to the periphery of the vehicle;
a virtual body placement means for placing a virtual body simulating an object existing around a vehicle at a position where the object exists within the three-dimensional space;
a viewpoint setting means for setting a viewpoint at a position in the three-dimensional space corresponding to a current vehicle position;
a shape acquisition means for acquiring shapes of the guide image and the virtual body viewed from the viewpoint in a state in which the guide image and the virtual body are arranged in the three-dimensional space ;
and an image display means for displaying the guide image of the shape acquired by the shape acquisition means and an object image, which is a transparent color image showing the shape of the virtual body acquired by the shape acquisition means, as a single image on an image display screen provided in the vehicle and displaying the scenery in front of the vehicle, thereby superimposing the guide image on the scenery in front of the vehicle so that it can be viewed by occupants of the vehicle, and for superimposing the object image at the position of the object in the scenery in front of the vehicle in a manner that cannot be viewed by the occupants of the vehicle. 3. The superimposed image display device according to claim 1, wherein the image display means displays the guide image at a display position on the image display surface for allowing the guide image to be visually recognized by being superimposed at a predetermined superimposed position. 3. The superimposed image display device according to claim 1, wherein the image display means displays the object image at a display position on the image display surface for visual recognition by superimposing the object image on the object. The superimposed image display device according to any one of claims 1 to 4, wherein the virtual body placement means places the virtual body having a size larger than that of the target object. The superimposed image display device according to any one of claims 1 to 5, wherein the image display means displays the object image corresponding to an object present in the vicinity of the vehicle that is visible to an occupant of the vehicle and overlaps with the guidance image. Computer,
a guide image arrangement means for arranging a guide image in a three-dimensional space corresponding to the periphery of the vehicle;
a virtual body placement means for placing a virtual body simulating an object existing around a vehicle at a position where the object exists within the three-dimensional space;
a viewpoint setting means for setting a viewpoint at a position in the three-dimensional space corresponding to a current vehicle position;
a shape acquisition means for acquiring shapes of the guide image and the virtual body viewed from the viewpoint in a state in which the guide image and the virtual body are arranged in the three-dimensional space ;
an image display means for displaying the guide image of the shape acquired by the shape acquisition means and an object image which is a black image showing the shape of the virtual body acquired by the shape acquisition means as an integrated image on an image display surface provided in the vehicle, and for reflecting the guide image and the object image which are integrated images displayed on the image display surface onto a windshield of the vehicle so that an occupant of the vehicle can view the image by superimposing a virtual image of the guide image on the scenery ahead of the vehicle, and for superimposing a virtual image of the object image on the position of the object in the scenery ahead of the vehicle in a manner that cannot be viewed by the occupant of the vehicle;
A computer program that makes something function. Computer,
a guide image arrangement means for arranging a guide image in a three-dimensional space corresponding to the periphery of the vehicle;
a virtual body placement means for placing a virtual body simulating an object existing around a vehicle at a position where the object exists within the three-dimensional space;
a viewpoint setting means for setting a viewpoint at a position in the three-dimensional space corresponding to a current vehicle position;
a shape acquisition means for acquiring shapes of the guide image and the virtual body viewed from the viewpoint in a state in which the guide image and the virtual body are arranged in the three-dimensional space ;
A computer program for functioning as an image display means that displays the guide image of the shape acquired by the shape acquisition means and an object image, which is a transparent color image showing the shape of the virtual body acquired by the shape acquisition means, as a single image on an image display screen provided in a vehicle and displays the scenery in front of the vehicle, thereby superimposing the guide image on the scenery in front of the vehicle so that it can be viewed by occupants of the vehicle, and superimposing the object image on the position of the object in the scenery in front of the vehicle in a manner that cannot be viewed by the occupants of the vehicle.

Images