JP2023150070A - Superimposed image display device - Google Patents

Abstract

To provide a superimposed image display device which prevents guide of superimposed image display to a wrong peripheral environment from being performed and prevents disadvantage from being given to an occupant of a vehicle even under such a situation that contradiction or deviation occurs between a result of image recognition of a photographed image and map information.SOLUTION: A superimposed image display device specifies a lane number included in a road on which a vehicle currently travels on the basis of a photographed image obtained by imaging a scenery around the vehicle, specifies a travel lane being a lane on which the vehicle currently travels on the road on which the vehicle currently travels by using the lane number specified on the basis of the photographed image when the lane number of the road on which the vehicle currently travels included in the map information does not match the lane number specified on the basis of the photographed image, specifies the travel lane by using the matching lane number when the lane number of the road on which the vehicle currently travels included in the map information matches the lane number specified on the basis of the photographed image, and displays a guide object on the basis of the specified travel lane.SELECTED DRAWING: Figure 12

Description

The present invention relates to a superimposed image display device that supports driving of a vehicle.

2. Description of the Related Art Conventionally, various means have been used as information providing means for providing vehicle occupants with various types of information for providing vehicle driving support, such as route guidance and warning of obstacles. For example, it is a display on a liquid crystal display installed in a vehicle, a sound output from a speaker, etc. In recent years, as one of such information providing means, there has been a device that provides information by displaying an image superimposed on the surrounding environment (scenery, real scene) of the passenger. For example, in addition to a head-up display, a window shield display, a method of displaying an image superimposed on a captured image of the surroundings of the vehicle displayed on a liquid crystal display, etc. are applicable.

Here, in order to provide accurate guidance by displaying an image superimposed on the surrounding environment, it is necessary to accurately identify the position on the road where the vehicle is traveling (more specifically, the lane in which it is traveling). This is very important. For example, Japanese Patent Laid-Open No. 2021-36226 discloses that when superimposing guidance on the foreground using a head-up display, external world information such as lane markings obtained using an image captured by a front camera and high-precision map information are combined. A technology is disclosed in which the location of the own vehicle on the road is specified by using the system, and an image superimposed on the surrounding environment is displayed to provide guidance.

JP 2021-36226 (Paragraph 0058-0064)

Here, in order to accurately identify the lane in which the vehicle is currently traveling (hereinafter referred to as the driving lane) using map information as in Patent Document 1, it is necessary to It is assumed that the information is accurate. However, there are many points on roads where the number of lanes increases or decreases, for example around intersections on general roads, branch points or junctions on expressways, and the number of lanes often changes due to road renovations. Therefore, it has been extremely difficult in reality to create an accurate database of information regarding lanes on all roads nationwide. As a result, it is expected that there will be situations where there will be inconsistencies or discrepancies between the image recognition results from the front camera and the map information, and in such situations, the results of identifying driving lanes will be unreliable and images will be superimposed on the wrong surrounding environment. There was also a risk that the information would be displayed.

The present invention has been made to solve the above-mentioned conventional problems, and when providing guidance using a guide object displayed superimposed on the scenery around the vehicle, it is possible to combine the results of image recognition of captured images with map information. Even in situations where there are contradictions or discrepancies between The purpose is to provide.

In order to achieve the above object, a superimposed image display device according to the present invention is a superimposed image display device that is mounted on a vehicle and allows a guide object that guides information to a passenger of the vehicle to be visually recognized by superimposing it on the scenery around the vehicle. a map information acquisition means for acquiring map information including information regarding the number of lanes on the road; and a map information acquisition means for determining the number of lanes included in the road on which the vehicle is currently traveling based on a captured image of the scenery around the vehicle. If the number of lanes of the road on which the vehicle currently traveling included in the map information is currently running does not match the number of lanes specified by the number of lanes specification means, the number of lanes specified by the number of lanes specification means is specified. While identifying the driving lane in which the vehicle is currently traveling on the road on which the vehicle is currently traveling using the number, the number of lanes on the road where the vehicle is currently traveling included in the map information and the lane number identifying means are identified. and a driving lane specifying means for specifying the driving lane using the matching number of lanes when the number of lanes that have been specified match, and displaying the guide object based on the driving lane specified by the driving lane specifying means. and object display means.
Note that the term "landscape" includes not only the scenery that is actually seen from the vehicle (actual scene), but also images of scenery, images that reproduce scenery, and the like.

According to the superimposed image display device according to the present invention having the above configuration, when guidance is provided using a guide object displayed superimposed on the scenery around the vehicle, the number of lanes of the road on which the vehicle currently traveling included in the map information is provided. If the number of lanes specified by the result of image recognition of the captured image does not match, the number of lanes specified by the result of image recognition is used to identify the lane in which the vehicle is currently traveling. Even in situations where there is a discrepancy or discrepancy between the results of the process and the map information, this system prevents the display of superimposed images on the surrounding environment from being displayed incorrectly, and prevents the vehicle occupants from being disadvantaged. do not have.

FIG. 1 is a block diagram showing a navigation device according to a first embodiment. 3 is a flowchart of a driving support processing program according to the first embodiment. FIG. 3 is a diagram showing an example of a first guide object displayed on a liquid crystal display. It is a figure showing an example of the 2nd guidance object displayed on a liquid crystal display. It is a flowchart of the sub-processing program of the 1st guidance object display position determination process. FIG. 2 is a diagram showing an example of recommended lanes when passing through a guidance branch point on an expressway. It is a diagram showing an example of recommended lanes when passing through a guidance branch point on a general road. FIG. 3 is a diagram illustrating a method for detecting the number of vehicle road lanes and the lane in which the vehicle is traveling based on image recognition results. FIG. 3 is a diagram illustrating a method for detecting the number of vehicle road lanes and the lane in which the vehicle is traveling based on image recognition results. FIG. 3 is a diagram illustrating a method for detecting the number of vehicle road lanes and the lane in which the vehicle is traveling based on image recognition results. FIG. 3 is a diagram illustrating link data included in map information. FIG. 3 is a diagram illustrating a case where a contradiction or gap occurs between map information and an image recognition result. FIG. 2 is a diagram illustrating a method for specifying a range in a landscape where a recommended lane exists and a range where a vehicle's driving lane exists. FIG. 3 is a diagram showing an example of a first guide object. FIG. 3 is a diagram showing an example of a first guide object. FIG. 6 is a diagram showing changes in the display mode of the first guide object. FIG. 3 is a diagram showing an example of a first guide object displayed on a liquid crystal display. FIG. 3 is a diagram showing an example of a first guide object displayed on a liquid crystal display. FIG. 3 is a diagram showing an example of a first guide object displayed on a liquid crystal display. FIG. 6 is a diagram illustrating an example of a first guide object displayed on a liquid crystal display especially when the reliability of image recognition results is low. FIG. 2 is a schematic configuration diagram of a superimposed image display device according to a second embodiment. FIG. 7 is a diagram illustrating a display example of a guide object in a superimposed image display device according to a second embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a first embodiment and a second embodiment in which a superimposed image display device according to the present invention is embodied in a navigation device will be described in detail with reference to the drawings.

[First embodiment]
First, a schematic configuration of a navigation device 1 according to a first embodiment will be described using FIG. 1. FIG. 1 is a block diagram showing a navigation device 1 according to the first embodiment.

As shown in FIG. 1, the navigation device 1 according to the first embodiment includes a current position detection unit 11 that detects the current position of a vehicle in which the navigation device 1 is mounted, and a data recording unit 12 that records various data. , a navigation ECU 13 that performs various calculation processes based on input information, an operation unit 14 that accepts operations from the user, and a liquid crystal display 15 that displays to the user an image of the actual scene taken in the forward direction of travel. Communication is performed between a speaker 16 that outputs voice guidance regarding route guidance, a DVD drive 17 that reads a DVD as a storage medium, and an information center such as a probe center or a VICS (registered trademark: Vehicle Information and Communication System) center. It has a communication module 18. Further, the navigation device 1 is connected to a front camera 19 and various sensors installed in the vehicle in which the navigation device 1 is mounted via an in-vehicle network such as CAN.

Below, each component included in the navigation device 1 will be explained in order.
The current position detection unit 11 includes a GPS 21, a vehicle speed sensor 22, a steering sensor 23, a gyro sensor 24, etc., and is capable of detecting the current vehicle position, direction, vehicle speed, current time, etc. . Here, in particular, the vehicle speed sensor 22 is a sensor for detecting the travel distance and vehicle speed of the vehicle, generates pulses according to the rotation of the drive wheels of the vehicle, and outputs the pulse signal to the navigation ECU 13. Then, the navigation ECU 13 calculates the rotational speed and travel distance of the drive wheels by counting the generated pulses. Note that it is not necessary for the navigation device 1 to include all of the above four types of sensors, and the navigation device 1 may be configured to include only one or more types of sensors.

The data recording unit 12 also includes a hard disk (not shown) as an external storage device and recording medium, and a driver for reading map information DB 31 and predetermined programs recorded on the hard disk and writing predetermined data on the hard disk. A recording head (not shown) is provided. Note that the data recording section 12 may be configured with a flash memory, a memory card, or an optical disk such as a CD or DVD instead of a hard disk. Alternatively, the map information DB 31 may be stored in an external server and acquired by the navigation device 1 through communication.

Here, the map information DB 31 includes, for example, link data 32 related to roads (links), node data 33 related to node points, branch point data 34 related to branch points, point data related to points such as facilities, and a map display for displaying a map. It is a storage means in which data, search data for searching for routes, search data for searching for points, etc. are stored.

In addition, the link data 32 includes the width, slope, cant, bank, road surface condition, number of road lanes, and traffic classification of each lane in the direction of travel for each link that constitutes the road network. , the presence or absence of oncoming lanes (whether it is a two-way traffic section or not), the presence or absence of a median strip, places where the width becomes narrow, railroad crossings, etc. Regarding corners, the data indicates the radius of curvature, intersections, T-junctions, corner entrances and exits. Regarding road attributes, data representing downhill roads, uphill roads, etc. are recorded, and regarding road types, data representing expressways and general roads (national highways, prefectural roads, narrow streets, etc.) are recorded.

In addition, the node data 33 includes coordinates (positions) of actual road branch points (including intersections, T-junctions, etc.) and node points set at predetermined distances on each road according to the radius of curvature, etc. Node attributes that indicate whether a node corresponds to an intersection, etc., a connected link number list that is a list of link numbers of links that connect to the node, and adjacency that is a list of node numbers of nodes that are adjacent to the node via links. A list of node numbers, data regarding the height (altitude) of each node point, etc. are recorded.

In addition, the branch point data 34 includes the intersection name of the branch point, corresponding node information that specifies the node forming the branch point, connecting link information that specifies the link connected to the branch point, and information on the link connected to the branch point. The corresponding direction name, information specifying the shape of the branch point, etc. are stored. Further, structures that can serve as landmarks when providing left/right turn guidance at a junction are also stored.

On the other hand, the navigation ECU (electronic control unit) 13 is an electronic control unit that controls the entire navigation device 1, and includes a CPU 41 as an arithmetic unit and a control device, and a working memory when the CPU 41 performs various arithmetic processes. In addition to the RAM 42 which stores route data etc. when a route is searched, the ROM 43 which stores a driving support processing program (see FIG. 2) to be described later in addition to the control program, which is read from the ROM 43 It includes an internal storage device such as a flash memory 44 that stores programs. Note that the navigation ECU 13 has various means as processing algorithms. For example, the map information acquisition means acquires map information including information regarding the number of lanes on a road. The lane number identifying means identifies the number of lanes included in the road on which the vehicle is currently traveling, based on a captured image of the scenery around the vehicle. The driving lane specifying means uses the number of lanes specified by the lane number specifying means when the number of lanes of the road on which the vehicle currently travels included in the map information does not match the number of lanes specified by the lane number specifying means. While identifying the driving lane in which the vehicle is currently traveling on the road on which the vehicle is currently traveling, it also identifies the number of lanes on the road where the vehicle is currently traveling included in the map information and the number of lanes identified by the lane number identifying means. If they match, the driving lane is identified using the matching number of lanes. The object display means displays the guide object based on the driving lane specified by the driving lane specifying means.

The operation unit 14 is operated when inputting a departure point as a travel start point and a destination as a travel end point, and has a plurality of operation switches (not shown) such as various keys and buttons. Then, the navigation ECU 13 performs control to execute various corresponding operations based on switch signals output by pressing each switch. Note that the operation unit 14 may have a touch panel provided on the front surface of the liquid crystal display 15. Alternatively, a configuration including a microphone and a voice recognition device may be used.

The liquid crystal display 15 also displays map images including roads, traffic information, operation guidance, operation menus, key guidance, a guidance route from the departure point to the destination, guidance information along the guidance route, news, weather forecasts, etc. The time, email, TV programs, etc. are displayed. In particular, in the first embodiment, when the vehicle approaches the guidance junction, the liquid crystal display 15 displays the image taken by the front camera 19, that is, the scenery (actual scene image) around the vehicle at the present moment (especially in front of the vehicle). The guide object is displayed superimposed on the scenery as necessary.

Here, the guide objects displayed superimposed on the scenery include information regarding the vehicle and various types of information used to support the driving of the passenger. For example, warnings to the occupants about objects (other vehicles, pedestrians, guide signs), guidance information based on the guide route or guide route set in the navigation device 1 (arrows indicating right/left turn directions, guide branches) icons indicating point markers, distance to guidance junctions, recommended lane positions for vehicles to drive, guidance to change lanes to recommended lanes, etc.), warnings displayed on the road surface (beware of rear-end collisions, speed limits, etc.), Includes lane markings, current vehicle speed, shift position, remaining energy, advertising images, facility information, guide signs, map images, traffic information, news, weather forecast, time, connected smartphone screen, etc. . In the first embodiment described below, the guide object is guide information for providing guidance at a guide branch point located ahead in the direction of travel of the vehicle. More specifically, it shows the arrow indicating the exit direction of the guidance junction along the guidance route, and the position of the recommended lane on the road where the vehicle is currently traveling in order to pass through the guidance junction along the guidance route. The guide image may be a guide image that shows the vehicle, a guide image that prompts the user to move to the recommended lane, or the like.

Furthermore, the speaker 16 outputs audio guidance for guiding travel along the guide route and traffic information based on instructions from the navigation ECU 13.

Further, the DVD drive 17 is a drive that can read data recorded on recording media such as DVDs and CDs. Then, based on the read data, music and video are played back, and the map information DB 31 is updated. Incidentally, instead of the DVD drive 17, a card slot for reading and writing a memory card may be provided.

Further, the communication module 18 is a communication device for receiving traffic information including various information such as congestion information, regulation information, traffic accident information, etc. transmitted from a traffic information center, such as a VICS center or a probe center, For example, this applies to mobile phones and DCMs.

Further, the front camera 19 is an imaging device having a camera using a solid-state imaging device such as a CCD, and is installed, for example, on the back side of a room mirror or on a front bumper with its optical axis facing forward in the direction of travel of the vehicle. Ru. The captured image captured by the front camera 19 is displayed on the liquid crystal display 15 as a scenery (actual scene image) around the vehicle (particularly in front of the vehicle), as described above. In addition, the captured image taken by the front camera 19 is used to detect the marking lines of the road on which the vehicle is traveling as described later, and to identify the total number of lanes on the road on which the vehicle is traveling and the lane in which the vehicle is currently traveling. Also used for

Next, a driving support processing program executed by the navigation ECU 13 in the navigation device 1 having the above configuration will be described based on FIG. 2. FIG. 2 is a flowchart of the driving support processing program according to the first embodiment. Here, the driving support processing program is executed after the vehicle's ACC power supply (accessory power supply) is turned on, and the driving support processing program is executed by making the guide object superimposed on the scenery around the vehicle displayed on the liquid crystal display 15 visible. This is a program that provides support. Note that the programs shown in the flowcharts in FIGS. 2 and 5 below are stored in the RAM 42 and ROM 43 included in the navigation device 1, and are executed by the CPU 41.

In the following description, an example will be described in which driving guidance of a vehicle along a guidance route set by the navigation device 1 is performed as driving support for a vehicle using a guide object. In addition, the guidance objects to be displayed are guidance information for providing guidance at the guidance junction located ahead in the direction of travel of the vehicle, and in particular, an arrow indicating the exit direction of the guidance junction, a guidance image indicating the position of the recommended lane, and A process for displaying a guide image that prompts movement to a recommended lane as a guide object will be described as an example. However, the navigation device 1 can also use guide objects to provide guidance and information other than the above-mentioned driving support. Further, the guide object to be displayed may be information other than the above-mentioned arrow or guide image. For example, warnings for objects (other vehicles, pedestrians, guide signs) that are subject to warnings for occupants as guidance objects, warnings displayed on the road surface (beware of rear-end collisions, speed limits, etc.), and distance to the next guidance junction. It is also possible to display current vehicle speed, shift position, remaining energy, advertising images, facility information, guide signs, map images, traffic information, news, weather forecast, time, connected smartphone screen, etc.

First, in step (hereinafter abbreviated as S) 1 of the driving support processing program, the CPU 41 specifies the current position of the vehicle based on the detection result of the current position detection section 11 and map information. Note that when identifying the current location of the vehicle, map matching processing is also performed to match the current location of the vehicle with map information. Thereafter, the guidance route set by the navigation device 1 is read out, and the distance from the specified current position of the vehicle to the next guidance branch point along the guidance route is calculated. Note that a guidance branch point refers to a guidance point such as a right or left turn instruction (including an instruction to enter a left turn ramp or a right turn ramp) when the navigation device 1 provides driving guidance according to the guide route set in the navigation device 1. This is the branch point (intersection) where the process is performed. Note that a junction (difficult intersection) that does not make a right or left turn but has a special shape also corresponds to a guidance junction.

Next, in S2, the CPU 41 determines whether the distance to the next guidance branch point calculated in S1 is less than a predetermined guidance start distance. Note that the guidance start distance is determined by the type of road on which the vehicle is traveling; for example, an expressway is 1 km, and a general road is 300 m, which is shorter than the expressway. However, the guidance start distance may not be a fixed value but may be a variable value. For example, if there is another branch point within 300 m in front of the guide branch point on a general road, the distance from the guide branch point to the other branch point may be used.

If it is determined that the distance to the next guidance branch point calculated in S1 is less than the guidance start distance (S2: YES), the process moves to S3. On the other hand, if it is determined that the distance to the next guidance branch point calculated in S1 is not less than the guidance start distance (S2: NO), the process returns to S1.

In S3, the CPU 41 determines whether the distance to the next guidance branch point is less than a predetermined exit direction guidance start distance. Note that the exit direction guidance start distance is a distance shorter than the guidance start distance that is the criterion for S2, and is determined by the type of road on which the vehicle is traveling. The distance is also short at 50m.

If it is determined that the distance to the next guidance branch point is less than the exit direction guidance start distance (S3: YES), the process moves to S6. On the other hand, if it is determined that the distance to the next guidance branch point is not less than the exit direction guidance start distance (S3: NO), the process moves to S4.

Next, in S4, the CPU 41 performs a first guide object display position determination process (FIG. 5), which will be described later. In the first guide object display position determination process, a guide image indicating the position of a recommended lane or a guide image urging movement to the recommended lane is displayed as a guide object (hereinafter referred to as a first guide object), and is displayed on the liquid crystal display 15. This process specifically determines the size and shape of the first guide object to be displayed and the position (range) in which the first guide object is displayed. The size and shape of the first guide object determined in step S4 and the position at which the first guide object is displayed may be determined by superimposing the guide object on the recommended lane in the scenery, the vehicle's driving lane, or the lanes in between, for example. This is a condition for making the vehicle visible to the passengers. In addition, the liquid crystal display 15 displays an image captured in advance by the front camera 19 before the distance from the vehicle to the guidance branch point becomes less than the guidance start distance, that is, a scene (actual scene image) of the current surroundings of the vehicle (especially in front of the vehicle). is displayed. Then, in S4, as described later, using image recognition processing and map information, a range in which the recommended lane and the own vehicle's driving lane exist in the scenery displayed on the liquid crystal display 15 is specified, and the area is superimposed on the specified range. The size and shape of the first guide object and the position at which the first guide object is displayed are determined so that the first guide object is displayed.

Subsequently, in S5, the CPU 41 generates an image of the first guide object having the size and shape determined in S4, further transmits a control signal to the liquid crystal display 15, and sends the image of the first guide object to the liquid crystal display 15. The image of the first guide object is drawn at the position (range) determined in S4. As mentioned above, the liquid crystal display 15 displays an image captured by the front camera 19 before the distance from the vehicle to the guidance branch point becomes less than the guidance start distance, that is, the scenery (actual scene) around the vehicle (especially in front of the vehicle) at the current moment. image) is displayed. As a result, it becomes possible for the occupant of the vehicle to visually recognize the first guide object superimposed on the scenery.

FIG. 3 is a diagram showing an example of the driving guide screen 51 displayed on the liquid crystal display 15 in S5. As shown in FIG. 3, the liquid crystal display 15 displays a current scene 52 in front of the vehicle captured by the front camera 19. Then, an image 53 of the first guide object is displayed superimposed on the scenery 52 in front of the vehicle.

Here, in the first embodiment, there are a plurality of types of guide objects used for guidance, and one or more types of guide objects selected depending on the content of the guide and the current situation are displayed. Furthermore, multiple types of guide objects may be displayed at the same time. The example shown in FIG. 3 is a driving guidance screen that is displayed when the current position of the vehicle approaches the guidance branch point less than the guidance start distance and closer to the exit direction guidance start distance (for example, 1 km to 500 m to the guidance junction). 51, the first guide image is a guide image that distinguishes the recommended lane 54, which is recommended for driving on the road where the vehicle is currently traveling, from other lanes and guides the vehicle in order to pass through a guide branch point along the guide route. A first display mode displayed as an image 53 of the object, and a guidance image that identifies the recommended lane 54 from other lanes and provides guidance and urges movement to the recommended lane 54 are displayed as the image 53 of the first guidance object. The second display mode shown in FIG.

As for the image 53 of the first guide object, as shown in FIG. It is displayed superimposed on the area spanning up to 54. Specifically, until the vehicle approaches the guidance junction within a threshold (for example, within 700 m on a highway), the vehicle is superimposed only on the recommended lane 54, regardless of whether or not the own vehicle is traveling on the recommended lane 54. (first display mode). At this stage, the purpose is to make the occupants aware of the existence of the recommended lane 54 rather than to encourage them to move to the recommended lane 54.

Thereafter, if the vehicle approaches the guidance branch point within a threshold value (for example, within 700 m on an expressway) without moving to the recommended lane 54 while the image 53 of the first guidance object is displayed, the vehicle will move forward. An image 53 of a new first guide object superimposed on the area spanning from the lane 55 to the recommended lane 54 is displayed (second display mode). Thereby, it becomes possible to urge the occupant who does not move to the recommended lane 54 to move to the recommended lane. Note that when providing guidance to encourage movement to the recommended lane, in the first embodiment, the image 53 of the first guidance object is displayed over a wide area including the own vehicle driving lane 55 to the recommended lane 54. Compared to the case where images such as arrows are displayed, it is possible to suppress guidance that feels forced to prompt a lane change to a recommended lane.

However, even if the vehicle approaches the guidance branch point within a threshold (for example, within 700m on a highway) and is traveling in the recommended lane 54, there is no need to prompt the vehicle to move to the recommended lane, so the first guidance object The image 53 is displayed superimposed only on the recommended lane 54 (which also corresponds to the own vehicle driving lane 55) (first display mode).

Moreover, the image 53 of the first guide object shown in FIG. It is displayed only when the reliability of the results of the number of lanes and the driving lane of the own vehicle is high), and is generally not displayed when the reliability is low. However, even if the reliability is low, it does not depend on the vehicle's driving lane like the second guide object described later (does not use the vehicle's driving lane, is not based on the vehicle's driving lane, does not affect the vehicle's driving lane) It is possible to provide guidance in the following manner (an arrow indicating only the direction of exit from a guidance junction)

Further, in the above embodiment, the first display mode is switched to the second display mode depending on the distance to the guidance branch point, but the first display mode is not displayed and the first display mode is always displayed. The display may be performed depending on the display mode. Further, the distance to the guidance branch point may be displayed on the image 53 of the first guidance object. Thereafter, the process returns to S3, and the image 53 of the first guide object is continuously displayed until the distance to the guide branch point becomes less than the exit direction guide start distance. The first guide object will be explained in more detail later.

On the other hand, in S6, the CPU 41 sets the arrow indicating the exit direction of the guidance branch point to be displayed as a guidance object (hereinafter referred to as a second guidance object), and determines the size and shape of the second guidance object to be displayed on the liquid crystal display 15. and determine the position (range) where the second guide object is displayed. In the first embodiment, the second guide object is located a predetermined distance ahead of the vehicle's current position (for example, 10 m ahead) and a predetermined height (for example, 1 m) above the road surface, and ahead in the direction of travel of the vehicle. Three arrows in a triangular shape indicate the direction of exit from a certain guidance branch point.

Subsequently, in S7, the CPU 41 generates an image of the second guide object having the size and shape determined in S6, further transmits a control signal to the liquid crystal display 15, and sends the image of the second guide object to the liquid crystal display 15. The image of the second guide object is drawn at the position (range) determined in S6. The liquid crystal display 15 displays an image taken in advance by the front camera 19 before the distance from the vehicle to the guidance branch point becomes less than the guidance start distance, that is, the current scenery around the vehicle (especially in front of the vehicle) (actual scene image). is displayed. As a result, it becomes possible for the occupant of the vehicle to visually recognize the second guide object superimposed on the scenery.

FIG. 4 is a diagram showing an example of the travel guide screen 51 displayed on the liquid crystal display 15 in S7. As shown in FIG. 4, the liquid crystal display 15 displays a current scene 52 in front of the vehicle captured by the front camera 19. When the current position of the vehicle approaches the guidance branch point within the exit direction guidance start distance (for example, within 500 m to the guidance junction), the image of the second guidance object is superimposed on the scenery 52 in front of the vehicle. 57 is displayed. More specifically, a plurality of arrows indicating the exit direction of the guidance branch point are displayed as the image 57 of the second guidance object at a position along the future course of the vehicle above the road on which the vehicle is currently traveling.

The image 57 of the second guide object includes images of a plurality of arrow-shaped objects, and the images of the plurality of objects are arranged at predetermined intervals along the future course of the vehicle above the road on which the vehicle is currently traveling. Display the position at . The direction of each arrow indicates the exit direction of the vehicle from the guidance branch point. Further, the image 57 of the second guide object is displayed in a manner in which the relative position with respect to the vehicle is fixed (hereinafter referred to as the first aspect) when the vehicle is away from the guidance junction, and when the vehicle is away from the guidance junction. When approaching a certain point, the display is switched to a mode (hereinafter referred to as a second mode) in which the relative position with respect to the guide branch point in the scenery 52 is fixed. In particular, in the second mode, a part of the image 57 of the second guide object is fixed in a state superimposed on the guide branch point. Therefore, when the vehicle occupant visually confirms the travel guidance screen 51, he or she can accurately grasp the course of the vehicle, the position of the guidance branch point to be turned left or right, and the exit direction at the guidance intersection. Thereafter, the image 57 of the second guide object is continuously displayed until the vehicle passes the guide branch point.

Thereafter, in S8, the CPU 41 determines whether the vehicle has passed through a guidance branch point. For example, the determination is made based on the current position of the vehicle detected by the current position detection unit 11 and map information.

If it is determined that the vehicle has passed the guidance branch point (S8: YES), a control signal is sent to the liquid crystal display 15, and the guidance object displayed on the liquid crystal display 15 is hidden. (S9). In addition, when hiding the guidance object, the transmittance of the image of the displayed guidance object is increased in stages according to the distance to the guidance junction, until the vehicle finally reaches the guidance junction. It is desirable that the transmittance be 100% at the timing of the change. In addition, regarding the captured image captured by the front camera 19, that is, the scenery (actual scene image) around the vehicle (especially in front of the vehicle) at the current moment, the map image is displayed continuously for a certain period of time even after the guide object is hidden. Switch to display.

On the other hand, if it is determined that the vehicle has not passed the guidance branch point (S8: NO), the process returns to S6 and the guidance object continues to be displayed.

Next, the sub-processing of the first guide object display position determining process executed in S4 will be described based on FIG. 5. FIG. 5 is a flowchart of a sub-processing program of the first guide object display position determination process.

First, in S11, the CPU 41 specifies a "recommended lane" which is a lane in which the vehicle is recommended to drive on the road on which the vehicle is currently traveling. Specifically, the CPU 41 recommends the lane in which the vehicle needs to travel in order to pass through the guidance branch point in front of the vehicle in the exit direction (guidance direction) along the guidance route, based on the map information and the guidance route. Identify as a lane. More specifically, the lane corresponding to the exit direction of the vehicle at the guidance branch point is acquired as the recommended lane. Note that the link data 32 included in the map information stores the traffic classification of each lane in the direction of travel, and the branch point data 34 stores the shape of the branch point, and these pieces of information are used to identify the recommended lane. do. For example, as shown in FIG. 6, if the exit direction (guidance direction) at the guidance branch point 60 located in front of the vehicle traveling on the expressway is diagonally to the left of the access road, Of the three lanes on the road, the lane furthest to the left is the recommended lane. On the other hand, as shown in FIG. 7, if the exit direction (guidance direction) at the guidance branch point 60 located in front of the vehicle traveling on the general road is to the right, the exit direction (guidance direction) is to the right. The lane located on the far right side corresponding to the right-turn traffic classification is the recommended lane. Note that the recommended lane is not necessarily only one lane, but may include two or more lanes.

Next, in S12, the CPU 41 performs image processing on the image captured by the front camera 19 to recognize (detect) a feature located around the vehicle. Specifically, the marking lines drawn on the road surface (including the road surface of the lane in which the own vehicle is traveling, as well as the road surface of lanes other than the lane in which the own vehicle is traveling), and the road edges (specifically, the edges of the roadway) If there is a sidewalk, the boundary between the road and the sidewalk) is targeted for recognition (detection).

Note that the features to be recognized (detected) in S12 above basically do not include features on the oncoming lane, but, for example, on a general road, the features between the lane corresponding to the direction of travel of the host vehicle and the oncoming lane. If there is no median strip and the vehicle is divided by a center line, or if there is no center line, features in the oncoming lane will also be included in the recognition (detection) targets. However, even in this case, the number of lanes specified in S15, which will be described later, is the number of lanes corresponding to the traveling direction of the host vehicle (excluding oncoming lanes). Note that it is desirable to also detect the color and type (solid line, broken line, etc.) of the lane markings. In addition, structures such as blocks, guardrails, and median strips installed at the road edge are basically detected as road edges, but for roads that do not have such structures at the road edge, asphalt is detected. A break or the outermost division line (if the vehicle is divided by an oncoming lane and a center line, the center line also becomes the road edge) may be detected as the road edge.

The marking line detection process in S12 will be briefly described below. First, in order to detect the marking line in the image taken by the front camera 19, the CPU 41 performs brightness correction between the road surface and the marking line based on the brightness difference. Thereafter, binarization processing to separate the marking line from the image, geometric processing to correct distortion, smoothing processing to remove image noise, etc. are performed to detect the boundary line between the road surface and the marking line. The existence and type of the lot line are identified based on the detected boundary line. Further, by extracting the image portion of the range where the lane markings are detected and performing color recognition (RGB value detection), the color of the lane markings can also be detected.

Next, the road edge detection process in S12 will be briefly described. First, in order to detect the road edge in the image captured by the front camera 19, the CPU 41 detects the road surface and structures provided at the road edge (blocks, guardrails, median strips, etc.) based on the brightness difference. Perform brightness correction. Thereafter, the boundary line between the road surface and the structure is detected by performing binarization processing to separate these structures from the image, geometric processing to correct distortion, smoothing processing to remove noise from the image, etc. The existence of the road edge is identified by the detected boundary line. Note that pattern matching processing using feature points or templates may be performed to detect lane markings and road edges. Since these image recognition processes are already known, details will be omitted. Furthermore, other vehicles located on the road other than the marking lines and road edges may also be detected.

Next, in S13, the CPU 41 determines how much the image recognition result of the image taken by the front camera 19 (more specifically, the number of lanes on the road identified by image recognition and the result of the driving lane of the own vehicle) is applied. Calculate the degree of reliability that indicates whether it is reliable. In particular, in this embodiment, (a) the right-side reliability is the reliability of the image recognition result of the lane markings and road edges on the right side with respect to the direction of travel of the vehicle, and (b) the reliability on the left side with respect to the direction of travel of the vehicle Calculate the left-side reliability, which is the reliability of the image recognition results of the lane markings and road edges, and (c) the overall reliability, which is the reliability that summarizes the right-side reliability and the left-hand reliability (total reliability). do.

Below, a method for calculating each reliability including right-side reliability, left-side reliability, and overall reliability will be explained by giving an example.
[Right side reliability]
First, the reliability of each marking line (hereinafter referred to as marking line reliability C) is calculated using the following equation (1) for each marking line existing on the right side of the driving lane of the own vehicle. For example, as shown in FIG. 8, when the vehicle is driving in the center lane on a highway with three lanes on each side, there is a lane boundary line 66 and a road outside line 65 on the right side of the lane in which the vehicle is traveling; A marking line reliability C is calculated for each marking line of the book. In addition, as shown in FIG. 9, when the vehicle is traveling in the rightmost lane on a highway with three lanes on each side, there is a roadway outside line 65 on the right side of the lane in which the vehicle is traveling, and that one marking line Compartment line reliability C is calculated for . In addition, as shown in FIG. 10, when the vehicle is traveling in the leftmost lane on a highway with three lanes on each side, there is a lane boundary line 63, a lane boundary line 66, and a roadway outer line 65 on the right side of the lane in which the vehicle is traveling. exist, and the marking line reliability C is calculated for each of these three marking lines.
C=(10×V1+V2+V3+V4)/13...(1)
Here, V1 is whether or not the lot line that is the target of calculation of the lot line reliability C could be detected by the image recognition process in S12 (whether the presence of the lot line could be recognized where it should exist). , if detected, it is set to "1", and if not detected, it is set to "0".
Further, V2 indicates whether or not the color of the partition line that is the target of calculation of the partition line reliability C could be detected by the image recognition process in S12, and is "1" if it was detected, and "1" if it could not be detected. is “0”. Note that even if the color of the marking line can be detected, it may be set to "0" if it does not match the color estimated using map information or the like.
Further, V3 indicates whether or not the type of lane marking (for example, solid line, broken line) that is the target of calculation of lane marking reliability C was detected in the image recognition process of S12, and if it was detected, it is set as "1". , if it cannot be detected, it is set to "0". Note that even if the type of lane marking can be detected, it may be set to "0" if it does not match the type estimated using map information or the like.
Further, V4 detects the inclination of the marking line, which is the object of calculation of the marking line reliability C, particularly with respect to the traveling direction of the road, by the image recognition process of S12, and if the detected inclination is less than a threshold value (for example, 10 degrees) It is set to "1" when the value is higher than the threshold value, and "0" when the value is equal to or higher than the threshold value.
Thereafter, the lane marking reliability C calculated for each lane marking on the right side with respect to the lane in which the vehicle is traveling is added to calculate the right side reliability CR. Note that when calculating the right side reliability CR, it is calculated by giving more importance to the lane marking reliability C of a lane marking located further away from the vehicle than the lane marking reliability C of a lane marking nearer to the vehicle. For example, as shown in FIG. 8, when the vehicle is traveling in the center lane on an expressway with three lanes on each side, the division calculated for the lane boundary line 66 that is closest to the lane in which the vehicle is traveling is calculated. Assuming that the line reliability C is C ₁ and the marking line reliability C calculated for the roadway outside line 65 located at the second closest position to the vehicle's driving lane is C ₂ , the following formula (2) is used. The right side reliability CR is calculated.
CR=(C ₁ +2×C ₂ )/3...(2)
In addition, as shown in FIG. 9, when the vehicle is driving in the rightmost lane on a highway with three lanes on each side, the lane line reliability calculated for the only outside road line 65 on the right side of the lane the vehicle is traveling in is also calculated. Assuming that the degree C is _C1 , the right side reliability CR is calculated by the following equation (3).
CR=C ₁ ...(3)
In addition, as shown in Fig. 10, when the vehicle is traveling in the leftmost lane on a highway with three lanes on each side, the calculation is made for the lane boundary line 63 that is closest to the lane in which the vehicle is traveling. The marking line reliability C is C ₁ , the marking line reliability C calculated for the lane boundary line 66 that is the second closest to the lane in which the vehicle is traveling is C ₂ , and the reliability C is 3 for the lane in which the vehicle is traveling. If the marking line reliability C calculated for the roadway outside line 65 located at the closest position is _C3 , the right side reliability CR is calculated by the following equation (4).
CR=(C ₁ +2×C ₂ +4×C ₃ )/7 (4)
In addition, in the above formulas (2) to (4), the coefficient multiplied by each marking line reliability C can be changed as appropriate, but It is desirable to set the line reliability C to be multiplied by a larger coefficient.
The right side reliability CR finally calculated by the above formulas (2) to (4) is a value between 0 and 1, and the closer it is to 1, the more the image of the marking line and road edge is on the right side with respect to the direction of travel of the vehicle. This indicates that the reliability of the recognition result, that is, the reliability of the lane in which the vehicle is traveling specified based on the right edge of the road (the number of lanes from the right edge of the road to the lane in which the vehicle is traveling) is high.
[Left side reliability]
First, for each marking line that is estimated to exist on the left side with respect to the driving lane of the host vehicle, the marking line reliability C for each marking line is similarly calculated using the above-mentioned formula (1). For example, as shown in FIG. 8, when the vehicle is driving in the center lane on a highway with three lanes on each side, there is a roadway outside line 62 and a lane boundary line 63 on the left side of the lane in which the vehicle is traveling; A marking line reliability C is calculated for each marking line of the book. In addition, as shown in FIG. 9, when the vehicle is traveling in the rightmost lane on a highway with three lanes on each side, there is a roadway outer line 62, a lane boundary line 63, and a lane boundary line 66 on the left side of the vehicle's driving lane. exist, and the marking line reliability C is calculated for each of these three marking lines. In addition, as shown in FIG. 10, when the vehicle is driving in the leftmost lane on a highway with three lanes on each side, there is a roadway outside line 62 on the left side of the lane in which the vehicle is traveling, and that one marking line Compartment line reliability C is calculated for . Note that the calculation method for the lane marking reliability C is the same as that for the right-side reliability described above, so a description thereof will be omitted.
Thereafter, the left side reliability CL is calculated by adding the lane marking reliability C calculated for each lane marking that is estimated to exist on the left side with respect to the lane in which the vehicle is traveling. Note that when calculating the left-side reliability CL, the calculation is performed with more emphasis on the lane marking reliability C of a lane marking located further away from the vehicle than the lane line reliability C of a lane marking nearer to the vehicle. For example, as shown in FIG. 8, when the vehicle is traveling in the center lane on an expressway with three lanes on each side, the division calculated for the lane boundary line 63 that is closest to the lane in which the vehicle is traveling is calculated. Assuming that the line reliability C is C ₁ and the marking line reliability C calculated for the roadway outside line 62 located at the second closest position to the vehicle's driving lane is C ₂ , the following formula (5) is used. The left side reliability CL is calculated.
CL=(C ₁ +2×C ₂ )/3 (5)
In addition, as shown in FIG. 9, when the vehicle is traveling in the rightmost lane on an expressway with three lanes on each side, the calculation is made for the lane boundary line 66 that is closest to the lane in which the vehicle is traveling. The marking line reliability C is C ₁ , the marking line reliability C calculated for the lane boundary line 63 at the second closest position to the lane in which the vehicle is traveling is C ₂ , and the reliability C is 3 for the lane in which the vehicle is traveling. If the marking line reliability C calculated for the roadway outside line 62 located at the closest position is _C3 , then the left side reliability CL is calculated by the following equation (6).
CL=(C ₁ +2×C ₂ +4×C ₃ )/7 (6)
In addition, as shown in FIG. 10, when the vehicle is driving in the leftmost lane on a highway with three lanes on each side, the lane marking reliability calculated for the only outside road line 62 on the left side of the lane in which the vehicle is traveling is calculated. Assuming that the degree C is _C1 , the left side reliability degree CL is calculated by the following equation (7).
CL=C ₁ ...(7)
In addition, in the above formulas (5) to (7), the coefficient multiplied by each marking line reliability C can be changed as appropriate, but It is desirable to set the line reliability C to be multiplied by a larger coefficient.
The left side reliability CL finally calculated by the above equations (5) to (7) is a value between 0 and 1, and the closer it is to 1, the more the image of the lane marking and road edge is on the left with respect to the vehicle's direction of travel. This indicates that the reliability of the recognition result, that is, the reliability of the lane in which the vehicle is traveling specified based on the left edge of the road (the number of lanes from the left edge of the road to the lane in which the vehicle is traveling) is high.
[Overall reliability]
Finally, the overall reliability CT is calculated by adding the right reliability CR and the left reliability CL calculated as described above. Note that when calculating the overall reliability CT, the right-side reliability CR and the left-side reliability CL may be simply added, or either one may be weighted and added. For example, the overall reliability CT is calculated using the following equation (8).
CT=(CR+CL)/2...(8)
The overall reliability CT finally calculated by the above formula (8) is a value between 0 and 1, and the closer it is to 1, the more reliable is the image recognition result of the image taken by the front camera 19 regardless of the left or right side. Indicates a high degree of

Subsequently, in S14, the CPU 41 determines whether the overall reliability CT calculated in S13 is greater than or equal to a predetermined reference value. Note that the reference value can be set as appropriate, but is set to 0.8, for example.

In S14, it may be determined whether the right side reliability CR or the left side reliability CL is equal to or higher than a reference value, instead of the overall reliability CT. More specifically, it is determined whether or not the reliability corresponding to the exit direction of the vehicle at the guidance branch point is greater than or equal to a reference value, out of the right side reliability CR and the left side reliability CL calculated in S13. In addition, the reliability corresponding to the exit direction of the vehicle is, for example, if the vehicle exits to the right at the guidance junction, the right reliability CR corresponds to the exit direction of the vehicle; If the vehicle exits to the left, the left side reliability CL corresponds to the exit direction of the vehicle. Here, as shown in FIGS. 6 and 7, the recommended lane basically exists in the exit direction of the guidance branch point. Therefore, for example, when the exit direction of a guidance junction is to the right, if the right side reliability CR, which is the reliability of the position of the own vehicle's lane specified with the right edge of the road as a reference, is high, at least the own vehicle's lane and the recommended lane The positional relationship is reliable. That is, when superimposing the actual scene, it is possible to accurately estimate the positions of the own vehicle driving lane and the recommended lane in the scenery. On the other hand, if the exit direction of the guidance junction is to the left, if the left side reliability CL, which is the reliability of the position of the own vehicle's lane identified with the left edge of the road as a reference, is high, at least the own vehicle's lane and the recommended lane The positional relationship is reliable. That is, when superimposing the actual scene, it is possible to accurately estimate the positions of the own vehicle driving lane and the recommended lane in the scenery.

If it is determined that the overall reliability CT calculated in S14 is equal to or higher than the reference value (S14: YES), then based on the image recognition result of S12 (more specifically, the image recognition result will be described later), The number of lanes on the road specified in S15, S18, and S20) are deemed to be sufficiently reliable, and the process moves to S15. On the other hand, if it is determined that the overall reliability CT calculated in S14 is less than the reference value (S14: NO), then the The number of lanes on the road specified in S15, S18, and S20, which will be described later, and the result of the lane in which the host vehicle is traveling) are deemed unreliable, and the process moves to S26.

Next, in S15, the CPU 41 specifies the "number of vehicle road lanes", which is the number of lanes of the road on which the vehicle is currently traveling, based on the result of the image recognition process performed in S12. Specifically, the number of vehicle lanes on the road is determined by the following process. Note that what is specified in S15 is the number of lanes corresponding to the traveling direction of the host vehicle, excluding oncoming lanes.

[Identification of number of vehicle road lanes]
For example, to explain the case where the vehicle is traveling on an expressway with three lanes on each side as shown in FIGS. 8 to 10, first, the CPU 41 obtains the recognition results of the road edge and marking line on the left side of the vehicle. Regarding the recognition result of the lot line, the color and type of the lot line are also obtained. As a result, for example, a road edge 61 is detected, a road outside line 62 which is a solid white line adjacent to the road edge 61 is detected, and a lane boundary line 63 which is a white dashed line or a solid line is detected between the road outside line 62 and the vehicle. If one line is detected, it can be determined that the vehicle is traveling in the second lane from the left as shown in FIG. Furthermore, the CPU 41 similarly acquires the recognition results of the road edge and lane markings on the right side. Regarding the recognition result of the lot line, the color and type of the lot line are also obtained. As a result, for example, a road edge 64 is detected, a road outside line 65 which is a solid white line adjacent to the road edge 64 is detected, and a lane boundary line 66 which is a white dashed line or a solid line is detected between the road outside line 65 and the vehicle. If one line is detected, it can be determined that the vehicle is traveling in the second lane from the right as shown in FIG. By combining the results, it can be determined that the vehicle is traveling in the second lane from the left and the second lane from the right, that is, the number of lanes on the vehicle road is three.
Similarly, if two lane boundary lines 63 are detected between the outer road line 62 and the vehicle, it can be determined that the own vehicle is running in the third lane from the left as shown in FIG. If the lane boundary line 66 is not detected between the outside line 65 and the vehicle, it can be determined that the vehicle is traveling in the rightmost lane as shown in FIG. By combining the results, it can be determined that the vehicle is traveling in the third and rightmost lane from the left, that is, the number of lanes on the vehicle road is three.
Similarly, if the lane boundary line 63 is not detected between the outer road line 62 and the vehicle, it can be determined that the own vehicle is traveling in the leftmost lane as shown in FIG. If two lane boundary lines 66 are detected between the vehicle and the vehicle, it can be determined that the vehicle is traveling in the third lane from the right as shown in FIG. By combining the results, it can be determined that the vehicle is traveling in the leftmost lane and the third lane from the right, that is, the number of lanes on the vehicle road is three.

Note that the detection result of other vehicles traveling on the road other than the marking lines and road edges may be used to specify the number of lanes of the own vehicle road. For example, even if lane markings cannot be clearly detected, it is possible to estimate the presence of lanes based on the positions of other vehicles. Moreover, by detecting an oncoming vehicle, especially in a two-way traffic section, it is possible to estimate whether the lane corresponds to the direction in which the own vehicle is traveling or whether it is an oncoming lane.

Next, in S16, the CPU 41 obtains the "number of vehicle road lanes", which is the number of lanes of the road on which the vehicle is currently traveling (excluding oncoming lanes), from the map information stored in the map information DB 31. Note that what is acquired in S16 is the number of lanes corresponding to the traveling direction of the host vehicle, excluding oncoming lanes.

Here, the link data 32 included in the map information includes information regarding each link constituting the road network, but when there is a point on the road where the number of lanes increases or decreases as shown in FIG. The links are divided at points where the number of lanes increases or decreases, and the number of lanes is associated with each divided link. For example, in the example shown in FIG. 11, the link is divided into link A and link B, and the link data 32 stores "3" as the number of lanes for link A and "2" as the number of lanes for link B. Therefore, in S16, the CPU 41 specifies the current position of the own vehicle based on the detection result of the current position detection unit 11, matches the current position of the own vehicle with map information, and links the current position of the own vehicle with the link. Get the number of linked lanes.

Subsequently, in S17, the CPU 41 determines whether the number of lanes of the own vehicle's road identified by the result of the image recognition process in S15 matches the number of lanes of the road on which the vehicle is currently traveling, which is included in the map information acquired in S16. Determine whether or not to do so.

Here, as described above, the link data 32 included in the map information has information on the number of lanes on the road, but this information may differ from the actual road. For example, as shown in Figure 12, the boundary between Link A and Link B in map information (the point at which the number of lanes increases or decreases in the map information) and the boundary where 2 lanes and 3 lanes change on the actual road (the point at which the number of lanes actually increases or decreases) ) may not match completely. In that case, when a vehicle travels between a point where the number of lanes increases or decreases in the map information and a point where the number of lanes actually increases or decreases, the detection result of the number of lanes based on the image taken by the front camera 19 is reflected in the map information. Situations that do not match may also occur.

If it is determined that the number of vehicle road lanes specified by the result of the image recognition process in S15 does not match the number of vehicle road lanes included in the map information acquired in S16 (S17: NO). ), the process moves to S18. On the other hand, if it is determined that the number of lanes of the own vehicle's road identified by the result of the image recognition process in S15 matches the number of lanes of the own vehicle's road included in the map information acquired in S16 (S17: If YES), the process moves to S20.

In S18, the CPU 41 considers that the map information is unreliable, and determines the "own vehicle driving lane" which is the driving lane in which the own vehicle is currently traveling based on the detection result of the image recognition process performed in S12 without using the map information. Identify. In the subsequent processing, the number of lanes of the own vehicle's road, which is the number of lanes of the road on which the own vehicle travels, is the number of lanes of the own vehicle's road identified by the result of the image recognition process in S15. Specifically, the lane in which the vehicle is traveling is specified by the following process.

Examples of methods for specifying the lane in which the vehicle is traveling include a method using the type and number of marking lines, and a method using the distance from the road edge and lane width.
[Identification of vehicle driving lane (pattern 1)]
The same method is used to identify the number of vehicle lanes on the road as described above. That is, first, the CPU 41 obtains the recognition results of the road edge and marking line on the left side of the own vehicle. Regarding the recognition result of the lot line, the color and type of the lot line are also obtained. As a result, for example, a road edge 61 is detected, a road outside line 62 which is a solid white line adjacent to the road edge 61 is detected, and a lane boundary line 63 which is a white dashed line or a solid line is detected between the road outside line 62 and the vehicle. If one line is detected, it can be determined that the vehicle is traveling in the second lane from the left as shown in FIG. Furthermore, the CPU 41 similarly acquires the recognition results of the road edge and lane markings on the right side. Regarding the recognition result of the lot line, the color and type of the lot line are also obtained. As a result, for example, a road edge 64 is detected, a road outside line 65 which is a solid white line adjacent to the road edge 64 is detected, and a lane boundary line 66 which is a white dashed line or a solid line is detected between the road outside line 65 and the vehicle. If one line is detected, it can be determined that the vehicle is traveling in the second lane from the right as shown in FIG. By combining the results, it can be determined that the vehicle is traveling in the second lane from the left and the second lane from the right on a three-lane road.
Similarly, if two lane boundary lines 63 are detected between the outer road line 62 and the vehicle, it can be determined that the own vehicle is running in the third lane from the left as shown in FIG. If the lane boundary line 66 is not detected between the outside line 65 and the vehicle, it can be determined that the vehicle is traveling in the rightmost lane as shown in FIG. By combining the results, it can be determined that the vehicle is traveling in the third lane from the left and the farthest right lane.
Similarly, if the lane boundary line 63 is not detected between the outer road line 62 and the vehicle, it can be determined that the own vehicle is traveling in the leftmost lane as shown in FIG. If two lane boundary lines 66 are detected between the vehicle and the vehicle, it can be determined that the vehicle is traveling in the third lane from the right as shown in FIG. By combining the results, it can be determined that the vehicle is traveling in the leftmost lane and the third lane from the right.
In addition, if there is a discrepancy between the identification result of the vehicle's driving lane from the right side of the road and the result of identifying the vehicle's driving lane from the left side of the road (for example, even though the number of vehicle's lanes is 2) When it is determined that the vehicle is traveling in the second lane from the left and the second lane from the right), the identification result from the road edge with the higher reliability of the right reliability CR and the left reliability CL is used. Prioritize.
[Identification of vehicle driving lane (pattern 2)]
Next, the method of using the distance from the road edge and the lane width will be explained using the example of driving on an expressway with three lanes on each side as shown in FIGS. 8 to 10. First, the CPU 41 is located on both sides of the vehicle. Partition lines (lane boundary line 63 and lane boundary line 66 in the example shown in FIG. 8, lane boundary line 66 and road outside line 65 in the example shown in FIG. 9, road outside line 62 and lane boundary line 63 in the example shown in FIG. 10) ) is detected and the width of the lane on which the vehicle is traveling is determined from the distance between the lane markings. Thereafter, the CPU 41 calculates the distance along the road width direction from the road edge 61 or outer road line 62 on the left side of the own vehicle to the own vehicle, and calculates how many lanes the calculated distance is wide. . As a result, if the distance from the road edge 61 or outer road line 62 to the own vehicle is the distance of one lane, the own vehicle is traveling in the second lane from the left as shown in Figure 8. It can be identified as Similarly, if the distance from the road edge 61 or outer road line 62 to the vehicle is the distance of two lanes, the vehicle is traveling in the third lane from the left as shown in Figure 9. It can be identified as If the distance from the road edge 61 or outer road line 62 to the vehicle is less than one lane, it can be determined that the vehicle is traveling in the leftmost lane as shown in FIG.
Further, the CPU 41 similarly calculates the distance from the road edge 64 or the outer road line 65 on the right side. As a result, if the distance from the road edge 64 or outer road line 65 to the own vehicle is the distance of one lane, the own vehicle is traveling in the second lane from the right as shown in Figure 8. It can be identified as Similarly, if the distance from the road edge 64 or outer road line 65 to the vehicle is the distance of two lanes, the vehicle is traveling in the third lane from the right as shown in Figure 10. It can be identified as If the distance from the road edge 64 or outer road line 65 to the vehicle is less than one lane, it can be determined that the vehicle is traveling in the rightmost lane as shown in FIG.
As described above, the CPU 41 determines the position of the lane in which the vehicle is traveling, which is specified with the left edge of the road as a reference (the number of lanes from the left edge of the road to the lane in which the vehicle is currently traveling), and the position of the lane in which the vehicle is traveling, which is specified with the right edge of the road as a reference. (the number of lanes from the right edge of the road to the lane in which the vehicle is currently traveling) is finally combined to identify the lane in which the vehicle is traveling. For example, in the example shown in FIG. 8, the lane in which the vehicle is traveling is specified as the second lane from the left and the second lane from the right. In the example shown in FIG. 9, the lane in which the vehicle is traveling is specified as the third lane from the left and the first lane from the right. In the example shown in FIG. 10, the lane in which the vehicle is traveling is specified as the first lane from the left and the third lane from the right.
In addition, if there is a discrepancy between the identification result of the vehicle's driving lane from the right side of the road and the result of identifying the vehicle's driving lane from the left side of the road (for example, even though the number of vehicle's lanes is 2) When it is determined that the vehicle is traveling in the second lane from the left and the second lane from the right), the identification result from the road edge with the higher reliability of the right reliability CR and the left reliability CL is used. Prioritize.

Note that detection results of other vehicles traveling on the road other than marking lines and road edges may be used to identify the lane in which the vehicle is traveling. For example, even if lane markings cannot be clearly detected, it is possible to estimate the presence of lanes based on the positions of other vehicles. Moreover, by detecting an oncoming vehicle, especially in a two-way traffic section, it is possible to estimate whether the lane corresponds to the direction in which the own vehicle is traveling or whether it is an oncoming lane.

Thereafter, in S19, the CPU 41 determines whether there is a lane in which the vehicle is traveling in the scenery around the vehicle, based on the detection result of the lane markings acquired in the image recognition process in S12 and the lane in which the vehicle is traveling identified in S18. Specify the range. Similarly, the range where the recommended lane exists in the scenery around the vehicle is also specified based on the lane markings detected in the image recognition process of S12 and the lane in which the vehicle is traveling specified in S18. Specifically, within the captured image captured by the front camera 19 (that is, the captured image displayed on the liquid crystal display 15), the range in which the own vehicle driving lane and the recommended lane exist is specified. Note that, as in S18, the map information is regarded as unreliable, and the range in which the own vehicle driving lane and the recommended lane exist is basically specified without using the map information.

For example, to explain a case where a vehicle is traveling on a road with three lanes on each side, the own lane is the center lane, and the recommended lane is the left lane, in S19, the captured image shown in FIG. When 71 is imaged by the front camera 19, each of the partition lines 72 to 74 is detected. However, regarding the marking line 74, the lane width is calculated from the detection results of the marking line 72 and marking line 73, and the position is determined based on the assumption that the marking line 74 is located on the left side of the marking line 72 and separated by the calculated lane width. may be predicted. As a result, since it has been determined that the recommended lane is the leftmost lane, a range 75 surrounded by the lane markings 72 and 74 can be identified as the range where the recommended lane exists. Further, since it has been determined that the lane in which the vehicle is traveling is the center lane, the range 76 surrounded by the marking lines 72 and 73 can be specified as the range in which the lane in which the vehicle is traveling exists. After that, the process moves to S22.

On the other hand, S20 is executed when it is determined that the number of vehicle road lanes specified by the result of the image recognition process in S15 matches the number of vehicle road lanes included in the map information acquired in S16. Then, the CPU 41 considers that the map information is reliable. Then, based on the detection result of the image recognition process performed in S12, the map information is also used to identify the "own vehicle lane" which is the lane in which the host vehicle is currently traveling. In addition, in the subsequent processing, the number of own vehicle road lanes, which is the number of lanes of the road on which the own vehicle travels, is determined to be the number of own vehicle road lanes that is determined to match.

In S20, the CPU 41 basically performs the same processing as the above-mentioned [Identification of own vehicle travel lane (pattern 1)] and [Identification of own vehicle travel lane (pattern 2)]. However, in S20, it is desirable to specify the lane in which the vehicle is traveling by taking into consideration map information in addition to the recognition results of road edges and lane markings. Note that the link data 32 included in the map information stores the number of lanes for each road, the presence or absence of oncoming lanes (whether or not it is a two-way traffic section), the presence or absence of a median strip, etc. With image recognition by the front camera 19, for example, when a solid or broken white line is detected near the center of a general road, it is difficult to determine whether the section line is a center line or a lane boundary line. , it becomes possible to easily make this determination by using map information. As a result, even in a section where there is an oncoming lane, it is possible to accurately identify the lane in which the vehicle is traveling, taking into account the presence of the oncoming lane. For example, if a vehicle is driving on a four-lane road with two lanes on each side without a median strip, and it is determined that the vehicle is traveling in the third lane from the right, the map information indicates that the vehicle is traveling in the third lane from the right. Since the second lane from the right can be determined to be the oncoming lane, it can be determined that the vehicle is traveling in the right lane of a road with two lanes on each side.

Thereafter, in S21, the CPU 41 determines whether there is a lane in which the vehicle is traveling in the scenery around the vehicle, based on the lane markings detected in the image recognition process in S12 and the lane in which the vehicle is traveling identified in S20. Specify the range. Similarly, the range where the recommended lane exists in the scenery around the vehicle is also specified based on the detection result of the lane markings obtained in the image recognition process of S12 and the lane in which the vehicle is running specified in S20. Basically, the same processing as in S19 is performed, but map information is also taken into account to specify the range where the own vehicle driving lane and the recommended lane exist.

For example, as mentioned above, if map information is used, it is possible to easily determine the presence or absence of oncoming lanes and the presence or absence of median strips, which are difficult to grasp with image recognition alone. can be specified more easily and accurately. Furthermore, the map information also includes information about the road shape (curve radius, etc.), road width, and lane width, so even in areas where it is difficult to accurately detect lane markings that are far away from your vehicle, the map information also includes information about the road shape and lane width. By estimating the position of the lane markings using the lane width, it is possible to more accurately identify the range where the recommended lane exists and the range where the vehicle is traveling. After that, the process moves to S22.

Subsequently, in S22, the CPU 41 determines whether the distance to the next guidance branch point is less than a threshold value. Note that the threshold value is determined depending on the type of road on which the vehicle travels; for example, an expressway is 700 m, and a general road is 150 m, which is shorter than the expressway.

If it is determined that the distance to the next guidance branch point is less than the threshold (S22: YES), the process moves to S24. On the other hand, if it is determined that the distance to the next guidance branch point is equal to or greater than the threshold value (S22: NO), the process moves to S23.

In S23, the CPU 41 calculates the size, shape, and display position of the first guide object 80 to be displayed on the liquid crystal display 15. Here, the first guide object 80 to be displayed in S23 has a rectangular shape as shown in FIG. 14, and its horizontal length is set to the lane width of the recommended lane and its vertical length is set to, for example, 100 m. Then, it is displayed so as to be superimposed on the road surface of the recommended lane within a range of 100 meters from the current position of the own vehicle. Therefore, in S23, the CPU 41 uses the result of S19 or S21 to specify the range of the recommended lane in the scenery displayed on the liquid crystal display 15, and further uses the range of the recommended lane from the current position of the own vehicle on the road surface of the specified recommended lane. The size, shape, and position of the first guide object 80 necessary to cover an area up to 100 meters ahead are calculated as the size, shape, and display position of the first guide object 80 to be displayed on the liquid crystal display 15. .

Thereafter, the process moves to S5, where an image of the first guide object 80 having the size and shape calculated in S23 is generated, and a control signal is sent to the liquid crystal display 15, so that the image of the first guide object 80 having the size and shape calculated in S23 is generated. The image of the first guide object 80 is drawn at the position (range) determined in S23. Note that the shape of the first guide object 80 to be displayed in S23 can be changed as appropriate, and may be any shape other than a rectangle as long as it can indicate the recommended lane. Additionally, the distance to the guide branch point may be drawn inside.

On the other hand, in S24, the CPU 41 determines whether or not the own vehicle traveling lane and the recommended lane match, based on the recommended lane and the own vehicle traveling lane specified in S11, S18, and S20, that is, the own vehicle is traveling in the recommended lane. Determine whether or not.

Then, if it is determined that the vehicle driving lane and the recommended lane match, that is, the vehicle is traveling in the recommended lane (S24: YES), the process moves to S23. In S23, as described above, the rectangular first guide object 80 that overlaps the recommended lane (which also coincides with the vehicle's travel lane) is set as the display target, and the size and shape of the first guide object 80 to be displayed on the liquid crystal display 15 are determined. , and calculate the display position. On the other hand, if it is determined that the vehicle driving lane and the recommended lane do not match, that is, the vehicle is not traveling in the recommended lane (S24: NO), the process moves to S25.

In S25, the CPU 41 calculates the size, shape, and display position of the first guide object 80 to be displayed on the liquid crystal display 15. Here, as shown in FIG. 15, the first guide object 80 to be displayed in S25 crosses over from the own vehicle driving lane to the recommended lane, and as the vehicle moves in the opposite direction to the traveling direction of the vehicle, the vehicle is currently traveling. It has a shape that gradually widens toward the lane. More specifically, the first guide object 80 includes a first area (rectangle) that is the road surface of the recommended lane up to, for example, 100 meters in front of the current position of the vehicle, and a first area that is between the recommended lane and the lane in which the vehicle is traveling. On the other hand, it has a shape that combines a second region (a right triangle) which is adjacent to the region and whose width gradually becomes narrower as it moves in the direction of travel of the vehicle. Then, it is displayed so as to be superimposed on the road surface of the recommended lane and the vehicle's driving lane within a range of 100 meters from the vehicle's current position. Therefore, in S25, the CPU 41 uses the results of S19 or S21 to specify the range of the recommended lane and the lane in which the vehicle is traveling within the scenery displayed on the liquid crystal display 15, and further specifies the range of the road surface of the specified recommended lane. The size of the first guide object 80 necessary to cover an area up to 100 meters ahead from the current position of the vehicle and a part of the range up to 50 meters ahead from the current location of the vehicle on the road surface of the vehicle driving lane; The shape and position are calculated as the size, shape, and display position of the first guide object 80 to be displayed on the liquid crystal display 15.

Thereafter, the process moves to S5, where an image of the first guide object 80 having the size and shape calculated in S25 is generated, and a control signal is sent to the liquid crystal display 15, so that the image of the first guide object 80 having the size and shape calculated in S25 is generated. The image of the first guide object 80 is drawn at the position (range) determined in S25. Note that the shape of the first guide object 80 to be displayed in S25 can be changed as appropriate, and may be any shape other than the shape that widens toward the end as long as it can indicate the recommended lane and encourage movement to the recommended lane. Also good. Additionally, the distance to the guide branch point may be drawn inside.

On the other hand, in S26, which is executed when it is determined in S14 that the reliability of the image recognition result is less than the reference value, the CPU 41 determines not to draw the guide object because there is a risk that incorrect guidance will be provided. . In that case, the display of the first guide object is not performed in the subsequent S5. As mentioned above, the liquid crystal display 15 displays an image taken in advance by the front camera 19 before the distance from the vehicle to the guidance branch point becomes less than the guidance start distance, that is, the current scenery around the vehicle (especially in front of the vehicle). (actual scene image) is displayed, but after deciding not to draw the guide object, you can continue to display only the captured image, or display the map image without displaying the captured image. You can also do it. Note that the processes from S11 onward are repeated at predetermined intervals, so even if the reliability of the image recognition result is once determined to be less than the reference value, the reliability of the image recognition result is subsequently determined to be greater than or equal to the reference value. At the time when the first guide object is drawn, the first guide object is drawn.

On the other hand, in S26, the first guidance is performed in a manner that does not depend on the lane in which the vehicle is traveling (does not use the lane in which the vehicle is traveling, is not based on the lane in which the vehicle is traveling, is not influenced by the lane in which the vehicle is traveling), which is different from S23 and S25. It is also possible to decide to display the object 80 and calculate the size, shape, and display position of the first guide object 80 to be displayed on the liquid crystal display 15. Here, the aspect that does not depend on the driving lane of the vehicle is, for example, an arrow indicating only the exit direction of a guidance branch point, and more specifically, an arrow that indicates only the exit direction of a guidance branch point, and more specifically, an arrow that is a predetermined distance ahead of the vehicle's current position (for example, 10 meters ahead) and on the road surface. Three arrows in a triangular shape indicate the exit direction of a guide branch point located at a predetermined height (for example, 1 m) above and ahead in the direction of travel of the vehicle. Thereafter, the process moves to S5, where an image of the first guide object 80 having the size and shape calculated in S26 is generated, and a control signal is sent to the liquid crystal display 15 to generate an image of the first guide object 80. The image of the first guide object 80 is drawn at the position (range) determined in S26. Note that the first guidance object 80 to be displayed in S26 is simply an arrow indicating only the exit direction of the guidance branch point, so even if the number of lanes on the road for which the vehicle is traveling or the lane in which the vehicle is traveling cannot be specified, the guidance will be incorrect. There is no risk.

As a result of the above processing, when the distance from the vehicle to the guidance branch point is less than the guidance start distance and greater than or equal to the exit direction guidance start distance, the travel guidance screen is displayed on the liquid crystal display 15 as the vehicle travels. 51 changes as shown in FIG. 16, and the screen becomes as shown in FIGS. 17 to 20.
Taking the example of a vehicle driving on an expressway, if the distance from the vehicle to the guidance junction is less than the guidance start distance (1 km on an expressway) but greater than the threshold (700m on an expressway), the "recommended" An image 53 (first display mode) of a first guide object that guides the position of the lane is displayed. The "image 53 of the first guide object guiding the position of the recommended lane" is displayed superimposed on the recommended lane 54 as shown in FIG. 17 (S23). As a result, it becomes possible for the occupant to clearly grasp the existence and position of the recommended lane 54.

In addition, when the distance from the vehicle to the guidance branch point is less than the threshold (700 m on expressways) but greater than the exit direction guidance start distance (500 m on expressways), the image of the first guidance object prompting movement to the recommended lane is displayed. 53 (second display mode)" is newly displayed in place of the above-mentioned "image 53 (first display mode) of the first guide object guiding the position of the recommended lane." However, the "image 53 of the first guide object that prompts movement to the recommended lane" is not displayed when the vehicle's driving lane 55 matches the recommended lane 54, and in that case, as shown in FIG. An image 53 of the first guide object is displayed superimposed on the recommended lane 54 (own vehicle travel lane 55) in which the car is traveling (first display mode). As a result, it becomes possible for the occupant to clearly understand that the vehicle is already traveling in the recommended lane 54. On the other hand, if the own vehicle driving lane 55 and the recommended lane 54 do not match, the image 53 of the first guide object is displayed superimposed on the area spanning from the own vehicle driving lane 55 to the recommended lane 54, as shown in FIG. (second display mode). As a result, it becomes possible for the occupant to clearly understand that the lane in which the vehicle is traveling is not the recommended lane and that it is necessary to move to the recommended lane. In addition, in the first embodiment, since the image 53 of the first guide object is displayed over a wide area including the own vehicle driving lane 55 to the recommended lane 54, the recommended lane is more It is possible to suppress guidance that feels forced to prompt a lane change.

Note that in a state where the distance from the vehicle to the guidance branch point is less than the threshold value (700 m on an expressway) and greater than the exit direction guidance start distance (500 m on an expressway), either the driving guidance screen 51 shown in FIG. 18 or FIG. 19 is displayed. (that is, whether to display in the first display mode or the second display mode) is determined by the positional relationship between the own vehicle driving lane 55 and the recommended lane 54 at that time. Therefore, for example, if the vehicle moves to the recommended lane 54 on the left side while the driving guidance screen 51 of FIG. 19 is being displayed on the liquid crystal display 15, the screen will switch to the driving guidance screen 51 of FIG. 18. Similarly, if the vehicle moves to a lane other than the recommended lane 54 while the driving guidance screen 51 of FIG. 18 is being displayed on the liquid crystal display 15, the screen will switch to the driving guidance screen 51 of FIG. 19.

Furthermore, the driving screens shown in FIGS. 17 to 19 are displayed only when the reliability of the image recognition results is determined to be equal to or higher than the reference value, and when the reliability of the image recognition results is less than the reference value. If it is determined that this is the case, as shown in FIG. 20, an image 53 of the first guide object, which is an arrow indicating the exit direction of the guide branch point, is displayed (third display mode). As a result, even if it is not possible to specify the lane in which the vehicle is traveling, it is possible for the occupants to at least know the exit direction of the guidance junction (which basically corresponds to the direction in which the recommended lane exists). Become.

If the distance from the vehicle to the guidance junction becomes less than the exit direction guidance start distance (500 m on an expressway), the image 53 of the first guidance object will be replaced with the exit direction of the guidance junction as described above. An image 57 of the second guide object, which is the indicated arrow, will be newly displayed (see FIG. 4).

In addition, in the above-described embodiment, the range in which the own vehicle driving lane and the recommended lane exist are specified in the image taken by the front camera 19 in S21, but if the map information is reliable, Based on this, a three-dimensional space corresponding to the vicinity of the vehicle's current position (particularly in front of the vehicle in the direction of travel) may be generated, and the range in which the own vehicle driving lane and the recommended lane exist may be specified within the three-dimensional space. In addition to roads, buildings, road signs, etc. may also be modeled in the three-dimensional space, or only roads may be modeled. Alternatively, it may be a simple blank three-dimensional space containing only the ground without modeling the road. An example using the above three-dimensional space will be described below.

First, the CPU 41 specifies the current position and orientation of the host vehicle in the generated three-dimensional space. In particular, the position of the front camera 19 installed on the vehicle is taken as the current position of the own vehicle, and the optical axis direction of the front camera 19 is taken as the orientation of the own vehicle. The position of the front camera 19 also corresponds to the position of the vehicle occupant, and the optical axis direction of the front camera 19 also corresponds to the line of sight direction of the vehicle occupant. Furthermore, the position of a guidance branch point located ahead in the direction of travel of the vehicle in the three-dimensional space is also specified.

Then, in S23 and S25, the CPU 41 generates the first guide object 80 to be displayed as a two-dimensional polygon, for example, and arranges the generated first guide object 80 in the generated three-dimensional space. Note that the position where the first guide object 80 is placed in the three-dimensional space is the position shown in FIGS. 14 and 15. Next, the CPU 41 selects a first guide object 80 that can be visually recognized when viewed from the vehicle's current position and the position at the height of the front camera 19 in the traveling direction of the vehicle in the three-dimensional space where the first guide object 80 is arranged. are calculated as the size and shape of the guide object to be displayed on the liquid crystal display 15. Here, the calculated size and shape of the guide object are the first guide object that can be visually recognized when the first guide object 80 arranged in three-dimensional space is viewed from the viewpoint of the current vehicle (more precisely, the front camera 19). 1. The size and shape of the guide object 80.

Thereafter, the CPU 41 displays the scenery displayed on the liquid crystal display 15 based on the current position of the vehicle in the three-dimensional space, the position of the guidance branch point, and the positions of the recommended lane and the lane in which the vehicle is traveling, identified in S11 and S20. The position of the recommended lane and the lane in which the vehicle is traveling within 52 is estimated, and the position where the first guide object 80 is displayed on the liquid crystal display 15 is determined.

As explained in detail above, according to the navigation device 1 according to the first embodiment and the computer program executed by the navigation device 1, based on the captured image of the scenery around the vehicle, the road on which the vehicle is currently traveling is based on the captured image of the scenery around the vehicle. (S15), and if the number of lanes of the road on which the vehicle currently travels included in the map information does not match the number of lanes identified based on the captured image, the number of lanes identified based on the captured image is determined. The number of lanes on which the vehicle is currently traveling is used to identify the driving lane in which the vehicle is currently traveling (S18). If the number of lanes specified based on the number of lanes matches, the driving lane is specified using the matching number of lanes (S20). Then, the guide object is displayed based on the specified driving lane (S23, S25), so even if there is a discrepancy or discrepancy between the image recognition result of the captured image and the map information, there will be no error in the surrounding area. To prevent a superimposed image display from being displayed in the environment, and to avoid causing disadvantages to vehicle occupants.
Also, based on the captured image, the lane markings around the vehicle and the road edge of the road on which the vehicle is currently traveling are detected (S12), and based on the detection results of the lane markings and road edge, the road edge on the left side is detected. The number of left lanes, which is the number of lanes from The number of lanes included in the road on which the vehicle is currently traveling is determined by combining the number of lanes on the right side with It is possible to specify the number of lanes, and furthermore, it is also possible to specify the lane in which the vehicle is currently traveling on the road on which the vehicle is currently traveling.
In addition, the reliability level indicating how reliable the image recognition of the captured image is is acquired (S13), and if the reliability level is higher than the standard, the number of lanes of the road on which the vehicle included in the map information is currently traveling is acquired. The driving lane is determined by determining whether or not the number of lanes and the number of lanes specified by the lane number identifying means match (S17, S18, S20), and a guidance object is displayed based on the specified driving lane (S23). , S25), the number of lanes is compared only when the number of lanes identified based on the captured image is reliable, that is, when it can be accurately compared with map information, and as a result, the number of lanes is compared based on the identified driving lane. Guide objects can now be displayed.
In addition, if the reliability is lower than the standard, the guide object is not displayed or the guide object is displayed in a manner that does not depend on the driving lane. In a situation where a certain driving lane cannot be accurately specified, by not providing guidance or not providing guidance that is likely to be erroneous, it is possible to prevent disadvantages to the occupants of the vehicle.
In addition, the recommended lane in which the vehicle is currently traveling is acquired (S11), and the range where the recommended lane exists in the scenery is specified based on the identification result of the vehicle's driving lane (S19, S21). Since the guide object is displayed superimposed on the range where the specified recommended lane exists, when the user visually recognizes the scenery around the vehicle, it becomes possible to visually recognize the guide object appropriately superimposed on the recommended lane.
In addition, if there is a guidance branch point within a predetermined guidance start distance ahead in the direction of travel of the vehicle, the lane corresponding to the exit direction of the vehicle at the guidance junction is acquired as a recommended lane (S11), and the reliability is determined as the right lane. The reliability level corresponds to the exit direction of the vehicle at the guidance junction, of the right side reliability level and the left side reliability level, including the right side reliability level, which is the reliability level of the number of left lanes, and the left side reliability level, which is the reliability level of the number of left lanes. Since it is determined whether or not is higher than the reference (S14), it is possible to accurately calculate the reliability of the vehicle's driving lane with the left and right road edges as a reference based on the result of image recognition of the captured image. Become. Even if there are concerns about identifying the number of lanes included in the road the vehicle is currently traveling on or the location of the driving lane, at least when the positional relationship between the driving lane and the recommended lane is reliable, guidance objects can provide accurate information. This makes it possible to provide detailed guidance.
In addition, by displaying the captured image on the image display device (15) mounted on the vehicle and displaying the guide object on the same image display device, the vehicle occupants can see the captured image superimposed on the scenery around the vehicle. Since the guide object is visually recognized, it is possible to provide appropriate guidance using the guide object superimposed on the scenery around the vehicle in the captured image displayed on the image display device.

[Second embodiment]
Next, a superimposed image display device according to a second embodiment will be described based on FIGS. 21 and 22. In the following description, the same reference numerals as those in the configuration of the superimposed image display device according to the first embodiment shown in FIGS. It shows.

The general configuration of the superimposed image display device according to the second embodiment is almost the same as the superimposed image display device according to the first embodiment. In addition, various control processes are almost the same as those of the superimposed image display device according to the first embodiment.
However, the superimposed image display device according to the first embodiment displays the captured image captured by the front camera 19 on the liquid crystal display 15 of the navigation device 1, and further displays the guide object on the liquid crystal display 15. , the guide object is displayed superimposed on the scenery around the vehicle, whereas the superimposed image display device according to the second embodiment uses a head-up display system as a means for displaying an image superimposed on the scenery around the vehicle. It's different.

A schematic configuration of a superimposed image display device according to a second embodiment will be described below using FIG. 21. FIG. 21 is a schematic configuration diagram of a superimposed image display device 101 according to the second embodiment.
As shown in FIG. 21, the superimposed image display device 101 basically includes a navigation device 103 mounted on a vehicle 102, and a front display 104 also mounted on the vehicle 102 and connected to the navigation device 103. Note that the front display 104 functions as a head-up display together with the windshield 105 of the vehicle 102, and serves as information providing means for providing various information to the occupants 106 of the vehicle 102.

Here, the front display 104 is a liquid crystal display that is installed inside the dashboard 107 of the vehicle 102 and has a function of displaying an image on an image display surface provided at the front. As the backlight, for example, a CCFL (cold cathode fluorescent lamp) or a white LED is used. Note that as the front display 104, in addition to a liquid crystal display, an organic EL display or a combination of a liquid crystal projector and a screen may be used.

The front display 104 functions as a head-up display together with the windshield 105 of the vehicle 102, and the image output from the front display 104 is reflected on the windshield 105 in front of the driver's seat so that the occupant 106 of the vehicle 102 can see it. It is configured as follows. Note that guide objects are displayed on the front display 104 as necessary. In addition, in the second embodiment described below, the guide objects include an arrow indicating the exit direction of the guide branch point along the guide route, and an arrow indicating the exit direction of the guide branch point along the guide route, as in the first embodiment. A guide image indicates the position of a recommended lane on which the vehicle is currently traveling, and a guide image urges the vehicle to move to the recommended lane.

Further, when the passenger 106 views the image displayed on the front display 104 by reflecting off the windshield 105, the passenger 106 may see the front display not at the position of the windshield 105 but at a distant position beyond the windshield 105. The image displayed on 104 is configured to be visually recognized as a virtual image 110. In addition, the virtual image 110 is displayed superimposed on the surrounding environment (landscape, real scene) in front of the vehicle, and for example, is superimposed on any object located in front of the vehicle (road surface, building, object to be warned, etc.). It is also possible to display the information in the same way.

Here, the position where the virtual image 110 is generated, more specifically the distance L from the occupant 106 to the virtual image 110 (hereinafter referred to as imaging distance), is determined by the position of the front display 104. For example, the imaging distance L is determined by the distance along the optical path (optical path length) from the position where the image is displayed on the front display 104 to the windshield 105. For example, the optical path length is set so that the imaging distance L is 1.5 m.

Further, a front camera 111 is installed above the front bumper of the vehicle, behind the room mirror, etc. The front camera 111 is an imaging device having a camera using a solid-state imaging device such as a CCD, and is installed with its optical axis facing forward in the direction of travel of the vehicle. Then, by performing image processing on the captured image captured by the front camera 111, the situation of the front environment (that is, the environment on which the virtual image 110 is superimposed) that is visually recognized by the occupant 106 through the windshield is detected. Ru. Note that a sensor such as a millimeter wave radar may be used instead of the front camera 111.

Furthermore, an in-vehicle camera 112 is installed on the top surface of the instrument panel of the vehicle. The in-vehicle camera 112 is an imaging device having a camera using a solid-state imaging device such as a CCD, and is installed with its optical axis directed toward the driver's seat. The range where the face of the passenger is generally expected to be located inside the vehicle is set as a detection range (imaging range of the in-vehicle camera 112), and the face of the passenger 106 sitting in the driver's seat is imaged. Then, image processing is performed on the captured image captured by the in-vehicle camera 112, thereby detecting the eye position (line-of-sight starting point) and line-of-sight direction of the occupant 106.

Then, the superimposed image display device according to the second embodiment displays the image 120 of the guide object on the front display 104 as shown in FIG. 22 in S5 and S7 of the driving support processing program (FIG. 2) described above. . As a result, as shown in FIG. 22, when the vehicle occupant visually recognizes the image 120 of the guide object displayed on the front display 104, a virtual image 121 of the image 120 of the guide object is superimposed on the scenery through the windshield 105. Visible.

As a result, similarly to the superimposed image display device according to the first embodiment, it is possible to accurately grasp the position of the recommended lane and the exit direction at the intersection. In the superimposed image display device according to the second embodiment, in the guide object display position determination processing in S4 and S6, the size and shape of the guide object to be displayed on the front display 104 and the position ( range).

It should be noted that the present invention is not limited to the above-described embodiments, and it goes without saying that various improvements and modifications can be made without departing from the gist of the present invention.
For example, as a means for displaying an image superimposed on the scenery around the vehicle, the first embodiment uses a liquid crystal display 15 displaying an actual scene image, and the second embodiment uses a head-up display system. A window shield display (WSD) that displays an image may also be used. In WSD, the windshield may be used as a screen to display images from a projector, or the windshield may be used as a transmissive liquid crystal display. The image displayed on the windshield by the WSD becomes an image superimposed on the scenery around the vehicle.

In the first and second embodiments, the guide objects include a first guide object that guides the vehicle to a recommended lane, and a second guide object that is an arrow that indicates the exit direction of the vehicle at a guide branch point located ahead in the direction of travel. However, only the first guide object may be used. In addition, the first guide object does not necessarily have to be a rectangle that overlaps the lane or a combination of a rectangle and a triangle; it can be any other shape as long as it can indicate the position of the recommended lane and encourage movement to the recommended lane. It may be in the shape of For example, it may be in the shape of an arrow. Further, it is not necessarily necessary to superimpose it on the road surface, but it may be displayed at a position separated from the road surface.

In addition, in the first and second embodiments, guidance objects are used to provide guidance to guidance junctions, but points to be guided by guidance objects are not limited to guidance junctions; for example, when a vehicle When passing through a point or a merging section, a guide object may be used to prompt the user to move to the recommended lane, that is, to guide the user to change lanes to the recommended lane.

In addition, in the first and second embodiments, when calculating the reliability indicating the degree of reliability of the image recognition result of the captured image captured by the front camera 19 in S13, the reliability is expressed as a numerical value. However, the reliability may be calculated using other than numerical values. Furthermore, in the case where the reliability is calculated using something other than numerical values, the reference value used as the determination criterion in S14 is also other than numerical values.

Further, in the first embodiment, the actual view image and the guide object captured by the front camera 19 are displayed on the liquid crystal display 15 of the navigation device 1, but the display for displaying the actual view image and the guide object is not provided inside the vehicle. Any display other than the liquid crystal display 15 may be used as long as the display is arranged.

Further, in the second embodiment, a virtual image is generated in front of the windshield 105 of the vehicle 102 by the front display 104, but a virtual image may be generated in front of a window other than the windshield 105. Further, the object on which the image is reflected by the front display 104 may be a visor (combiner) installed around the windshield 105 instead of the windshield 105 itself.

Further, in the first and second embodiments, the navigation ECU 13 of the navigation device 1 executes the processing of the driving support processing program (FIG. 2), but the execution entity can be changed as appropriate. For example, it may be configured to be executed by a control unit of the liquid crystal display 15, a vehicle control ECU, or other on-vehicle equipment.

DESCRIPTION OF SYMBOLS 1...Navigation device, 15...Liquid crystal display, 19...Front camera, 41...CPU, 42...RAM, 43...ROM, 51...Driving guide screen, 52...Scenery, 53...Image of first guide object, 54...Recommended lane , 55... Vehicle driving lane, 57... Image of second guide object, 60... Guide branch point, 61, 64... Road edge, 62, 63, 65, 66... Partition line

Claims (7)

A superimposed image display device that is mounted on a vehicle and allows a guide object that guides information to a passenger of the vehicle to be visually recognized by superimposing it on the scenery around the vehicle,
map information acquisition means for acquiring map information including information regarding the number of lanes on a road;
Lane number identifying means for identifying the number of lanes included in the road on which the vehicle is currently traveling based on a captured image of the scenery around the vehicle;
If the number of lanes of the road on which the vehicle currently travels included in the map information does not match the number of lanes specified by the number of lanes specifying means, the number of lanes specified by the number of lanes specifying means is used to move the vehicle. While identifying the driving lane in which the vehicle is currently traveling on the road on which the vehicle is currently traveling, the number of lanes on the road on which the vehicle is currently traveling included in the map information and the number of lanes identified by the lane number identifying means are determined. a driving lane specifying means for specifying the driving lane using the matching number of lanes if they match;
A superimposed image display device comprising: object display means for displaying the guide object based on the driving lane specified by the driving lane specifying means. The lane number identifying means includes:
Based on the captured image, each of the marking lines around the vehicle and the edge of the road on which the vehicle is currently traveling is detected;
Based on the detection results of the partition line and road edge, the number of left lanes, which is the number of lanes from the left road edge to the lane in which the vehicle is currently traveling, and the lane number from the right road edge to the lane in which the vehicle is currently traveling. Calculate the number of right lanes, which is the number of
The superimposed image display device according to claim 1, wherein the number of lanes included in the road on which the vehicle is currently traveling is specified by combining the number of left lanes and the number of right lanes. comprising a reliability obtaining means for obtaining a reliability indicating how reliable the identification result of the lane number identification means is;
The driving lane specifying means determines whether the number of lanes of the road on which the vehicle currently traveling included in the map information matches the number of lanes specified by the lane number specifying means when the reliability is higher than a reference. determining the driving lane;
The superimposed image display device according to claim 1 or 2, wherein the guide object is displayed based on the driving lane specified by the driving lane specifying means. The superimposed image display device according to claim 3, wherein when the reliability is lower than a reference, the guide object is not displayed or the guide object is displayed in a manner independent of the driving lane. Recommended lane acquisition means for acquiring a recommended lane in which a vehicle is currently traveling on a road in which it is recommended to drive;
a range specifying means for specifying a range in the landscape in which the recommended lane exists based on the driving lane specified by the driving lane specifying means;
5. The superimposed image display device according to claim 3, wherein the object display means displays the guide object in a superimposed manner in a range where the specified recommended lane exists. The recommended lane acquisition means acquires, as the recommended lane, a lane corresponding to the exit direction of the vehicle at the guidance junction when there is a guidance junction within a predetermined guidance start distance ahead in the direction of travel of the vehicle;
The lane number identifying means includes:
Based on the captured image, each of the marking lines around the vehicle and the edge of the road on which the vehicle is currently traveling is detected;
Based on the detection results of the partition line and road edge, the number of left lanes, which is the number of lanes from the left road edge to the lane in which the vehicle is currently traveling, and the lane number from the right road edge to the lane in which the vehicle is currently traveling. Calculate the number of right lanes, which is the number of
Identifying the number of lanes included in the road on which the vehicle is currently traveling by combining the number of left lanes and the number of right lanes,
The reliability includes a right-side reliability that indicates the reliability of the number of right-hand lanes, and a left-side reliability that indicates the reliability of the number of left-hand lanes,
The superimposed image display device according to claim 5, wherein it is determined whether or not the reliability corresponding to the exit direction of the vehicle at the guidance branch point is higher than a reference among the right side reliability and the left side reliability. comprising display means for displaying the captured image on an image display device mounted on the vehicle;
The object display means displays the guide object on the image display device on which the captured image is displayed, so that an occupant of the vehicle can visually recognize the guide object by superimposing it on the scenery around the vehicle in the captured image. The superimposed image display device according to any one of claims 1 to 6.

Images