JP6136238B2 - Driving support system, driving support method, and computer program - Google Patents

Description

  The present invention relates to a driving support system, a driving support method, and a computer program that support driving by performing guidance based on the line of sight of a vehicle occupant.

  In recent years, as one of the driving assistance for vehicle occupants, a camera mounted on the vehicle is used to image the surrounding environment ahead of the vehicle in the traveling direction, and obstacles such as people and other vehicles located in the traveling direction forward from the captured image. A technique for detecting and guiding has been proposed (for example, JP 2007-73011 A). In order to appropriately guide the obstacle, it is important to specify where the occupant who is the target of guidance is currently viewing.

  For example, when the occupant is not visually recognizing an obstacle, it is desirable to provide guidance for notifying the occupant of the obstacle. On the other hand, when the occupant has already visually recognized the obstacle, guidance for notifying the occupant of the obstacle is unnecessary, and it is desirable to provide details regarding the obstacle visually recognized by the occupant. And as a means for identifying where the occupant is currently viewing, for example, in Japanese Patent Laid-Open No. 2006-242859, the surrounding environment ahead of the traveling direction of the vehicle is imaged by a camera mounted on the vehicle. A technique has been proposed in which the sight line direction of an occupant is detected by a camera installed in the vehicle, and the portion of the captured image that the occupant is viewing is specified.

Japanese Patent Laying-Open No. 2004-322771 (page 5-7, FIG. 2) JP-A-2006-242859 (page 9-10, FIG. 4)

  However, with the technique described in Patent Document 1, it is possible to provide guidance regarding an obstacle such as a person or other vehicle located in the forward direction of travel, but where the occupant who is the target of guidance is currently viewing. Because there is no consideration about whether or not it is, there have been cases where unnecessary guidance for passengers is provided or necessary guidance is not provided. In the technique described in Patent Document 2, it is possible to specify where the occupant is currently viewing, but what kind of obstacle the occupant visually recognizes is the obstacle. It was not possible to determine how the object was viewed. In particular, the guidance required to be performed differs depending on whether the occupant is viewing an obstacle or not. However, in Patent Document 1 and Patent Document 2 described above, a visual aspect of such an obstacle is described. Nothing was considered.

  The present invention has been made in order to solve the above-described conventional problems, and detects a feature in front of the vehicle in the traveling direction, and the occupant visually recognizes the feature as an aspect of the occupant to the feature. A driving support system, a driving support method, and a computer program capable of performing necessary guidance for the current occupant at a necessary timing without performing unnecessary guidance for the occupant by determining whether or not The purpose is to provide.

In order to achieve the above object, the driving support system (1) according to claim 1 of the present application stores a surrounding environment imaging means (13) that images the surrounding environment ahead of the traveling direction of the vehicle (51) and an image of a feature object. The feature image information acquisition means (13) for acquiring the feature image information (33), the line-of-sight detection means (13) for detecting the line-of-sight start point and the line-of-sight direction of the vehicle occupant (52), and the surrounding environment The sight line of the occupant is positioned with respect to the surrounding environment ahead of the traveling direction of the vehicle based on the captured image captured by the imaging unit and the sight line start point and the sight line direction of the occupant detected by the sight line detection unit. gaze position, the line-of-sight position specifying means (13) for identifying at in said captured image, the shooting was included the gaze position specified by the line-of-sight position specifying means within the past predetermined time from the present time Based on the line-of-sight area specifying means (13) for specifying the range on the image as the line-of-sight area, and the captured image captured by the surrounding environment imaging means and the feature object image information, the vehicle is ahead of the traveling direction of the vehicle. A feature detection unit (13) for detecting the feature as a forward feature, a feature region specifying unit (13) for specifying a region surrounding the front feature on the captured image as a feature region, and the line of sight Based on the positional relationship between the area and the feature area, the visual aspect specifying means (13) for specifying the visual aspect of the occupant with respect to the front feature, and the visual aspect of the occupant specified by the visual aspect specifying means And guidance means (13) for performing guidance on the forward feature with the contents according to the above.

Further, the driving support system (1) according to claim 2 is the driving support system according to claim 1, wherein the visual recognition mode specifying means (13) determines whether the occupant is gazing at the front feature. And when the passenger is determined to be gazing at the front feature, the guiding means (13) is more than the case where it is determined that the occupant is not gazing at the front feature. Further, detailed guidance of the front feature is performed.

A travel support system (1) according to claim 3 includes travel determination means (13) for determining whether or not the forward feature is a moving object in the travel support system according to claim 2, guide means (13) is a case where the forward characteristic object is judged to be moving body by the movement determining means, identified as gazing the front characteristic object the by visually recognizable identification means (13) If it is, the information identifying the relationship between the forward feature and the vehicle is guided, and the movement determining means determines that the forward feature is a moving body, and the visual aspect If it is identified as not gazing at the front characteristic object by specifying means, characterized in that to guide the information identifying the forward characteristics thereof.

In addition, the driving support system (1) according to claim 4 includes, in the driving support system according to claim 2, movement determination means (13) for determining whether or not the forward feature is a moving body, The guiding means (13) is specified when the forward determining object is determined not to be a moving object by the movement determining means, and is specified as gazing at the forward characteristic by the visual recognition mode specifying means (13). The auxiliary information for assisting the occupant's understanding of the content of the forward feature is guided, and the movement determination means determines that the forward feature is not a moving body, and the visual recognition If it is identified as not gazing at the front characteristic object by aspects specifying means is characterized in that the guidance of the identification information for identifying the contents of the front characteristics thereof.

  Further, the driving support system (1) according to claim 5 is the driving support system according to any one of claims 1 to 4, further comprising type identifying means (13) for identifying the type of the feature. The feature object image information acquisition means (13) acquires the feature object image information (33) in which an image of the feature object is stored for each type of the feature object, and the type identification means includes the feature object image information. The type of the front feature is identified based on the feature image information acquired by the information acquisition unit, and the guide unit is a guide content according to the type of the feature identified by the type identification unit. It is characterized by providing guidance on the characteristic object.

  A travel support system (1) according to claim 6 is the travel support system according to any one of claims 1 to 5, wherein the distance detection means (10) detects the distance from the vehicle to the front feature. 13), and when the distance from the vehicle to the front feature is determined to be equal to or greater than a predetermined distance, the guide means (13) guides around the front feature or the front feature. It is characterized by performing travel guidance with an object as a destination.

  The driving support system (1) according to claim 7 is the driving support system according to any one of claims 1 to 6, wherein the feature is an occupant of the vehicle (51) traveling on a road ( 52) is an object to be recognized.

In addition, the driving support system (1) according to claim 8 is the driving support system according to any one of claims 1 to 7, wherein the visual recognition mode specifying means (13) includes the line-of-sight region and the characteristic object. The aspect of visual recognition of the occupant with respect to the front feature is specified based on the overlapping ratio with the region.
In the driving support method according to claim 9, the surrounding environment imaging means images the surrounding environment ahead of the vehicle (51) in the traveling direction, and the feature object image information acquisition means stores the feature image. The step of acquiring the feature object image information, the step of the line-of-sight detection means detecting the line-of-sight start point and the line-of-sight direction of the vehicle occupant (52), and the line-of-sight position specifying means are imaged by the surrounding environment imaging means. Based on the captured image and the sight line start point and the sight line direction of the occupant detected by the sight line detection means, the sight line position where the sight line of the occupant is located with respect to the surrounding environment ahead of the traveling direction of the vehicle is captured. identifying in the image, line-of-sight region specifying means, the imaging field that includes the sight line position specified by the line-of-sight position specifying means within the past predetermined time from the present time The step of specifying the upper range as a line-of-sight region, and the feature detection unit is located forward of the vehicle in the traveling direction based on the captured image and the feature image information captured by the surrounding environment imaging unit. A step of detecting a feature as a front feature, a step in which a feature region specifying unit specifies a region surrounding the front feature as a feature region on the captured image, and a visual recognition mode specifying unit is the line-of-sight region And the step of identifying the occupant's visual aspect with respect to the front feature based on the positional relationship between the visual feature and the feature area, and the guiding means according to the occupant's visual aspect identified by the visual aspect identifying means And a step of providing guidance on the front feature in terms of contents.

Further, the computer program according to claim 10 is a feature that obtains feature object image information in which the computer captures the surrounding environment imaging means for imaging the surrounding environment ahead of the traveling direction of the vehicle (51) and the image of the feature object. Image information acquisition means, gaze detection means for detecting the gaze start point and gaze direction of the vehicle occupant (52), the captured image taken by the surrounding environment imaging means, and the occupant detected by the gaze detection means based starting point of the line of sight and line of sight direction, a line-of-sight position of the occupant's line of sight is positioned relative to the direction of travel of the surrounding environment of the vehicle, a line-of-sight position specifying means for specifying the in said captured image, from the present sight territory identifying the range on the captured image includes the sight line position identified as line-of-sight region by the gaze position determining means within the past predetermined time A feature detection unit that detects, as a forward feature, the feature in front of the vehicle in the traveling direction based on the captured image captured by the surrounding environment imaging unit and the feature image information. The feature region specifying means for specifying the region surrounding the front feature as a feature region on the captured image, and the occupant with respect to the front feature based on the positional relationship between the line-of-sight region and the feature region A visual recognition mode specifying means for specifying the visual recognition mode of the vehicle, and a guidance means for guiding the front feature with contents corresponding to the visual recognition mode of the occupant specified by the visual recognition mode specifying means. Features.

According to the driving support system according to claim 1 having the above-described configuration, the feature in front of the traveling direction of the vehicle is detected based on the feature image information, and the occupant visually recognizes the feature . By performing guidance according to the contents, it is possible to provide guidance necessary for the current occupant at a necessary timing without performing unnecessary guidance for the occupant. In addition, it is possible to appropriately specify information necessary for the current occupant from the visual aspect of the occupant's characteristic object. In particular, when the occupant is visually recognizing a characteristic object, it is possible to provide more detailed information about the characteristic object that the occupant is interested in by performing guidance regarding the characteristic object.

  Further, according to the driving support system of the second aspect, when it is determined that the occupant is gazing at the characteristic object, more detailed guidance of the characteristic object is provided than when it is determined that the occupant is not gazing at the characteristic object. As a result, it is possible to provide more detailed guidance to the features that the occupant is more interested in, that is, the features that require explanation.

  According to the driving support system according to claim 3, when it is determined that the feature is a moving body and the occupant is gazing at the feature, the relationship between the feature and the vehicle is determined. When the information to be specified is guided and it is determined that the feature is a moving body and the occupant is not gazing at the feature, the information for specifying the feature is guided. Information required by the occupant can be estimated from the visual recognition mode, and the feature can be guided based on the estimated information. As a result, information required by the occupant can be provided.

  According to the driving support system of the fourth aspect, when it is determined that the feature is not a moving object and the occupant is gazing at the feature, the understanding of the occupant to the feature is assisted. If it is determined that the feature is not a moving object and the occupant is not gazing at the feature, information for identifying the content of the feature is guided. Information required by the occupant can be estimated from the visual recognition mode, and the feature can be guided based on the estimated information. As a result, information required by the occupant can be provided.

  Further, according to the driving support system of the fifth aspect, the guidance content is changed depending on the type of the feature, so that it is possible to appropriately select and guide information necessary for guidance of the feature visually recognized by the occupant. It becomes. For example, when the characteristic object is a road sign, it is possible to provide guidance on the restriction contents of the road sign. Further, when the characteristic object is a vehicle, it is possible to provide guidance on the vehicle type and number.

  According to the driving support system of the sixth aspect, when it is determined that the distance from the vehicle to the feature is greater than or equal to the predetermined distance, the guidance around the feature or the feature is used as the destination. Since the traveling guidance is performed, it is possible to perform the guidance in consideration of the intention that the occupant visually recognizes the characteristic object.

  Further, according to the driving support system of the seventh aspect, since the characteristic object to be guided is an object to be observed by the passenger of the vehicle traveling on the road, there is a risk of affecting the traveling of the vehicle. It is possible to let the occupant know in advance the existence of a certain object and drive it appropriately.

According to the driving support system of the eighth aspect, it is possible to identify the occupant's visual aspect with respect to the front feature based on the overlapping ratio between the sight line region and the feature region of the occupant.
According to the driving support method of the ninth aspect, the feature in front of the traveling direction of the vehicle is detected based on the feature image information, and the content according to the visual aspect of the occupant on the feature. By performing the guidance, it is possible to perform guidance necessary for the current occupant at a necessary timing without performing unnecessary guidance for the occupant. In addition, it is possible to appropriately specify information necessary for the current occupant from the visual aspect of the occupant's characteristic object. In particular, when the occupant is visually recognizing a characteristic object, it is possible to provide more detailed information about the characteristic object that the occupant is interested in by performing guidance regarding the characteristic object.

Further, according to the computer program of the tenth aspect, the feature object ahead of the traveling direction of the vehicle is detected based on the feature object image information, and the guidance is provided with the content according to the visual recognition mode of the occupant on the feature object. By performing the above, it is possible to cause the current occupant to perform necessary guidance at a necessary timing without causing the occupant to perform unnecessary guidance. In addition, it is possible to appropriately specify information necessary for the current occupant from the visual aspect of the occupant's characteristic object. In particular, when the occupant is visually recognizing a characteristic object, it is possible to provide more detailed information about the characteristic object that the occupant is interested in by performing guidance regarding the characteristic object.

It is the block diagram which showed the navigation apparatus which concerns on this embodiment. It is the figure which showed the front camera and driver camera which were installed in the vehicle. It is a flowchart of the driving assistance processing program concerning this embodiment. It is a flowchart of the sub process program of a gaze circle specific process. It is a figure explaining the detection method of a gaze position. It is a figure explaining the detection method of a gaze position. It is a figure explaining the identification method of the gaze circle. It is a figure explaining the identification method of the gaze circle. It is a flowchart of the sub process program of a visual field area | region specific process. It is a figure explaining the identification method of the visual field area. It is the figure which showed the identified visual field area and the imaging range of a front camera. It is a flowchart of the sub process program of a feature object detection process. It is a figure explaining the detection method of the characteristic object contained in a captured image. It is a flowchart of the sub process program of visual recognition mode specific processing. It is a figure explaining the detection method of the duplication aspect of a feature object rectangle and a gaze circle. It is a figure explaining each pattern which classified a visual recognition mode to a driver's characteristic object. It is a flowchart of the sub process program of a 1st guidance process. It is the figure which showed the determination classification of the urgency level. It is a flowchart of the sub process program of a 2nd guidance process.

  Hereinafter, a driving support system according to the present invention will be described in detail with reference to the drawings based on an embodiment embodied in a navigation device. First, a schematic configuration of the navigation device 1 according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a navigation device 1 according to this embodiment.

  As shown in FIG. 1, the navigation device 1 according to the present embodiment includes a current position detection unit 11 that detects a current position of a vehicle on which the navigation device 1 is mounted, a data recording unit 12 that records various data, Based on the input information, the navigation ECU 13 that performs various arithmetic processes, the operation unit 14 that receives operations from the user, the liquid crystal display 15 that displays a map around the vehicle to the user, and the voice related to route guidance. A speaker 16 that outputs guidance and guidance on a feature located in front of the traveling direction of the vehicle, a DVD drive 17 that reads a DVD as a storage medium, a probe center, a VICS (registered trademark: Vehicle Information and Communication System) center, and the like And a communication module 18 for communicating with the information center. The navigation device 1 is connected to a front camera 19 and a driver camera 20 installed on a vehicle on which the navigation device 1 is mounted via an in-vehicle network such as CAN.

Below, each component which comprises the navigation apparatus 1 is demonstrated in order.
The current position detection unit 11 includes a GPS 21, a vehicle speed sensor 22, a steering sensor 23, a gyro sensor 24, and the like, and can detect the current vehicle position, direction, vehicle traveling speed, current time, and the like. . Here, in particular, the vehicle speed sensor 22 is a sensor for detecting a moving distance and a vehicle speed of the vehicle, generates a pulse according to the rotation of the driving wheel of the vehicle, and outputs a pulse signal to the navigation ECU 13. And navigation ECU13 calculates the rotational speed and moving distance of a driving wheel by counting the generated pulse. Note that the navigation device 1 does not have to include all the four types of sensors, and the navigation device 1 may include only one or more types of sensors.

  The data recording unit 12 reads an external storage device and a hard disk (not shown) as a recording medium, a map information DB 31 and a captured image DB 32 recorded on the hard disk, a predetermined program, and the like, and stores predetermined data on the hard disk. And a recording head (not shown) as a driver for writing. The data recording unit 12 may be configured by a memory card, an optical disk such as a CD or a DVD, instead of the hard disk.

  Here, the map information DB 31 includes, for example, link data relating to roads (links), node data relating to node points, facility data relating to facilities, search data used for route search processing, map display data for displaying a map, and each intersection. This is a storage means for storing intersection data regarding, search data for searching points, and the like.

The captured image DB 32 is a storage unit that stores captured images captured by the front camera 19 and the driver camera 20. In the captured image DB 32, an image of a feature and an arrangement pattern of feature points are defined for each type of feature described later (for example, landmark, vehicle, person, traffic light, road sign, guide board, etc.). The feature object image information 33 is also stored. Further, the feature object image information 33 also stores detailed information related to the landmarks such as position information and names of the landmarks, in particular, the landmarks. In addition, regarding the vehicle, detailed information related to the vehicle such as a vehicle name, a vehicle type, and a brand name is also stored. As for road signs, the contents indicating the kind of sign (guidance sign, warning sign, regulation sign, instruction sign, etc.) and what kind of regulation or instruction the road sign indicates are also stored. Further, the captured image DB 32 also stores guidance information (guidance text and guidance voice for guiding the feature object) regarding the feature object for each type of the feature object.
Then, the navigation ECU 13 determines the feature located in the forward direction of the vehicle (hereinafter referred to as the forward feature), the driver's line-of-sight position, and the like based on the captured image and the feature object image information 33 stored in the captured image DB 32. To detect. Further, guidance information associated with the detected forward feature is read from the captured image DB 32 and guidance is performed based on the read guidance information. The feature object image information 33 is not stored in the navigation device 1 but may be stored in an external server and acquired by the navigation device 1 from the external server.

  On the other hand, the navigation ECU (Electronic Control Unit) 13 is an electronic control unit that controls the entire navigation device 1. The CPU 41 as an arithmetic device and a control device, and a working memory when the CPU 41 performs various arithmetic processes. Read out from the ROM 43 and the ROM 43 in which a RAM 42 that stores route data when a route is searched, a control program, a driving support processing program (see FIG. 3) described later, and the like are recorded. And an internal storage device such as a flash memory 44 for storing the program. The navigation ECU 13 constitutes various means as processing algorithms. For example, the surrounding environment imaging unit images the surrounding environment ahead of the vehicle in the traveling direction by the front camera 19. The feature object image information acquisition means acquires feature object image information storing a feature object image and guidance information related to the feature object. The line-of-sight detection means detects the line-of-sight start point and the line-of-sight direction of the vehicle occupant. The line-of-sight position specifying means is based on the captured image picked up by the front camera 19 and the occupant's line-of-sight start point and line-of-sight direction detected by the line-of-sight detection means. The line-of-sight position where is located is specified in the captured image. The feature object detection means detects a feature object ahead of the traveling direction of the vehicle as a front feature object based on the captured image captured by the front camera 19 and the feature object image information. The visual recognition determination unit determines whether or not the occupant is viewing the front feature based on the captured image captured by the front camera 19 and the visual line position specified by the visual line position specifying unit. The guidance means performs guidance related to the characteristic object when it is determined by the visual recognition determination means that the occupant is visually recognizing the front characteristic object. The gaze determination unit determines whether or not the occupant is gazing at the front feature based on the captured image captured by the front camera 19 and the line-of-sight position specified by the line-of-sight position specifying unit. The movement determination means determines whether or not the characteristic object is a moving body. The type identifying means identifies the type of feature. The distance detection means detects the distance from the vehicle to the forward feature.

  The operation unit 14 is operated when inputting a starting point as a travel start point and a destination as a travel end point, and includes a plurality of operation switches (not shown) such as various keys and buttons. Then, the navigation ECU 13 performs control to execute various corresponding operations based on switch signals output by pressing the switches. The operation unit 14 can also be configured by a touch panel provided on the front surface of the liquid crystal display 15. Moreover, it can also be comprised with a microphone and a speech recognition apparatus.

  The liquid crystal display 15 includes a map image including a road, traffic information, operation guidance, operation menu, key guidance, guidance route from the departure point to the destination, guidance information along the guidance route, news, weather forecast, Time, mail, TV program, etc. are displayed.

The speaker 16 outputs voice guidance for guiding traveling along the guidance route based on an instruction from the navigation ECU 13 and traffic information guidance. In the present embodiment, guidance relating to a characteristic object (such as a person or another vehicle) positioned in front of the traveling direction of the vehicle is also output.
As a guide means for the user, instead of the liquid crystal display 15 and the speaker 16, a video can be projected on a head mounted display (HMD) configured to be mounted on the head of the driver of the vehicle or on the windshield of the vehicle. A head-up display (HUD) may be provided at the top of the dashboard.

  The DVD drive 17 is a drive that can read data recorded on a recording medium such as a DVD or a CD. Based on the read data, music and video are reproduced, the map information DB 31 is updated, and the like.

  The communication module 18 is a communication device for receiving traffic information composed of information such as traffic jam information, regulation information, and traffic accident information transmitted from a traffic information center, for example, a VICS center or a probe center. For example, a mobile phone or DCM is applicable.

  Further, as shown in FIG. 2, the front camera 19 is installed near the boundary between the ceiling of the vehicle 51 on which the navigation device 1 is mounted and the windshield so that the traveling direction of the vehicle 51 becomes the imaging direction of the front camera 19. The Then, the surrounding environment ahead of the traveling direction of the vehicle 51 is photographed through the windshield of the vehicle 51. Further, the installation position of the front camera 19 is adjusted so as to be substantially the same as the position of the eyes of the driver 52 (gaze start point). Furthermore, the angle of view of the front camera 19 is set to be wider than the viewing angle of the driver. As a result, a range including at least a range (visual field region) of the surrounding environment in front of the traveling direction of the vehicle 51 that can be visually recognized through the windshield by the driver 52 facing the front is configured to be imaged. And navigation ECU13 detects the characteristic objects (a person, other vehicles, etc.) located ahead of the advancing direction of vehicles 51 based on the picturized picture picturized with front camera 19 as mentioned below.

  Further, the driver camera 20 uses a solid-state image sensor such as a CCD, for example, and is attached to the upper surface of the instrument panel 53 of the vehicle 51 as shown in FIG. The Then, the face of the driver 52 sitting on the driver's seat is imaged. Further, the navigation ECU 13 detects the eye position (gaze start point) and the gaze direction of the driver 52 from the captured image captured by the driver camera 20 as described later.

  Next, a driving support processing program executed by the CPU 41 in the navigation device 1 according to this embodiment having the above-described configuration will be described with reference to FIG. FIG. 3 is a flowchart of the driving support processing program according to the present embodiment. Here, the driving support processing program is executed after the ACC of the vehicle is turned on, and a visual aspect of the feature in the forward direction of the vehicle based on the captured image captured by the front camera 19 or the driver camera 20 is displayed. It is a program which performs various guidance based on a driver | operator's visual recognition aspect to the detected characteristic object while it detects. The programs shown in the flowcharts of FIGS. 3, 4, 9, 12, 14, 17, and 19 below are stored in the RAM 42, ROM 43, and the like provided in the navigation ECU 13, and are executed by the CPU 41. Executed.

  First, in the driving support processing program, in step (hereinafter abbreviated as “S”) 1, the CPU 41 acquires captured images captured by the front camera 19 and the driver camera 20 most recently.

  Next, in S2, the CPU 41 performs a line-of-sight circle specifying process (FIG. 4) described later. The line-of-sight circle specifying process is a process for specifying a line-of-sight circle (line-of-sight range) that is a range in which the driver's line of sight is located based on a captured image captured by the driver camera 20.

  Subsequently, in S3, the CPU 41 performs a visual field region specifying process (FIG. 9) described later. The visual field area specifying process is a process for specifying the driver's visual field area and blind spot area in the captured image captured by the front camera 19.

  Thereafter, in S4, the CPU 41 performs a feature detection process (FIG. 12) described later. The feature object detection process is a process for detecting a feature object ahead of the traveling direction of the vehicle based on the captured image captured by the front camera 19. The “characteristic object” is an object that is recognized by the driver of the vehicle traveling on the road, and includes, for example, other vehicles, people, traffic lights, road signs, information boards, landmarks, and the like.

  Next, in S <b> 5, the CPU 41 determines whether or not at least one feature in front of the traveling direction of the vehicle has been detected from the captured image captured by the front camera 19 in the feature detection process in S <b> 4.

  And when it determines with having detected at least 1 or more characteristic object ahead of the advancing direction of a vehicle (S5: YES), it transfers to S6. On the other hand, if it is determined that the feature in the forward direction of the vehicle cannot be detected (S5: NO), the process proceeds to S10.

  In S6, the CPU 41 specifies a rectangle surrounding each feature detected by the feature detection process in S4 (hereinafter referred to as a feature rectangle) on the captured image. Then, the specified feature rectangle is stored in a storage area such as the RAM 42. Note that the position and size of the feature object rectangle are specified by the coordinate system on the captured image of the front camera 19. When a plurality of feature objects are detected by the feature object detection process in S4, a feature object rectangle is specified for each feature object.

  Thereafter, in S7, the CPU 41 executes a line-of-sight mode specifying process (FIG. 14) described later. Note that the line-of-sight identification process identifies the driver's visual aspect of the feature ahead of the vehicle in the traveling direction based on the line-of-sight circle identified in S2 and the feature rectangle identified in S6. It is processing.

  Next, in S8, the CPU 41 executes a first guidance process (FIG. 17) described later. The first guidance process is based on the feature of the vehicle ahead in the traveling direction of the vehicle based on the driver's visual aspect of the feature identified in S7, the visual field area of the driver identified in S3, and the like. This is a process for guiding the driver to recognize the existence.

  Subsequently, in S9, the CPU 41 executes a second guidance process (FIG. 19) described later. The second guidance process provides the driver with the details of the feature in the forward direction of the vehicle based on the driver's visual aspect of the feature identified in S7, the type of the subject, and the like. Is a process for providing guidance.

  On the other hand, in S10, the CPU 41 determines whether or not the driver is gazing at any point in the surrounding environment ahead of the traveling direction. Specifically, when the driver's line-of-sight circle specified in S <b> 2 is within the captured image range captured by the front camera 19, the driver gazes at any point in the surrounding environment ahead of the traveling direction. It is determined that Further, when the driver's line-of-sight circle cannot be specified in S2 or when the line-of-sight circle is outside the captured image range captured by the front camera 19, the driver is gazing at the surrounding environment ahead of the traveling direction. Judge that there is no.

  If it is determined that the driver is gazing at any point in the surrounding environment ahead of the traveling direction (S10: YES), the process proceeds to S11. On the other hand, when the driver is not gazing at the surrounding environment ahead of the traveling direction (S10: NO), that is, even if the driver is not gazing or gazing, it is other than the surrounding environment ahead of the traveling direction. If it is determined that a point (eg, meter, rearview mirror, door mirror, etc.) is being watched, the process returns to S1 without performing guidance.

  In S <b> 11, the CPU 41 identifies the location where the driver is gazing in the surrounding environment ahead of the traveling direction (hereinafter, the gazing location) based on the line-of-sight circle identified in S <b> 2 and the captured image captured by the front camera 19. To do. And guidance based on the specified gaze point is performed. For example, when the driver is gazing at the sky, guidance on the future weather is given. In addition, when gazing at the topography of the sea, mountains, etc., the name of the terrain being watched (the name of the mountain, the name of the bay, etc.) is guided. Alternatively, a place name (city name, etc.) in a direction in which the driver is gazing may be guided. Further, the guidance may be performed by voice guidance using the speaker 16 or display guidance using the liquid crystal display 15. Further, guidance may be performed using HMD or HUD. Thereafter, the driving support processing program ends.

  Next, a sub-process of the line-of-sight circle specifying process executed in S2 will be described with reference to FIG. FIG. 4 is a flowchart of a sub-processing program for the line-of-sight circle specifying process.

  First, in S21, the CPU 41 detects the driver's gaze start point (eye position) and gaze direction based on the captured image of the driver camera 20 acquired in S1. The driver camera 20 is installed on the instrument panel of the vehicle as described above, and is installed with the imaging direction facing the driver's seat (FIG. 2), and the captured image includes the driver's face. As a method for detecting the line-of-sight start point and the line-of-sight direction, there is a method of detecting using the center position of the pupil or the Purkinje image measured by the corneal reflection method, for example. Since these methods are already known techniques, details are omitted.

Next, in S22, the CPU 41 specifies the line-of-sight position where the driver's line of sight is located with respect to the surrounding environment ahead of the traveling direction of the vehicle based on the driver's line-of-sight start point and line-of-sight direction detected in S21. .
Specifically, as shown in FIG. 5, a line-of-sight position Q is an intersection of a straight line extending from the line-of-sight start point P in the line-of-sight direction α and the detection surface 54. The line-of-sight position Q is specified by the X coordinate and the Y coordinate with the detection surface 54 as a coordinate system. In particular, in the present embodiment, the detection surface 54 is the imaging surface of the front camera 19. That is, the line-of-sight position specified in S22 corresponds to the line-of-sight position specified in the captured image captured by the front camera 19.

  Subsequently, in S23, the CPU 41 corrects the driver's line-of-sight position specified in S22 so as to be a more accurate position based on the calibration situation, the driving situation, the degree of confidence, and the like.

  Thereafter, in S24, the CPU 41 reads the driver's line-of-sight position (stored in S25 described later) detected during the sub-process of the line-of-sight circle identification process executed last time from the RAM 42 or the like, and newly reads this line-of-sight position and this time. The detected line-of-sight position of the driver (the position after the correction in S23) is compared, and it is determined whether or not the deviation between the positions is equal to or greater than a predetermined distance. The predetermined distance is a threshold (for example, 10 cm) that can be arbitrarily set, and is stored in the RAM 42 or the like.

  If it is determined that the deviation between the previously detected line-of-sight position and the newly detected line-of-sight position is greater than or equal to the predetermined distance (S24: YES), the process proceeds to S27. On the other hand, when it is determined that the difference between the previously detected line-of-sight position and the newly detected line-of-sight position is less than the predetermined distance (S24: NO), the process proceeds to S25.

In S25, the CPU 41 identifies the midpoint between the previously detected line-of-sight position and the newly detected line-of-sight position (the position after the correction in S23), and the driver who has newly detected the position of the identified midpoint this time. The line-of-sight position. That is, the driver's line-of-sight position specified in S22 is corrected once in S23, further corrected to the coordinates of the midpoint specified in S25, and finally determined. The determined coordinates of the driver's line-of-sight position are cumulatively stored in a storage medium such as the RAM 42.
For example, as shown in FIG. 6, when the driver's line-of-sight position detected last time is position A and the current line-of-sight position corrected in S23 is position B, the position A is not the position B but the position A. The position C of the midpoint with B is finally determined as the driver's line-of-sight position detected this time.

  Next, in S <b> 26, the CPU 41 reads the driver's line-of-sight position history stored in a storage medium such as the RAM 42, and the line-of-sight circle is a range in which the driver's line of sight is located with respect to the surrounding environment ahead of the traveling direction of the vehicle. Specify (line-of-sight range).

  Here, even if a person continues to look at the same location, the line-of-sight position is not completely fixed, and although the movement distance is small, the vicinity of the location is always viewed while moving the line-of-sight position (so-called “so-called”). Fixation fine movement). The line-of-sight position also moves due to the vibration of the vehicle and the change in the driving posture. Therefore, in order to accurately specify the driver's visual aspect with respect to the surrounding environment ahead of the traveling direction of the vehicle, it is desirable to specify the line-of-sight position by a range having a certain width instead of a point.

  Therefore, in this embodiment, in S26, the CPU 41 specifies a range on the detection surface including the history of the line-of-sight position within the latest predetermined time as the line-of-sight circle where the driver's line of sight is located. And the driver | operator's visual recognition mode is specified by comparing the position of the characteristic object contained in the picked-up image of the front camera 19, and a gaze circle so that it may mention later. Note that the predetermined time is a time after the previous line-of-sight circle is initialized in S27 (to be described later) (that is, a time since the driver started visual recognition around the visual recognition position currently viewed by the driver). The line-of-sight circle specified in S26 is defined by, for example, the coordinates of the center point and the radial distance, and is cumulatively stored in a storage medium such as the RAM 42.

For example, as shown in FIG. 7, when the position D is first detected as the line-of-sight position and then the position E is detected as the line-of-sight position, a circle 55 that is the smallest circle including the position D and the position E is set as the line-of-sight circle. As specified. Thereafter, when the position F is further detected as the line-of-sight position, the circle 56 that is the smallest circle including the position D, the position E, and the position F is specified as the line-of-sight circle. As a result, the line-of-sight circle is updated from the circle 55 to the circle 56. However, when the distance between the position E and the position F is far apart as will be described later, it is presumed that the subject to be visually recognized by the driver has changed, and the line-of-sight circle is initialized once, and the position F is a new starting point. A line-of-sight circle is formed. Further, as described above, since the human visual line position always moves, the longer the driver visually recognizes the same part for a long time, the more visual line positions included in the visual line circle. Become.
In particular, in the present embodiment, the detection surface on which the line-of-sight circle is specified is the imaging surface of the front camera 19 as described above. That is, the line-of-sight circle specified in S26 corresponds to the line-of-sight circle specified in the captured image captured by the front camera 19. In this embodiment, the line-of-sight range, which is the range in which the driver's line of sight is located with respect to the surrounding environment ahead of the traveling direction of the vehicle, is specified as a circular line of sight, but the line-of-sight range is specified by a shape other than a circle. You may do it.

  On the other hand, in S27, the CPU 41 estimates that the object visually recognized by the driver has changed, and initializes the existing line-of-sight circle. Further, the history of the line-of-sight positions detected in the past stored in the RAM 42 is also initialized. Then, the coordinates of the driver's line-of-sight position (position corrected in S23) newly detected this time are stored in the RAM 42 as the line-of-sight position history. As a result, in the process of S26 executed after the next time, a new line-of-sight circle is formed starting from the line-of-sight position of the driver newly detected this time (the position after the correction in S23).

  For example, as shown in FIG. 8, when the position G is first detected as the line-of-sight position, then the position H is detected as the line-of-sight position, and then a position I that is far away from the position H is detected as the line-of-sight position. The line-of-sight circle 57 including the existing position G and position H is once initialized. Thereafter, when the position J is further detected as the line-of-sight position, the circle 58, which is the smallest circle including the position I and the position J, is newly identified as the line-of-sight circle.

  Next, the sub-process of the visual field area specifying process executed in S3 will be described with reference to FIG. FIG. 9 is a flowchart of a sub-processing program of the visual field area specifying process.

First, in S31, the CPU 41 determines the vehicle based on the captured image of the front camera 19 acquired in S1, the driver's gaze start point (eye position), the gaze direction, and the driver's viewing angle detected in S21. The driver's field of view with respect to the surrounding environment ahead in the traveling direction is specified in the captured image of the front camera 19. The viewing angle of the driver may be a fixed value or may be changed according to the age of the driver.
Specifically, as shown in FIG. 10, a cutting plane in which the range of the viewing angle β around the straight line extending from the line-of-sight start point P in the line-of-sight direction α is cut by the imaging surface 60 of the front camera 19. A region of view (that is, a region in the captured image included in the view angle) is defined as a view region 61. Further, the position and size of the identified visual field region 61 are defined by the coordinate system on the imaging surface 60.

  Here, FIG. 11 is a diagram illustrating an imaging range 62 of an image captured by the front camera 19 and a driver's visual field area 61. As described above, the installation position of the front camera 19 is substantially the same as the eye position (gaze start point) of the driver 52, and the forward direction of the vehicle is taken as the imaging direction (FIG. 2). Further, the angle of view of the front camera 19 is set to be wider than the viewing angle of the driver, and the driver's viewing area 61 that is basically facing the front is included in the imaging range 62 of the front camera 19. It becomes. On the other hand, a range that is not included in the visual field region 61 in the imaging range 62 is a blind spot region 63 that the driver cannot visually recognize in the surrounding environment ahead of the traveling direction of the vehicle.

  Thereafter, in S32, the CPU 41 cuts out an image included in the visual field area of the driver specified in S31 from the captured image of the front camera 19 acquired in S1.

  In S33, the CPU 41 captures the image captured by the front camera 19 acquired in S1 and includes an image included in the driver's field of view cut out in S32 and an image not included in the remaining driver's field of view. (Ie, an image of a blind spot area) and is stored in the captured image DB 32.

  Next, the sub-process of the feature object detection process executed in S4 will be described with reference to FIG. FIG. 12 is a flowchart of the sub-processing program for the feature object detection process.

In the following sub-process of the feature detection process, the CPU 41 compares the captured image captured by the front camera 19 with the feature image stored in advance for each feature type as the feature image information 33. A feature included in the captured image (that is, a feature located forward in the traveling direction of the vehicle) is detected.
Specifically, first, in S41, the CPU 41 performs conversion processing of the captured image of the front camera 19 acquired in S1. Specifically, edge extraction or the like is performed to detect feature points of the feature object included in the captured image.

  Next, in S42, the CPU 41 detects a feature point included in the captured image. Further, the feature recognition image information 33 stored in the captured image DB 32 is read, and the detected feature point arrangement pattern is compared with the arrangement pattern previously defined for each type of feature in the feature object image information 33. Process.

  Subsequently, in S43, the CPU 41 identifies the type of the feature included in the captured image based on the result of the pattern recognition process in S42. Specifically, when an arrangement pattern of feature points that matches a predefined pattern is included in the captured image, it is recognized that a feature object of a type corresponding to the arrangement pattern is included in the captured image. The types of “characteristics” include vehicles, people, traffic lights, road signs, information boards, landmarks, and the like. Further, when a plurality of feature objects are included in the captured image, the CPU 41 identifies the type of each feature object. Furthermore, when a landmark is included in the captured image, the CPU 41 acquires detailed information related to the landmark such as the position information and name of the landmark based on the feature object image information 33. Further, when a vehicle is included in the captured image, detailed information related to the vehicle such as the vehicle name, vehicle type, and brand name of the vehicle is acquired based on the feature object image information 33. In addition, when a road sign is particularly included in the captured image, the type of road sign (guidance sign, warning sign, regulation sign, instruction sign, etc.), and what kind of road sign is based on the feature image information 33 Detailed information related to road signs such as contents indicating whether the sign indicates a regulation or an instruction is acquired. In the second guidance process (FIG. 19) described later, when it is determined that the driver is visually recognizing a landmark, vehicle, or road sign that is a characteristic object, the acquired detailed information is guided.

  Thereafter, in S44, the CPU 41 determines whether or not at least one or more features are included in the captured image captured by the front camera 19 based on the detection result in S43.

  If it is determined that at least one or more features are included in the captured image captured by the front camera 19 (S44: YES), the process proceeds to S45. On the other hand, when it is determined that no feature is included in the captured image captured by the front camera 19 (S44: NO), the process proceeds to S5.

  After that, in S45, the CPU 41 detects the contour of the feature whose type is identified in S43 based on the feature points included in the captured image and the contour pattern previously defined for each type of the feature. For example, when the feature point 65 is detected in the captured image with the arrangement shown in FIG. 13, it is identified that the traffic light is included in the captured image as a feature object. Then, the contour 66 of the traffic light is detected. As a result, the position, shape, and the like of the feature included in the captured image of the front camera 19 are specified, and a feature rectangle 67 (see FIG. 13) surrounding the feature can be set in subsequent S6. It becomes. Note that the CPU 41 detects the outline of each feature object when the captured image includes a plurality of feature objects.

  Next, the sub process of the visual recognition mode specifying process executed in S7 will be described with reference to FIG. FIG. 14 is a flowchart of a sub-processing program for visual recognition mode specifying processing.

  First, in S51, the CPU 41 overlaps the feature rectangle and the line-of-sight circle in the captured image captured by the front camera 19 based on the feature rectangle specified in S6 and the line-of-sight circle specified in S3. Aspect is detected. Specifically, as shown in FIG. 15, the ratio of the area of the region 68 that overlaps the line-of-sight circle 55 to the total area of the feature (in this embodiment, the area of the feature rectangle 67 surrounding the feature is used) (hereinafter referred to as the area 68). , Referred to as overlap ratio). When a plurality of feature objects are detected in the feature object detection in S4, the CPU 41 calculates an overlap ratio for each feature object. Further, as the area of the feature object, the area surrounded by the contour detected in S45 may be used instead of the area of the feature object rectangle 67. Although the calculation process is complicated, it is possible to calculate a more accurate overlap ratio.

  Next, in S52, the CPU 41 calculates the gaze dwell time of the gaze circle specified in S3. Note that the gaze dwell time is the time after the current gaze circle is newly created after the previous gaze circle is initialized in S27 described above (that is, the gaze staying time around the viewing position that the driver currently gazes at. Time from the start of

  Subsequently, in S53, the CPU 41 corrects the overlapping ratio calculated in S51 based on the line-of-sight residence time calculated in S52. Specifically, the overlap ratio is added when the line-of-sight residence time is equal to or greater than the threshold, and the added value is increased as the line-of-sight residence time is longer. For example, the overlap ratio is added by 10% when the line-of-sight residence time is 3 seconds or more, and the overlap ratio is added by 20% when the line-of-sight residence time is 5 seconds or more.

  Here, as described above, as the driver visually recognizes the same part for a long time, the line-of-sight position included in the line-of-sight circle increases, and the line-of-sight circle basically increases (see FIG. 7). However, even if the driver visually recognizes the same part for a long time, the line-of-sight circle may not increase. Therefore, by correcting the overlap ratio according to the line-of-sight dwell time in S53, it is possible to specify a more appropriate visual aspect of the driver's feature in S54 described later.

Next, in S54, the CPU 41 identifies the driver's visual aspect of the feature ahead of the traveling direction of the vehicle based on the overlap ratio calculated in S51 and corrected in S53. More specifically, the visual recognition mode is specified in any of the following four patterns (1) to (4) defined in advance.
(1) The driver does not visually recognize the characteristic object.
(2) The driver is visually observing the feature but not gazing. The visual recognition is only slightly made.
(3) The driver visually recognizes the feature but does not pay attention. Visible more firmly than (2) above.
(4) The driver is watching the feature.

  For example, in the present embodiment, as shown in FIG. 16, when the overlap ratio is 0%, the visual aspect is specified in the pattern (1), and when the overlap ratio is 1 to 40%, the pattern (2) is specified. A visual recognition mode is specified, and when the overlapping ratio is 41 to 80%, the visual recognition mode is specified for pattern (3), and when the overlapping ratio is 81% or more, the visual recognition mode is specified for pattern (4). . The visual recognition mode specified in S54 is stored in the RAM 42 or the like in association with the target feature rectangle 67. Note that the numerical value of the overlapping ratio shown in FIG. 16 can be changed as appropriate.

  Next, the sub-process of the first guidance process executed in S8 will be described with reference to FIG. FIG. 17 is a flowchart of a sub-processing program of the first guidance process.

  First, in S61, the CPU 41 determines whether or not the driver is gazing at the feature based on the driver's visual aspect toward the feature ahead of the traveling direction of the vehicle identified in the visual aspect identification process of S7. Determine whether. Specifically, when the visual recognition mode of the pattern (4) among the patterns of (1) to (4) above is specified as the visual recognition mode for the feature in the visual recognition mode identification process of S7, When it is determined that the driver is gazing at the feature, and the visual recognition mode of the patterns (1) to (3) is specified as the visual aspect to the feature, the driver is not gazing at the feature. Is determined.

  If it is determined that the driver is gazing at the feature (S61: YES), it is assumed that the driver has already recognized the presence of the feature, and guidance regarding the feature is provided. It moves to S9 without it. On the other hand, if it is determined that the driver is not gazing at the feature (S61: NO), it is estimated that the driver may not recognize the presence of the feature, and the process proceeds to S62. To do.

  In S62, the CPU 41 determines (a) a distance from the vehicle to the feature, (b) a relative speed between the vehicle and the feature, and (c) the feature that is determined not to be watched by the driver. The visual recognition time of the feature is detected. Note that (a) the distance from the vehicle to the feature and (b) the relative speed between the vehicle and the feature are detected using a captured image taken by the front camera 19, a distance sensor mounted on the vehicle, or the like. The The visual recognition time of the feature corresponds to the visual line dwell time calculated in S52.

  Next, in S63, the CPU 41 determines the characteristics based on (a) the distance from the vehicle to the feature, (b) the relative speed between the vehicle and the feature, and (c) the visual recognition time of the feature detected in S62. Determine the urgency of guidance for an object. Basically, it is determined that the urgency is higher as the distance from the vehicle to the feature is shorter. Further, it is determined that the urgency level is higher as the relative speed at which the vehicle approaches the feature object earlier. Moreover, it is determined that the urgency level is higher as the visual recognition time of the feature object is shorter. Specifically, the degree of urgency is determined as “high”, “medium”, or “low” in the manner shown in FIG. The level of urgency depends on the necessity for the vehicle occupant to recognize the front features and the necessity for the vehicle occupant to correct the operation of the vehicle according to the recognition of the front features. It is classified in advance as “low”.

FIG. 18 shows the urgency of “high”, “medium”, and “low” with the feature recognition time as the x-axis, the relative speed between the vehicle and the feature as the y-axis, and the distance from the vehicle to the feature as the z-axis. It is the figure which showed each division determined with.
Specifically, the determination is made based on the following conditions.
Urgency “Low”: At least the following condition (A) is satisfied.
(A) The relative speed at which the feature moves away from the vehicle (y ≧ 0) (that is, the traveling direction of the feature is the same as the traveling direction of the vehicle, and the moving speed is faster than the vehicle).
Urgency “medium”: satisfies the following condition (A) and satisfies one or both of the conditions (B) and (C).
(A) The relative speed at which the feature approaches the vehicle (y <0) (that is, the traveling direction of the feature is opposite to or the same as the traveling direction of the vehicle, and the moving speed is slower than the vehicle).
(B) The distance from the vehicle to the feature is a predetermined distance (for example, 10 m) or more (z ≧ β).
(C) The visual recognition time of the characteristic object is a predetermined time (for example, 3 sec) or more (x ≧ α).
Urgency “High”: All the following conditions (A) to (C) are satisfied.
(A) The relative speed at which the feature approaches the vehicle (y <0) (that is, the traveling direction of the feature is opposite to or the same as the traveling direction of the vehicle, and the moving speed is slower than the vehicle).
(B) The distance from the vehicle to the feature is less than a predetermined distance (for example, 10 m) (z <β).
(C) The visual recognition time of the feature is less than a predetermined time (for example, 3 sec) (x <α).

If the feature is a road sign, the urgency level may be determined based on the type of road sign. Specifically, the CPU 41 first identifies the type of road sign detected as a feature ahead of the traveling direction of the vehicle, based on the captured image of the front camera 19 and the feature object image information 33 stored in the captured image DB 32. To do. The feature object image information 33 stores a road sign image for each type of road sign as described above. Then, based on the specified type of road sign, the degree of urgency is determined as “high”, “medium”, or “low”.
For example, when the feature is a restriction sign, particularly a sign with a red border, the urgency is determined as “high”.
If the feature is a warning sign (a black sign on a yellow background), the degree of urgency is determined as “medium”.
Also, if the feature is another road sign (a sign or regulation sign, especially a sign with no red border (white sign on a blue background), etc.) judge.
Further, in addition to the type of road sign, the degree of urgency may be determined in consideration of the road sign visual recognition time, the relative speed between the vehicle and the road sign, and the distance from the vehicle to the road sign. For example, when the distance from the vehicle to the road sign is within a predetermined distance (for example, 10 m) and the road sign is a restriction sign, in particular, a sign with a red border, the urgency level is “high”. You may judge.

  Subsequently, in S64, the CPU 41 determines whether or not the urgency level is determined to be “low” as a result of the determination of the urgency level in S63.

  If the urgency level is determined to be “low” (S64: YES), there is no significant influence on the driving even if the driver does not recognize the feature, that is, the driver of the vehicle has the front feature. It is presumed that there is no need to recognize the object and that the driver does not need to correct the operation of the vehicle, and the process proceeds to S9 without providing guidance regarding the characteristic object. On the other hand, when the urgency level is determined to be “medium” or “high” (S64: NO), the process proceeds to S65.

  In S65, the CPU 41 determines whether or not the urgency level is determined to be “medium” as a result of the determination of the urgency level in S63.

  If the urgency is determined to be “medium” (S65: YES), there is a possibility that the driving may be affected when the driver does not recognize the feature, that is, the driver of the vehicle It is estimated that there is a need to recognize the forward feature and the driver does not need to correct the operation of the vehicle, and guidance is provided to allow the driver to recognize the presence of the feature in the forward direction of the vehicle (S66). ). For example, a sound “Please pay attention to the front” is output from the speaker 16. In addition, when the feature is a road sign in particular, a sound “Please pay attention to the road sign” is output from the speaker 16. The guidance may be display guidance on the liquid crystal display 15, or guidance using HMD or HUD. Further, guidance information (guidance text and guidance voice) output in S66 and subsequent S68 and S69 is read from the captured image DB 32. In the captured image DB 32, guidance information is stored in association with each feature type and each urgency level.

  On the other hand, when the urgency level is determined to be “high” (S65: NO), the process proceeds to S67.

  In S67, the CPU 41 determines whether or not the feature located in the forward direction of the vehicle is within the driver's visual field based on the driver's visual field identified in S3. Specifically, if a feature is detected in the captured image of the field of view saved in S33, it is determined that the feature located in the forward direction of the vehicle is within the driver's field of view. If a feature is detected in the captured image of the blind spot area stored in S33, it is determined that the feature located in the forward direction of the vehicle is not in the driver's field of view.

  If it is determined that the feature is within the driver's field of view (S67: YES), there is a possibility that the driving will be greatly affected if the driver does not recognize the feature, that is, The vehicle occupant is required to recognize the front feature and the driver needs to correct the operation of the vehicle, and the driver is made aware of the presence of the feature in the forward direction of the vehicle. At the same time, guidance is provided for prompting the driver to correct the operation of the vehicle in accordance with the recognition of the presence of a characteristic object ahead of the traveling direction of the vehicle (S68). The guidance content at that time is a guidance in which the feature is emphasized as compared with S66. For example, the voice of “XX (type of characteristic object) is approaching. Be careful ahead” is output from the speaker 16. In addition, when the feature is a road sign in particular, guidance on the contents of the road sign regulation (for example, vehicle entry prohibition, speed limit guidance), for example, when a vehicle is likely to enter a vehicle entry prohibited road Will give guidance to the driver, such as "There is a road where vehicles are prohibited from entering. Please make a right or left turn to avoid entering a road where vehicles are prohibited." Further, the guidance may be display guidance by the liquid crystal display 15 or guidance using HMD or HUD. Furthermore, when using HMD or HUD, the feature object rectangle specified in S6 may be displayed around the feature object.

  On the other hand, when it is determined that the feature is not in the driver's field of view, that is, in the blind spot area (S67: NO), the driver recognizes that the feature is present in the driver's blind spot. (S69). The guidance content at that time is guidance in which the feature is emphasized as compared with S66, and guides the position of the feature with respect to the visual field region. For example, the speaker 16 outputs the sound “XX (type of characteristic object) is located in the blind spot. Further, when the feature is a road sign in particular, guidance on the restriction contents of the road sign in the blind spot (for example, vehicle entry prohibition, speed limit guidance) is performed. Further, the guidance may be display guidance by the liquid crystal display 15 or guidance using HMD or HUD.

  In the first guidance process, when a plurality of feature objects are detected in the feature object detection in S4, the CPU 41 may execute the processes in S61 to S69 for each feature object. The processes of S61 to S69 may be executed only for the feature closest to the vehicle.

  Next, the sub-process of the second guidance process executed in S9 will be described with reference to FIG. FIG. 19 is a flowchart of a sub-processing program for the second guidance process.

  First, in S71, the CPU 41 recognizes at least the feature (gazes) the driver based on the driver's visual aspect of the feature in the forward direction of the vehicle identified in the visual aspect identification process in S7. Whether it is included) or not. Specifically, in the visual recognition mode specifying process of S7, among the patterns (1) to (4), the visual modes of the patterns (2) to (4) are particularly specified as the visual mode for the feature. In this case, when it is determined that the driver is visually recognizing the feature, and the visual aspect of the pattern (1) is specified as the visual aspect of the feature, the driver does not visually recognize the characteristic. Is determined.

  And when it determines with the driver | operator visually recognizing the characteristic object (S71: YES), it transfers to S72. On the other hand, when it is determined that the driver is not visually recognizing the feature (S71: NO), the driving support processing program is terminated without performing guidance regarding the feature.

  Next, in S72, the CPU 41 determines whether or not the distance from the vehicle detected in S62 to the feature is equal to or greater than a predetermined distance. The predetermined distance as the determination criterion in S72 is a distance set for determining whether or not the feature is on the road along which the vehicle travels or along the road, and is set to 1000 m, for example.

  If it is determined that the distance from the vehicle to the feature is greater than or equal to the predetermined distance (S72: YES), the feature is not a feature on or along the road on which the vehicle travels, but in the distance. Presumably a large facility or building that would be a landmark. Then, in S73, the CPU 41 provides a sightseeing guide for the surroundings as information about the characteristic object, or shifts to a destination setting screen. Then, when the screen shifts to the destination setting screen, the feature is set as the destination after that, and the travel guidance using the feature as the destination is performed.

  On the other hand, if it is determined that the distance from the vehicle to the feature is less than the predetermined distance (S72: NO), the feature is estimated to be a feature on or along the road on which the vehicle travels. , The process proceeds to S74.

  In S74, the CPU 41 determines whether or not the feature is a moving object currently moving based on the relative speed between the vehicle and the feature detected in S62.

  If it is determined that the feature is a moving object (S74: YES), the feature is estimated to be a moving pedestrian or another vehicle, and the process proceeds to S75. On the other hand, when it is determined that the feature is not a moving object (S74: NO), the feature is estimated to be a pedestrian or other vehicle that is stopped, or a feature such as a road sign or a traffic light, The process proceeds to S78.

  In S75, the CPU 41 determines whether or not the driver is gazing at the feature. Note that the specific processing contents are the same as in S61, and thus the details are omitted.

  If it is determined that the driver is not gazing at the feature (S75: NO), information specific to the feature for identifying the feature is provided as information provision on the feature (S76). . For example, the location of the feature, the type of the feature, etc. are guided. Thereby, it is possible to provide the driver with more detailed information about the feature visually recognized by the driver.

  On the other hand, if it is determined that the driver is gazing at the feature (S75: YES), the information providing information about the information feature is provided from S76 such as information for specifying the relationship between the feature and the vehicle. The detailed information is also guided (S77). Specifically, the relative speed of the feature with respect to the host vehicle, the distance between the host vehicle and the feature, the moving direction of the feature with respect to the host vehicle (whether approaching or moving away), etc. are guided. If the feature is a vehicle, the vehicle name, vehicle type, brand name, etc. are read from the captured image DB 32 and guided. In addition, when the characteristic object approaches the vehicle, the estimated time until contact is also guided. As a result, it is possible to provide the driver with more detailed information about the feature that the driver has watched. In S77, it is estimated that the driver has a strong interest in the feature. Therefore, the guidance content is more detailed than the guidance when the driver is not gazing at the feature (S76). It becomes the contents to guide about.

  On the other hand, in S78, the CPU 41 determines whether or not the feature is a feature based on the type of the feature detected in S43. A feature is a structure installed on or along a road, such as a road sign, a guide board, or a facility.

  If it is determined that the feature is a feature (S78: YES), the process proceeds to S79. On the other hand, when it is determined that the feature is not a feature (S78: NO), the feature is estimated to be a pedestrian or another vehicle that is stopped, and the process proceeds to S82.

  In S79, the CPU 41 determines whether or not the driver is gazing at the feature. Note that the specific processing contents are the same as in S61, and thus the details are omitted.

  If it is determined that the driver is not gazing at the feature (S79: NO), the information about the feature is provided as identification information for identifying the feature (S80). For example, when the characteristic object is a guide plate, an image of the guide plate is displayed on the liquid crystal display 15. Further, when the feature is a road sign, guidance on the content of the road sign (for example, guidance on speed limit, one-way traffic, traveling direction-specific traffic classification) is performed. Thereby, it is possible to provide the driver with more detailed information about the feature visually recognized by the driver.

  On the other hand, when it is determined that the driver is gazing at the feature (S79: YES), the information such as the auxiliary information for assisting the passenger in understanding the feature is provided as information on the feature. More detailed information than S80 is guided (S81). Specifically, more detailed information on the feature is read from the captured image DB 32. For example, when the feature is a guide plate, guidance on the connected road ahead is performed. If the feature is a facility such as a landmark, the facility name and genre are provided. When the feature is a road sign, guidance is provided to explain the type of road sign and what kind of regulations and instructions the road sign indicates. Accordingly, it is possible to provide the driver with more detailed information about the feature that the driver has watched, and to allow the passenger who cannot understand the content of the feature to understand the content of the feature. In S81, it is presumed that the driver is strongly interested in the feature. Therefore, the guidance content is more detailed than the guidance (S80) when the driver is not gazing at the feature. It becomes the contents to guide about.

  On the other hand, in S82, the CPU 41 determines whether or not the driver is watching the feature. Note that the specific processing contents are the same as in S61, and thus the details are omitted.

  If it is determined that the driver is not gazing at the feature (S82: NO), the information about the feature is provided as identification information for identifying the feature (S83). For example, when the characteristic object is another vehicle, the presence / absence of a driver in the vehicle (that is, whether or not there is a possibility of starting traveling) is provided. Thereby, it is possible to provide the driver with more detailed information about the feature visually recognized by the driver.

  On the other hand, if it is determined that the driver is gazing at the feature (S82: YES), the supplementary information for assisting the occupant's understanding of the content of the feature is provided as information on the feature. More detailed information than S83 is guided (S84). Specifically, more detailed information on the feature is read from the captured image DB 32. For example, when the feature is a vehicle, guidance on the vehicle type and year is performed. In addition, when the characteristic object is a pedestrian, guidance of gender and age group is performed. Accordingly, it is possible to provide the driver with more detailed information about the feature that the driver has watched, and to allow the passenger who cannot understand the content of the feature to understand the content of the feature. In S84, it is estimated that the driver has a strong interest in the feature. Therefore, the guidance content is more detailed than the guidance when the driver is not gazing at the feature (S83). It becomes the contents to guide about.

  In the second guidance process, the CPU 41 may execute the processes of S71 to S84 for each characteristic object when a plurality of characteristic objects are detected in the characteristic object detection of S4. The processes of S71 to S84 may be executed only for the feature closest to the vehicle.

As described above in detail, in the navigation device 1 according to the present embodiment, the driving support method using the navigation device 1, and the computer program executed by the navigation device 1, the driver's line-of-sight position from the captured image captured by the driver camera, A line-of-sight circle and a visual field region are detected (S2, S3), and a feature in front of the traveling direction of the vehicle is detected from the captured image and feature image information 33 captured by the front camera 19 (S4). Based on this, the visual aspect of the driver to the feature ahead of the traveling direction of the vehicle is specified (S7), and guidance on the characteristic is performed based on the specified visual aspect (S8, S9). Therefore, it is possible to provide necessary guidance for the current driver at a necessary timing without performing unnecessary guidance. In addition, it is possible to appropriately specify information necessary for the current driver from the aspect of visual recognition of the driver's features.
Further, since guidance regarding the feature is performed based on the type of the feature ahead of the traveling direction of the vehicle and the occupant's visual recognition mode on the feature, information necessary for the passenger according to the type of the feature is particularly guided. It becomes possible.
In addition, since the feature object is guided based on whether or not the driver is gazing at the feature object, it is possible to perform appropriate guidance according to the visual recognition mode of the driver to the feature object. For example, when the driver is not gazing at the feature object, it is possible to urge the feature object to be gazed and cause the feature object to be gazed. Further, when the driver is gazing at the feature, it is possible to provide detailed information regarding the feature that the driver is gazing at (that is, the occupant is interested).
In addition, since the feature is guided based on whether or not the driver is visually recognizing the feature, it is possible to perform appropriate guidance according to the driver's visual aspect to the feature. For example, when the driver does not visually recognize the feature object, the driver can be made to appropriately recognize the presence of the feature object. In addition, when the driver visually recognizes the characteristic object, it is possible to provide information regarding the characteristic object visually recognized by the driver.
In addition, since the feature object to be guided is the object that the driver of the vehicle traveling on the road observes, the driver knows in advance the presence of an object that may affect the vehicle traveling. And it is possible to drive appropriately.

Note that the present invention is not limited to the above-described embodiment, and various improvements and modifications can be made without departing from the scope of the present invention.
For example, in this embodiment, the line-of-sight range including the driver's line-of-sight position is specified by a circular line-of-sight circle, but may be specified by a shape other than a circle (for example, a square shape or a hexagonal shape).

  Further, in this embodiment, the visual recognition mode for the driver's feature is classified according to the four patterns shown in FIG. 16, but the number of patterns to be classified may be 3 or less or 5 or more. Further, the visual aspect may be specified in consideration of the area of the line-of-sight circle in addition to the overlapping ratio. Specifically, the larger the area of the line-of-sight circle, the longer it can be estimated that the feature is viewed longer, so when the area of the line-of-sight circle is greater than or equal to a predetermined value, the driver gazes at the feature. It may be determined that

  Moreover, the guidance content regarding the characteristic object demonstrated by the 2nd guidance process (FIG. 19) is an example, The content can be changed suitably. For example, the shape and moving speed of the feature may be guided.

  In the present embodiment, both the first guidance process (FIG. 17) and the second guidance process (FIG. 19) are executed in the driving support processing program, but only one of them may be executed.

1 Navigation device 13 Navigation ECU
19 Front camera 20 Driver camera 41 CPU
42 RAM
43 ROM
51 Vehicle 52 Driver

Claims (10)

Surrounding environment imaging means for imaging the surrounding environment ahead of the traveling direction of the vehicle;
Feature object image information acquisition means for acquiring feature object image information storing an image of the feature object;
Line-of-sight detection means for detecting the line-of-sight start point and line-of-sight direction of the vehicle occupant;
The line of sight of the occupant with respect to the surrounding environment ahead of the direction of travel of the vehicle based on the captured image captured by the surrounding environment imaging unit and the sight line start point and the line of sight direction of the occupant detected by the line-of-sight detection unit A line-of-sight position specifying means for specifying the line-of-sight position in which the image is located in the captured image;
A line-of-sight area specifying unit that specifies a range on the captured image including the line-of-sight position specified by the line-of-sight position specifying unit within a past predetermined time from the current time as a line-of-sight area;
Based on the captured image captured by the surrounding environment imaging unit and the feature image information, the feature detection unit that detects the feature in front of the traveling direction of the vehicle as a forward feature;
A feature region specifying means for specifying, on the captured image, a region surrounding the front feature as a feature region;
Based on the positional relationship between the line-of-sight area and the feature area, a visual aspect specifying unit that specifies the visual aspect of the occupant with respect to the front feature;
A driving support system comprising: guidance means for performing guidance related to the front feature with contents according to the visual aspect of the occupant identified by the visual aspect identification means. The visual recognition mode specifying unit specifies whether the occupant is gazing at the front feature,
When the occupant determines that the occupant is gazing at the front feature, the guide means is more detailed than when determining that the occupant is not gazing at the front feature. The driving support system according to claim 1, wherein the guidance is performed. Movement determining means for determining whether or not the front feature is a moving object;
The guiding means includes
When it is determined by the movement determining means that the front feature is a moving body, and when the visual feature specifying means determines that the front feature is being watched, the front feature And information that identifies the relationship between the vehicle and the vehicle,
When it is determined by the movement determination means that the front feature is a moving body, and when it is specified by the visual recognition mode specifying means that the front feature is not being watched, the front feature The driving support system according to claim 2, wherein information for identifying the vehicle is guided. Movement determining means for determining whether or not the front feature is a moving object;
The guiding means includes
When it is determined by the movement determination means that the front feature is not a moving body, and when it is specified by the visual recognition mode specifying means that the front feature is being watched, Guide auxiliary information to assist the passengers in understanding the content,
When it is determined by the movement determination means that the front feature is not a moving body, and when it is specified by the visual recognition mode specifying means that the front feature is not being watched, The driving support system according to claim 2, wherein identification information for identifying contents is guided. Having a type identifying means for identifying the type of the feature,
The feature object image information acquisition means acquires the feature object image information storing an image of the feature object for each type of the feature object,
The type identifying means identifies the type of the forward feature based on the feature image information acquired by the feature image information acquiring means,
The travel according to any one of claims 1 to 4, wherein the guide unit performs guidance related to the feature object according to a guide content corresponding to a type of the feature object identified by the type identification unit. Support system. A distance detecting means for detecting a distance from the vehicle to the front feature,
When it is determined that the distance from the vehicle to the front feature is equal to or greater than a predetermined distance, the guidance means provides guidance around the front feature or travel guidance with the front feature as a destination. The driving support system according to any one of claims 1 to 5, wherein the driving support system is performed.   The travel support system according to any one of claims 1 to 6, wherein the characteristic object is an object to be observed by an occupant of the vehicle traveling on a road.   The said visual recognition mode specification means specifies the said visual recognition mode of the said passenger | crew with respect to the said front feature based on the overlapping ratio of the said visual line area | region and the said feature object area | region, Any of Claim 1 thru | or 7 characterized by the above-mentioned. The driving support system described in Crab. A step in which the surrounding environment imaging means images the surrounding environment ahead of the traveling direction of the vehicle;
A feature object image information acquisition means for acquiring feature object image information storing an image of the feature object; and
A step of detecting a line-of-sight start point and a line-of-sight direction of an occupant of the vehicle;
The line-of-sight position specifying means is based on the captured image picked up by the surrounding environment image pickup means and the occupant's line-of-sight start point and line-of-sight direction detected by the line-of-sight detection means in the surrounding environment ahead of the vehicle traveling direction. A line of sight position where the line of sight of the occupant is located in the captured image;
A line-of-sight area specifying unit specifying, as a line- of-sight area, a range on the captured image including the line-of-sight position specified by the line-of-sight position specifying unit within a past predetermined time from the present time;
A feature detection unit that detects the feature in front of the traveling direction of the vehicle as a forward feature based on the captured image and the feature image information captured by the surrounding environment imaging unit;
A feature object specifying unit specifying a region surrounding the front feature as a feature object region on the captured image;
A step of identifying a visual aspect of the occupant with respect to the front feature based on a positional relationship between the line-of-sight region and the feature region;
And a step of guiding the front characteristic object with a content according to the visual aspect of the occupant specified by the visual aspect specifying means. Computer
Surrounding environment imaging means for imaging the surrounding environment ahead of the traveling direction of the vehicle;
Feature object image information acquisition means for acquiring feature object image information storing an image of the feature object;
Line-of-sight detection means for detecting the line-of-sight start point and line-of-sight direction of the vehicle occupant;
The line of sight of the occupant with respect to the surrounding environment ahead of the direction of travel of the vehicle based on the captured image captured by the surrounding environment imaging unit and the sight line start point and the line of sight direction of the occupant detected by the line-of-sight detection unit A line-of-sight position specifying means for specifying the line-of-sight position in which the image is located in the captured image;
A line-of-sight area specifying unit that specifies a range on the captured image including the line-of-sight position specified by the line-of-sight position specifying unit within a past predetermined time from the current time as a line-of-sight area;
Based on the captured image captured by the surrounding environment imaging unit and the feature image information, the feature detection unit that detects the feature in front of the traveling direction of the vehicle as a forward feature;
A feature region specifying means for specifying, on the captured image, a region surrounding the front feature as a feature region;
Based on the positional relationship between the line-of-sight area and the feature area, a visual aspect specifying unit that specifies the visual aspect of the occupant with respect to the front feature;
Guidance means for performing guidance on the front feature with contents according to the visual aspect of the occupant identified by the visual aspect identification means;
Computer program to make it function.

Images