JP2019164611A - Traveling support device and computer program - Google Patents

Abstract

To provide a traveling support device capable of further accurately determining risk factors in a peripheral environment of a mobile, and a computer program.SOLUTION: A traveling support device is configured to: acquire a picked-up image capturing a peripheral environment of a vehicle; acquire a map information image that is a three-dimensionally representing map image indicating the same range as a range of the picked-up image in the same direction as an imaging direction of the picked-up image in a map image; set determination target areas subjected to determination of risk factors in the peripheral environment of the vehicle; and input the picked-up image and the map information image to machine learning as input images, thereby extracting the risk factors in the determination target areas on the basis of learning data.SELECTED DRAWING: Figure 7

Description

  The present invention relates to a travel support apparatus and a computer program that support travel of a moving object.

  In recent years, for example, as one of driving support for a moving body such as a vehicle, a risk factor around the moving body is extracted and the extracted risk factor is guided. A risk factor is a factor to be noted when a moving object travels. For example, an obstacle such as another vehicle or a pedestrian, a crossing road, a pedestrian crossing, etc. that are in a position that is difficult to see from the moving object (the same applies hereinafter) Road signs, lane increase / decrease sections, and entrances to buildings facing the road. As a means for determining the risk factor as described above, for example, the determination is made by comparing the current position and orientation of the moving object with map information, or using a camera, sensor or communication device mounted on the moving object. Is possible.

  For example, in JP2012-192878A, the position and moving speed of obstacles such as other vehicles in the vicinity are detected in real time using a camera, a sensor, and a communication device installed in the vehicle while the vehicle is running. It is disclosed that when a vehicle has a blind spot area and it is determined that the risk of the blind spot area is high, driving assistance is performed to avoid an obstacle in the blind spot area.

JP2012-192878A (page 8-11)

  However, in Patent Document 1 above, the blind spot area is determined from a bird's-eye view based on the positional relationship between the host vehicle and the other vehicle. However, since the surrounding environment that is actually visible to the driver is not taken into consideration, The area determined to be the area may be different from the actual driver's blind spot area. Moreover, although the risk of the blind spot area is determined based on the behavior of the preceding vehicle and the oncoming vehicle, even if the preceding vehicle or the oncoming vehicle is determined to have a high risk, the blind spot area is actually dangerous. It was difficult to accurately determine whether or not. Therefore, a blind spot area that does not become a risk factor is also determined as a risk factor and may be a support target.

  The present invention has been made to solve the above-described conventional problems, and by setting a determination target region, it is possible to more accurately determine risk factors in the surrounding environment of the moving body. An object is to provide an apparatus and a computer program.

In order to achieve the above object, a driving support apparatus according to the present invention includes a surrounding environment imaging unit that acquires a captured image obtained by imaging a surrounding environment of a moving body, and an imaging range of the captured image in a map image that represents a map in three dimensions. A map information image acquisition means for acquiring a map image indicating the same range from the same direction as the imaging direction of the captured image as a map information image, and a determination target region for determining a risk factor in the surrounding environment of the mobile object Region setting means, and risk factor extraction means for extracting a risk factor in the determination target region based on learning data by inputting the captured image and the map information image as input images to machine learning. Have.
The “moving body” is not limited to a vehicle, and may be anything that moves on a road such as a pedestrian or a bicycle.
The “risk factor” is a factor to be noted when the moving body travels. For example, an obstacle such as other vehicles or pedestrians, intersections, etc. that are difficult to see from the moving body (the same applies hereinafter). There are road signs such as roads, pedestrian crossings, lane increase / decrease sections, and entrances to buildings facing the road.

  The computer program according to the present invention is a program for supporting traveling of a moving object. Specifically, the surrounding environment imaging means for acquiring a captured image obtained by imaging the surrounding environment of the moving body, and the captured image having the same range as the imaging range of the captured image in the map image expressing the map in three dimensions. A map information image acquiring means for acquiring a map image shown from the same direction as the imaging direction of the image as a map information image, an area setting means for setting a determination target area for determining a risk factor in the surrounding environment of the mobile body, By inputting the captured image and the map information image as input images to machine learning, the risk factor extraction means for extracting the risk factor in the determination target region based on the learning data is caused to function.

  According to the driving support device and the computer program according to the present invention having the above-described configuration, a map image expressing a map in three dimensions and a captured image around the moving body are input as input images to machine learning, and risk factors By setting the determination target region to be determined in advance, it becomes possible to more accurately determine the risk factor in the surrounding environment of the moving body based on the learning data.

It is the block diagram which showed the navigation apparatus which concerns on 1st Embodiment. It is a flowchart of the machine learning processing program which concerns on 2nd Embodiment. It is the figure which showed the imaging range of the captured image. It is the figure which showed the map information image produced | generated based on a captured image. It is a figure explaining an example of the determination object area | region. It is a figure explaining an example of the determination object area | region. It is a figure explaining an example of machine learning concerning a 1st embodiment. It is the figure which compared the captured image and the map information image. It is a flowchart of the risk factor determination processing program which concerns on 1st Embodiment. It is a figure explaining an example of machine learning concerning a 1st embodiment.

  Hereinafter, a driving support device according to the present invention will be described in detail with reference to the drawings based on a first embodiment and a second embodiment embodied in a navigation device.

[First Embodiment]
First, a schematic configuration of the navigation device 1 according to the first embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a navigation device 1 according to the first embodiment.

  As shown in FIG. 1, the navigation device 1 according to the first embodiment includes a current position detection unit 11 that detects a current position of a vehicle on which the navigation device 1 is mounted, and a data recording unit 12 that records various data. The navigation ECU 13 that performs various arithmetic processes based on the input information, the operation unit 14 that receives operations from the user, and the information on the map around the vehicle and the guidance route set in the navigation device 1 for the user Etc., a speaker 16 that outputs voice guidance regarding route guidance, warnings about risk factors, etc., a DVD drive 17 that reads DVD as a storage medium, a probe center and VICS (registered trademark: Vehicle Information and Communication System) A communication module that communicates with an information center such as a center. Has a 8, a. In addition, the navigation device 1 is connected to an out-of-vehicle camera 19 installed on a vehicle on which the navigation device 1 is mounted via an in-vehicle network such as CAN.

Below, each component which the navigation apparatus 1 has is demonstrated in order.
The current position detection unit 11 includes a GPS 21, a vehicle speed sensor 22, a steering sensor 23, a gyro sensor 24, and the like, and can detect the current vehicle position, direction, vehicle traveling speed, current time, and the like. . Here, in particular, the vehicle speed sensor 22 is a sensor for detecting a moving distance and a vehicle speed of the vehicle, generates a pulse according to the rotation of the driving wheel of the vehicle, and outputs a pulse signal to the navigation ECU 13. And navigation ECU13 calculates the rotational speed and moving distance of a driving wheel by counting the generated pulse. Note that the navigation device 1 does not have to include all the four types of sensors, and the navigation device 1 may include only one or more types of sensors.

  The data recording unit 12 reads an external storage device and a hard disk (not shown) as a recording medium, a map information DB 31 and a captured image DB 32 recorded on the hard disk, a predetermined program, and the like, and stores predetermined data on the hard disk. And a recording head (not shown) as a driver for writing. The data recording unit 12 may include a memory card, an optical disk such as a CD or a DVD, instead of the hard disk. Further, the map information DB 31 and the captured image DB 32 may be stored in an external server, and the navigation device 1 may acquire them by communication.

  Here, the map information DB 31 stores two-dimensional map information 33 and three-dimensional map information 34, respectively. The two-dimensional map information 33 is map information used in a general navigation device 1. For example, link data related to roads (links), node data related to node points, facility data related to facilities, and search data used for route search processing. , Map display data for displaying a map, intersection data for each intersection, search data for searching for a point, and the like.

  On the other hand, the three-dimensional map information 34 is information relating to a map image representing a map in three dimensions. In particular, in the first embodiment, the information is related to a map image expressing the outline of a road in three dimensions. In addition, it is good also as a map image expressing also information other than the outline of a road. For example, a facility image, a road marking line, a road sign, a signboard, and the like may be used as a map image expressed in three dimensions. Further, as the 3D map information 34, the map itself in which the road outline is arranged in the 3D space may be stored, or information necessary for expressing the map in 3D (3D of the road outline). Coordinate data, etc.) may be stored. When information necessary for representing a map in three dimensions is stored, the navigation apparatus 1 uses the information stored as the three-dimensional map information 34 at the necessary timing to select a target area 3 It is possible to generate a map expressed in dimensions.

  The navigation device 1 uses the two-dimensional map information 33 for general functions such as displaying a map image on the liquid crystal display 15 and searching for a guide route. Further, as will be described later, the process relating to the risk factor determination is performed using the three-dimensional map information 34.

  The captured image DB 32 is a storage unit that stores the captured image 35 captured by the camera 19 outside the vehicle. The captured images 35 captured by the camera 19 outside the vehicle are cumulatively stored in the captured image DB 32 and are deleted in order from the oldest image.

  On the other hand, the navigation ECU (Electronic Control Unit) 13 is an electronic control unit that controls the entire navigation device 1. The CPU 41 as an arithmetic device and a control device, and a working memory when the CPU 41 performs various arithmetic processes. In addition to the RAM 42 for storing route data when a route is searched, a control program, a machine learning processing program (see FIG. 2) and a risk factor determination processing program (see FIG. 9) described later. ) Etc. are recorded, and an internal storage device such as a flash memory 44 for storing a program read from the ROM 43 is provided. The navigation ECU 13 has various means as processing algorithms. For example, the surrounding environment imaging unit acquires a captured image obtained by imaging the surrounding environment of the moving body (vehicle). The map information image acquisition means acquires, as a map information image, a map image that shows the same range as the imaging range of the captured image from the same direction as the imaging direction of the captured image in the map image that represents the map in three dimensions. The area setting means sets a determination target area for determining a risk factor in the surrounding environment of the moving body. The risk factor extraction means extracts the risk factor in the determination target region based on the learning data by inputting the captured image and the map information image as input images to machine learning.

  The operation unit 14 is operated when inputting a starting point as a travel start point and a destination as a travel end point, and has a plurality of operation switches (not shown) such as various keys and buttons. Then, the navigation ECU 13 performs control to execute various corresponding operations based on switch signals output by pressing the switches. The operation unit 14 may have a touch panel provided on the front surface of the liquid crystal display 15. Moreover, you may have a microphone and a speech recognition apparatus.

  The liquid crystal display 15 includes a map image including a road, traffic information, operation guidance, operation menu, key guidance, guidance route set in the navigation device 1, guidance information along the guidance route, news, weather forecast. , Time, mail, TV program, etc. are displayed. In place of the liquid crystal display 15, HUD or HMD may be used. In the first embodiment, guidance on risk factor determination results is also displayed.

  The speaker 16 outputs voice guidance for guiding traveling along the guidance route based on an instruction from the navigation ECU 13 and traffic information guidance. In the first embodiment, the guidance of the risk factor determination result is also output.

  The DVD drive 17 is a drive that can read data recorded on a recording medium such as a DVD or a CD. Based on the read data, music and video are reproduced, the map information DB 31 is updated, and the like. A card slot for reading / writing a memory card may be provided instead of the DVD drive 17.

  The communication module 18 is a communication device for receiving traffic information transmitted from a traffic information center such as a VICS center or a probe center, and corresponds to a mobile phone or DCM, for example.

  The vehicle exterior camera 19 is constituted by a camera using a solid-state imaging device such as a CCD, for example, and is installed on the rear side of a vehicle rearview mirror, a front bumper, etc. The And the camera 19 outside a vehicle images the surrounding environment ahead of the advancing direction of a vehicle. In addition, the navigation ECU 13 determines a risk factor around the vehicle by inputting the captured image captured as described later to the machine learning as an input image of a plurality of channels together with the image of the three-dimensional map information 34. In addition, you may comprise the vehicle exterior camera 19 so that it may arrange | position also to the side and back of a vehicle. Further, the installation position of the outside camera 19 is substantially the same as the position of the driver's eyes (the line-of-sight start point), and the optical axis direction is adjusted to be substantially the same as the driver's line-of-sight direction at the normal time. desirable. As a result, the image captured by the vehicle camera 19 matches the driver's field of view, and the risk factor can be determined more appropriately.

  The risk factor determined by the machine learning in the navigation device 1 according to the first embodiment is a factor that should be noted (the guidance should be performed) when the vehicle travels. For example, there are obstacles such as other vehicles and pedestrians, road signs such as cross roads and pedestrian crossings, lane increase / decrease sections, entrances to buildings facing the road, etc. . For example, the “entrance / exit of a building facing the road” is a point where a pedestrian may newly appear on the road, and is a place to be noted when the vehicle travels. The “lane increase / decrease section” is a point where another vehicle may change lanes, and is a place to be careful when the vehicle travels.

  In the first embodiment, as described above, the three-dimensional map information 34 is information related to a map image that expresses a road contour line in three dimensions. Therefore, the risk factor to be determined by inputting the map image of the three-dimensional map information 34 into the machine learning is a factor related to the road, for example, “a road in a position that is difficult to see from the vehicle”. However, by increasing the number of information included in the three-dimensional map information 34, other types of risk factors can be determined. For example, by including lane marking information in the three-dimensional map information 34, “a lane increase / decrease section at a position that is difficult to see from the vehicle” is determined as a risk factor. Furthermore, by including information on the shape of the facility in the three-dimensional map information 34, “a doorway of a building facing a road at a position that is difficult to see from the vehicle” is determined as a risk factor.

  Next, a machine learning process program executed by the CPU 41 in the navigation device 1 according to the first embodiment having the above configuration will be described with reference to FIG. FIG. 2 is a flowchart of the machine learning processing program according to the first embodiment. Here, the machine learning processing program is executed after the ACC of the vehicle is turned on, and performs machine learning for determining a risk factor based on the captured image captured by the outside camera 19 and the three-dimensional map information 34. Is a program for setting a teacher signal for learning (correct answer value). The machine learning processing program (FIG. 2) is executed in parallel with a risk factor determination processing program (FIG. 9) for determining risk factors described later. That is, while determining risk factors by the risk factor determination processing program (FIG. 9), a more appropriate learning teacher signal (correct value) for determining risk factors by the machine learning processing program (FIG. 2) is set. To do. 2 and 7 are stored in the RAM 42, the ROM 43, and the like provided in the navigation ECU 13, and are executed by the CPU 41.

  First, in the machine learning processing program, in step (hereinafter abbreviated as S) 1, the CPU 41 acquires the current position and direction of the vehicle based on the detection result of the current position detection unit 11. Specifically, position coordinates on the map indicating the current position of the vehicle are acquired using the two-dimensional map information 33. Note that when detecting the current position of the vehicle, a map matching process for matching the current position of the vehicle with the two-dimensional map information 33 is also performed. Furthermore, the current position of the vehicle may be specified using a high-precision location technique. Here, the high-accuracy location technology detects the white line and road surface paint information captured from the camera behind the vehicle by image recognition, and further compares the white line and road surface paint information with a previously stored map information DB, thereby driving the vehicle. This is a technology that makes it possible to detect lanes and highly accurate vehicle positions. The details of the high-accuracy location technology are already known and will be omitted. It is desirable that the current position and direction of the vehicle are finally specified on the map of the three-dimensional map information 34.

  Next, in S2, the CPU 41, among the 3D map information 34 stored in the map information DB 31, particularly 3D map information around the current position of the vehicle specified in S1 (for example, around 300 m from the current position of the vehicle). 34 is acquired.

  Subsequently, in S <b> 3, the CPU 41 obtains a captured image that was most recently captured by the outside camera 19 from the captured image DB 32. The captured image captured by the camera 19 outside the vehicle corresponds to the driver's line-of-sight start point (viewpoint) and the line-of-sight direction in the forward direction of the vehicle, that is, the front environment (driver's field of view) visually recognized by the driver. This is a captured image.

  Thereafter, in S4, the CPU 41 acquires the imaging range of the captured image acquired in S3. Here, as shown in FIG. 3, the imaging range of the captured image 35 can be specified by the position of the focal point P, the optical axis direction α, and the angle of view φ of the outside camera 19 at the time of imaging. The angle of view φ is a fixed value determined in advance by the camera 19 outside the vehicle. On the other hand, the position of the focal point P is determined based on the current position of the vehicle acquired in S1 and the installation position of the outside camera 19 in the vehicle. Further, the optical axis direction α is determined based on the direction of the vehicle acquired in S1 and the installation direction of the outside camera 19 in the vehicle.

  Next, in S <b> 5, the CPU 41 uses the 3D map information 34 acquired in S <b> 2 to map the same range as the imaging range of the captured image acquired in S <b> 4 from the same direction as the imaging direction of the captured image. An overhead image of the image (hereinafter referred to as a map information image) is generated. Note that the map information image itself is the same two-dimensional image as the captured image.

  Here, FIG. 4 is a diagram showing an example of the map information image 52 generated corresponding to the captured image 51. As shown in FIG. 4, the map information image 52 is an image in which a line indicating the contour 53 of the road included in the imaging range of the captured image 51 (that is, within the driver's field of view) is drawn. In the captured image 51, the contour of the road hidden by an obstacle such as another vehicle is also drawn in the map information image 52.

  Thereafter, in S6, the CPU 41 sets a determination target area for determining risk factors in the surrounding environment of the vehicle, and acquires a distance from the own vehicle to the determination target area. In the following description, an example will be described in which a connection road (a road connected at a branch point) located at a position that is in front of the vehicle in the traveling direction and difficult to be visually recognized as a risk factor is described. In this case, it is desirable that the determination target region for determining the risk factor is a region including at least a part of the road on which the vehicle travels and a connecting road connected at a branch point around the vehicle.

  For example, in the example shown in FIG. 4, roads 58 and 59 connected to the branch point 57 ahead of the traveling direction of the vehicle correspond to the connection road. Therefore, the risk factor is determined for the roads 58 and 59, and the determination target area for determining the risk factor is set to an area including at least a part of the roads 58 and 59. For example, as shown in FIG. 5, a peripheral region (for example, a rectangular region having a width of 3 m and a height of 2 m) centered on the end portion X located on the most vehicle side where the roads 58 and 59 are connected to the branch point 57 is determined. The target area 61 is set. Alternatively, as shown in FIG. 6, the road 58, 59 is 3 m in a direction away from the connecting branch point 57 along the roads 58, 59, with the end X located closest to the host vehicle connecting with the branch point 57 as the origin. It is also possible to set a rectangular area of 2 m in the height direction as the determination target area 61. In this embodiment, the roads 58 and 59 connected in the direction intersecting with the road on which the vehicle travels are particularly targeted. However, the road 60 connected in the traveling direction may also be included in the connection road.

  In S <b> 6, the CPU 41 is positioned as the distance from the own vehicle to the determination target region, so that the road on which the vehicle travels and the connection road connecting at the branch points around the vehicle are located closest to the own vehicle connecting to the branch point. The distance to the edge (edge X in the examples shown in FIGS. 5 and 6) is acquired. Specifically, it is calculated based on the current position of the vehicle acquired in S1 and the three-dimensional map information 34. In addition, when the vehicle exterior camera 19 is a stereo camera, it is also possible to detect from a captured image. Further, if there is a point serving as a reference for the determination target region other than the end X, the distance to the point may be acquired.

  In addition, although 1st Example demonstrates the example which determines the connection road (road connected by a branch point) in the position which is ahead of the advancing direction of a vehicle and is difficult to visually recognize from a vehicle as a risk factor, Different types of risk factors (for example, entrances to buildings, pedestrians moving around roads, other vehicles, etc.) can be determined. Then, it is desirable to appropriately set the determination target area according to the type of risk factor to be determined.

  Thereafter, in S7, the CPU 41 inputs the captured image acquired in S3 and the map information image generated in S5 as input images of a plurality of channels to machine learning. In addition, since the captured image is input to machine learning as a three-channel input image expressed in three colors of RGB, it becomes a four-channel input image including the map information image. In the first embodiment, machine learning (Deep Learning) using a convolutional neural network having a multilayer structure is used.

  As shown in FIG. 7, when a captured image 51 and a map information image 52 expressed in three colors of RGB are input to a learning model as 4-channel input images, first, an image based on a convolutional neural network (hereinafter referred to as convolution CNN) 65. Processing is performed. In the convolution CNN 65, the “convolution layer” and the “pooling layer” are repeated a plurality of times, and a particularly important feature map 66 for determining a risk factor is output in the process.

  The “convolution layer” is a layer that filters (convolves) an input image. A pattern (feature) in an image can be detected by convolution of the image. In addition, a plurality of filters are convolved. By using a plurality of filters, various features of the input image can be captured. Further, the size of the output image is reduced by applying the filter. The output image is also called a feature map. Moreover, the filter used for this convolution layer does not need to be set by the designer, and can be acquired by learning. As the learning progresses, a filter suitable for extracting a particularly important feature for determining a risk factor is set. The teacher signal for learning (correct value) set by the machine learning processing program includes the filter.

  Furthermore, in S8, the CPU 41 also inputs information for specifying the determination target area set in S6 to the machine learning. In particular, in the navigation device 1 according to the first embodiment, the input is executed by performing mask processing on the feature map 66 output from the convolution CNN 65 based on the input distance information. Masking is performed in an area other than the determination target area.

  Specifically, the CPU 41 first corresponds to the determination target area 61 in the feature map 66 (that is, the determination target area in the captured image 51 and the map information image 52) based on the distance information to the determination target area 61 acquired in S6. ) Range coordinates (coordinates in the coordinate system set for the map (image)) are specified. Then, based on the specified coordinates, mask processing is performed on areas other than the area specified as the determination target area in the feature map 66. Note that the features extracted in the masked area are excluded from the characteristics for determining the risk factor (that is, the risk factor is determined outside the masked area).

  On the other hand, the “pooling layer” is placed immediately after the convolution layer, and lowers the position sensitivity of the extracted features. Specifically, the difference due to a slight shift of the image is absorbed by roughly re-sampling the output of the convolution. Even in the pooling layer, the size of the output image is smaller than that of the input image.

  Then, after repeating the “convolution layer” and the “pooling layer” a plurality of times, a multilayer perceptron that is finally fully coupled is output from the feature map 66. The multilayer perceptron is one-dimensional vector data.

  Thereafter, the fully coupled multilayer perceptron output by the convolution CNN 65 is input to an input layer of a neural network (hereinafter referred to as risk determination CNN) 67 for risk factor determination. In the risk determination CNN 67, data obtained by multiplying each neuron, which is output data after processing in the input layer, by a weight (weight coefficient), is input to the next intermediate layer. Similarly, in the intermediate layer, data obtained by multiplying each neuron by a weight (weighting factor), which is output data after processing in the intermediate layer, is input to the next output layer. Then, the final risk factor is determined using data input from the intermediate layer in the output layer, and the determination result (that is, the extracted risk factor) is output. The risk determination CNN 67 is appropriately changed and set to a more suitable value for the weight (weight coefficient) as learning progresses. That is, the learning teacher signal (correct value) set by the machine learning processing program includes the weight (weight coefficient).

  Further, in particular, in the first embodiment, by inputting the captured image 51 and the map information image 52, which are input targets, by overlapping them on the same channel, the correlation between the same pixels (that is, the same area around the vehicle), that is, It is possible to easily identify a difference area where there is a difference between the captured image and the map information image. The CPU 41 is an area where there is a difference area between the captured image and the map information image that exists on the map image but has disappeared (not captured) for some reason in the captured image. It is estimated that the area is a blind spot from the passenger. Accordingly, by determining the risk factor from the feature portion extracted by machine learning, particularly in the difference area, it is possible to further reduce the processing related to the risk factor determination. Note that the feature portion extracted by machine learning includes an object that is included in the map information image but not included in the captured image, that is, an object that is in the driver's field of view but cannot be seen. Will be included.

  For example, when the captured image 51 (three images actually expressed in RGB) and the map information image 52 shown in FIG. 8 and the map information image 52 are input to the machine learning, the contour of the road existing in the map information image 52 is displayed. Is lost in the captured image 51 by another vehicle. That is, in the example shown in FIG. 8, the area surrounded by the broken line is the difference area 68. In addition, the feature portion extracted by the machine learning is another vehicle that covers the road outline, or a lost outline. Since these feature portions are in the difference area 68, by making the difference area 68 easily identifiable, it becomes possible to facilitate the processing relating to the extraction of the feature portions.

  Furthermore, in the first embodiment, as described above, information for specifying the determination target region 61 is also input to machine learning. As a result, risk factors are determined from the feature portions extracted by machine learning, particularly in the determination target region 61. As a result, when there is a risk factor candidate that should cause the driver to know the presence of a connected road or the like that cannot be visually recognized by the driver, it becomes easy to determine the connected road as a risk factor by machine learning. In the example shown in FIG. 8, the outline of the road disappears from the captured image 51 due to the other vehicle, but may disappear due to a building or the like other than the other vehicle.

  Thereafter, in S9, the CPU 41 sets a learning teacher signal (correct answer value), which is learning data, based on the learning results in S7 and S8. Note that the learning teacher signal includes a filter used for the convolution layer and a weight (weighting coefficient) used for the risk determination CNN 67 as described above. As a result, by repeating the machine learning processing program, a filter suitable for extracting particularly important features for identifying risk factors is set, and more accurate risk factors can be determined from the surrounding situation (scene) of the vehicle. A determination can be made. Particularly in the first embodiment, the learning teacher signal includes an object that is included in the map information image but is not included in the captured image (that is, an object that is in the driver's field of view but cannot be seen). Learning data for determining as a risk factor.

  Next, a risk factor determination processing program executed by the CPU 41 in the navigation device 1 according to the first embodiment will be described with reference to FIG. FIG. 9 is a flowchart of the risk factor determination processing program according to the first embodiment. Here, the risk factor determination processing program is executed after the ACC of the vehicle is turned on, and determines the risk factor around the vehicle based on the captured image captured by the outside camera 19 and the three-dimensional map information 34, It is a program that outputs the judgment result. The risk factor determination processing program (FIG. 9) is executed in parallel with the above-described machine learning processing program (FIG. 2). That is, while determining risk factors by the risk factor determination processing program (FIG. 9), a more appropriate learning teacher signal (correct value) for determining risk factors by the machine learning processing program (FIG. 2) is set. To do.

  First, in the risk factor determination processing program, in S <b> 11, the CPU 41 acquires the current position and direction of the vehicle based on the detection result of the current position detection unit 11. The details are the same as in S1, and will be omitted.

  Next, in S12, the CPU 41, among the 3D map information 34 stored in the map information DB 31, in particular, the 3D map information around the current position of the vehicle specified in S11 (for example, around 300 m from the current position of the vehicle). 34 is acquired.

  Subsequently, in S <b> 13, the CPU 41 acquires a captured image captured by the vehicle exterior camera 19 most recently from the captured image DB 32. The captured image captured by the camera 19 outside the vehicle corresponds to the driver's line-of-sight start point (viewpoint) and the line-of-sight direction in the forward direction of the vehicle, that is, the front environment (driver's field of view) visually recognized by the driver. This is a captured image.

  Thereafter, in S14, the CPU 41 acquires the imaging range of the captured image acquired in S3. The details are the same as in S4, and will be omitted.

  Next, in S15, the CPU 41 uses the three-dimensional map information 34 acquired in S12, and the map when the same range as the imaging range of the captured image acquired in S14 is viewed from the same direction as the imaging direction of the captured image. An overhead image (map information image) of the image is generated.

  Thereafter, in S <b> 16, the CPU 41 sets a determination target region for determining risk factors in the surrounding environment of the vehicle, and acquires a distance from the vehicle to the determination target region. The details are the same as S6, and will be omitted.

  Subsequently, in S17, the CPU 41 inputs the captured image acquired in S13 and the map information image generated in S15 to machine learning as a multi-channel input image. In addition, since the captured image is input to machine learning as a three-channel input image expressed in three colors of RGB, it becomes a four-channel input image including the map information image. In the first embodiment, machine learning (Deep Learning) using a convolutional neural network having a multilayer structure is used.

  In the first embodiment, machine learning (Deep Learning) using a convolutional neural network having a multilayer structure is used. In addition, the latest learning teacher signal (correct value) set by the machine learning processing program (FIG. 2) described above is used.

  Furthermore, in S18, the CPU 41 also inputs information for specifying the determination target area set in S16 to the machine learning. Since the contents of machine learning have been described in S7 and S8, details are omitted.

  In S19, the CPU 41 determines risk factors in the surrounding environment of the vehicle based on the machine learning results in S17 and S18. As described above, in the first embodiment, a correlation between the same pixels (that is, the same area around the vehicle), that is, imaging is performed by inputting the captured image to be input and the map information image by overlapping them on the same channel. It becomes possible to easily identify the difference area where there is a difference between the image and the map information image. Further, risk factors are determined from feature parts extracted by machine learning in a predetermined determination target region. Thereby, it is possible to more accurately and lighten the processing relating to the determination and extraction of risk factors. The learning teacher signal includes, as a risk factor, an object that is included in the map information image but not included in the captured image (that is, an object that is in the driver's field of vision but cannot be visually recognized). Since it includes learning data for determination, when there is a risk factor candidate such as a connection road that cannot be visually recognized by the driver in the surrounding environment of the vehicle, the connection road is considered dangerous based on the learning data such as a learning teacher signal. It becomes possible to determine accurately as a factor.

  If the accuracy is further improved by repeated learning, it is possible to determine a risk factor that the driver cannot notice.

  Thereafter, the navigation apparatus 1 outputs the risk factor determination result (ie, the extracted risk factor). Specifically, when it is determined that there is a risk factor, the user may be informed that the risk factor exists, or vehicle control that avoids the risk factor may be performed. For example, the guidance “Please pay attention to the front of the vehicle” is performed. Also, if machine learning can determine what kind of risk factor it is, the guidance that identifies the risk factor more specifically (for example, "Please pay attention to the intersection road in the front blind spot of the vehicle" )). Moreover, you may guide also about the position of a risk factor. On the other hand, when performing vehicle control, deceleration control is performed, for example. It can also be applied to an autonomous driving vehicle. In that case, control such as setting a travel route that avoids risk factors is possible.

As described in detail above, in the navigation device 1 and the computer program executed by the navigation device 1 according to the first embodiment, a captured image obtained by capturing the surrounding environment of the vehicle is acquired (S13), and the map is expressed in three dimensions. A map information image, which is a map image showing the same range as the captured range of the captured image from the same direction as the captured direction of the captured image in the acquired map image (S15), and a determination target for determining risk factors in the surrounding environment of the vehicle A region is set (S16, S18), and the risk factor in the determination target region is extracted based on the learning data by inputting the captured image and the map information image as input images to machine learning (S17, S18). Based on the learning data, it becomes possible to more accurately determine the risk factor in the surrounding environment of the moving body.
Further, in the prior art (for example, Japanese Patent Application Laid-Open No. 2012-192878), the position and the moving speed of a preceding preceding vehicle and an oncoming vehicle are always detected using a camera, a sensor, and a communication device installed in the vehicle while the vehicle is running. If the number of applicable vehicles increases, there is a problem that the processing burden related to the determination of risk factors becomes very heavy. On the other hand, in the first embodiment, it is possible to reduce the processing burden relating to the risk factor determination as compared with the conventional case.

[Second Embodiment]
Next, a navigation device according to a second embodiment will be described with reference to FIG. In the following description, the same reference numerals as those of the configuration of the navigation device 1 according to the first embodiment in FIGS. 1 to 9 indicate the same or corresponding parts as the configuration of the navigation device 1 according to the first embodiment. It is.

The schematic configuration of the navigation device according to the second embodiment is substantially the same as that of the navigation device 1 according to the first embodiment. Various control processes are also substantially the same as those in the navigation apparatus 1 according to the first embodiment.
However, the feature map output in the convolution CNN 65 when the navigation device 1 according to the first embodiment inputs information for specifying the determination target region to the machine learning in the learning process for determining the risk factor (S8, S18). The navigation apparatus according to the second embodiment is different from the navigation apparatus 1 according to the first embodiment in that information for specifying the determination target region is added as a neuron while the mask process is performed on 66. .

  Below, the learning process (S7, S8, S17, S18) which determines the risk factor performed in the navigation apparatus which concerns on 2nd Embodiment is demonstrated based on FIG.

  As shown in FIG. 10, when a captured image 51 and a map information image 52 expressed in three colors of RGB are input to a learning model as a four-channel input image, first, an image based on a convolutional neural network (hereinafter referred to as convolution CNN) 65. Processing is performed. In the convolution CNN 65, the “convolution layer” and the “pooling layer” are repeated a plurality of times, and a particularly important feature map 66 for determining a risk factor is output in the process. However, in the second embodiment, the mask process for the feature map 66 is not performed.

  Then, after repeating the “convolution layer” and the “pooling layer” a plurality of times, a multilayer perceptron that is finally fully coupled is output from the feature map 66. The multilayer perceptron is one-dimensional vector data.

  Thereafter, the fully coupled multilayer perceptron output by the convolution CNN 65 is input to the input layer of the risk determination CNN 67 for risk factor determination. At that time, the CPU 41 inputs information specifying the determination target region 61 to the risk determination CNN 67 as a new neuron (feature data).

  Specifically, the CPU 41 first determines the determination target region 61 in the feature map 66 (that is, the determination target region in the captured image 51 and the map information image 52 based on the distance information to the determination target region acquired in S6 or S16. Equivalent) coordinates (coordinates in the coordinate system set for the map (image)) are specified. Then, the specified coordinate data is input to the danger determination CNN 67 as a new neuron. If there are a plurality of determination target regions 61, a plurality of neurons can be added.

  In the risk determination CNN 67, data obtained by multiplying each neuron, which is output data after processing in the input layer, by a weight (weight coefficient), is input to the next intermediate layer. Similarly, in the intermediate layer, data obtained by multiplying each neuron by a weight (weighting factor), which is output data after processing in the intermediate layer, is input to the next output layer. Then, the final risk factor is determined using data input from the intermediate layer in the output layer, and the determination result (that is, the extracted risk factor) is output. In addition, by adding a neuron that identifies the determination target region 61, it is possible to determine a risk factor particularly for a feature portion extracted in the determination target region 61. Thereby, it is possible to more accurately and lighten the processing relating to the determination and extraction of risk factors.

  As described above in detail, in the navigation device 1 and the computer program executed by the navigation device 1 according to the second embodiment, a captured image obtained by capturing the surrounding environment of the vehicle is acquired (S13) and the map is expressed in three dimensions. A map information image, which is a map image showing the same range as the captured range of the captured image from the same direction as the captured direction of the captured image in the acquired map image (S15), and a determination target for determining risk factors in the surrounding environment of the vehicle A region is set (S16, S18), and the risk factor in the determination target region is extracted based on the learning data by inputting the captured image and the map information image as input images to machine learning (S17, S18). Based on the learning data, it becomes possible to more accurately determine the risk factor in the surrounding environment of the moving body.

Note that the present invention is not limited to the above-described embodiment, and various improvements and modifications can be made without departing from the scope of the present invention.
For example, in the above-described embodiment, the three-dimensional map information 34 is information regarding a map image that expresses the contour of the road in three dimensions, but may be map information that expresses information other than the contour of the road. For example, it may be a map image expressing the facility shape, road lane markings, road signs, signboards, and the like. In that case, factors other than the cross road can be determined as risk factors. For example, by including lane marking information in the three-dimensional map information 34, it is possible to determine “a lane increase / decrease section at a position that is difficult to see from the vehicle” as a risk factor. Further, by including the facility shape information in the three-dimensional map information 34, it is possible to determine “a doorway of a building at a position that is difficult to see from the vehicle” as a risk factor.

  Moreover, in the said embodiment, as a determination object area | region which determines the risk factor in the surrounding environment of a vehicle, "The area | region which contains at least a part of the connection road connected with the branch point around the road where a vehicle drive | works, and a vehicle" However, if the type of risk factor to be determined changes, it is desirable to appropriately change the determination target region accordingly. For example, when determining “entrance / exit of a building that is difficult to see from the vehicle” as a risk factor, “region including at least a part of the entrance / exit of the building facing the road on which the vehicle travels” is to be determined. It can be set as an area.

  In the above embodiment, machine learning (Deep Learning) using a multi-layered convolutional neural network is used as the machine learning. However, other machine learning can also be used.

  In the above embodiment, the machine learning processing program (FIG. 2) and the risk factor determination processing program (FIG. 9) are executed in parallel. However, it is not always necessary to execute each program at the same time. The machine learning processing program (FIG. 2) may be executed after executing the program (FIG. 9).

  In the above embodiment, the machine learning processing program (FIG. 2) and the risk factor determination processing program (FIG. 9) are implemented by the navigation device 1, but may be configured by an on-vehicle device other than the navigation device. Moreover, it is good also as an external server performing some processes instead of an onboard equipment performing all the processes.

  Moreover, although the embodiment which actualized the driving support device according to the present invention has been described above, the driving support device can also have the following configuration, and in that case, the following effects can be obtained.

For example, the first configuration is as follows.
Peripheral environment imaging means (41) that acquires a captured image (51) that captures the surrounding environment of the moving body, and imaging of the captured image in the same range as the captured image of the captured image in a map image that represents a map in three dimensions A map information image acquisition means (41) for acquiring a map image shown from the same direction as the direction as a map information image (52), and a determination target region (61) for determining a risk factor in the surrounding environment of the moving body are set. Area setting means (41) for performing, and inputting the captured image and the map information image as input images to machine learning, thereby extracting a risk factor in the determination target area based on learning data (41).
According to the driving support apparatus having the above configuration, a determination target region that is a risk factor determination target while inputting a map image representing a map in three dimensions and a captured image around the moving object to machine learning as an input image By setting in advance, it becomes possible to more accurately determine risk factors in the surrounding environment of the moving body based on the learning data.

The second configuration is as follows.
The risk factor extraction means (41) extracts risk factors in the determination target region based on learning data by inputting information for specifying the determination target region (61) to machine learning.
According to the driving support apparatus having the above-described configuration, by inputting information for specifying the determination target area to machine learning, it is possible to determine the risk factor by focusing on the determination target area. Therefore, it is possible to accurately determine the risk factor, and to further reduce the processing load related to the determination.

The third configuration is as follows.
The determination target region (61) is a region including at least a part of a connection road that is a road connected to a road on which the mobile body moves and a branch point around the mobile body.
According to the driving support apparatus having the above-described configuration, it is possible to accurately determine risk factors in the vicinity of a connection road connected to a branch point around the moving body.

The fourth configuration is as follows.
The determination target region (61) is a peripheral region around the end where the connection road and the branch point are connected.
According to the driving support apparatus having the above-described configuration, it is possible to accurately determine a risk factor in the vicinity of a connection point between a branch point around the mobile body and a connection road connected to the branch point.

The fifth configuration is as follows.
The determination target region (61) is a region including a road portion within a predetermined distance along the connection road from the end connected to the branch point in the connection road.
According to the driving support apparatus having the above-described configuration, it is possible to accurately determine a risk factor in the vicinity of a connection point between a branch point around the mobile body and a connection road connected to the branch point.

The sixth configuration is as follows.
Road distance acquisition means (41) for acquiring a road distance that is a distance from the mobile body to the determination target region, wherein the risk factor extraction means (41) includes the captured image (51) and the map information image. The determination target area in (52) is specified based on the road distance, and risk factors are extracted in the determination target area.
According to the driving support apparatus having the above configuration, the risk factor in the determination target region can be more accurately determined based on the learning data by acquiring the distance to the determination target region that is the risk factor determination target. It becomes possible.

The seventh configuration is as follows.
The risk factor extracting means (41) determines a risk factor in the surrounding environment of the moving body by machine learning using a neural network having a multilayer structure.
According to the driving support apparatus having the above-described configuration, image features can be learned at a deeper level and machine features can be recognized with very high accuracy by machine learning using a multi-layered neural network. Judgment is possible. In particular, it is possible to set an optimum filter by learning without setting a filter for extracting features on the user side.

The eighth configuration is as follows.
The risk factor extracting means (41) masks a region not corresponding to the determination target region with respect to the feature map generated in the machine learning.
According to the driving support device having the above-described configuration, by performing mask processing on the feature map, it is possible to determine the risk factor by focusing on the determination target region, and the risk factor in the determination target region can be more accurately compared to the conventional case. In addition, it is possible to reduce the processing load related to the determination.

The ninth configuration is as follows.
The risk factor extracting means (41) adds information for specifying the determination target region to the neural network as a new neuron.
According to the driving support apparatus having the above configuration, by performing processing for adding information for specifying a determination target region as a new neuron, it is possible to determine risk factors by focusing on the determination target region. It is possible to accurately determine risk factors in the determination target area, and further reduce the processing load related to the determination.

1 Navigation device 13 Navigation ECU
19 Outside camera 31 Map information DB
32 Captured image DB
33 2D map information 34 3D map information 41 CPU
42 RAM
43 ROM
51 Captured image 52 Map information image 61 Target area 65 Convolution CNN
67 Danger judgment CNN

Claims (10)

A surrounding environment imaging means for acquiring a captured image obtained by imaging the surrounding environment of the moving body;
Map information image acquisition means for acquiring, as a map information image, a map image showing a map image representing the same range as the captured image of the captured image from the same direction as the captured direction of the captured image in a map image representing a map in three dimensions;
Area setting means for setting a determination target area for determining a risk factor in the surrounding environment of the moving body;
A driving support device comprising: a risk factor extracting unit that extracts a risk factor in the determination target region based on learning data by inputting the captured image and the map information image as input images to machine learning.   The driving support device according to claim 1, wherein the risk factor extraction unit extracts risk factors in the determination target region based on learning data by inputting information specifying the determination target region into machine learning.   The travel support according to claim 1, wherein the determination target region is a region including at least a part of a connection road that is a road connected to a road on which the mobile body moves and a branch point around the mobile body. apparatus.   The travel support device according to claim 3, wherein the determination target region is a peripheral region around an end portion where the connection road and the branch point are connected.   The travel support device according to claim 3, wherein the determination target region is a region including a road portion within a predetermined distance along the connection road from an end connected to the branch point in the connection road. Road distance acquisition means for acquiring a road distance that is a distance from the mobile body to the determination target region;
The risk factor extracting means includes
The determination target region in the captured image and the map information image is specified based on the road distance,
The driving support device according to any one of claims 1 to 5, wherein risk factors are extracted in the determination target region.   The driving support apparatus according to claim 1, wherein the risk factor extracting unit extracts a risk factor in the surrounding environment of the moving body by machine learning using a neural network having a multilayer structure.   The driving support apparatus according to claim 7, wherein the risk factor extracting unit masks a region that does not correspond to the determination target region with respect to the feature map generated in the machine learning.   The driving support apparatus according to claim 7, wherein the risk factor extracting unit adds, as a new neuron, information specifying the determination target region to the neural network. Computer
A surrounding environment imaging means for acquiring a captured image obtained by imaging the surrounding environment of the moving body;
Map information image acquisition means for acquiring, as a map information image, a map image showing a map image representing the same range as the captured image of the captured image from the same direction as the captured direction of the captured image in a map image representing a map in three dimensions;
Area setting means for setting a determination target area for determining a risk factor in the surrounding environment of the moving body;
A risk factor extracting means for extracting a risk factor in the determination target region based on learning data by inputting the captured image and the map information image as input images into machine learning;
Computer program to make it function.