JP2018173861A - Travel support system and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a travel support system and a computer program capable of outputting a determination result of risk factors without causing a user feel annoying.SOLUTION: The travel support system is configured to input pick-up images of the perimeter circumstance of a vehicle into a learning model which is sequentially generated by means of machine learning; to recognize the peripheral situation generated in the perimeter circumstance of the vehicle at each imaging timing of the input pick-up images; to determine risk factors existing in the peripheral circumstance of a moving body based on the recognized peripheral situation; and to output the determination result of the risk factors after changing the output mode based on whether any risk factor is generated in the peripheral situation similar to a previous peripheral situation in the perimeter circumstance of vehicle.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a travel support apparatus and a computer program that support travel of a moving object.

  In recent years, for example, as one of driving assistance for a moving body such as a vehicle, a risk factor around the moving body is determined, and the determination result is guided. A risk factor is a factor that should be noted when a moving object travels.For example, an obstacle such as another vehicle or a pedestrian that is difficult to see from the moving object, a pedestrian near an intersection, or an approach from an intersection road There are other vehicles, lane increase / decrease sections, building entrances facing the road, etc. As a means for determining the risk factor as described above, for example, the determination is made by comparing the current position and orientation of the moving object with map information, or using a camera, sensor or communication device mounted on the moving object. Is possible.

  For example, in JP-A-2006-126756, a current position image around a vehicle captured by an in-vehicle camera, a drive recorder, a smartphone, or the like installed in the vehicle while the vehicle is moving is acquired and registered in advance in the server device. It is disclosed that the degree of danger in the current situation is determined from the similarity between the danger point image and the current point image.

JP, 2006-126756, A (15-15, 21 pages)

  Here, in Patent Document 1, the similarity between the current point image and the dangerous point image is determined, and if the current point image and the dangerous point image are similar, it is determined that the degree of risk is high. However, in addition to elements that are highly likely to be risk factors (for example, pedestrians located at intersections and other vehicles that run on intersection roads), elements that are less likely to be risk factors (such as pedestrians located at intersections and other vehicles traveling on intersection roads) For example, a pedestrian walking on a sidewalk and other vehicles traveling on the opposite lane). In Patent Document 1, when determining the similarity between the current point image and the dangerous point image, even if the elements that are likely to be risk factors are not similar, the elements that are less likely to be risk factors are similar. As a result, there is a problem that the risk is determined to be high even when there is a high possibility that the risk factor does not exist.

  The present invention has been made to solve the above-described conventional problems, and more accurately determines risk factors by inputting captured images obtained by capturing the surrounding environment into a learning model generated by machine learning. An object of the present invention is to provide a driving support device and a computer program that can also change the output mode in a situation in which the driver can recognize in advance that the risk factor determination results are dangerous. To do.

In order to achieve the above object, a driving support apparatus according to the present invention includes a surrounding environment imaging unit that acquires captured images obtained by sequentially capturing the surrounding environment of a moving body over time, and sequentially acquires the captured images by machine learning. By inputting into the generated learning model, the mobile unit recognizes the peripheral situation occurring in the peripheral environment of the mobile object at each imaging timing of the input captured image, and based on the recognized peripheral situation Comparing risk factors determining means for determining risk factors in the surrounding environment, judgment result output means for outputting the judgment results of the risk factor judging means, and surrounding conditions recognized at each imaging timing of the captured image Similarity determination means for determining whether or not the risk factor in the surrounding situation similar to the past has occurred in the surrounding environment of the moving object Has the determination result output means changes an output mode in accordance with the determination result of the similarity determination unit.
The “moving body” is not limited to a vehicle, and may be anything that moves on a road such as a pedestrian or a bicycle.
The “risk factor” is a factor to be noted when the moving body travels. For example, other vehicles and obstacles such as pedestrians that are difficult to see from the moving body, pedestrians near the intersection, etc. There are other vehicles that enter from crossing roads, lane increase / decrease sections, and entrances to buildings facing the road.

  The computer program according to the present invention is a program for supporting traveling of a moving object. Specifically, the computer inputs peripheral environment imaging means for acquiring captured images obtained by sequentially capturing the surrounding environment of the moving body over time, and the acquired captured images are sequentially input to a learning model generated by machine learning. By recognizing the surrounding situation occurring in the surrounding environment of the moving body at each imaging timing of the input captured image, the risk factors in the surrounding environment of the moving body based on the recognized surrounding situation By comparing a peripheral situation recognized at each imaging timing of the captured image, and a risk factor determining unit that determines the risk factor, a determination result output unit that outputs a determination result of the risk factor determination unit, When functioning as similarity determination means for determining whether or not the risk factor in a surrounding situation similar to the past has occurred in the environment On the determination result output means changes an output mode in accordance with the determination result of the similarity determination unit.

  According to the driving support apparatus and the computer program according to the present invention having the above-described configuration, a current moving body is obtained by sequentially inputting captured images obtained by capturing the surrounding environment as time passes into a learning model generated by machine learning. This makes it possible to more accurately determine risk factors according to the situation.

It is the block diagram which showed the navigation apparatus which concerns on this embodiment. It is a flowchart of the risk factor determination processing program according to the present embodiment. It is the figure which showed the outline of the learning model which concerns on this embodiment. It is a figure explaining the calculation method of the whole similarity of surrounding environment. It is a figure explaining the calculation method of the feature similarity of surrounding environment.

  Hereinafter, a driving support device according to the present invention will be described in detail with reference to the drawings based on an embodiment embodied in a navigation device. First, a schematic configuration of the navigation device 1 according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a navigation device 1 according to this embodiment.

  As shown in FIG. 1, the navigation device 1 according to the present embodiment includes a current position detection unit 11 that detects a current position of a vehicle on which the navigation device 1 is mounted, a data recording unit 12 that records various data, Based on the input information, the navigation ECU 13 that performs various arithmetic processes, the operation unit 14 that receives operations from the user, information about a map around the vehicle for the user, information on the guidance route set in the navigation device 1, etc. A liquid crystal display 15 that displays voice guidance for route guidance, warnings for risk factors, and the like, a DVD drive 17 that reads a DVD as a storage medium, a probe center and VICS (registered trademark: Vehicle Information and Communication) System) Communication module 1 that communicates with an information center such as a center. And, the has. In addition, the navigation device 1 is connected to an out-of-vehicle camera 19 installed on a vehicle on which the navigation device 1 is mounted via an in-vehicle network such as CAN.

Below, each component which the navigation apparatus 1 has is demonstrated in order.
The current position detection unit 11 includes a GPS 21, a vehicle speed sensor 22, a steering sensor 23, a gyro sensor 24, and the like, and can detect the current vehicle position, direction, vehicle traveling speed, current time, and the like. . Here, in particular, the vehicle speed sensor 22 is a sensor for detecting a moving distance and a vehicle speed of the vehicle, generates a pulse according to the rotation of the driving wheel of the vehicle, and outputs a pulse signal to the navigation ECU 13. And navigation ECU13 calculates the rotational speed and moving distance of a driving wheel by counting the generated pulse. Note that the navigation device 1 does not have to include all the four types of sensors, and the navigation device 1 may include only one or more types of sensors.

  The data recording unit 12 reads an external storage device and a hard disk (not shown) as a recording medium, a map information DB 31 and a captured image DB 32 recorded on the hard disk, a predetermined program, and the like, and stores predetermined data on the hard disk. And a recording head (not shown) as a driver for writing. The data recording unit 12 may include a memory card, an optical disk such as a CD or a DVD, instead of the hard disk. Further, the map information DB 31 and the captured image DB 32 may be stored in an external server, and the navigation device 1 may acquire them by communication.

  Here, the map information DB 31 is a storage unit that stores map information used for route search and travel guidance in the navigation device 1. For example, link data related to roads (links), node data related to node points, facility data related to facilities, search data used for route search processing, map display data for displaying a map, intersection data related to each intersection, and points are searched. Search data and the like.

  The captured image DB 32 is a storage unit that stores captured images 35 sequentially captured by the vehicle exterior camera 19 over time. The captured images 35 captured by the camera 19 outside the vehicle are cumulatively stored in the captured image DB 32 and are deleted in order from the oldest image.

  On the other hand, the navigation ECU (Electronic Control Unit) 13 is an electronic control unit that controls the entire navigation device 1. The CPU 41 as an arithmetic device and a control device, and a working memory when the CPU 41 performs various arithmetic processes. From the ROM 43 and the ROM 43 in which a RAM 42 storing route data when a route is searched, a control program, a risk factor determination processing program (see FIG. 2) described later, and the like are recorded. An internal storage device such as a flash memory 44 for storing the read program is provided. The navigation ECU 13 has various means as processing algorithms. For example, the surrounding environment imaging unit acquires a captured image obtained by sequentially imaging the surrounding environment of the vehicle with the passage of time. The risk factor determination means recognizes the surrounding situation occurring in the surrounding environment of the vehicle at each imaging timing of the input captured image by sequentially inputting the acquired captured image into a learning model generated by machine learning, Risk factors in the surrounding environment of the vehicle are determined based on the recognized surrounding situation. The determination result output means outputs the determination result of the risk factor determination means. The similarity determination unit determines whether or not there is a risk factor in the surrounding situation similar to the past in the surrounding environment of the vehicle by comparing the surrounding situation recognized at each imaging timing of the captured image.

  The operation unit 14 is operated when inputting a starting point as a travel start point and a destination as a travel end point, and has a plurality of operation switches (not shown) such as various keys and buttons. Then, the navigation ECU 13 performs control to execute various corresponding operations based on switch signals output by pressing the switches. The operation unit 14 may have a touch panel provided on the front surface of the liquid crystal display 15. Moreover, you may have a microphone and a speech recognition apparatus.

  The liquid crystal display 15 includes a map image including a road, traffic information, operation guidance, operation menu, key guidance, guidance route set in the navigation device 1, guidance information along the guidance route, news, weather forecast. , Time, mail, TV program, etc. are displayed. In place of the liquid crystal display 15, HUD or HMD may be used. In the present embodiment, the guidance of the risk factor determination result is also displayed.

  The speaker 16 outputs voice guidance for guiding traveling along the guidance route based on an instruction from the navigation ECU 13 and traffic information guidance. Further, in this embodiment, in particular, guidance on risk factor determination results is also output.

  The DVD drive 17 is a drive that can read data recorded on a recording medium such as a DVD or a CD. Based on the read data, music and video are reproduced, the map information DB 31 is updated, and the like. A card slot for reading / writing a memory card may be provided instead of the DVD drive 17.

  The communication module 18 is a communication device for receiving traffic information transmitted from a traffic information center such as a VICS center or a probe center, and corresponds to a mobile phone or DCM, for example.

  The vehicle exterior camera 19 is constituted by a camera using a solid-state imaging device such as a CCD, for example, and is installed on the rear side of a vehicle rearview mirror, a front bumper, etc., and installed with the optical axis direction downward from the horizontal by a predetermined angle. The And the camera 19 outside a vehicle images the surrounding environment ahead of the advancing direction of a vehicle. Moreover, navigation ECU13 determines the risk factor around a vehicle by inputting into the machine learning the captured image imaged as mentioned later. In addition, you may comprise the vehicle exterior camera 19 so that it may arrange | position also to the side and back of a vehicle. Further, it is desirable to adjust the installation position of the outside camera 19 so as to be substantially the same as the position of the driver's eyes (the line-of-sight start point). Thereby, it becomes possible to determine risk factors more appropriately.

  The risk factor determined by the machine learning in the navigation device 1 according to the present embodiment is a factor that should be noted (guided) when the vehicle travels. For example, there are obstacles such as other vehicles and pedestrians that are difficult to see from the vehicle, pedestrians near the intersection, other vehicles entering from the intersection road, lane increase / decrease sections, building entrances facing the road, etc. . For example, the “entrance / exit of a building facing the road” is a point where a pedestrian may newly appear on the road, and is a place to be noted when the vehicle travels. The “lane increase / decrease section” is a point where another vehicle may change lanes, and is a place to be careful when the vehicle travels.

  Next, a risk factor determination processing program executed by the CPU 41 in the navigation device 1 according to this embodiment having the above-described configuration will be described with reference to FIG. FIG. 2 is a flowchart of the risk factor determination processing program according to the present embodiment. Here, the risk factor determination processing program is repeatedly executed at predetermined time intervals (for example, 1 sec) after the ACC (accessory power supply) of the vehicle is turned on, and around the vehicle based on the captured image captured by the outside camera 19. This program determines a certain risk factor and outputs the determination result. 2 is stored in the RAM 42, the ROM 43, and the like provided in the navigation ECU 13, and is executed by the CPU 41.

  First, in the risk factor determination processing program, in step (hereinafter abbreviated as S) 1, the CPU 41 acquires a captured image most recently captured by the outside camera 19 from the captured image DB 32 and generates a learning model generated by machine learning. To enter. Here, FIG. 3 is a diagram showing an outline of the learning model according to the present embodiment. In particular, in this embodiment, machine learning (Deep Learning) using a multilayered neural network (CNN) is used as machine learning.

  As shown in FIG. 3, when a captured image 51 captured by the outside camera 19 is input to a learning model, first, image processing based on a convolutional neural network (hereinafter referred to as convolution CNN) 52 is performed. The convolution CNN 52 outputs a particularly important feature map 53 for determining risk factors after repeating the “convolution layer” and the “pooling layer” a plurality of times.

  Here, the “convolution layer” is a layer that filters (convolves) an input image. A pattern (feature) in an image can be detected by convolution of the image. In addition, a plurality of filters are convolved. By using a plurality of filters, various features of the input image can be captured. Further, the size of the output image is reduced by applying the filter. The output image is also called a feature map. Moreover, the filter used for this convolution layer does not need to be set by the designer, and can be acquired by learning. As the learning progresses, a filter suitable for extracting a particularly important feature for determining a risk factor is set.

  On the other hand, the “pooling layer” is placed immediately after the convolution layer, and lowers the position sensitivity of the extracted features. Specifically, the difference due to a slight shift of the image is absorbed by roughly re-sampling the output of the convolution. Even in the pooling layer, the size of the output image is smaller than that of the input image.

  Thereafter, the feature map 53 output by the convolution CNN 52 is input to an input layer of a neural network (hereinafter referred to as risk determination CNN) 54 for risk factor determination. The risk determination CNN 54 inputs data obtained by multiplying each neuron, which is output data after processing in the input layer, by a weight (weight coefficient), to the next intermediate layer. Similarly, in the intermediate layer, data obtained by multiplying each neuron by a weight (weighting factor), which is output data after processing in the intermediate layer, is input to the next output layer. Then, the final risk factor is determined using the data input from the intermediate layer in the output layer, and the determination result is output. The risk determination CNN 54 is appropriately changed and set to a more suitable value of the weight (weighting factor) as learning progresses. In the present embodiment, in particular, the intermediate layer is a layer that recognizes the surrounding situation (scene) of the vehicle, and the output layer is a layer that determines risk factors from the surrounding situation (scene) of the vehicle.

  In S <b> 2, the CPU 41 recognizes a surrounding situation (scene) generated in the surrounding environment of the vehicle at the imaging timing of the captured image 51 input in S <b> 1 in the intermediate layer of the risk determination CNN 54 of the learning model described above. For example, as shown in Fig. 3, there are neurons in the middle layer such as "There are people waving hands", "There are blind spots", "There are convenience stores", "There are many people", "There are vehicles ahead", etc. The scene is recognized by specifying the neuron value for each neuron. A neuron value is specified between 0 and 1, and a neuron having a high neuron value is an important element (characteristic element) in the surrounding environment, while a neuron having a low neuron value is an insignificant element ( This indicates that the scene is a non-characteristic element. For example, in the example shown in FIG. 3, since a high neuron value is specified for “there is a person waving” and “there is a blind spot”, the person or the blind spot waving near the vehicle in the surrounding environment of the vehicle On the other hand, low neuron values are specified for “There are convenience stores”, “Many people”, and “There are vehicles ahead”, so there are no convenience stores around the vehicle. It indicates that there is not much traffic and there is no vehicle ahead. The scene recognition result in S2 is stored in the flash memory 44 or the like for a certain period in association with the recognized date and time.

  Subsequently, in S <b> 3, the CPU 41 reads out the scene recognition result performed in S <b> 2 from the flash memory 44 during a past fixed period (for example, the latest 20 seconds). Then, as shown in FIG. 4, the latest scene is recognized by comparing the scene recognized in S2 (hereinafter referred to as the latest scene) and the read past scene (hereinafter referred to as the past scene). An overall similarity indicating the degree of overall similarity between the scene and the past scene is calculated. That is, the overall similarity calculated in S3 is a value obtained by determining the similarity of the entire surrounding environment without considering risk factors occurring in the surrounding environment of the vehicle. For the past scene, the process of S3 is executed for all scenes recognized in the past certain period.

The method for calculating the overall similarity in S3 will be described below.
First, the CPU 41 squares the difference between the neuron values of the latest scene and the past scene acquired for every neuron included in the intermediate layer, and sums up all the neurons. Thereafter, a value obtained by dividing the total value by the number (N) of all neurons included in the intermediate layer is calculated. Further, a value obtained by subtracting the calculated value from 1 and multiplying by 100 is defined as the overall similarity (%). For example, in the example shown in FIG. 4, it calculates with the following formula | equation (1)-(3).
(0.9−0.8) ² + (0.8−0.9) ² + (0.0−0.0) ² + (0.1−0.2) ² +... = X ₁ (1)
X ₁ / N = X ₂ (2)
(1.0−X ₂ ) × 100 = Overall similarity [%] (3)

  Thereafter, in S4, the CPU 41 determines whether or not the overall similarity calculated in S3 is less than or equal to a threshold value. The threshold can be changed as appropriate, but is set to 70%, for example.

  If it is determined that the overall similarity calculated in S3 is less than or equal to the threshold value (S4: YES), that is, it is determined that no similar situation has occurred in the past certain period in the surrounding environment of the vehicle. For the case, the process proceeds to S5.

  On the other hand, when it is determined that there is a past scene in which the overall similarity calculated in S3 is larger than the threshold (S4: NO), that is, a surrounding situation similar in the past certain period occurs in the surrounding environment of the vehicle. In the case where it is determined that, the driver presumes that the current scene has already been recognized as dangerous by a warning in a similar scene in the past. Therefore, the risk factor determination processing program is terminated without outputting the risk factor determination result.

  In S <b> 5, the CPU 41 determines risk factors based on the surrounding situation (scene) of the vehicle input from the intermediate layer in the output layer of the risk determination CNN 54 of the learning model shown in FIG. 3 described above. The risk determination CNN 54 is a case where the weight (weighting factor) between the intermediate layer and the output layer is appropriately changed to a more suitable value as learning progresses, so that the same scene is input from the intermediate layer. Even if there are different learning stages, different determination results may be obtained. For example, in the initial stage, risk factors are determined by setting weights (weighting factors) that place importance on pedestrians, but after that, as the learning proceeds, setting of weights (weighting factors) that place importance on other vehicles is performed. May be changed. In that case, even if it is the same scene, it may be determined that there is no risk factor at an early stage, but it may be determined that there is a risk factor at a later stage.

  Thereafter, in S6, the CPU 41 determines whether or not there is at least one risk factor in the surrounding environment of the vehicle as a result of the determination of the risk factor in S5.

  And when it determines with there being a risk factor in the surrounding environment of a vehicle (S6: YES), it transfers to S7. On the other hand, when it is determined that there is no risk factor in the surrounding environment of the vehicle (S6: NO), the risk factor determination processing program is terminated without outputting the determination result of the risk factor determination.

  In S7, the CPU 41 obtains the neuron value of each neuron in the intermediate layer and the weight (weighting factor) between the intermediate layer and the output layer in the risk factor determination process of the risk determination CNN 54 performed most recently.

  Thereafter, in S8, the CPU 41 reads from the flash memory 44 the scene recognition result performed in S2 in the past fixed period (for example, the latest 20 seconds). The risk determined to exist in the vicinity of the vehicle, particularly in the risk factor determination process of S6, with respect to the scene (latest scene) recognized in S2 and the read past scene (past scene) read this time. By comparing the neurons corresponding to the factors, the feature similarity indicating the degree of similarity between the latest scene and the past scene is calculated. Note that the feature similarity calculated in S8 differs from S3 in that it is similar to a scene related to a risk factor occurring in the surrounding environment of the vehicle (that is, whether a risk factor similar to the past has occurred). The determined value. That is, in S8, even if there is no situation similar to the entire surrounding environment in the past certain period, similar risk factors (for example, there are pedestrians at the intersection, other vehicles entering the intersection from the intersection road) If there is a certain feature), a high feature similarity is calculated. For the past scene, the process of S8 is executed for all the scenes recognized in the past certain period.

The method for calculating the feature similarity in S8 will be described below.
First, the CPU 41 obtains the neuron values of the neurons whose weight (weight coefficient) between the output layer and the output layer is greater than or equal to the threshold for the latest scene and the past scene. Note that the weight (weighting factor) between the intermediate layer and the output layer indicates the connection with the risk factor determination in the output layer, and a neuron with a high weight strongly influences the risk factor determination result, that is, It becomes a neuron corresponding (strongly related) to the risk factor determined to be present around the vehicle in the risk factor determination process of S6. The weight threshold is set to 0.6, for example. Note that the value of the weight for each neuron is appropriately changed as learning progresses, so that the neuron whose weight is equal to or greater than the threshold also changes according to the learning stage. For example, in the example shown in FIG. 5, “there is a person waving”, “there is a blind spot”, “there is a convenience store”, and “there is a lot of traffic”, the weight (weight coefficient) between the output layer is greater than the threshold Become neurons.

Next, the CPU 41 squares the difference between the neuron values acquired for each neuron, and sums up all the target neurons. Thereafter, a value obtained by dividing the total value by the number of neurons from which the neuron value is obtained (4 in the example shown in FIG. 5) is calculated. Further, a value obtained by subtracting the calculated value from 1 and multiplying by 100 is defined as the feature similarity (%). For example, in the example shown in FIG. 5, it calculates with the following formula | equation (4)-(6).
^{(0.9-0.8) 2} + ^{(0.8-0.9) 2} + ^{(0.0-0.0) 2} + ^(0.1-0.2) 2 = 0.03 ...・ (4)
0.03 / 4 = 0.0075 (5)
(1.0−0.0075) × 100 = 99.25 (feature similarity [%]) (6)

  However, the feature similarity calculated in S8 may be corrected according to the overall similarity calculated in S3. That is, even if strong similarity is shown in the part related to the risk factor, if the similarity is not similar as a whole, correction for lowering the feature similarity may be performed. For example, when the overall similarity is lower than 70% (for example, 50%), the feature similarity may be lowered by a value obtained by multiplying the feature similarity by the difference (70% −50% = 20%). .

  Further, even if the similarity is low in the part related to the risk factor, if the similarity is high as a whole, a correction for increasing the feature similarity may be performed. For example, when the overall similarity is higher than 70% (for example, 90%), the feature similarity may be increased by a value obtained by multiplying the feature similarity by the difference (90% −70% = 20%). .

  Next, in S9, the CPU 41 selects an output mode of the risk factor determination result based on the highest feature similarity value among the feature similarities for each past scene calculated in S8. Specifically, when the feature similarity is high, it is determined that a risk factor in the surrounding situation similar to the past has occurred in the surrounding environment of the vehicle. Therefore, the driver estimates that there is a high possibility that the current scene has already been recognized as dangerous by a warning in a similar scene in the past, and if the feature similarity is low, that is, in the surrounding environment of the vehicle, The output mode is simpler than the case where no risk factor is generated in the surrounding situation similar to. For example, when the feature similarity is less than 70%, the display by the liquid crystal display 15 and the sound by the speaker 16 are selected as the risk factor output mode. On the other hand, when the feature similarity is 70% or more, only the display on the liquid crystal display 15 is selected as the risk factor output mode.

  Thereafter, in S10, the CPU 41 outputs a risk factor determination result. Specifically, it is performed according to the output mode selected in S9. For example, when the feature similarity calculated in S8 is less than 70%, a warning mark indicating the presence of a risk factor is displayed on the liquid crystal display 15 and “Please be careful in front of the vehicle”. Voice guidance is performed. On the other hand, when the feature similarity calculated in S8 is 70% or more, only a warning mark that the risk factor exists is displayed on the liquid crystal display 15.

  In addition, if machine learning can determine what kind of risk factor, guidance for specifying the risk factor more specifically (for example, "Please pay attention to the intersection road in the front blind spot of the vehicle" )). Moreover, you may guide also about the position of a risk factor. On the other hand, vehicle control may be performed as the output of risk factors. In this case, for example, deceleration control may be performed when the feature similarity is 70% or more, and stop control that is stronger control may be performed when the feature similarity is less than 70%. It can also be applied to an autonomous driving vehicle. In that case, control such as setting a travel route that avoids risk factors is possible.

  As described in detail above, in the navigation device 1 and the computer program executed by the navigation device 1 according to the present embodiment, captured images obtained by capturing the surrounding environment of the vehicle are sequentially input to a learning model generated by machine learning. By recognizing the surrounding situation occurring in the surrounding environment of the vehicle at each imaging timing of the input captured image, and determining the risk factor in the surrounding environment of the moving body based on the recognized surrounding situation (S5), Since the risk factor in the surrounding environment similar to the past has occurred in the surrounding environment of the vehicle, the output mode is changed and the risk factor determination result is output (S10). An accurate risk factor can be determined. Further, when the risk factor determination result is output, the output mode can be changed in a situation where the user can recognize in advance that the risk is dangerous, and the user is not bothered.

Note that the present invention is not limited to the above-described embodiment, and various improvements and modifications can be made without departing from the scope of the present invention.
For example, in this embodiment, machine learning (Deep Learning) using a convolutional neural network having a multilayer structure is used as the machine learning, but other machine learning can also be used.

  In this embodiment, when the overall similarity is equal to or greater than the threshold value in the surrounding environment of the vehicle in S4 (S4: YES), the risk factor determination result is not output. May be output. However, it is desirable that the output mode be a simple mode as compared with S10.

  In the present embodiment, the risk factor determination processing program (FIG. 2) is executed by the navigation device 1, but may be configured by an on-vehicle device other than the navigation device. Moreover, it is good also as an external server performing some processes instead of an onboard equipment performing all the processes.

  Moreover, although the embodiment which actualized the driving support device according to the present invention has been described above, the driving support device can also have the following configuration, and in that case, the following effects can be obtained.

For example, the first configuration is as follows.
Peripheral environment imaging means (41) that acquires captured images (51) that sequentially capture the surrounding environment of the moving body over time, and the acquired captured images are sequentially input to a learning model generated by machine learning. To recognize a surrounding situation occurring in the surrounding environment of the moving body at each imaging timing of the input captured image and determine a risk factor in the surrounding environment of the moving body based on the recognized surrounding situation The risk factor determination means (41) that performs the determination, the determination result output means (41) that outputs the determination result of the risk factor determination means, and the peripheral situation recognized at each imaging timing of the captured image, Similarity determination means (41) for determining whether or not a risk factor in a surrounding situation similar to the past has occurred in the surrounding environment of the moving object, Judgment result output means changes an output mode in accordance with the determination result of the similarity determination unit.
According to the driving support apparatus having the above-described configuration, a captured image obtained by capturing the surrounding environment as time elapses is sequentially input to a learning model generated by machine learning. Risk factors can be determined. Further, when the risk factor determination result is output, the output mode can be changed in a situation where the user can recognize in advance that the risk is dangerous, and the user is not bothered.

The second configuration is as follows.
The machine learning is machine learning using a neural network having a multilayer structure, and the similarity determination means (41) compares the neuron value in the intermediate layer with a past imaging timing, thereby moving the movement. It is determined whether or not there are risk factors in the surrounding environment similar to the past in the surrounding environment of the body.
According to the driving support apparatus having the above-described configuration, image features can be learned at a deeper level and machine features can be recognized with very high accuracy by machine learning using a multi-layered neural network. Judgment is possible. Further, by comparing the values of neurons in the intermediate layer, it is possible to accurately determine whether or not a surrounding situation similar to the past has occurred in the surrounding environment of the moving body.

The third configuration is as follows.
The similarity determination means (41) compares the neuron value corresponding to the risk factor determined by the risk factor determination means (41) among the neurons in the intermediate layer with the past imaging timing.
According to the driving support apparatus having the above-described configuration, it is determined whether or not a surrounding situation similar to the past has occurred by comparing with the determined risk factor and an element that is strongly associated with the past. It is possible to accurately determine whether or not a surrounding situation similar to is occurring.

The fourth configuration is as follows.
The similarity determination unit (41) sets a neuron having a weight coefficient between the output layer and a neuron corresponding to the risk factor determined by the risk factor determination unit among the neurons in the intermediate layer.
According to the driving support device having the above configuration, since it is determined whether or not a peripheral situation similar to the past has occurred by comparison with a neuron in the intermediate layer that is particularly strongly associated with the determined risk factor, It is possible to accurately determine whether or not a surrounding situation similar to the past has occurred in the surrounding environment of the moving object.

The fifth configuration is as follows.
The similarity determination means (41) calculates the similarity with the surrounding situation recognized at the past imaging timing by comparing the value of the neuron in the intermediate layer with the past imaging timing. If the similarity is equal to or greater than the threshold value, it is determined that a risk factor in the surrounding situation similar to the past has occurred in the surrounding environment of the moving body.
According to the driving support apparatus having the above configuration, by using the similarity calculated based on the comparison of the neuron values, it is accurately determined whether or not a surrounding situation similar to the past has occurred in the surrounding environment of the moving body. It becomes possible to do.

The sixth configuration is as follows.
The determination result output means (41) does not output a determination result when it is determined that a risk factor in a surrounding situation similar to the past has occurred in the surrounding environment of the moving body, or the surrounding environment of the moving body Is output in a simplified manner as compared with the case where it is determined that the risk factors in the surrounding situation similar to the past have not occurred.
According to the driving support device having the above configuration, by changing the output mode of the risk factor to a simpler output mode in a situation where the user is likely to recognize the current surrounding situation as dangerous in advance. , The user does not feel bothered.

1 Navigation device 13 Navigation ECU
19 Outside camera 31 Map information DB
32 Captured image DB
33 2D map information 34 3D map information 41 CPU
42 RAM
43 ROM
51 Captured image 52 Convolution CNN
54 Danger judgment CNN

Claims (7)

A surrounding environment imaging means for acquiring a captured image obtained by sequentially capturing the surrounding environment of the moving body over time;
The acquired captured images are sequentially input to a learning model generated by machine learning, thereby recognizing and recognizing the surrounding situation occurring in the surrounding environment of the moving body at each imaging timing of the input captured image. A risk factor judging means for judging a risk factor in the surrounding environment of the mobile body based on the surrounding situation;
Determination result output means for outputting the determination result of the risk factor determination means;
Similarity determination means for determining whether or not the risk factor in the surrounding situation similar to the past has occurred in the surrounding environment of the mobile body by comparing the surrounding situation recognized at each imaging timing of the captured image; Have
The determination result output unit is a driving support device that changes an output mode according to a determination result of the similarity determination unit. The machine learning is machine learning using a neural network having a multilayer structure,
The similarity determination means determines whether or not the risk factor in the surrounding situation similar to the past has occurred in the surrounding environment of the moving body by comparing the neuron value in the intermediate layer with the past imaging timing. The driving support device according to claim 1 for determination.   The driving support according to claim 2, wherein the similarity determination unit compares a value of a neuron corresponding to the risk factor determined by the risk factor determination unit among neurons in the intermediate layer with a past imaging timing. apparatus.   4. The similarity determination unit according to claim 3, wherein among the neurons in the intermediate layer, a neuron whose weight coefficient between the output layer and the output layer is greater than or equal to a threshold value is a neuron corresponding to the risk factor determined by the risk factor determination unit. Driving support device. The similarity determination means includes
By comparing the neuron value in the intermediate layer with the past imaging timing, the similarity with the surrounding situation recognized at the past imaging timing is calculated,
The travel according to any one of claims 2 to 4, wherein when the calculated similarity is equal to or greater than a threshold value, it is determined that the risk factor in the surrounding situation similar to the past has occurred in the surrounding environment of the moving body. Support device.   The determination result output means does not output the determination result when it is determined that the risk factor in the surrounding situation similar to the past has occurred in the surrounding environment of the moving body, or the past in the surrounding environment of the moving body The driving support device according to any one of claims 1 to 5, wherein the output is simplified and output as compared with a case where it is determined that the risk factor is not generated in a surrounding situation similar to the above. Computer
A surrounding environment imaging means for acquiring a captured image obtained by sequentially capturing the surrounding environment of the moving body over time;
The acquired captured images are sequentially input to a learning model generated by machine learning, thereby recognizing and recognizing the surrounding situation occurring in the surrounding environment of the moving body at each imaging timing of the input captured image. A risk factor judging means for judging a risk factor in the surrounding environment of the mobile body based on the surrounding situation;
Determination result output means for outputting the determination result of the risk factor determination means;
Similarity determination means for determining whether or not the risk factor in the surrounding situation similar to the past has occurred in the surrounding environment of the mobile body by comparing the surrounding situation recognized at each imaging timing of the captured image; A computer program to make it function,
The said determination result output means is a computer program which changes an output mode according to the determination result of the said similarity determination means.