JP5516561B2 - Vehicle driving support device - Google Patents

Description

  The present invention relates to a vehicle driving support device that provides information about an approaching object to a vehicle driver when the vehicle is started from a stopped state, for example.

  Conventionally, various systems for detecting an object approaching a vehicle or detecting an obstacle around the vehicle are known. For example, Patent Document 1 describes a system that detects an approaching object. In this system, by using optical flow processing based on multiple images that move back and forth in time series, image parts that have moved in a specific direction in time series are extracted from these images, and each detected image part is specified. By performing processing based on dynamic programming along a direction different from the direction of the image, the enlargement ratio of the image portion in the subsequent image is obtained in a time series with respect to that image portion in the previous image in time series . Then, it is determined whether or not the moving object corresponding to the image portion is approaching according to the enlargement ratio.

  Patent Document 2 describes an apparatus that detects an approaching vehicle on a road when traveling from a parking space to a road. In the apparatus of Patent Document 2, a vanishing point of a road is obtained from a camera-captured image, and a plurality of inspection lines including a horizontal line passing through the ordinate position of the vanishing point and parallel lines arranged in the vicinity thereof are set. . Then, in each inspection line, a spatiotemporal image in which line images of a predetermined number of frames are stacked is created, and a moving object is detected on the spatiotemporal image.

  Furthermore, Patent Document 3 describes a periphery monitoring device that detects an obstacle (a three-dimensional object) around a vehicle and detects whether the obstacle is stationary or moving. In this periphery monitoring device, a captured image of a camera installed in a vehicle is converted into a bird's-eye view image, and alignment of bird's-eye view images obtained at different times is performed based on image data. At that time, the obstacle with the height is displaced and a difference area is formed. Therefore, the obstacle is detected based on the difference area. Furthermore, it classifies whether the obstacle is a stationary object or a moving object based on the movement amount of the point where the obstacle is located on the ground.

JP 2010-152873 A JP 2005-217482 A JP 2008-219063 A

  For example, when a parked vehicle is started forward or backward, a contact accident with an object such as a pedestrian or a vehicle approaching the vehicle may occur. This type of accident occurs mainly because the driver trying to start the vehicle is less likely to notice an approaching object because there is a blind spot around the vehicle. It is done. Therefore, as in Patent Documents 1 to 3 described above, various systems for detecting an object approaching the vehicle and notifying the driver based on an image taken by a camera mounted on the vehicle have been proposed. .

  However, with current image sensing technology, it is not possible to zero out detection errors such as false detection and non-detection due to various disturbances (noise, occlusion, motion blur, specular highlights, changes in ambient light, etc.). Is possible. Therefore, in the binarization type device that switches between notification / non-notification according to the presence / absence of an approaching object as described in Patent Documents 1 to 3, a driver is presented by erroneous information presented due to erroneous detection or non-detection. May be confusing.

  The present invention has been made in view of the above-described points, and is a vehicle capable of reducing the undetected approaching object and suppressing the driver from being confused by providing information based on erroneous detection. An object of the present invention is to provide a driving support apparatus.

In order to achieve the above object, a vehicle driving support apparatus according to claim 1 is provided.
Photographing means mounted on the subject vehicle so as to photograph the periphery of the subject vehicle;
In the first image and the second image taken at different times by the photographing means, the same feature point is extracted, the first coordinate position of the feature point in the first image, and the second coordinate of the feature point in the second image First calculating means for calculating a position;
Second calculating means for calculating a movement vector of the photographing means by movement of the host vehicle in the translational direction between the photographing time of the first image and the photographing time of the second image;
When the feature point indicates an approaching object based on the first coordinate position and the second coordinate position calculated by the first calculation unit and the movement vector calculated by the second calculation unit , Using the probability distribution of the values that can be taken by the two coordinate positions, the probability distribution that can be taken as the moving speed, and the probability distribution of the positions that should be present in the depth direction taken by the photographing means , A likelihood calculating means for calculating a likelihood representing a probability indicating an approaching object approaching the vehicle;
Expected value for calculating the expected value of likelihood that represents the likelihood that each object is an approaching object based on the likelihood of multiple grouped feature points while grouping the feature points for which likelihood is calculated A calculation means;
When notifying an occupant of the own vehicle that there is an approaching object, and informing the occupant of the own vehicle of the presence of the approaching object according to the expected value of the likelihood calculated by the expected value calculating means. And a notification means for changing a notification form.

  As described above, in the vehicle driving support device according to claim 1, the likelihood calculating unit calculates the likelihood indicating the probability that the feature point in the image indicates the approaching object, and further calculates the expected value. The likelihood-calculated feature points are grouped for each object, and the expected likelihood value representing the likelihood that each object is an approaching object is calculated from the likelihood of the grouped feature points. The That is, instead of binaryly determining whether or not each object corresponding to a plurality of grouped feature points is an approaching object, the probability of being an approaching object is continuously determined by the expected value of likelihood. As a typical value. Therefore, if there is an object that is unlikely to be an approaching object, it is not detected as an approaching object, but can be detected as an approaching object having an expected value with a low likelihood. For this reason, it can reduce that it does not detect while an approaching object exists.

  On the other hand, the lower the expected likelihood value, the higher the possibility that the object is not necessarily an approaching object. For this reason, the notification means changes the notification form when notifying the presence of the approaching object according to the expected value of likelihood. As a result, the driver can recognize the certainty when the approaching object is notified, so that even if it is not actually the approaching object, it can suppress the occurrence of confusion. it can.

  As described in claim 2, the expected value calculation means may perform grouping on feature points having a likelihood equal to or greater than a predetermined threshold. Thereby, since the expected value calculation means can narrow down the feature points to be processed, the processing load can be reduced and the processing can be executed quickly. In this case, in order to suppress the non-detection of the approaching object as much as possible, the likelihood threshold is set so that the non-detection rate becomes extremely small.

  According to a third aspect of the present invention, the expected value calculation means may calculate the expected value of likelihood by averaging the likelihoods of a plurality of grouped feature points. By such an averaging process, the expected value of likelihood can be calculated by comprehensively considering the likelihood of a plurality of grouped feature points.

  In the case of calculating the likelihood expectation value by averaging the likelihoods of a plurality of feature points, as described in claim 4, the expectation value calculation means has a higher weight as the reliability belonging to the same group is higher. A weighted average process for assigning In grouping, feature points are close to each other, motion vectors of feature points share a common vanishing point, and the distance between the vehicle and the three-dimensional point indicated by each feature point is zero. It is performed from various viewpoints such as a similar collision time. However, as will be described in detail later, the feature point motion vector referred to here is obtained by removing the motion vector component generated by the rotational motion of the vehicle. For this reason, there may be a difference in the reliability belonging to the same group at each feature point. In this case, it is possible to calculate the expected value of the likelihood of each object with higher accuracy by assigning a greater weight to the likelihood of the feature point having higher reliability belonging to the group and performing the averaging process.

  As described in claim 5, when the number of grouped feature points is less than a predetermined number, the expectation value calculating means regards that the set of grouped feature points corresponds to the object. Alternatively, the expected value of likelihood may not be calculated. This can prevent erroneous detection as an approaching object when a feature point is accidentally extracted due to noise or the like.

  As described in claim 6, the likelihood calculating means calculates at least a pedestrian, a bicycle, and a car as attributes of an approaching object, calculates the likelihood of a feature point for each attribute of the approaching object, and calculates an expected value. The means may calculate an expected value of likelihood for each attribute of the approaching object, and determine an attribute indicating the highest expected value of likelihood as the attribute of the approaching object. In this way, not only the presence of an approaching object can be detected, but also attribute information of the approaching object can be obtained.

  Alternatively, as described in claim 7, the likelihood calculating means calculates at least the likelihood of the feature point for each attribute of the approaching object by setting at least a pedestrian, a bicycle, and a car as the attributes of the approaching object. The value calculating means calculates the expected value of likelihood for each attribute of the approaching object, and also calculates the probability that the approaching object is a pedestrian, the probability of being a bicycle, and the probability of being a vehicle for each prior probability distribution. The posterior probability is calculated by multiplying the expected value of the likelihood and the probability for each attribute of the approaching object, and the attribute having the highest posterior probability is determined as the attribute of the approaching object. May be. By using prior probabilities in this way, it is possible to set different prior probability distributions, for example, depending on the location where the vehicle is stopped, and it is possible to determine the attributes of approaching objects with higher accuracy. . For example, in a vehicle-dedicated parking lot, it can be said that the approaching object has a high probability of being a pedestrian and a vehicle, and the probability of being a bicycle is low. On the other hand, when the vehicle is stopped in a parking space provided across the sidewalk from the road, the approaching object is likely to be a pedestrian or a bicycle, and the probability of being a vehicle is low. Therefore, by considering not only the expected likelihood value but also such a prior probability distribution, it is possible to further improve the accuracy of determining the attribute of the approaching object.

  According to the eighth aspect of the present invention, when the host vehicle moves in the rotation direction between the shooting time of the first image and the shooting time of the second image, the first calculation unit performs shooting on the second image. It is desirable to calculate the second coordinate position from which the influence of the rotational movement of the means is removed. When the host vehicle moves in the rotation direction between the shooting time of the first image and the shooting time of the second image, the shooting means moves in the rotation direction together with the host vehicle at the coordinate position of the feature point in the second image. This is because the influence of the rotational motion component of is included.

  According to a ninth aspect of the present invention, the notifying unit has a display that displays an image captured by the capturing unit, and superimposes and displays a frame surrounding the approaching object on the image displayed on the display. It is preferable to enlarge the frame with the passage of time at an enlargement rate corresponding to the expected value of likelihood. Accordingly, it is possible to notify the driver of the host vehicle in an easy-to-understand manner visually that there is an approaching object approaching the host vehicle. In particular, by enlarging the frame with an enlargement rate corresponding to the expected likelihood value, the larger the expected likelihood value, the larger the frame is enlarged over time. The difference is obvious, and the driver can be alerted to pay more attention to an approaching object with a higher likelihood expectation.

As described in claim 10, the notifying means determines the value of the optical information τ so as to decrease as the expected value of likelihood increases, and expands the frame in accordance with an enlargement ratio determined by (1 + τ ⁻¹ ). You may do it. Here, the optical information τ is defined as a value that correlates with the time for the approaching object to reach the retina (including the plane). Then, in order to display the frame corresponding to the approaching object on the screen so as to approach according to the optical information τ, it is only necessary to enlarge the frame with the passage of time using (1 + τ ⁻¹ ) as an enlargement ratio. As a result, the expected value of likelihood is large, and therefore a frame corresponding to an approaching object for which a small value is set as the optical information τ can be displayed so as to be greatly enlarged as time passes.

  As described in claim 11, the notifying means may change the thickness of the line of the frame according to the enlargement ratio. That is, when the frame is enlarged and displayed with the passage of time, the thickness of the line of the frame may be increased as the size of the enlargement increases. As a result, the larger the frame is enlarged, the more clearly the frame can be displayed.

  As described in claim 12, the notification means may display the frame in a translucent manner so that the background portion can be visually recognized. As a result, the driver can visually recognize the entire image around the vehicle while displaying the frame superimposed.

  As described in claim 13, the notification means may display the shadow of the frame on the back side of the frame. Thereby, when enlarging a frame, a three-dimensional effect can be given and the feeling that the frame is approaching a driver can be given. As in the case of the frame, the notification means may display the shadow of the frame in a translucent manner so that the background portion can be visually recognized.

  As described in the fifteenth aspect, the display device can perform a three-dimensional display, and the notification unit may enlarge the frame while displaying the frame three-dimensionally. By performing such a three-dimensional display, a feeling that the frame is approaching the driver can be given more clearly.

  According to a sixteenth aspect of the present invention, it is desirable that the notification means repeatedly executes the enlargement of the frame so as to return to the state in which the size of the frame is not enlarged when the enlargement of the frame is performed for a predetermined time. This is because the effect of alerting can be enhanced by repeatedly executing the enlargement of the frame. In addition, there is a limit in enlarging the size of the frame due to restrictions on the display range of the display device.

  According to a seventeenth aspect of the present invention, the notifying means is a case where frames corresponding to a plurality of approaching objects are superimposed and displayed on the image at the same time, and the frames corresponding to the plurality of approaching objects overlap. It is preferable to display a frame corresponding to an approaching object with a high degree of expected value in front. Thereby, when there are a plurality of approaching objects approaching the vehicle, it is possible to perform display according to the priority that the driver should pay attention to.

  As described in claim 18, when the attribute of the approaching object is determined by the expected value calculation means, the notifying unit may change the form of the frame according to the determined attribute of the approaching object. good. For example, the shape of the frame may be a shape of a pedestrian, bicycle, or vehicle, or the color of the frame may be a color corresponding to the attribute of the approaching object. Accordingly, it is possible to notify the driver of the attribute of the approaching object using the frame display form.

  According to a nineteenth aspect of the present invention, the notifying means has a display for displaying an image taken by the photographing means, and a partial image showing an approaching object in the image is enlarged according to an expected value of likelihood. Thus, it may be displayed so as to expand with the passage of time. That is, the partial image showing the approaching object may be enlarged together with the frame or instead of displaying the frame.

  Claims 20 to 23 are the same as claims 10 and 15 to 17 except that the object to be enlarged is a partial image showing an approaching object, and thus the description thereof is omitted.

1 is a configuration diagram illustrating an overall configuration of a vehicle driving support apparatus according to an embodiment. It is the flowchart which showed the likelihood calculation process. It is explanatory drawing for demonstrating likelihood calculation processing. It is explanatory drawing for demonstrating the subject in a prior art. It is explanatory drawing for demonstrating the threshold value contrasted with the likelihood calculated for every feature point. It is explanatory drawing for demonstrating a grouping process. It is a figure which shows an example of the alerting | reporting form of an approaching object. It is a figure for demonstrating optical information (tau).

  Hereinafter, a vehicle driving support apparatus according to an embodiment of the present invention will be described in detail with reference to the drawings.

  As shown in FIG. 1, a vehicle driving support apparatus 100 according to this embodiment includes a camera 10 as a photographing unit mounted on the own vehicle, a vehicle for detecting information related to the moving direction and the moving amount of the own vehicle. An information detection unit 20, an image processing ECU 30 that processes an image captured by the camera 10, and a display 40 that displays an image processed by the image processing ECU 30 are provided.

  The camera 10 is constituted by, for example, a CCD camera, and is installed in the vicinity of the ceiling in the vehicle interior of the vehicle or at the front end (front bumper) and / or the rear end (rear bumper) of the vehicle. Moreover, you may install the camera 10 so that the side of a vehicle can be image | photographed. During installation on the vehicle, the installation angle of the camera 10 is adjusted so that the optical axis (Z axis) of the camera 10 is parallel to the ground and the horizontal axis of the image taken by the camera 10 is also parallel to the ground. Is done. As a result, the vertical axis of the image coincides with the direction perpendicular to the ground. The camera 10 periodically captures images around the vehicle at least while attempting to start from a state where the vehicle is stopped, and outputs the captured images to the image processing ECU 30.

  The vehicle information detection unit 20 includes, for example, a speed sensor that detects the traveling speed of the vehicle, a yaw rate sensor that detects the change speed of the rotation angle of the host vehicle in the turning direction, and a steering angle sensor that detects the steering angle of the steering. Then, information relating to the moving direction and moving amount of the host vehicle is detected and output to the image processing ECU 30. In the image processing ECU 30, based on the information detected by the vehicle information detection unit 20, the amount of movement in the rotational direction of the camera 10 due to the movement of the host vehicle in the turning direction and the translation direction of the host vehicle between the shooting times of the two images. The movement vector of the camera 10 by the movement of is calculated. The amount of movement of the camera 10 in the rotation direction and the movement vector of the camera 10 are used for image processing, which will be described in detail later.

  The image processing ECU 30 stores an A / D converter for converting an image output from the camera 10 into a digital image, and a converted digital image, in addition to a microcomputer including a CPU, ROM, RAM, etc. (not shown). An image memory is provided. Note that the image memory has a storage capacity capable of storing a plurality of images.

This image processing ECU 30 exhibits various functions as shown in FIG. 1 by executing various processes according to a program stored in advance. That is, FIG. 1 shows functions provided in the image processing ECU 30 by function blocks. In the following, each function of the image processing ECU 30 will be described. First, the motion vector calculation unit 31 extracts feature points from two digital images taken at different times and stored in an image memory. For this feature point extraction, for example, a technique described in J. Shi and C. Tomasi, “Good Features to Track”, IEEE CVPR, pp. 593-600, 1994 can be used. Then, by associating the same feature points in the two digital images, a motion vector (optical flow) of the feature points is calculated. That is, it is assumed that a certain feature point is observed at the two-dimensional coordinate i ₀ of the previously captured digital image, and the same feature point is moved to the two-dimensional coordinate i _{1 ′} of the later-captured digital image. In this case, i _{1 ′} -i ₀ is the motion vector of the feature point. For example, the LK method (BDLucas and T. kanade, “An iterative Image Registration Technique with an Application to Stereo Vision”, IJCAI, pp674-679, 1981) can be used to calculate the motion vector.

When the digital image has lens distortion, the motion vector calculation unit 31 needs to perform distortion correction. As the distortion correction, there are a case where correction is performed on a digital image and a case where correction is performed on the coordinates (i ₀ and i _{1 ′} ) of feature points without performing correction on the digital image. If parameters relating to lens distortion are given, the correspondence between the coordinates of the image with distortion and the coordinates of the image without distortion can be determined in advance. Using this correspondence, it is possible to convert the digital image itself into an image without distortion. Furthermore, it is also possible to extract feature points from a distorted digital image and convert the feature point coordinates to coordinates in an image without distortion.

Next, the rotation component removal unit 32 will be described. When the host vehicle performs a rotational motion in the turning direction, the motion vector i _{1 ′} −i ₀ = (v _x , v _y ) ^T calculated by the motion vector calculation unit 31 is generated by the rotational motion of the camera 10. Ingredients will be included. If such a rotational motion component of the camera 10 is included in the motion vector (v _x , v _y ) ^T , it becomes a factor that the determination accuracy regarding the approaching object described later is lowered. Therefore, the rotational component removal unit 32 removes the rotational motion component of the camera 10 from the motion vector (v _x , v _y ) ^T.

Here, with respect to an image coordinate (x, y) ^T, the coordinate change amount when given any rotation about the focus of the camera _{10 (r x, r y)} T is the formula 1 below It is known that it can be sought.

However, Ω = (Ω _x , Ω _y , Ω _z ) ^T is the rotational movement amount of the camera 10 between the shooting times of two digital images, and f is the focal length of the camera 10.

The rotation component removal unit 32 calculates the amount of rotational movement of the camera 10 between the shooting times of two digital images based on the information obtained from the vehicle information detection unit 20. For example, if the rotational movement amount about the Y axis is a value Ω _y obtained from the yaw rate sensor and the rotations about the X axis and Z axis are ignored, the rotational movement amount of the camera 10 is Ω = (0, Ω _y , 0). ^T can be set. Then, the coordinate change amount (r _x , r _y ) ^T due to the rotational movement of the camera 10 is calculated using Equation 1 for the coordinate i _{1 ′} of the feature point in the digital image taken later. Further, as shown in the following Equation 2, the rotation component removal unit 32 uses the coordinate change amount (r _x , r _y ) ^T from the motion vector (v _x , v _y ) ^T output from the motion vector calculation unit 31. To obtain a motion vector (u _x , u _y ) ^{T from} which the influence of the rotational motion component of the camera 10 is removed.

Further, the rotation component removing unit 32 performs the subsequent digital processing based on the motion vector (u _x , u _y ) ^T calculated in this way and the coordinate i ₀ of the feature point in the previously captured digital image. The coordinates i ₁ of the feature point in the image excluding the influence of the rotational motion component of the camera 10 are calculated. Note that the coordinate i ₁ of the feature point excluding the influence of the rotational motion component of the camera 10 is the coordinate i _{1 ′} of the feature point before the influence is eliminated and the coordinate change amount (r _x , r _y ). It can also be calculated from ^T.

  Next, the likelihood calculating unit 33 will be described. FIG. 2 is a flowchart showing a likelihood calculation process executed in the likelihood calculation unit 33.

  In step S100, one feature point is selected from all the feature points extracted by the motion vector calculation unit 31. In step S110, a likelihood representing the probability that the point indicated by the selected feature point in the actual space indicates an approaching object approaching the host vehicle is calculated. This likelihood calculation will be described in detail below.

First, as shown in FIG. 3, the focal point of the camera 10 is the origin, the optical axis of the camera 10 is the Z axis, the axes orthogonal to the Z axis and parallel to the ground are the X axis, the Z axis, and the X axis. The axis that is orthogonal and that faces the ground is taken as the Y axis. At this time, the ZX plane is parallel to the ground, and the Y axis is perpendicular to the ground. The Y axis is positive in the direction toward the ground. Further, it is assumed that the lens distortion has been corrected and a so-called pinhole camera model can be applied. Further, it is assumed that the ground is a plane and the camera 10 moves in parallel with the ground. Therefore, the horizon (a set of points on the ground at infinity) is projected onto a straight line of y = 0 on the two-dimensional coordinates in the digital image. The perspective projection is given by Equation 3 below.

The likelihood calculating unit 33 first obtains the movement vector V _O of the camera 10 based on the translational movement of the host vehicle between the shooting times of the two digital images based on the information obtained from the vehicle information detecting unit 20. Since the camera 10 moves in parallel with the ground, this movement vector V _O is composed of an X component and a Z component in the ZX plane (V _O = (V _OX , V _OZ )). Therefore, the movement vector V _O can be calculated mainly from the moving direction of the host vehicle calculated from the detection result by the steering angle sensor or the yaw rate sensor and the movement distance calculated from the detection result by the speed sensor. Incidentally, the rotation component removing unit 32, a motion rotational movement component of the camera has been removed vectors _{(u x, u} ^{y) T,} more motion vector _{(u x, u} ^y) using ^T feature coordinates _{i 1} And the direction (optical axis) of the camera 10 at the time of capturing each digital image is considered to be parallel.

Since the coordinates i ₀ and i ₁ of the feature point in the image indicate the difference between the points indicated in the real space, the height of the point from the ground does not change, so that the two-dimensional structure is composed of the X component and the Z component. Vector V _T = (V _TX , V _TZ ). The vector V _T is for indicating the moving speed of the point indicated by the feature point, is an unknown variable. Further, the three-dimensional coordinates of the point indicated by the feature point coordinate i ₀ are (X ₀ , Y, Z ₀ ), and the three-dimensional coordinates of the point indicated by the feature point coordinate i ₁ are (X ₁ , Y, Z ₁ ). And The point in the real space corresponding to the feature point is on the surface of a moving object such as a pedestrian or a vehicle, or a stationary object, and the amount of change in the Y coordinate can be ignored between two digital image capturing times It is said.

In the following description, only feature points pointing to points on the ground side of the horizon, that is, Y> 0 are considered. If Y> 0, as shown in FIG. 3, the Z-axis direction of the coordinates of a point corresponding to the feature point is because it can be handled as existing in the range of Z <Z _G. Note that the coordinate components X ₀ , Y, Z ₀ , X ₁ , and Z ₁ of the point indicated by the feature points i ₀ and i ₁ are unknown, but if one coordinate component is determined due to the perspective projection, the remainder is left. Four coordinate components are also determined.

Likelihood calculation unit 33, based on the movement vector V _O of the coordinate i _1, the camera 10 of the feature point in the digital image coordinate i ₀ of the feature points, was taken after the digital images taken previously, the feature point The likelihood representing the probability that the point indicated by indicates the approaching object is calculated for each feature point by the following Equation 4. Incidentally, represents in Equation 4, the state is approaching object and H _1. Further, in Equation 4, the likelihood to be obtained indicates the unknown variable, the coordinate (depth) Z _{0 in} the Z-axis direction of the point indicated by the coordinate i ₀ of the feature point, and the moving speed of the point indicated by the feature point. It is shown in marginalized form by the vector V _T.

The first probability distribution in the integral of the right-hand side of Equation 4, the depth Z ₀ point indicated by the feature point i _0, when the vector V _T which indicates the moving speed of the point is given, the point is approaching In the case of an object, the probability distribution of values that can be taken by the coordinate i _{1 of the} feature point is shown. This is expressed by a Dirac delta function as shown in Equation 5 below from the perspective projection perspective.

As shown in FIG. 3, the second probability distribution in the integral on the right side of Equation 4 shows the probability distribution of the moving speed that can be taken when the point indicated by the feature point i ₀ is an approaching object. . This probability distribution is as shown in Equation 6, modeled as a Gaussian distribution, previously determined and average mu _T and covariance matrix sigma _T of the Gaussian distribution in advance.

Note that Gaussian distribution in equation 6, i.e. the average mu _T and covariance matrix sigma _T defining the Gaussian distribution is preferably determined for each attribute of the approaching object. This is because the moving speed varies depending on the attribute of the approaching object. The attributes of the approaching object include at least a pedestrian, a bicycle, and a vehicle (such as an automobile and a motorcycle). When the Gaussian distribution in Equation 6 is determined for each attribute of the approaching object, the likelihood of the feature point is calculated for each attribute of the approaching object by Equation 9 described later. In this case, the attribute indicating the highest likelihood among the calculated likelihoods can be determined as the attribute of the approaching object.

The probability distribution of the third in the integration of the right-hand side of Equation 4, as shown in FIG. 3, the point indicated by the feature point i ₀ shows the probability distribution of the depth that can be taken in the case of an approaching object. This probability distribution can be expressed by Equation 7 below.

In Equation 7 above, the variable Z _G is the Z coordinate of the intersection of the FIG. 3, connecting points indicated by the coordinates i ₀ of the feature points between the line segment and the earth surface. With a simple calculation, Z _G = f / i ₀ (2) * H. Here, i ₀ (k) is the k-th component of i ₀ , and H is the distance from the camera 10 to the ground. Z _i is a constant set in advance as a lower limit distance where an approaching object exists. For example, Zi = 0 is set. _{Then, p (Z 0 | H 1} ) is a point of interest is in the direction in which the Z-coordinate is positive, and takes a probability of only non-zero when present above the ground. In addition, the average μ _Z and variance σ _Z ² of the Gaussian distribution in Expression 7 are constants that are determined in advance. For example, when considering the case where the vehicle is starting from a parking lot, for example, mu _Z is 2m, sigma _Z ² may be set to 1.5 m. Further, in Equation 7, erf represents a Gaussian error function.

By combining the above formulas 4 to 7, the following formula 8 is obtained.

By substituting the coordinates i ₀ and i ₁ of the feature points and the movement vector V _O of the camera 10 into the above equation 8, for each feature point, the point indicated by each feature point in the real space indicates an approaching object. It is possible to calculate a likelihood representing the certainty that is a thing.

  When the likelihood of the selected feature point is calculated in this way, it is determined in step S120 of the flowchart of FIG. 2 whether or not the calculation of the likelihood has been completed for all the extracted feature points. In this determination process, if it is determined that the calculation of the likelihood of all feature points has been completed, the process proceeds to step S130, and if it is determined that the calculation of the likelihood of all feature points has not yet been completed, the step The processing from S100 is repeated.

  In step S130, it is determined whether or not there is a likelihood equal to or greater than a threshold among the calculated likelihoods. Here, in the prior art, as shown in FIG. 4, when a certain index exceeds a threshold, it is determined that a detection target such as an approaching object or a moving object exists, and if it falls below the threshold, there is no detection target. judge. However, when the presence / absence of the detection target is determined in a binary manner based on the magnitude relationship between the index and the threshold value, in general, many detection errors such as non-detection and erroneous detection occur in the vicinity of the threshold value.

  On the other hand, in this embodiment, as described above, the likelihood representing the probability that the point indicated by each feature point indicates the approaching object is calculated. Therefore, unlike the prior art, it is not necessary to determine the magnitude relationship with the threshold in the sense of determining the presence or absence of the detection target. In this embodiment, the purpose of comparing the likelihood with the threshold is to eliminate the likelihood that does not clearly indicate the approaching object that is the detection target, thereby reducing the subsequent processing load and executing the processing quickly. This is to make it possible. Therefore, in the present embodiment, as conceptually shown in FIG. 5, the threshold is set so that the undetected rate of the approaching object is extremely small. This threshold value is set based on, for example, an experimental result of actual measurement.

  However, if this is done, the ratio of erroneously detecting an approaching object increases, but the likelihood of feature points when such erroneous detection occurs is usually calculated low. For this reason, as will be described in detail later, in this embodiment, by contriving a notification form of an approaching object, it is possible to prevent the driver from being confused even when notification regarding an erroneously detected approaching object is performed. Like to do.

  In step S130, when it is determined that there is a likelihood greater than or equal to the threshold value, the process proceeds to step S140, and a grouping process for grouping the approaching objects for the likelihood equal to or greater than the threshold value is performed. On the other hand, if it is determined that there is no likelihood greater than or equal to the threshold value, the process proceeds to step S160, and the subsequent τ calculation unit 34 is output that there is no approaching object. In this case, an enlarged display of a frame or a partial image for notifying the presence of an approaching object, which will be described later, is not performed, and the display device 40 simply displays a surrounding image of the vehicle.

  Here, the grouping process in step S140 will be described in detail. In this grouping process, feature points that have a likelihood greater than or equal to a threshold are extracted, and feature points that can be regarded as indicating the same approaching object under various conditions are extracted and grouped. This is because the feature point is merely a point, and as it is, it is difficult to understand information on the approaching object (the number, the magnitude of the likelihood when there are a plurality of approaching objects, etc.). Also, feature points themselves may be erroneously extracted due to various disturbances, and feature points due to such disturbances can be eliminated by collecting the feature points for each approaching object.

  The first condition for performing grouping is that feature points exist in the vicinity of each other. In this neighborhood determination, as shown in FIG. 6A, for each feature point, a square area or the like centering around each feature point is given, and the feature points where these regions overlap are in the vicinity of each other. Group by giving the same label (first label). On the other hand, isolated points whose regions do not overlap with other feature point regions are considered to be due to disturbance and are excluded from grouping targets.

The second condition for performing the grouping is that the vanishing points of the motion vectors (u _x , u _y ) ^T from which the rotational motion components of the cameras are removed match. Therefore, the same label (second label) is given to the feature point having the motion vector (u _x , u _y ) ^T having the same vanishing point in the set of feature points having the same first label. The reason for this processing is based on the fact that, as shown in FIG. 6B, the feature points of the same approaching object translate in parallel with each other, so that their motion vectors share a vanishing point. . When the object moves in parallel with the ground, the vanishing point exists on the horizon.

However, the target approaching object is limited to that approaching the camera 10 (Z ₁ <Z ₀ ). As shown in FIG. 6B, this corresponds to the motion vector (u _x , u _y ) ^{T of} each feature point having a direction of radially diffusing from the vanishing point toward the feature point. In other words, the feature point having the direction from the feature point toward the vanishing point with the motion vector (u _x , u _y ) ^T is excluded from the grouping target and the label is deleted. When the target object has a motion such that the Z coordinate does not change, the motion vectors (u _x , u _y ) ^T of the feature points are parallel to each other, so that the vanishing point is damaged at infinity. Become. When this state is observed, these feature points may be grouped as the same object. In other words, it is not necessary to provide an upper limit and a lower limit for the vanishing point coordinates.

In the vanishing point coincidence determination, as shown in FIG. 6B, the distribution state of the vanishing point coordinates is obtained for each group to which the first label is assigned, and it is considered that the same vanishing point is shared from the distribution state. And feature points that are not considered to share the same vanishing point. Alternatively, whether or not the vanishing point is shared may be determined by whether or not the vanishing point is within a predetermined threshold from the center of the distribution. Furthermore, considering the error of the motion vector (u _x , u _y ) ^T , the fluctuation range of each vanishing point under this error is obtained, and the same second label is assigned to those whose fluctuation ranges overlap each other. May be.

As another example of the grouping process, consider a case where an object does not necessarily translate in parallel with the ground. In this case, since the vanishing point does not necessarily exist on the horizon, it is necessary to obtain the two-dimensional vanishing point coordinates (x _∞ , y _∞ ).

Here, the inclination of the straight line obtained by extending the motion vector (u _x , u _y ) ^T on the image plane of the feature point k is a _k , the intercept is b _k, and the distance in the y direction between the straight line and the vanishing point is If r _k is set, y−a _k x−b _k = r _k holds. This equation holds for the number of feature points. Then, for each group of the first label has been applied, the value of the vanishing point that minimizes the square sum of r _k, is calculated using the least squares method. Against vanishing point and the calculated feature point having a value larger r _k as an absolute value, it is considered as not sharing vanishing point, as outliers (outliers), excluded the like thresholding. The vanishing point coordinates can be estimated by applying the least square method again to the remaining feature points.

Note that a RANSAC (RANdom SAmple Consensus) method can also be used as a method combining the least square method and outlier removal. The RANSAC method is a commonly used method, and DA. Forsyth, J. Ponce, “Computer Vision: A Modern Approach” Detailed explanation on Prentice Hall, etc. Further, in the above example, the error r _k was placed with a distance in the y direction, described as the distance in the direction perpendicular to the straight line, it can be formulated as a linear least squares problem. This is a method called Total Least Squares. Note that Total Least Squares is also a commonly used technique, and is detailed in http://en.wikipedia.org/wiki/Total least squares, so a description thereof is omitted here.

  A third condition for performing grouping is that the collision times TTC of the feature points are similar. Therefore, the same label (group label) is given to the feature points having the same collision time TTC in the set of feature points having the same second label. The collision time TTC is defined as the time until an object existing in front of the camera 10 (Z> 0) reaches the image plane (substantially, the time until it collides with the host vehicle).

Assuming that the collision time TTC is ΔT, the collision time ΔT is expressed as follows using the motion vector (u _x , u _y ) ^T from which the rotational motion component of the camera 10 is removed and the vanishing point coordinates (x _∞ , y _∞ ). It can be calculated by the following formula 9.

Using Equation 9, the collision time ΔT is obtained for each feature point. Specifically, for each feature point, vanishing point coordinates (x _∞ , y _∞ ) and motion vector (u _x , u _y ) ^T are substituted into Equation 9 to calculate the collision time ΔT. The collision time ΔT is calculated as a time having a certain range in consideration of a motion vector calculation error. Then, the same second label is assigned, and among the grouped feature points, those having the same collision time ΔT are given the same group label. Further, in consideration of possible errors of vanishing points, the range of the collision time ΔT under these errors may be determined.

  Here, the collision time ΔT of the same approaching object is considered to be similar as long as general detection targets (people, bicycles, vehicles, etc.) are taken into consideration, although there are some deviations depending on the part of the object. Is natural. Accordingly, with respect to the collision time ΔT obtained for each feature point with the same label assigned under the conditions 1 and 2 as a target, the similarity of the collision time ΔT as shown in FIG. 6C. Judgment is made. That is, if an error bar having a preset time width is given and the collision time ΔT matches within the error bar range, it is determined that the collision time ΔT is similar, and if there is a feature point where the collision time ΔT does not match within the error bar range , Exclude from group.

  In the grouping of feature points, it is preferable to set a lower limit on the number of feature points having the same group label. If the number of grouped feature points is less than a predetermined number corresponding to the lower limit, the set of grouped feature points is not regarded as corresponding to the object, and the group label is deleted. You may do it. Then, since each group of feature points includes at least a predetermined number of feature points corresponding to the lower limit, it is possible to make a reliable determination as the same object. For example, when a feature point is accidentally extracted due to noise or the like, it can be prevented from erroneously detecting it as an approaching object.

  However, if the predetermined number is too large, for example, because it is far away, there is a possibility that an approaching object having a small number of feature points may be undetected because the area shown in the image is small. Therefore, the predetermined number needs to be set to an appropriate value. For example, in practice, the predetermined number is preferably set to 2 to 4, but other numerical values may be used.

  In addition, in the process of step S110, when the attribute of the approaching object is determined for each feature point in addition to the calculation of the likelihood, the grouping process in step S140 performs grouping in consideration of the attribute information. May be. That is, feature points having the same attribute information as attributes of approaching objects may be grouped by giving the same label as indicating the same approaching object.

  When the above-described grouping process is completed, the likelihood calculating unit 33 proceeds to step S150 in the flowchart of FIG. 2, and regarding the objects represented by the grouped feature points, the likelihood of the plurality of grouped feature points. Based on the above, an expected value of likelihood representing the likelihood that the object is an approaching object is calculated. The expected value of the likelihood becomes the output of the likelihood calculating unit 33.

  The expected value of likelihood can be calculated by, for example, averaging the likelihood of a plurality of grouped feature points. This is because the expected value of likelihood can be calculated by comprehensively considering the likelihood of a plurality of grouped feature points by such averaging processing. This averaging process may be a simple average, or may be a weighted average in which a feature point is weighted. An example of how to obtain this weight will be described below.

  Even if the vanishing point coordinates and the collision time TTC described above are relatively small errors, the characteristic points closer to the horizon are more affected by the error. For this reason, the feature points closer to the horizon are more likely to have errors in the grouping result. Therefore, the reliability of the grouping of each feature point can be quantified according to the distance from the horizon. Then, in the set of feature points having the same group label, the value indicating the reliability of each feature point can be normalized and used as the above weight. In addition, in order to quantify the reliability of grouping, it is possible to take the reciprocal of the vanishing point range of each feature point.

  In addition, when the likelihood of the feature point is calculated for each attribute of the approaching object, the expected value of likelihood is calculated for each attribute of the approaching object, and the expected value of the highest likelihood is indicated. The attribute is preferably determined as the attribute of the approaching object. In this way, it is possible not only to detect the presence of an approaching object, but also to determine the attributes of the approaching object.

  Next, the τ calculation unit 34 of the image processing ECU 30 will be described. First, in this embodiment, as shown in FIG. 7, in order to notify the driver of an approaching object to the host vehicle, a frame surrounding the approaching object is superimposed and displayed on the vehicle peripheral image, and the frame and the frame The partial image including the enclosed approaching object is enlarged as time passes to alert the driver about the approaching object. At that time, in order to change the notification form for notifying the driver of the presence of the approaching object according to the expected value of likelihood representing the probability that each object is an approaching object, the frame and the partial image are represented by the likelihood. It will be expanded at an expansion rate according to the expected value. Specifically, the enlargement ratio is determined so that the frame and the partial image are enlarged as the expected value of likelihood increases.

  In this way, the optical information τ is used in the present embodiment in order to determine the enlargement ratio according to the expected value of likelihood. As shown in FIG. 8, the optical information τ is τ = θ / (dθ / dt), where θ is an angle formed by two straight lines from one point in the crystalline lens of the eye to two points on the contour portion on the object. Defined. Note that dθ / dt represents time differentiation of the angle θ.

  This optical information τ is known to be approximately equal to the time for the approaching object to reach the retina (including the plane) when the object is an approaching object. For this reason, the optical information τ is also defined as a value that correlates with the time for the approaching object to reach the retina (including the plane).

  Humans (and other animals) are known to have neurons, called τ neurons, that ignite when an object with a value of τ is viewed. Information on other movements (distance, speed, It is known that the above-mentioned arrival time can be perceived more intuitively than the acceleration). For example, when you look at a ball that is facing you, you cannot recognize the distance to the ball, the speed or acceleration of the ball (at least in an instant), but the amount of time it takes for the ball to reach you It is possible to recognize. Thus, the optical information τ is information that is very easy for humans to recognize. Therefore, it can be said that the optical information τ is closely related to the degree of attention drawn by humans. For example, human beings are more strongly alerted to objects that have small optical information τ and that reach themselves in a shorter period of time, in other words, objects that come toward themselves at a faster speed.

  For this reason, the τ calculation unit 34 calculates optical information τ that decreases as the expected value of likelihood output from the likelihood calculation unit 33 increases. That is, the optical information τ becomes a decreasing function with respect to the expected value of likelihood. For example, the optical information τ can be calculated from the reciprocal of the expected value of likelihood. In this case, the coefficient is experimentally determined in advance. Further, the optical information τ may be calculated by a linear function having a slope that is negative with respect to the expected value of likelihood. In this case, the slope and intercept are experimentally determined in advance. Needless to say, the optical information τ may be calculated by a more complicated function other than the reciprocal or linear function.

The frame / partial image drawing unit 35 uses the optical information τ calculated by the τ calculation unit 34 to determine the enlargement ratio of the frame and the partial image. Generate a partial image. Here, when the enlargement factor is determined using the optical information τ, the enlargement factor is (1 + τ ⁻¹ ). By determining the enlargement ratio in this way, the frame and the partial image corresponding to the approaching object can be displayed on the screen so as to approach according to the optical information τ. Therefore, the expected value of the likelihood is large, and therefore, the frame and the partial image corresponding to the approaching object for which a small value is set as the optical information τ is greatly enlarged as time passes.

  Further, as illustrated in FIG. 7, the frame / partial image drawing unit 35 returns the frame and the partial image to a state in which the sizes of the frame and the partial image are not enlarged when the enlargement of the frame and the partial image is executed for a predetermined time. Then, from that state, the frame and the partial image are again enlarged at an enlargement rate corresponding to the optical information τ calculated by the τ calculation unit 34. That is, the frame / partial image drawing unit 35 repeatedly executes enlargement of the frame and the partial image every predetermined time. In this way, by repeatedly executing the enlargement of the frame and the partial image, the enlarged display of the frame and the partial image can be continuously performed, and the alerting effect can be enhanced. In addition, due to the limitation of the display range of the display device 40, there is a limit to enlarge the size of the frame and the partial image. Can also be cleared.

  The image drawing unit 36 generates an image to be displayed on the display 40 based on image information captured by the camera 10. The synthesizing unit 37 synthesizes these images by superimposing the frame and the partial image generated by the frame / partial image drawing unit 35 on the image generated by the image drawing unit 36. is there. The display 40 is provided in the passenger compartment and displays the image synthesized by the synthesis unit 37.

  In the present embodiment, as described above, when an approaching object is present around the vehicle, a frame surrounding the approaching object and a partial image surrounded by the frame are displayed so as to be enlarged with the passage of time. For this reason, the presence of an approaching object can be reported to the driver of the host vehicle in an easy-to-understand manner visually. Particularly, since the frame and the partial image are enlarged at an enlargement rate corresponding to the expected value of likelihood, the frame and the partial image are greatly enlarged as the expected value of likelihood is larger. For this reason, the difference in the magnitude of the expected value of likelihood is obvious at a glance, and the driver can be alerted to pay more attention to an approaching object with a higher expected value of likelihood. .

  In the above-described embodiment, the frame / partial image drawing unit 35 may perform the following display regarding the display of the frame.

  First, the frame / partial image drawing unit 35 may change the thickness of the line of the frame according to the enlargement ratio at which the frame and the partial image are enlarged. That is, when the frame is gradually enlarged and displayed with the passage of time, the thickness of the line of the frame may be increased as the size of the enlargement increases. Thereby, the larger the frame is enlarged, the more clearly the frame can be displayed, and the driver's attention can be more strongly alerted.

  Further, the frame / partial image drawing unit 35 may display the frame translucently so that the background portion can be visually recognized. As a result, the driver can visually recognize the entire image around the vehicle while displaying the frame superimposed.

  Further, the frame / partial image drawing unit 35 may display the shadow of the frame on the back side of the frame. Thereby, when enlarging a frame, a three-dimensional effect can be given and the feeling that the frame is approaching a driver can be given. Note that the shadow of the frame may be displayed in a translucent manner so that the background portion can be visually recognized, as in the case of the frame.

  Also, the frame / partial image drawing unit 35 or the combining unit 37 is a case where frames corresponding to a plurality of approaching objects need to be superimposed and displayed on the vehicle peripheral image at the same time, and the frames corresponding to the plurality of approaching objects overlap. In such a case, it is preferable to generate an image showing a frame or synthesize an image showing a frame so that a frame corresponding to an approaching object having a high likelihood value is displayed in front. Thereby, when there are a plurality of approaching objects approaching the vehicle, it is possible to perform display according to the priority that the driver should pay attention to.

  Further, the frame / partial image drawing unit 35 may determine the shape of the frame according to the determined attribute of the approaching object when the likelihood calculation unit 33 determines the attribute of the approaching object. good. For example, the shape of the frame may be a shape of a pedestrian, bicycle, or vehicle, or the color of the frame may be a color corresponding to the attribute of the approaching object. Accordingly, it is possible to notify the driver of the attribute of the approaching object using the frame display form.

  Further, the image processing ECU 30 may enlarge the frame while displaying the frame three-dimensionally when the display device 40 is capable of performing the three-dimensional display. By performing such a three-dimensional display, a feeling that the frame is approaching the driver can be given more clearly.

  The preferred embodiments of the present invention have been described above. However, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

  For example, in the above-described embodiment, when the likelihood calculation unit 33 calculates the likelihood of the feature point for each attribute of the approaching object, the attribute indicating the highest value among the expected values of the calculated likelihoods. Is determined as the attribute of the approaching object.

  However, for example, at least pedestrians, bicycles, and automobiles are attributes of approaching objects, the probability that the approaching object is a pedestrian, the probability of being a bicycle, the probability of being a vehicle is determined in advance as respective prior probability distributions, For each attribute of the approaching object, the posterior probability may be calculated by multiplying the expected value of the likelihood and the prior probability, and the attribute having the highest posterior probability may be determined as the attribute of the approaching object. By using prior probabilities in this way, it is possible to set different prior probability distributions, for example, depending on the location where the vehicle is stopped, and it is possible to determine the attributes of approaching objects with higher accuracy. . For example, in a vehicle-dedicated parking lot, it can be said that the approaching object has a high probability of being a pedestrian and a vehicle, and the probability of being a bicycle is low. On the other hand, when the vehicle is stopped in a parking space provided across the sidewalk from the road, the approaching object is likely to be a pedestrian or a bicycle, and the probability of being a vehicle is low. Therefore, by considering not only the expected likelihood value but also such a prior probability distribution, it is possible to further improve the accuracy of determining the attribute of the approaching object.

  In the above-described embodiment, the frame surrounding the approaching object and the partial image including the approaching object surrounded by the frame at the enlargement rate according to the expected value of the likelihood representing the probability of being the approaching object The display was enlarged with the passage of time.

  However, only the frame may be displayed in an enlarged manner, or only the partial image may be displayed in an enlarged manner without displaying the frame.

10 Camera 20 Vehicle Information Detection Unit 30 Image Processing ECU
40 indicator

Claims (23)

Photographing means mounted on the subject vehicle so as to photograph the periphery of the subject vehicle;
In the first image and the second image taken at different times by the photographing means, the same feature point is extracted, the first coordinate position of the feature point in the first image, and the feature in the second image. First calculating means for calculating a second coordinate position of the point;
Second calculating means for calculating a movement vector of the photographing means by movement of the host vehicle in a translational direction between the photographing time of the first image and the photographing time of the second image;
The feature point indicates an approaching object based on the first coordinate position and the second coordinate position calculated by the first calculation unit and the movement vector calculated by the second calculation unit. In addition, using the probability distribution of values that can be taken by the second coordinate position, the probability distribution that can be taken as a moving speed, and the probability distribution of positions that should be present in the depth direction to be photographed by the photographing means, the feature points are: A likelihood calculating means for calculating a likelihood representing a probability that indicates an approaching object approaching the host vehicle in a real space;
Expectation to calculate the expected value of likelihood indicating the likelihood that each object is an approaching object based on the likelihood of the plurality of feature points grouped while grouping the feature points for which the likelihood is calculated A value calculating means;
Informing the occupant of the host vehicle of the presence of an approaching object, and informing the occupant of the host vehicle of the presence of an approaching object according to the expected value of likelihood calculated by the expected value calculation means. A vehicle driving support device, comprising: a notification unit configured to change a notification mode when performing the operation.   The vehicle driving support apparatus according to claim 1, wherein the expected value calculation unit performs grouping on feature points having a likelihood equal to or greater than a predetermined threshold.   The vehicle driving according to claim 1, wherein the expectation value calculating means calculates the expectation value of the likelihood by averaging the likelihoods of a plurality of grouped feature points. Support device.   The expectation value calculation means performs weighted average processing for assigning larger weights as the reliability belonging to the same group is higher when averaging the likelihoods of the plurality of feature points. The vehicle driving support device according to claim 1.   If the number of feature points grouped is less than a predetermined number, the expectation value calculation means does not consider the grouped feature point set corresponding to an object, and the expected value of the likelihood The vehicle driving support device according to any one of claims 1 to 4, wherein the calculation is not performed. The likelihood calculating means sets at least a pedestrian, a bicycle, and a car as an attribute of an approaching object, and calculates the likelihood of the feature point for each attribute of the approaching object,
The expected value calculation means calculates the expected value of the likelihood for each attribute of the approaching object, and determines an attribute indicating the expected value of the highest likelihood as the attribute of the approaching object. The vehicle driving support device according to any one of claims 1 to 5. The likelihood calculating means sets at least a pedestrian, a bicycle, and a car as an attribute of an approaching object, and calculates the likelihood of the feature point for each attribute of the approaching object,
The expected value calculation means calculates the expected value of the likelihood for each attribute of the approaching object, and regarding the approaching object, the probability that it is a pedestrian, the probability that it is a bicycle, and the probability that it is a vehicle, respectively. For each attribute of the approaching object, the posterior probability is calculated by multiplying the expected value of the likelihood and the probability, and the attribute having the highest posterior probability is determined as the approaching object. The vehicle driving support apparatus according to claim 1, wherein the driving support apparatus is determined as an attribute of the vehicle.   The first calculating means rotates the photographing means in the second image when the host vehicle moves in the rotation direction between the photographing time of the first image and the photographing time of the second image. The vehicle driving support apparatus according to claim 1, wherein the second coordinate position from which the influence of movement is removed is calculated.   The informing means has a display for displaying an image photographed by the photographing means, and a frame surrounding the approaching object is superimposed on the image displayed on the display, and the expected value of the likelihood The vehicle driving support device according to any one of claims 1 to 8, wherein the frame is enlarged with the passage of time at an enlargement rate corresponding to. The notification means determines the value of the optical information τ so as to decrease as the expected value of the likelihood increases, and enlarges the frame according to an enlargement ratio determined by (1 + τ ⁻¹ ). Item 10. The vehicle driving support device according to Item 9.   The vehicle driving support apparatus according to claim 9 or 10, wherein the notification unit changes a thickness of a line of the frame according to the enlargement ratio.   The vehicle driving support device according to claim 9, wherein the notification unit displays the frame in a translucent manner so that a background portion of the frame can be visually recognized.   The vehicle driving support apparatus according to any one of claims 9 to 12, wherein the notification unit displays a shadow of the frame on the back side of the frame.   The vehicular driving support apparatus according to claim 13, wherein the war record notification means displays the shadow of the frame in a translucent manner so that the background portion can be visually recognized.   15. The display device according to claim 9, wherein the display unit is capable of performing a three-dimensional display, and the notification unit enlarges the frame while displaying the frame three-dimensionally. The vehicle driving support device according to claim.   16. The method according to claim 9, wherein when the enlargement of the frame is performed for a predetermined time, the notifying unit repeatedly executes the enlargement of the frame so that the size of the frame is returned to a state where the frame is not enlarged. The vehicle driving support device according to claim 1.   The notification means is a case where frames corresponding to a plurality of approaching objects are simultaneously superimposed and displayed on the image, and when the frames corresponding to the plurality of approaching objects overlap, an approach having a large expected value of the likelihood The vehicle driving support device according to claim 9, wherein a frame corresponding to the object is displayed in front.   The notification means, when the attribute of the approaching object is determined by the expected value calculation means, makes the form of the frame according to the determined attribute of the approaching object. Item 18. A vehicle driving support device according to any one of Items 9 to 17.   The notification means has a display for displaying an image photographed by the photographing means, and displays a partial image showing an approaching object in the image with an enlargement ratio corresponding to the expected value of the likelihood as time passes. The vehicle driving support device according to any one of claims 1 to 18, wherein the vehicle driving support device is displayed in an enlarged manner. The informing means determines the value of the optical information τ so as to decrease as the expected value of the likelihood increases, and enlarges the partial image according to an enlargement ratio determined by (1 + τ ⁻¹ ). The vehicle driving support device according to claim 19.   21. The display device according to claim 19 or 20, wherein the display device is capable of performing a three-dimensional display, and the notifying unit enlarges the partial image while displaying the three-dimensional image. Vehicle driving support device.   The said notification means repeatedly executes the enlargement of the partial image so as to return to the state where the partial image is not enlarged when the enlargement of the partial image is executed for a predetermined time. The vehicle driving support device according to any one of the above. The notification means is a case where partial images corresponding to a plurality of approaching objects are simultaneously displayed on the image, and when the partial images corresponding to the plurality of approaching objects overlap, the expected value of the likelihood is large. The vehicle driving support device according to any one of claims 19 to 22 , wherein a partial image corresponding to the approaching object is displayed in front.

Images