JP2014026555A - Mobile-object recognition system, mobile-object recognition program, and mobile-object recognition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a mobile-object recognition system for removing an outlier of an optical flow by using a smaller operation quantity.SOLUTION: A mobile-object recognition system includes: a connected optical flow calculation part 20, disposed in a vehicle, for calculating a connected optical flow by extracting a feature point mutually corresponding between images of plural frames and connecting a plurality of unit optical flows interconnecting the extracted feature points of adjacent frames; an outlier removal part 51 for removing an outlier on the basis of linearity of the connected optical flow and removing the outlier on the basis of a degree of similarity of impact time between the plurality of connected optical flows with different numbers of unit optical flows for the same feature point; and a group determination part 52 for grouping, as plural feature points on a mobile object, plural feature points whose extended optical flows converge onto one vanishing point, among plural feature points remaining after a connected optical flow is removed as an outlier.

Description

  The present invention relates to a moving object recognition system, a moving object recognition program, and a moving object recognition method for recognizing a moving object using a single viewpoint image.

  2. Description of the Related Art Conventionally, a moving body recognition system that recognizes a moving body around a vehicle and performs notification or warning to a driver or automatic braking of the vehicle (hereinafter simply referred to as “notification”) is known. The distance from the host vehicle to the moving body can be directly measured using a distance sensor such as a millimeter wave radar, or can be measured by performing image processing using a stereo camera. By analyzing the measured distances in time series, the moving direction of the moving body and the relative speed with respect to the host vehicle can be obtained. And by calculating | requiring the moving direction of a moving body and the relative speed with respect to the own vehicle, the possibility (risk) that the moving body collides with a vehicle can be calculated | required, and notification can be performed based on the risk. .

  As a simpler and cheaper mobile object recognition system, a system that recognizes a mobile object with a monocular camera having only one optical system and an image sensor is also known. This system obtains images of a plurality of single viewpoints arranged in time series by continuously photographing with a monocular camera. A technique described in Patent Document 1 is known as a technique for detecting a moving object from a plurality of single viewpoint images.

  In the technique described in Patent Document 1, a group of feature points is grouped for each moving object based on feature points extracted from each of a plurality of single viewpoint images arranged in time series and their motion vectors (optical flow). This grouping is performed according to the convergence of the optical flow to the vanishing point and the variation of the external division ratio. Thereby, even if optical flows calculated inaccurately are mixed, it is possible to reliably detect a moving body.

Japanese Patent No. 4919036

  Although the technique of Patent Document 1 is robust against optical flow outliers, it is still rare that multiple optical flows have the same vanishing point and external division ratio even though they are outliers. May have in range. In this case, since the outlier is detected, an erroneous notification is made.

  It is an object of the present invention to provide a mobile object recognition system that removes an optical flow outlier with a small amount of computation.

  The moving body recognition system of the present invention is a feature that is installed in a vehicle and that captures images of a plurality of consecutive frames, and that corresponds to each other between each of the plurality of frame images from the plurality of frame images. And a connected optical flow calculation unit that calculates a connected optical flow obtained by connecting a plurality of single optical flows that connect the extracted feature points of adjacent frames, and an outlier is removed using the connected optical flow. Of the plurality of feature points remaining after the connected optical flow is removed as an outlier by the outlier removing unit, and the extended connected optical flow converges to one vanishing point. Group determination unit for grouping a plurality of feature points as a plurality of feature points on the moving body And it has a configuration with a.

  According to this configuration, since the outlier is removed using the connected optical flow, it is possible to perform mobile object recognition with a small amount of calculation and with little recognition error.

  In the above moving body recognition system, the outlier removal unit may remove the outlier based on the straightness of the connected optical flow.

  When feature points are accurately tracked over a plurality of frames, a plurality of single optical flows will be connected in a straight line. Therefore, according to this configuration, the outlier can be determined and removed effectively with a small amount of calculation of examining the linearity when a plurality of single optical flows are connected.

  In the above moving body recognition system, the outlier removal unit may perform the outlier based on similarity of collision times of a plurality of connected optical flows having different numbers of single optical flows to be connected with respect to the same feature point. Liar may be removed.

  When the feature point is accurately tracked over a plurality of frames, the collision times (TTC) of the plurality of connected optical flows having different numbers of connected single optical flows are similar to each other. Therefore, according to this configuration, the outlier can be determined and removed effectively with a small amount of calculation of examining the TTC similarity of such a plurality of connected optical flows.

  In the mobile object recognition system, the outlier removal unit obtains a vanishing point range that each of the plurality of connected optical flows can take, and the vanishing point range is a vanishing point range of another connected optical flow. Non-overlapping connected optical flows may be removed as the outlier.

  When a feature point is accurately tracked over a plurality of frames, the ranges of vanishing points that can be taken by the connected optical flow of the feature point match. Therefore, according to this configuration, the outlier can be determined and removed effectively with a small amount of calculation of examining the overlap of vanishing point ranges of the connected optical flows between a plurality of feature points.

  In the above moving object recognition system, the outlier removal unit determines a range that can be taken by the intersection of the connected optical flow and an extension line of a vector within an error range thereof and a horizon defined in the image. It may be obtained as a range.

  According to this configuration, a vanishing point for each feature point can be obtained by a simple method without requiring information on other feature points.

  In the above mobile object recognition system, the outlier removal unit obtains a range of collision times that each of the plurality of connected optical flows can take, and the collision time does not overlap with a range of collision times of other connected optical flows. The connected optical flow may be removed as the outlier.

  When a feature point is accurately tracked over a plurality of frames, the TTC ranges that can be taken by the connected optical flow of the feature point match. Therefore, according to this configuration, the outlier can be determined and removed effectively with a small amount of calculation of examining the overlap of the TTC range of the connected optical flow between a plurality of feature points.

  In the above mobile object recognition system, the outlier removal unit determines a range of vanishing points that can be taken by an intersection of the connected optical flow and an extension line of a vector within an error range thereof and a horizon defined in the image. The collision time range may be obtained based on the vanishing point range and the connection optical flow.

  According to this configuration, the TTC for each feature point can be obtained by a simple method without requiring information on other feature points.

  The moving object recognition program of the present invention extracts a feature point corresponding to each other between the plurality of frame images from each of a plurality of consecutive frame images photographed by a camera installed in a vehicle. A connected optical flow calculation unit that calculates a connected optical flow that connects a plurality of single optical flows that connect the extracted feature points of adjacent frames, an outlier removal unit that removes an outlier using the connected optical flow, Among the plurality of feature points remaining after the connected optical flow is removed as an outlier in the outlier removal unit, the plurality of feature points where the extended connected optical flow converges to one vanishing point Functions as a group determination unit that groups multiple feature points on a moving object That.

  Also with this configuration, since the outlier is removed using the connected optical flow, it is possible to perform mobile object recognition with a small amount of calculation and with little misrecognition.

  The moving body recognition method of the present invention corresponds to a shooting step of capturing images of a plurality of consecutive frames with a camera installed in a vehicle and a plurality of frames from each of the images of the plurality of frames. A connected optical flow calculation step of calculating a connected optical flow by connecting a plurality of single optical flows connecting the extracted feature points of adjacent frames, and an outlier using the connected optical flow Of the plurality of feature points remaining after the connected optical flow is removed as an outlier in the outlier removing unit, and the extended connected optical flow becomes one vanishing point. A grouping of a plurality of convergent feature points as a plurality of feature points on the moving body. And it has a configuration in which the-loop determining step.

  Also with this configuration, since the outlier is removed using the connected optical flow, it is possible to perform mobile object recognition with a small amount of calculation and with little misrecognition.

  According to the present invention, since the outlier is removed using the connected optical flow, it is possible to perform mobile object recognition with a small amount of calculation and with little recognition error.

The block diagram which shows the structure of the mobile body recognition system in the 1st Embodiment of this invention. The figure explaining the connection optical flow in the 1st Embodiment of this invention The figure explaining the definition of the coordinate in the 1st Embodiment of this invention The figure which shows the distance which the point cloud of the three-dimensional space in the 1st Embodiment of this invention moves in translation in unit time The figure explaining the vanishing point in the 1st Embodiment of this invention (A) Graph showing the cost function of robust estimation in the first embodiment of the present invention (b) Graph showing the influence function of robust estimation in the first embodiment of the present invention The figure which shows the example of the image which an outlier tends to generate | occur | produce in the 1st Embodiment of this invention The block diagram which shows the structure of the mobile body recognition system in the 2nd Embodiment of this invention. The figure for demonstrating calculation of the vanishing point range in the 2nd Embodiment of this invention The graph which shows the vanishing point range of the some feature point in the 2nd Embodiment of this invention The graph which shows the TTC range of the some feature point in the 2nd Embodiment of this invention

(First embodiment)
Hereinafter, a mobile object recognition system according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the mobile object recognition system according to the first embodiment of this invention. The mobile object recognition system 101 of this embodiment is a system that recognizes a mobile object from an image. The moving body recognition system 101 includes a camera 10, a connected optical flow calculation unit 20, a rotational movement amount / vanishing point estimation unit 30, a background point removal unit 40, a grouping unit 50, and a result output unit 60. In addition, the structure which consists of the connection optical flow calculation part 20, the rotational movement amount and vanishing point estimation part 30, the background point removal part 40, and the grouping part 50 performs a moving body recognition program of embodiment of this invention by a computer. This is realized. The moving body recognition program may be stored in a storage medium, read from the storage medium by a computer, and executed by the computer.

  The camera 10 is mounted on a vehicle and photographs the periphery of the vehicle. The camera 10 is installed in the vehicle so as to capture the traveling direction of the host vehicle. The camera 10 may be installed, for example, on the back side (front side) of the rearview mirror so as to photograph the front of the vehicle. Since the vehicle can also move backward, the camera 10 may be installed, for example, near the rear license plate so that the camera 10 captures the rear of the vehicle. The camera 10 is a monocular camera provided with a single optical system and an image sensor. The camera 10 continuously captures images at a predetermined cycle (for example, every 1/30 seconds) and outputs an image signal.

  The connected optical flow calculation unit 20 extracts feature points corresponding to each other between the images of the plurality of frames from the images of the plurality of frames obtained by the camera 10, and calculates an optical flow for each feature point (J. shi). and C. Tomasi, "Good features to track," IEEE CVPR, pp. 593-600, 1994). In this embodiment, the LK method is used as an optical flow calculation algorithm (BD Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” IJCAI, pp. 674-679, 1981. reference).

  In calculating the optical flow, it is desirable that there are few outliers. The outlier generally refers to an unexpected calculation result, and the optical flow outlier particularly refers to a trajectory tracked in error. Although the technique of the above-mentioned Patent Document 1 is robust against an optical flow outlier, it is still rare that a plurality of optical flows are outliers and have the same vanishing point and external division ratio with an allowable error. It may have in the range of. In this case, since an outlier is detected, the risk level is erroneously calculated. In this embodiment, an outlier removal method with a low calculation amount is provided.

  In order to improve the accuracy of the optical flow, in the present embodiment, instead of using the optical flow between two adjacent frames, a concatenation of a plurality of optical flows between adjacent frames is used. In this specification, an optical flow between two adjacent frames, that is, a vector from a feature point of one frame to the same (corresponding) feature point in the next frame of the frame is referred to as “single optical flow. And the same as the feature point in the frame obtained by connecting a plurality of single optical flows, that is, the feature point in one frame at a predetermined number of frames of 1 or more from that frame (corresponding The vector to the feature point is called “connected optical flow”.

  FIG. 2 is a diagram for explaining a connection optical flow in the embodiment of the present invention. In FIG. 2, the left column is a sequence of images sequentially acquired from the camera 10 in order from the top, the central column shows a single optical flow, and the right column is A consolidated optical flow is shown in which three single optical flows are connected. The connected optical flow calculation unit 20 connects single optical flows for a plurality of frames of the same (corresponding) feature points to calculate a connected optical flow.

  As shown in FIG. 2, a single optical flow is generated by connecting the same feature points between two adjacent frames, and a connected optical flow is generated by connecting a plurality of single optical flows. In the example of FIG. 2, a connected optical flow is generated by connecting three (n = 3) optical flows.

  The rotational movement amount / vanishing point estimation unit 30 estimates the vanishing point from the connected optical flow. In general, when a three-dimensional point cloud translates at a constant speed, the trajectory of a two-dimensional point cloud that is a perspective projection of these point clouds has a feature that the extension lines intersect at one point. . This intersection is the vanishing point. In this specification, the term vanishing point is used to mean the intersection of the extension lines of the locus of points on the image plane. When there are a plurality of moving objects, vanishing points are estimated for each moving object.

  First, the coordinates used in the following calculation are defined. FIG. 3 is a diagram for explaining the definition of coordinates in the embodiment of the present invention. When the road surface is flat, it is assumed that the optical axis of the camera 10 and the road surface are parallel. The optical axis of the camera 10 is the Z axis, the vertically downward direction is the Y axis, and the X axis is defined by the right hand coordinate system. The origin (0, 0, 0) is the optical center of the camera 10. In this embodiment, it is assumed that the optical axis of the camera 10 is parallel to the road surface. However, even when the optical axis of the camera 10 and the road surface are not parallel, it can be easily generalized by introducing an appropriate rotation matrix. is there. Considering a perspective projection model, the image coordinates are given by (x, y) = (fX / Z, fY / Z). Here, f is the focal length of the camera 10 and is known.

  FIG. 4 is a diagram showing the distance (the velocity of the point group in the three-dimensional space) that the point group (point on the moving body) in the three-dimensional space moves in translation in unit time. When this velocity V is (ΔX, ΔY, ΔZ), the image coordinates x (t) = f (X + tΔX) / (Z + tΔZ) projected at time t, y (t) = f (Y + tΔY) / (Z + tΔZ) Converges to (fΔX / ΔZ, fΔY / ΔZ) in the limit of t → −∞ or t → ∞. Therefore, the two-dimensional coordinates of the vanishing point are given as (fΔX / ΔZ, fΔY / ΔZ). The point cloud that converges from t → −∞ is a point cloud that moves away from the camera, and the point cloud that converges at t → ∞ is a point cloud that approaches the camera.

  FIG. 5 is a diagram illustrating vanishing points. In the example of FIG. 5, the extension lines of the connected optical flows COF 1 to 6 of the feature points FP 1 to 6 intersect at one point, and this point becomes the vanishing point VP. The rotational movement amount / vanishing point estimation unit 30 extends the connection optical flow of each feature point, searches for a point where a plurality of connection optical flows intersect, and estimates it as a vanishing point. Considering that the optical flow includes the outlier, the optical flow of the moving object, and the optical flow of the background object, it is appropriate to apply the robust estimation to the vanishing point estimation. Robust estimation refers to parameter estimation that is robust against outliers. In this embodiment, M estimation (see P. J. Huber and E. M. Ronchetti, “Robust Statistice, 2nd Edition,” Wiley Interscience) is applied as robust estimation.

  If the above-described technique cannot be applied because, for example, the calculation capability is limited, the rotational movement amount / vanishing point estimation unit 30 may calculate the vanishing point of the connected optical flow by the following simple process. Good. In this method, the rotational movement amount / vanishing point estimation unit 30 records the position of the horizon in the image of the camera 10 in advance, and the intersection of the horizon and the connected optical flow is used as the vanishing point of the locus of this point. . This method is based on the fact that when the moving object and the camera 10 are moving parallel to the flat ground, the vanishing point of the moving object's trajectory exists on the horizon. However, if the slope of the connected optical flow is small (when it is almost parallel to the horizon), the error of the connected optical flow can greatly change the value of the vanishing point. If the range is too large, this point is excluded from the detection candidates as an exceptional process.

ここで、ｘ＝ｆＸ／Ｚ、ｙ＝ｆＹ／Ｚであり、ｖ_x ^r、ｖ_y ^rは、以下の式（２）で与えられるカメラ１０の回転量の一次の項からなるオプティカルフローの成分である。

Further, the rotational movement amount / vanishing point estimation unit 30 estimates an optical flow component caused by the rotational movement of the camera 10 and removes it from the connected optical flow. Now, a three-dimensional point (X, V, _X , V _y , V _z ) and a three-dimensional point (X, V) projected by a camera having a rotational movement amount Ω = (Ω _x , Ω _y , Ω _z ). If the optical flow of Y, Z) is v = (v _x , v _y ), this optical flow v is expressed by the following equation (1).

Here, x = fX / Z, y = fY / Z, and v _x ^r and v _y ^r are components of an optical flow consisting of a first-order term of the rotation amount of the camera 10 given by the following equation (2). It is.

ここで、Ｒ（ｐ_F，Ω）は、パラメタ空間内の誤差関数であり、理想状態において０をとる。式（３）は即ち、回転成分を取り除いた静止点のオプティカルフローを延長すると、画像上の１点（即ち、拡張焦点：ＦＯＥ：Focus Of Expansion）で交わることを意味している。 By replacing p _F = (x _F , y _F ) = (fV _x / V _z , fV _y / V _z ) from the expression (2) and eliminating the depth components Z and V _z , the following expression (3) is obtained: can get.

Here, R (p _F , Ω) is an error function in the parameter space, and takes 0 in the ideal state. That is, the expression (3) means that when the optical flow of the stationary point from which the rotation component is removed is extended, it intersects at one point on the image (that is, FOE: Focus Of Expansion).

ここで、ρ（Ｒ）は最小値に０を持つ、Ｒ＝０に対して対称の関数である。はずれ値に対する頑健性の度合いは、影響関数ψ（Ｒ）＝∂ρ／∂Ｒによって特徴付けられる。 Since the outlier value of the optical flow and the point of the moving object are generally considered to have a large R value, in this embodiment, the rotational movement amount / vanishing point estimation unit 30 performs the M estimation of Expression (4). Is used.

Here, ρ (R) is a function having a minimum value of 0 and symmetric with respect to R = 0. The degree of robustness against outliers is characterized by the influence function ψ (R) = ∂ρ / ∂R.

FIG. 6A is a graph showing a cost function for robust estimation, and FIG. 6B is a diagram showing an influence function for robust estimation. 6A and 6B, the broken line indicates the L2 norm, and the solid line indicates the Cauchy function. In this embodiment, the Cauchy influence function defined by FIG. 6 and the following equation (5) is used.

  Since the influence function of Couchy is not a monotonically increasing function at R> 0 but starts to decrease at the extreme value, the influence on the input solution having a large error can be suppressed to a low level. The constant C in equation (5) is C = 2.385. This value is set so as to give 95% efficiency of the least squares method of Gaussian distribution with an average of 0.

The rotational movement amount / vanishing point estimation unit 30 uses an iteratively reweighted least squares (IRLS) method as a solution for M estimation. This is a method in which the cost function of Equation (4) is transformed into a weighted error sum of squares, and the least square method and weight update are alternately repeated until the solution converges. This weight is given by ω = ψ (R) / R using an influence function. The IRLS algorithm is shown below.

  The denominator 1.48 mad (R) in the rescaling in step 3) is set to be equal to the standard deviation when R follows a normal distribution. The reason for using the median absolute deviation (mad) instead of the standard deviation is to suppress the fluctuation of the scale due to the mixture of outliers. Since the error is expressed in the form of the product of the FOE coordinates and the rotational movement amount, step 4) is a nonlinear least square method. In the present embodiment, the rotational movement amount / vanishing point estimation unit 30 obtains a solution by the Newton method. The rotational movement amount / vanishing point estimation unit 30 removes a component caused by the obtained rotational movement amount from the connected optical flow.

  The background point removal unit 40 removes the connected optical flow as an object that does not move with respect to the background, that is, the ground, when a straight line obtained by extending the connected optical flow passes through the vanishing point of the background within the allowable error range. That is, since the moving body that moves relative to the ground has a vanishing point at a position different from the background, in order to detect a feature point group having such a vanishing point as a moving body, the background point removing unit 40 includes: The feature point having the vanishing point of the background and its connected optical flow are removed.

  Next, the grouping unit 50 will be described. The connected optical flow generated by the connected optical flow calculation unit 20, the component due to the rotational movement amount removed by the rotational movement amount / vanishing point estimation unit 30, and removed as the background by the background removal unit 40 is Either an outlier or an optical flow of feature points on a correctly tracked moving object. The grouping unit 50 performs grouping of the feature points and the connected optical flows while removing the outliers using the connected optical flows. The grouping unit 50 robustly estimates the vanishing point again for the remaining feature points and the connected optical flows.

  The grouping unit 50 includes an outlier removal unit 51 and a group determination unit 52. The outlier removal unit 51 includes a straightness determination unit 511 and a TTC similarity determination unit 512. Hereinafter, it demonstrates in order.

  The straightness determination unit 511 removes the outlier based on the straightness of the connected optical flow. If a feature point is tracked correctly, the locus will be a straight line as long as the point moves in a uniform linear motion in a three-dimensional space. In general, the constant velocity linear motion model is a good approximation when the position of a moving object is observed at sufficiently short time intervals. Therefore, it is effective for removing outliers to determine whether or not a plurality of single optical flows are arranged linearly.

FIG. 7 is a diagram illustrating an example of an image in which an outlier is likely to occur. As shown in FIG. 7, if there are trees in the image and the image contains many similar leaves and branches, that is, if the image contains multiple similar patterns, multiple frames are supported. When a feature point is detected, points FP _t1 , FP _t2 , FP _t3 , and FP _t4 that do not originally correspond are detected as corresponding feature points. Such feature points and the optical flow connecting them are outliers. At this time, when the connected optical flow COF of the feature points FP _t1 , FP _t2 , FP _t3 , and FP _t4 detected as the outliers is obtained, as shown in FIG. 7, the difference between the connected optical flow COF and each single optical flow is obtained. Becomes larger.

  In order to determine and remove such outliers, the straightness determination unit 511 extracts a component orthogonal to the connected optical flow of each single optical flow, and quantifies the degree of straightness from the extracted component. The outlier is determined by comparing the value with the threshold. Specifically, the straightness determination unit 511 determines, for each feature point, whether or not each single optical flow constituting the connected optical flow is linear in the following manner.

Now, singly optical flows constituting the n-linked optical flow are time-sequentially set (V _x (i), V _y (i)), i = 1, 2,..., N, and the n-linked optical flow is (V _x ( 1: n) and V _y (1: n)). First, the straightness determination unit 511 calculates a unit vector orthogonal to the connected optical flow. Specifically, this unit vector is given by (V _y (1: n), −V _x (1: n)) / sqrt (V _x (1: n) ² + V _y (1: n) ² ). .

  Next, the absolute value of the inner product of each unit optical flow constituting the n-connected optical flow and this unit vector is calculated, the sum of n values is taken, and this value is used as a linearity index. The straightness determination unit 511 compares this straightness index with a predetermined threshold value, and excludes feature points having a linearity index of connected optical flow larger than the threshold value and the connected optical flow as outliers.

  The TTC similarity determination unit 512 removes outliers based on the similarity of the collision times of a plurality of connected optical flows having different numbers of connected single optical flows for the same feature point. The TTC similarity determination unit 512 calculates a collision time (TTC: Time To Collision) based on the connected optical flow of each feature point. TTC refers to the time until a point on the image plane reaches the image plane when a three-dimensional point is approaching the camera. In calculating the collision time, the image plane has an infinite extent. Also, TTC when the point moves away from the camera takes a negative value.

ここで、ｆはカメラ１０の焦点距離である。速度の二乗の項は無視できるオーダーであると考えると、下式（７）を得る。

ここで、消失点のｘ座標ｘ_∞ は、時間を無限にさかのぼった（点が接近している場合）ときに収束する位置であるため、下式（８）のように表される。

Now, the two-dimensional vector representing a consolidated optical flow and u, respectively the x and y components, u _x, When u _y, satisfies the ^{_{u 2 = u x 2 + u}} y 2, u x is the three-dimensional It is expressed by the following equation (6) by the coordinates X, Z and the speeds ΔX, ΔZ.

Here, f is the focal length of the camera 10. Considering that the term of the square of velocity is an order that can be ignored, the following equation (7) is obtained.

Here, the x-coordinate x _∞ of the vanishing point is a position that converges when the time goes back infinitely (when the point is approaching), and is expressed as the following equation (8).

式（７）と式（９）を連立して解くと、下式（１０）が得られる。

ここで、ｐは特徴点の画像座標であり、ｐ_∞は点の軌跡の消失点の画像座標であり、「・」は内積演算子である。 Here, when TTC is expressed as ΔT, ΔT is given by Z / (− ΔZ) (ΔZ <0 when the feature points approach). Since the same derivation can be performed for the y component u _y of the optical flow, the following equation (9) is obtained.

When the equations (7) and (9) are solved simultaneously, the following equation (10) is obtained.

Here, p is the image coordinate of the feature point, p _∞ is the image coordinate of the vanishing point of the locus of the point, and “·” is an inner product operator.

The TTC similarity determination unit 512 calculates the collision time ΔT _n from the n-linked optical flow (V _x (1: n), V _y (1: n)) of each feature point using Equation (10). The TTC similarity determination unit 512 calculates ΔT _n−1 , ΔT _n−2 ,... Of the same feature points by the same method as ΔT _n . When a feature point is correctly tracked, ΔT _n , ΔT _n−1 ,... Are equal to each other as long as the point is in a constant velocity linear motion in a three-dimensional space. As described above, when a moving body is observed at a sufficiently short time interval, the constant velocity linear motion model is a good approximation, so that ΔT _n , ΔT _n−1 ,... Can be said to be similar to each other. When the similarity of ΔT _n , ΔT _n−1 ,... Is broken, it is a result of the point being mistakenly tracked and can be determined to be an outlier.

The TTC similarity determination unit 512 obtains the similarity of the collision times ΔT _n , ΔT _n−1 ,... For each of the feature point groups constituting the n-connected optical flow, and compares it with a predetermined threshold value. Feature points that are smaller than the threshold and their connected optical flows are excluded from the temporary group. Specifically, the grouping unit 50 quantifies the deviation of TTC as D _j = | ΔT _nj −ΔT _n | (j = 1,..., N−1), and _j satisfies D _j > D _th. If there is one, the feature point and its connected optical flow are excluded from the group. Here, D _th is a predetermined threshold value.

  The group determination unit 52 includes a plurality of features in which the extended connected optical flow converges to one vanishing point among a plurality of feature points remaining after the connected optical flow is removed as an outlier by the outlier removal unit 51. The points are grouped as a plurality of feature points on the moving body. A feature point of a plurality of connected optical flows grouped into one group is a feature point of one moving body.

  As described above, the grouping unit 50 detects moving objects by performing grouping of feature points while removing outliers. Note that the grouping unit 50 may remove the outliers by only one of the method based on the linearity and the method based on the TTC similarity.

  The result output unit 60 receives the result of grouping by the grouping unit 50 and outputs the result. Specifically, the result output unit 60 superimposes and displays the grouped feature points, that is, the frame surrounding the moving body, on the image captured by the camera 10. The grouping result may be used for another process such as calculation of the degree of risk that the vehicle will collide with the moving object.

  As described above, according to the moving object recognition system 101 of the present embodiment, feature points are extracted to calculate a connected optical flow, and the straightness determination unit 511 outputs an outlier based on the straightness of the connected optical flow. Therefore, it is possible to recognize a moving object by effectively removing feature points detected in error. Furthermore, since the TTC similarity determination unit 512 removes outliers based on the TTC similarity, it is possible to effectively remove feature points detected in error and recognize the moving object.

  Note that the outlier removal by the straightness determination unit 511 may be performed at any stage after calculating the connected optical flow. For example, the outlier of the straightness determination unit 511 is immediately after the connected optical flow calculation unit 20. Removal may be performed.

(Second Embodiment)
FIG. 8 is a block diagram illustrating a mobile object recognition system according to the second embodiment. In the mobile object recognition system 102 in FIG. 8, the same components as those in the mobile object recognition system 101 according to the first embodiment are denoted by the same reference numerals, and description thereof is omitted. The moving body recognition system 102 according to the present embodiment includes a camera 10, a connected optical flow calculation unit 20, a rotational movement amount / vanishing point estimation unit 30, a background point removal unit 40, a grouping unit 50, and a result output unit 60. Yes.

  The grouping unit 50 includes an outlier removal unit 51 and a grouping unit 52 as in the first embodiment. The outlier removal unit 51 of the present embodiment includes a vanishing point range calculation unit 513, a vanishing point match determination unit 514, and a TTC match determination unit 515. The vanishing point range calculation unit 513 calculates a possible range of vanishing points for each feature point. FIG. 9 is a diagram for explaining calculation of the vanishing point range. As described above, the vanishing point is obtained as a point where the optical flows of a plurality of feature points are extended to intersect each other, but by defining the horizon in the image, the optical flow intersects with the horizon. Can also be a vanishing point.

The vanishing point range calculation unit 513 sets a circle having a predetermined radius with the end point of the connected optical flow v ₀ as the center as the error range R, and the motion vector having the end point in the error range R and having the maximum inclination Set v _a and a motion vector v _b with the minimum slope. If the motion vectors v _a and v _b are extended and the points that can intersect the horizon are point a and point b, the vanishing point range calculation unit 53 sets the range from point b to point a as the vanishing point range. And The vanishing point range calculation unit 513 calculates a vanishing point range for each feature point.

  The vanishing point match determination unit 514 determines and removes feature point outliers based on the vanishing point range of each feature point. FIG. 10 is a graph showing a vanishing point range of a plurality of feature points. In FIG. 10, the horizontal axis is the feature point number, and the vertical axis is the vanishing point range (x coordinate). The vanishing point match determination unit 54 groups feature points whose vanishing point ranges overlap with vanishing point ranges of other feature points, and the feature points that are not grouped, that is, the vanishing point range is the vanishing point range of other feature points. A feature point that does not overlap is determined to be an outlier and is removed. When the number of feature points to be grouped falls below a predetermined value (for example, 3), all the feature points of such a group are removed so that the group determination unit 52 does not perform grouping.

  The TTC coincidence determination unit 515 determines and removes feature point outliers based on the TTC range of each feature point. Using the vanishing point range of each feature point calculated by the vanishing point range calculation unit 153, the TTC match determination unit 55 calculates the TTC range of each feature point using Equation (10). FIG. 11 is a graph showing a TTC range of a plurality of feature points. In FIG. 11, the horizontal axis is the feature point number, and the vertical axis is the TTC range. The TTC coincidence determination unit 515 groups feature points whose TTC range overlaps with the TTC range of other feature points, and features that are not grouped, that is, features whose TTC range does not overlap with the TTC range of other feature points. Judge the point as an outlier and remove it. When the number of feature points to be grouped falls below a predetermined value (for example, 3), all the feature points of such a group are removed so that the group determination unit 52 does not perform grouping.

  As in the first embodiment, the grouping unit 52 has an extended connected optical flow among the plurality of feature points remaining after the connected optical flow is removed as an outlier by the outlier removing unit 51. A plurality of feature points that converge to one vanishing point are grouped as a plurality of feature points on the moving object. A feature point of a plurality of connected optical flows grouped into one group is a feature point of one moving body.

  As described above, in the mobile object recognition system 102 according to the second embodiment, the vanishing point range of the connected optical flow is obtained for each feature point, and further, the TTC range is obtained from the vanishing point range. Since features that do not overlap with other feature points are determined to be outliers and removed, feature points detected in error can be effectively removed to recognize a moving object.

  The grouping unit 50 may remove the outlier only by either the vanishing point match determination by the vanishing point match determination unit 514 and the TTC match determination by the TTC match determination unit 515. Further, in addition to the configuration of the second embodiment described above, the outlier removal based on the straightness of the first embodiment and / or the outlier removal based on the TTC similarity may be performed. .

  The present invention eliminates the outlier by using the connected optical flow, and thus has the effect of being able to perform mobile object recognition with a small amount of computation and less misrecognition. It is useful as a mobile object recognition system that recognizes

DESCRIPTION OF SYMBOLS 101,102 Mobile body recognition system 10 Camera 20 Connection optical flow calculation part 30 Rotation movement amount and vanishing point estimation part 40 Background point removal part 50 Grouping part 51 Outlier removal part 511 Straightness judgment part 512 TTC similarity judgment part 513 Vanishing point Range calculation unit 514 Vanishing point match determination unit 515 TTC match determination unit 52 Group determination unit 60 Result output unit

Claims (9)

A camera installed in a vehicle and taking images of a plurality of consecutive frames;
Linking a plurality of single optical flows connecting the extracted feature points of adjacent frames by extracting feature points corresponding to each other between the plurality of frame images from each of the plurality of frame images. A consolidated optical flow calculation unit for calculating an optical flow;
An outlier removal unit that removes an outlier using the connected optical flow;
Among the plurality of feature points remaining after the connected optical flow is removed as an outlier in the outlier removal unit, the plurality of feature points where the extended connected optical flow converges to one vanishing point is moved. A group determination unit for grouping as a plurality of feature points on the body;
A mobile object recognition system comprising:   The mobile body recognition system according to claim 1, wherein the outlier removal unit removes the outlier based on the straightness of the connected optical flow.   The outlier removal unit removes the outlier based on the similarity of the collision times of a plurality of connected optical flows having different numbers of connected single optical flows for the same feature point. The moving body recognition system according to claim 1.   The outlier removal unit obtains a vanishing point range that each of the plurality of connected optical flows can take, and outputs a connected optical flow in which the vanishing point range does not overlap with a vanishing point range of another connected optical flow. The moving body recognition system according to claim 1, wherein the moving body recognition system is removed as a liar.   The outlier removal unit obtains, as the vanishing point range, a possible range of an intersection of the connected optical flow and an extension line of a vector within an error range thereof and a horizon defined in the image. The moving body recognition system according to claim 4.   The outlier removal unit obtains a collision time range that each of the plurality of connected optical flows can take, and a connection optical flow that does not overlap with a collision time range of another connected optical flow is defined as the outlier. The moving body recognition system according to claim 1, wherein the moving body recognition system is removed.   The outlier removal unit obtains, as a vanishing point range, a range that can be taken by an intersection of the connected optical flow and a vector extension line within the error range and a horizon defined in the image, and the vanishing point range. The moving object recognition system according to claim 6, wherein a range of the collision time is obtained based on the connection optical flow. Computer
The feature points corresponding to each other between the images of the plurality of frames are extracted from each of the images of the plurality of consecutive frames photographed by the camera installed in the vehicle, and the feature points extracted from adjacent frames are extracted. A consolidated optical flow calculation unit for calculating a consolidated optical flow by connecting a plurality of single optical flows connecting
An outlier removal unit that removes an outlier using the connection optical flow, and the extended feature points of the plurality of feature points remaining after the connection optical flow is removed as an outlier in the outlier removal unit. A group determining unit for grouping a plurality of feature points at which the connected optical flow converges to one vanishing point as a plurality of feature points on the moving body;
A moving body recognition program characterized by functioning as A shooting step of shooting images of a plurality of consecutive frames with a camera installed in the vehicle;
Linking a plurality of single optical flows connecting the extracted feature points of adjacent frames by extracting feature points corresponding to each other between the plurality of frame images from each of the plurality of frame images. A consolidated optical flow calculation step for calculating an optical flow;
An outlier removal unit that removes an outlier using the connected optical flow;
Among the plurality of feature points remaining after the connected optical flow is removed as an outlier in the outlier removal unit, the plurality of feature points where the extended connected optical flow converges to one vanishing point is moved. A group determination step for grouping as a plurality of feature points on the body;
A moving body recognition method comprising:

Images