JP3452188B2 - Tracking method of feature points in 2D video - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for tracking feature points in a two-dimensional moving image such as a video camera. 2. Description of the Related Art A moving image of a stationary object captured by a video camera or the like contains important information relating to the movement of an observer and the structure of a target. For example, if data is taken by moving around a building, which is a video camera while walking in a city, data is acquired, and a three-dimensional structure can be restored later by a computer and viewed from any angle. Therefore, restoration of a three-dimensional structure from such a two-dimensional moving image is one of the important issues of computer vision, and if this technology is established,
It can be applied to many fields such as three-dimensional modeling, tracking, passive navigation, and robot vision. [0003] Research in this area can be broadly classified into two types. One is a means for using the correspondence between points on the image obtained at different times, and the other is a means for using the speed (optical flow) on the image. The means of using optical flow compared to the former is
(1) The optical flow can be obtained more easily than the corresponding point on the image, and (2) the speed can be obtained from the optical flow but cannot be obtained from the corresponding point. Further, as means for restoring an object structure from an optical flow on an image, there are (1) a method using a parallel projection image and (2) a method using a perspective projection image. The former is an approximation of the latter, and this approximation only holds when the object is far from the camera. Therefore, the latter means using the perspective projection image can obtain higher accuracy. As a means for restoring the structure of an object from an optical flow of a perspective projection image, it has conventionally been necessary to solve a system of nonlinear equations by an iterative method unless special assumptions are made. A special assumption is that when the point to be observed lies on a plane, the movement is only rotation or only translation. Therefore, in general, it is necessary to solve a system of nonlinear equations. However, in this case, there are problems that the uniqueness of the solution is not guaranteed and that a search by an iterative method is required. [0006] In order to solve these problems, the inventors of the present invention have developed a structure simply by solving a linear equation using an optical flow image obtained by perspective projection from a point performing rigid motion. A method for restoration is proposed (“Reconstruction of 3D motion parameters and structure by linear calculation from optical flow image”, Transactions of the Society of Instrument and Control Engineers, Vol.3
4, No. 5, 438/444 (1998)). According to this method, it is not necessary to solve the nonlinear equation, the uniqueness of the solution is guaranteed, and the accuracy can be easily improved by increasing the number of observation points. As described above, tracking the movement of a feature point (optical flow) through a moving image requires the reconstruction of an object, pattern recognition, and calibration for obtaining a camera position. It is extremely important as preprocessing. However, the tracking of the optical flow has conventionally been performed using the correlation of the area around the characteristic point before and after the short time has elapsed, and thus there has been a problem that the tracking accuracy is low. For example, FIG. 7 shows an image of a cube at a certain moment.
I _A, Figure 8 is an image I _B after the short, 9, 10
Are images obtained by performing image processing on the images in FIGS. The numbers in the figure are the corresponding points of the image I _B obtained by correlation with the feature points of the image I _A. As seen from comparison between FIGS. 7 and 9 and Figure 8 and 10, a point on the image I _A 8,13,21 (FIG. 1
(Indicated by an arrow at 0)). As described above, when a corresponding point on a moving image is obtained using only the correlation, an erroneous correspondence occurs (correspondence is mistaken) as shown in FIG. In some cases, the position accuracy was poor. When these problems occur, the accuracy of pre-processing such as restoration of three-dimensional structures (three-dimensional modeling, tracking, passive navigation, and robot vision) using the above-mentioned optical flow, pattern recognition, and calibration for obtaining camera positions is improved. And the problem of reduced usefulness arises. [0010] Note that the term "correlation" is an area corresponding points of the feature points in the image I _A is expected to be present in the image I _B taken in size, most point image is closer by comparing the image (Point at which the correlation becomes maximum) is referred to as a corresponding point. The present invention has been made to solve the above-mentioned problems. That is, the object of the present invention is:
Provided is a tracking method of a feature point in a two-dimensional moving image, which can correct an error of a corresponding point that can occur when a corresponding point on a moving image is obtained using only correlation and can improve the position accuracy of the corresponding point. It is in. It is difficult to solve the above-mentioned problems only by image processing. However, these disadvantages can be avoided by supplementing the information by adding an angular velocity sensor to the camera. The present invention is based on such a new idea. That is, according to the present invention, a sensor for measuring an angular velocity is integrally attached to a camera for capturing a moving image of a stationary object, and the moving image and the angular velocity data are recorded in synchronization with each other. A method of tracking feature points in a two-dimensional moving image, characterized by tracking characteristic points in a two-dimensional moving image by using the obtained moving image and angular velocity data in the following steps (S1 to S5). (S1) and the instant of the image I _A with the in the moving image data,
The next moment the image I _B of the image I _A, rotates the image I _A → image I as much image I _A and the rotation included in the _B and the image I _A 'by using the angular velocity data. (S2) for each of the feature points of the image I _A ', finds the corresponding points in the image I _B by correlating a two-dimensional region. (S3) The procedure from the following is repeated many times, and the error r _i ²
Is the focal FOE selected at this stage when the median value of is the smallest. Randomly select p corresponding points.
A focus FOE candidate is obtained by minimizing the evaluation function E (x ^FOE ) for the selected point. Where the evaluation function E
The (x ^FOE), the point x ^FOE and the image I _A 'feature points of the image
'Draw a straight line passing through the center of gravity g _i and the point of image I _B corresponding to the feature points of the straight line and the image I _A' I _A feature point or image I _B of
The sum of the squares of the distances to the corresponding points of the P corresponding points selected at random is used. The error r _i ² (i = 1, 2,..., P) of the focus FOE candidate is calculated. Here, the error is obtained for the candidate focus FOE, using the square of the distance between corresponding points of the feature point or image I _B of the straight line and the image I _A '. (S4) A corresponding point having a small error r _i ² is selected from the whole with respect to the focus FOE candidate, and a more accurate focus FOE is obtained again by using E (x ^FOE ). (S5) by correlating on a straight line connecting the feature points in the focal FOE image I _A ', finds the corresponding points in the image I _B more accurately. According to the method of the present invention, first, a moving image of a stationary object is photographed while moving a camera provided with an angular velocity sensor, and the image and the output of the angular velocity sensor are captured in synchronization. When considering tracking to this from the moment of the image I _A with a moving picture next moment an image I _B, since the angular velocity data is obtained from the output of the angular velocity sensor, or camera is rotated much by integrating the angular velocity Is required. Using this value, rotate the image I _A obtains a new image I _'A. Then, the image I _A instead of obtaining the movement of the points of the image I _B, obtains the movement of a point of the image I _B from the image I _'A. [0015] While use of correlation as in the conventional to determine the candidate points in I _B corresponding to the feature point in the I _'A, this corresponds to the often contains incorrect correspondence. Therefore,
Since the movement of the image I _'A to image I _B is in a translation by the flow of the "points theoretically FOE (Focus of Expan
at a single point called the sion). Based on this property, a focus FOE of an optical flow of a moving image is obtained by a robust method using candidate corresponding points, and then the image I ' _A
By correlating the original point on a line connecting the feature points and the focal FOE in, it is possible to determine the corresponding points in the image I _B more accurately. Embodiments of the present invention will be specifically described below. Figure 1 is an image of obtaining the optical flow from image I _B of the moment of the next image I _A moment with the moving image of the object is stationary. That is, in FIG.
White line indicates the optical flow in the image I _B from the image I _A. Note that FIG. 2 is obtained by legible by image processing the image of FIG. 1, the optical flow from image I _A to the image I _B is indicated by a black line. According to the method of the present invention, a sensor (for example, a gyro sensor) for measuring an angular velocity is integrally attached to a camera for capturing a moving image of a stationary object, and the moving image and the angular velocity data are recorded in synchronization. I do. Since the angular velocity can be obtained from the sensor (gyro sensor),
By integrating the angular velocity, it is possible to determine how much the camera has rotated. Using this value, rotate the image I _A obtains a new image I _'A. Then, obtained from the image I _A instead of obtaining the movement of the points of the image I _B, the image I _'A movement of a point of the image I _B (the optical flow). This optical flow is shown by a thin black line in FIG. 1 and a broken line in FIG. [0019] Since the movement of the points from the image I _'A to image I _B such translation only intersect at Think Theoretically infinity position in the three-dimensional space. It intersects at a certain point on the image, as in perspective. This point is due to the FOE (Focus of
Expansion). That is, the image is rotated I _A to 'seek _A, image I' new image I by only translate the optical flow in the image I _B from _A, with the corresponding point candidate, robust (robust, FOE can be obtained in a robust manner. Then, by taking the correlation between the original point on a line connecting the feature points and the FOE in the image I _'A, we obtain the corresponding points in the image I _B more accurately. FIG. 3 is a flowchart illustrating the method of the present invention. The method of the present invention will be described below in detail with reference to FIG. [0021] Description Introduction The image processing method of the image I _A → image I picture I _A is subjected to rotation applied at _B in step 1 of FIG. 3 (method performs rotation to the image) (S1) in the image I _'A I do. The image processing by the motion of the image I _B from the image I _'A (optical flow) rotation is removed, the translation only. The position of a feature point in an image is set to u = (u, v) ^T.
If this is represented by homogeneous coordinates, m = (u, v, 1) ^T. That is,
When u ₁ = m ₁ / m ₃ and u ₂ = m ₂ / m ₃ , m is u
Are homogeneous coordinates. Using the homogeneous coordinates, the vector from the camera center to the three-dimensional position of the feature point can be written as equation (1). _{Here, u 0, v 0, f} x, f y is the specific parameters to the camera, and has been obtained in advance by the calibration of the camera. By applying a matrix R representing the rotation of the camera obtained from the gyro sensor to this vector q and returning it as an inverse matrix P ⁻¹ of P, homogeneous coordinates m ′ with the rotation added can be obtained. That is, equation (2) holds. Note that the inverse matrix P ⁻¹ of P is represented by Expression (3). From homogeneous coordinates m ',
The position u 'of the rotated feature point is u ₁ ' = m ₁ '/ m ₃ ', u ₂ '= m ₂ '
It is obtained from the homogeneous coordinate relationship of / m ₃ '. ## EQU1 ## [0024] Step 2 in (S2) in FIG. 3, for each point in the I _A ', finds the corresponding points in the I _B by correlating a two-dimensional region. As shown in FIGS. 8 and 10, in this correlation, almost correct correspondence is obtained, but there may be erroneous correspondence and poor position accuracy. Next, in S3, F
An evaluation function is introduced to obtain the OE, and the LMedS method (Least Median of S
quares method). (Evaluation Function) FIG. 4 is an explanatory diagram of the evaluation function. FOE from point correspondence
As an evaluation function for obtaining a, as schematically shown in this figure, the center of gravity g of the corresponding points in the I _B and I _A 'from the candidates of the FOE
a straight line is drawn to _i, used after taking the sum with respect to three or more corresponding points of the square of the distance between corresponding points of the straight line and the image I _B. X _{B; i} is positive for all i considered
If it is determined properly , the straight line connecting x'A _{; i} and xB _{; I}
Since all the points intersect at one point, the point where this evaluation function is 0 is positive.
New FOE. If there is an erroneous response, set the evaluation function to 0
However, the point to be minimized is a candidate for FOE. The distance between the corresponding point and the straight line l _i of the image I _B image
It is equal to the distance between the corresponding points and the straight line I _A ', is used only in the evaluation function one. The position of the corresponding point i of the image I _A '
x _{'A; i,} position x _B of the corresponding point i of the corresponding image I _{B; i,} the point x'
Assuming that the center of gravity of _{A; i} and point xB _{; i} is g _i , the evaluation function is represented by Expression (4). Where l _i (x ^FOE , g _i ) is the point x ^FOE and the center of gravity g
a straight line passing through ^{_{i, dist 2 (x B;}} i, l i) the point x _{B; i} and the line l _i
Is the square of the Euclidean distance to Let n _{i be} the normalized normal vector of l _i and a _i = x _{B; i} -g _i ,
Expressions (5) and (6) are obtained. [Mathematical formula-see original document] [0028] determine the x ^FOE to minimize the evaluation function E (x ^FOE). For this purpose, some numerical calculation method such as a conjugate gradient method is used. This requires a gradient, which can be calculated as follows: That is, Expression (7) and Expression (8)
Then, the gradient can be expressed by Expressions (9) and (10). [Equation 3] (LMedS Method for Dealing with Error in Correspondence) Next, in order to correctly obtain the FOE when the correspondence between points is incorrect, the LMedS method (Least Median of Squares method)
Introduce d). From the whole correspondence of the points obtained by the first matching, a part of the correspondence of the p points (for example, 5 points) is randomly taken out, and this is taken as ｛x ₁ ^sub , x ₂ ^sub , ..., x _p
^sub ｝. Also, x _i ^sub in image I ′ _A is x _{A; i} ^sub , image
The x _i ^sub in I _B x _B; denoted as _i ^sub. For this subset, the FOE candidate x is calculated using the evaluation function E (x ^FOE ) described above.
^Ask for ^FOE . An error {r ₁ ² , r ₂ ² ,..., R _p ² } of each point in the subset is obtained for this candidate. Here, equation (11) holds. g _i ^sub is the center of gravity of x _{A; i} ^sub and x _{B; i} ^sub . x _{B; i} ^sub (i = 1,2, ..., p)
If it is found more correctly, the errors r _i ² are all 0 and
Become. If there is an erroneous response, it does not become 0, but LMe
Based on the dS method, when the median of the error r _i ² is minimized
The obtained FOE is the FOE obtained robustly. There
Then, this action is repeated many times, and x ^FOE when the median of r _i ² is minimized, that is, x ^FOE when equation (12) is minimized, is adopted as the robustly obtained FOE. [Equation 4] In step 3 (S3) of FIG. 3, a result from which a corresponding error is eliminated is obtained. However, the position error is not an optimal value because few points are used. Therefore, in step 4 (S4), a correspondence that does not contradict the FOE obtained above is selected.
Find E. Specifically, r _i ² is calculated for all point correspondences, and E (x ^FOE ) is again calculated for the selected correspondence.
X ^FOE that minimizes This is the final F
OE. [0033] (matching with FOE) FOE and image I 'a point in _A x'_A; there should _{i also; i} and a point in the image I _B on the straight line connecting x _B. All points are FOE
Or x _{A; i} 'on the same side of the straight line. Therefore, in step 5 (S5), the average value and the standard deviation of the flow size are obtained for the pair of points used when finally obtaining the FOE, that is, the pair considered to be almost correctly obtained, and the average value is obtained. Matching is performed on a straight line while moving the range before and after within a constant multiple of the standard deviation (usually about three times). However, since this range does not theoretically exceed x _{A; i} ′ or x ^FOE , if the range exceeds this, no matching is performed for the excess part. Thereby, the final corresponding point xB _{; i} is obtained. [0034] As EXAMPLES image I _A using the image shown in FIG. If determined corresponding points in the I _B using only correlation without using gyro sensors, it occurs erroneous correspondence as shown in FIG. 8 and 10. On the other hand, when a correspondence was obtained by the method of the present invention using a gyro sensor, a correct corresponding point could be obtained as shown in FIGS. It should be noted that the present invention is not limited to the above-described embodiment, but can be variously modified without departing from the gist of the present invention. Further, the line drawings shown in FIGS. 2, 6, 9, and 10 are for directly viewing the images and are not directly related to the present invention. As described above, according to the method for tracking feature points in a two-dimensional moving image of the present invention, errors in corresponding points that may occur when corresponding points on a moving image are obtained using only correlations. Was corrected,
In addition, there is an excellent effect that the positional accuracy of the corresponding point can be improved.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a halftone image I _A on the moment of the display with a stationary object. FIG. 2 is a halftone image on a display obtained by performing image processing on FIG. 1; FIG. 3 is a flowchart illustrating a method of the present invention. FIG. 4 is an explanatory diagram of an evaluation function. 5 is a halftone image I _B on the display by shortly after the invention of the image I _A. 6 is a halftone image on a display obtained by performing image processing on FIG. 5; 7 is a halftone image I _A on the moment of the cube in the display. 8 is a halftone image I _B on shortly after the display of the image I _A. FIG. 9 is a halftone image on the display obtained by performing image processing on FIG. 7; 10 is a halftone image on the display obtained by performing image processing on FIG. 9;

──────────────────────────────────────────────────の Continuing on the front page (72) Inventor Noboru Noboru 2271-130 Science Park Research and Development Center, Shimo-Shidami Tateji, Moriyama-ku, Nagoya-shi, Aichi Pref. RIKEN Bio-Mimetic Control Research Center (56) Reference Document JP-A-9-249083 (JP, A) JP-A-9-212648 (JP, A) JP-A-9-86314 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06T 7/00-7/60 G06T 1/00 H04N 5/232 H04N 7/18

Claims (1)

(57) [Claims] [Claim 1] A sensor for measuring an angular velocity is integrally attached to a camera for capturing a moving image of a stationary object,
The moving image and the angular velocity data are recorded in synchronization with each other, and using the obtained moving image and angular velocity data, the following steps (S
A method of tracking feature points in a two-dimensional moving image, characterized by tracking feature points in a two-dimensional moving image according to 1 to S5). (S1) and the instant of the image I _A with the in the moving image data,
The next moment the image I _B of the image I _A, rotates the image I _A → image I as much image I _A and the rotation included in the _B and the image I _A 'by using the angular velocity data. (S2) for each of the feature points of the image I _A ', finds the corresponding points in the image I _B by correlating a two-dimensional region. (S3) The procedure from the following is repeated many times, and the error r _i ²
Is the focal FOE selected at this stage when the median value of is the smallest. Randomly select p corresponding points.
A focus FOE candidate is obtained by minimizing the evaluation function E (x ^FOE ) for the selected point. Where the evaluation function E
The (x ^FOE), the point x ^FOE and the image I _A 'feature points of the image
'Draw a straight line passing through the center of gravity g _i and the point of image I _B corresponding to the feature points of the straight line and the image I _A' I _A feature point or image I _B of
The sum of the squares of the distances to the corresponding points of the P corresponding points selected at random is used. The error r _i ² (i = 1, 2,..., P) of the focus FOE candidate is calculated. Here, the error is obtained for the candidate focus FOE, using the square of the distance between corresponding points of the feature point or image I _B of the straight line and the image I _A '. (S4) A corresponding point having a small error r _i ² is selected from the whole with respect to the focus FOE candidate, and a more accurate focus FOE is obtained again by using E (x ^FOE ). (S5) by correlating on a straight line connecting the feature points in the focal FOE image I _A ', finds the corresponding points in the image I _B more accurately.