JP2017151797A - Geometry verification device, program and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a geometry verification device capable of taking validity of a generated sample into account for geometry verification of an epipolar geometry model.SOLUTION: A geometry verification device 10 includes a geometry model generation part 22 which generates a matrix as an epipolar geometry model based on a main point of a first camera and a second camera based on a local characteristic correspondence of a given correspondence between a local characteristic extracted from a first image that is generated by the first camera and a local characteristic extracted from a second image that is generated with the second camera, a parameter extraction part 23 for extracting a parameter related to the first camera and the second camera from the matrix, and an evaluation part 24 which evaluates validity of the matrix as the epipolar geometry model based on the parameter.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a geometric verification apparatus, program, and method for performing geometric verification using epipolar geometry as a model between two images of the same three-dimensional object.

  Conventionally, as disclosed in Non-Patent Document 1, for example, there is a technique for searching for an object using local features extracted from an image. According to this technique, an object is photographed by a camera, and local features are extracted from a photographed image (question (query) image) in which the object is photographed. Then, the object is searched by comparing the local feature of the question image with the local feature of an image (reference (reference) image) in which an object registered in advance is captured. In addition, as disclosed in Non-Patent Document 2 and the like, a search is performed by performing geometric verification between a reference image and a query image having the highest N similarity (N is a predetermined number). Increases accuracy. Then, the number of feature point correspondence groups (inliers) satisfying the conditions in the geometric verification is set as a new inter-image similarity.

  On the other hand, there is a case where it is desired to determine whether or not the same object is included in both the reference image and the question image using a threshold value based on the similarity. This is because the higher the threshold of similarity between images, the higher the recognition precision.

  The geometric constraint model that can be used in geometric verification differs depending on the target object. That is, for a planar object or a three-dimensional object having a partially planar shape, a homography matrix (Homography matrix) representing a planar projection relationship can be used. On the other hand, epipolar geometry, that is, a fundamental matrix or an essential matrix is used as a geometric constraint model for a three-dimensional object.

  Here, the epipolar geometry is a geometry that represents a relative posture between cameras that capture two images. The base matrix can be used when the camera is calibrated, and the base matrix can be used even when the camera is uncalibrated. For example, in Patent Document 1, a basic matrix is used for geometric verification of a three-dimensional object photographed by an uncalibrated camera.

  When an outlier is included in the point correspondence set of local features, geometric verification is performed using a robust estimation method. This is a method of performing estimation so that the influence of the outlier is minimized even when the outlier is included in the measured value. For example, in Non-Patent Document 1, a RANSAC (RANdom SAmple Consensus) algorithm is used.

  In geometric verification using the RASNAC algorithm, samples are randomly extracted from all point correspondence sets, a temporary geometric model is generated from the samples, and the number of point correspondences (inliers) along the geometric model is repeatedly determined. . The maximum number of inliers obtained in all iterations is the geometric verification score.

JP-A-2015-201123

Philbin. J, Chum. O, Isard. M, Sivic. J, and Zisserman. A, "Object retrieval with large vocabularies and fast spatial matching," In Computer Vision and Pattern Recognition, pp. 1-8, 2007. Jegou, Herve, Matthijs Douze, and Cordelia Schmid. "Hamming embedding and weak geometric consistency for large scale image search." Computer Vision-ECCV 2008. Springer Berlin Heidelberg, 2008. 304-317. Kento Yamada, et al. "Latest algorithm for 3D reconstruction from images." IPSJ SIG (2009): 1-8.

  However, in the conventional technique described above, the validity of the sample that generated the epipolar geometry has not been considered.

  An inlier to be obtained by geometric verification is a correspondence between points at the same position of an object. Epipolar constraints are ambiguous because they are constraints based on the distance between epipolar lines and points, that is, constraints between straight lines and points. Therefore, even a correspondence (outlier) of a point that is not at the same position of the object may meet the constraint condition by chance and be erroneously determined as an inlier. As a result, in the prior art, a high geometric verification score may be given to an incorrect image pair.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a geometric verification apparatus, program, and method that can solve the above-described problems of the prior art and can consider the validity of a sample that has generated epipolar geometry.

  In order to achieve the above object, the present invention provides a geometric verification device, which comprises a local feature extracted from a first image generated from a first camera and a local feature extracted from a second image generated from a second camera. A geometric model generation unit that generates a matrix as an epipolar geometric model based on the principal points of the first camera and the second camera based on local feature correspondence to which correspondence is given between the features, and the first from the matrix A parameter extraction unit that extracts parameters related to the camera and the second camera, and an evaluation unit that evaluates the validity of the matrix as the epipolar geometric model based on the parameters.

  Further, the present invention is a program for causing a computer to function as the geometric verification device.

  Further, the present invention is a geometric verification method, wherein a local feature extracted from a first image generated from a first camera and a local feature extracted from a second image generated from a second camera are used. A geometric model generation step of generating a matrix as an epipolar geometric model based on principal points of the first camera and the second camera from the local feature correspondence to which the correspondence is given, and the first camera and the second from the matrix A parameter extracting step of extracting a parameter related to the camera; and an evaluation step of evaluating the validity of the matrix as the epipolar geometric model based on the parameter.

  According to the present invention, the validity of a matrix as an epipolar geometric model can be evaluated by evaluation based on the extracted parameters relating to the first camera and the second camera.

It is a functional block diagram of the geometric verification apparatus which concerns on one Embodiment. It is a flowchart of operation | movement of the geometric verification apparatus which concerns on one Embodiment. It is a figure for demonstrating epipolar geometry. It is a figure which shows the pinhole camera model for demonstrating a focal distance evaluation part. It is a figure which classifies and shows the angle which the optical axis of two cameras makes for explaining a positional relationship evaluation part. It is a figure which shows the example of the positive / negative distinction of the depth of a three-dimensional restoration point for demonstrating a restoring property evaluation part. It is a flowchart of operation | movement of the geometric verification apparatus which concerns on one Embodiment. It is a functional block diagram of the geometric verification apparatus which concerns on one Embodiment. It is a flowchart of operation | movement of the geometric verification apparatus which concerns on one Embodiment.

  FIG. 1 is a functional block diagram of a geometric verification apparatus according to an embodiment. As illustrated, the geometric verification apparatus 10 includes a local feature extraction unit 11, a correspondence acquisition unit 12, a weak geometric verification unit 13, an iterative processing unit 20, and an additional processing unit 31. The iterative processing unit 20 includes a selection unit 21, a geometric model generation unit 22, a parameter extraction unit 23, an evaluation unit 24, and a verification unit 25. Further, the evaluation unit 24 includes a focal length evaluation unit 1, a positional relationship evaluation unit 2, a consistency evaluation unit 3, and a restoration property evaluation unit 4.

  FIG. 2 is a flowchart of the operation of the geometric verification apparatus 10 according to an embodiment. Hereinafter, the processing content of each functional unit in FIG. 1 will be described while explaining each step in FIG. 2.

[Step S1: Prepare for local features]
In step S1, the local feature extraction unit 11, the correspondence acquisition unit 12, and the weak geometry verification unit 13 perform each process in this order, so that the correspondence between the local features of both images as an output from the input question image and the reference image After obtaining (corresponding relationship between each local feature of the query image and each local feature of the reference image), the process proceeds to step S2. Therefore, specifically, in step S1, each of the functional units 11 to 13 performs the following processing in this order.

  In step S1, the local feature extraction unit 11 first extracts local features from the input question image and reference image, and outputs them to the correspondence acquisition unit 12. The local feature extraction processing is common to the question image and the reference image, and the following feature point detection processing and local feature extraction processing are performed in this order.

  That is, first, feature points (points such as edges) are detected from the input images (each of the reference image and question image) using a feature point detector. Furthermore, the local feature of the image at the feature point position detected using the feature quantity detector is extracted in a vector format called a local feature. Local feature extraction algorithms include commonly known SIFT (Scale invariant feature transform), SURF (Speeded Up Robust Features), ORB (oriented BRIEF), FREAK (Fast Retina Keypoint), etc. Can be used. These local features are characterized by coordinates x = (u, v), direction θ, scale s, and feature vector f in the image. Normally, a feature group detected from one image is stored in an array on a program (a program for realizing the geometric verification device 10 including the local feature extraction unit 11), and the subscript of the array is used as a feature point. It can be identified as an ID.

  In the present invention, there is a relationship between the question image and the reference image that is the correct answer to the question image, in which both images are obtained by photographing the same three-dimensional object with different camera arrangements. To do. Therefore, it is assumed that there is a relationship given by epipolar geometry between feature points corresponding to the same point of the same three-dimensional object extracted from both the images. Here, with reference to FIG. 3, the well-known epipolar geometry as a relationship between the question image assumed in the present invention and the corresponding correct reference image will be briefly described.

  FIG. 3 shows the case where the question image G1 and the correct reference image G2 are taken from the object OB (for example, a house) as the same three-dimensional object in the three-dimensional space with different camera arrangements. For any one feature point P of the plurality of feature points, the corresponding point P1 on the image G1 and the corresponding point P2 on the image G2 are in a relationship described by epipolar geometry. Here, the lens principal points of the cameras that photograph the images G1 and G2 in different positions are the points C1 and C2, respectively, as shown in the figure. An epipolar plane EP is formed as a plane on which the triangle C1-C2-P rides, the intersections of the straight line C1-C2 and the images G1, G2 are the epipolar points E1, E2, respectively, and the feature points P1, P2 are the straight lines C1- It is the intersection of P and image G1, and the intersection of straight line C2-P and image G2. The straight line E1-P1 and the straight line E2-P2 are referred to as epipolar lines, and there is an ambiguity on the epipolar line when applied to object recognition or the like as described above.

The homogeneous coordinates of the feature points P1 and P2 having such an epipolar geometry relationship are expressed as a 3 × 1 matrix (vertical vector), and x1 = (u ₁ , v ₁ , 1) ^T , x2 = (u ₂ , v ₂ , 1) As ^T (T represents a transpose operation), as is well known, with respect to points P1 and P2 corresponding to the same point P, a basic matrix F = (F _ij ) of size 3 × 3 The epipolar equation (the following equation (1)) is established using (1 ≦ i, j ≦ 3).

  In addition, as is well known, in order to calculate the basic matrix F specifically, seven or eight corresponding pairs of points P1 and P2 corresponding to the same point P are required (different as the point P of the three-dimensional object) 7 or 8 is necessary), and can be calculated using a well-known 7-point algorithm or 8-point algorithm. Hereinafter, the description returns to step S1 in FIG.

  Next, in step S1, the correspondence acquisition unit 12 acquires the correspondence between the local features in both images from the local features of the question image and the reference image obtained as described above in the local feature extraction unit 11, and weak geometry Output to the verification unit 13. Specifically, between each local feature of the query image and each local feature of the reference image, a feature vector among the above-mentioned coordinates x, direction θ, scale s, and feature vector f, which is the constituent data of the local feature Corresponding relations may be set for those having the smallest distance between f. As the distance between the feature vectors f, a predetermined type of distance among Euclidean distance and other arbitrary types of distances may be used. When it is determined that the minimum distance is large in the threshold determination, the correspondence relationship may not be set for the local feature.

  Finally, in step S1, the weak geometric verification unit 13 performs weak geometric verification on the local feature correspondence obtained as described above in the correspondence acquisition unit 12, and the local feature correspondence that passed the verification is repetitively processed 20 To the selection unit 21. That is, the weak geometric verification unit 13 has a function of selecting the local feature correspondence acquired by the correspondence acquisition unit 12 by weak geometric verification, and the local feature correspondence that has not been selected is not output to the selection unit 21.

  For the weak geometric verification process, a well-known process such as that disclosed in Non-Patent Document 2 and the like described above can be adopted, and the direction θ in the local feature not considered in the correspondence acquisition unit 12 and Sorting can be performed based on the scale s. Specifically, it is as follows.

First, as a premise for explanation, in the correspondence acquisition unit 12, the local feature FG1 _i (i = 1, 2,..., N) of the question image G1 and the local feature FG2 _i (i = 1, 2, ..., N) is assumed to have a corresponding relationship. That is, it is assumed that the distance between the corresponding feature vector f1 _i (i = 1, 2,..., N) and the feature vector f2 _i (i = 1, 2,..., N) is minimum. Note that i is an identification index of local features for which correspondence relations are set, and it is assumed that N correspondence relations are set in total.

In the weak geometric verification, as the first procedure, the respective directions θ1 _i (i = 1, 2,..., N) and the directions θ2 _i (i = _i ) in the local features of the query image G1 and the reference image G2 for which the corresponding relationship is set. 1,2, ..., N), scale s1 _i (i = 1,2, ..., N) and scale s2 _i (i = 1,2, ..., N) _i = θ1 _i -θ2 _i (i = 1, 2,..., N) and scale ratio Δs _i = s1 _i / s2 _i (i = 1, 2,..., N) are obtained. Further, as a second procedure, regarding the set of difference of the obtained direction {Δθ _i | i = 1,2, ..., N} and the set of scale ratio {Δs _i | i = 1,2, ..., N}, respectively. The peak value Δθ _peak of the direction difference and the peak value Δs _peak of the scale ratio are obtained by taking a histogram using a predetermined bin or fitting a value distribution with a predetermined function. These peak values Δθ _peak and Δs _peak are called weak geometric parameters.

In the weak geometric verification, as a third procedure, the direction difference Δθ _{i of} the local feature pairs FG1 _i and FG2 _i (i = 1, 2,..., N) for which the correspondence relationship is set is the peak value Δθ _peak . Is within the threshold range (ie, | Δθ _i −Δθ _peak | <THθ due to the threshold THθ), and the scale ratio Δs _i is within the threshold range compared to the peak value Δs _peak (ie, the threshold It is determined by THs that | Δs _i −Δs _peak | <THs) that passed the weak geometric verification, and is output to the selection unit 21.

The significance of the weak geometry verification as described above is as follows, and with regard to the inlier determination performed in step S2 described later, there is an effect of excluding those that are likely to be outliers instead of inliers in advance. is there. That is, when the correspondence of the local features is the correspondence between the inliers, it is considered that the difference in the direction represents the rotation angle of the local region of the object shown in the image, and the scale ratio represents the change in the scale. Therefore, the direction difference and scale ratio extracted from the inliers in the correct image pair represent the rotation angle and scale change of the entire object between the image pairs, respectively, and thus are considered to be within a certain range. It is done. That is, the weak geometric parameters Δθ _peak and Δs _peak described above are considered to reflect such a rotation angle and scale change of the entire object. On the contrary, when the correspondence of the local features is the correspondence between the outliers, the values are considered to be random, and it is highly likely that the values do not fall within the certain range. From these considerations, weak geometry verification is effective for preselection of inliers.

In the above description, selection by weak geometric verification is performed using both the direction difference Δθ _i and the peak ratio Δs _i , but weakness using only one of the direction difference Δθ _{i and the} peak ratio Δs _i. Geometric verification may be performed.

  In the above, step S1 of FIG. 2 was demonstrated. Subsequent steps S2 to S7 are repeatedly performed by the iterative processing unit 20 as surrounded by steps S2 and S7 which mean repeated control. Here, the framework of the iterative process itself is an iterative process in accordance with the well-known RANSAC algorithm, but in the present invention, in particular, the steps S4 and S5 etc. are woven into the iterative process to change the epipolar geometry to the RANSAC geometry. It is possible to realize highly accurate geometric verification when used as a parameter fitting model used in the above. Details of each step will be described below.

[Step S2: Selection unit 21]
In step S2, local feature correspondence between the question image obtained in step S1 by the weak geometry verification unit 13 and the reference image (a plurality of local feature correspondences exist between one question image and one reference image). The selection unit 21 randomly selects a predetermined number of local feature correspondences and outputs them to the geometric model generation unit 22 before proceeding to step S3.

  Note that when there are many local feature correspondences, the probability is low, but the following may be performed in step S2. That is, the current step S2 corresponds to the mth repetition of steps S2 to S7 (loop processing), and the selection unit 21 has already selected in any past nth (n <m) step S2. If the same selection result is obtained at the current nth time, the same result is discarded, and the selection unit 21 may repeat the selection until a result that has never been selected in the past is obtained. . In this case, the selection unit 21 may determine whether the current selection result does not match any of the past selection results by storing the selection result at each time of the loop processing.

[Step S3: Geometric model generator 22]
In step S3, the geometric model generation unit 22 generates a temporary geometric model using the predetermined number of local feature correspondences selected by the selection unit 21 in step S2, and outputs the model to the parameter extraction unit 23. To step S4.

Specifically, the temporary geometric model to be generated is the basic matrix F in the epipolar geometric model as described with reference to FIG. That is, the coordinates x1i = (u _1i , v _1i ) (i = 1, 2,..., K: k is the selected predetermined number) and the reference image in the selected predetermined number of mutually corresponding question images G1 Coordinate x2i = (u _2i , v _2i ) (i = 1,2, ..., k) in the local feature of G2 This is the basic matrix F that appears in the epipolar equation (formula (1) above) under the assumption that it was taken in the image.

  Here, the “temporary” geometric model (basic matrix F) is referred to because it is a candidate for what is finally obtained as an optimal geometric model. That is, a “temporary” geometric model F (m) is calculated at each of the loop processes S2 to S7 (m = 1, 2,...) In the framework of the RANSAC algorithm, and the calculated series of geometric models F (m ) Determined to be optimal is determined by the verification unit 25 in step S8 described later.

  In step S3, the geometric model generation unit 22 may calculate the basic matrix F using a known 7-point algorithm or 8-point algorithm as described above. In step S2, the selection unit 21 may adopt the minimum number necessary for the geometric model generation unit 22 to calculate the basic matrix F as the predetermined number k for selecting the local feature correspondence. That is, seven local feature correspondences may be selected when the 7-point algorithm is used and eight when the 8-point algorithm is used.

[Step S4: Parameter extraction unit 23]
In step S4, the parameter extraction unit 23 extracts various parameters from the geometric model (fundamental matrix F) obtained by the geometric model generation unit 22 in step S3 and outputs the various parameters to the evaluation unit 24, and then proceeds to step S5. Proceed with

The various parameter extraction processes by the parameter extraction unit 23 are listed as follows (1) to (4).
(1) Extract from the basic matrix F the focal length f1 of the camera that captured the query image G1 and the focal length f2 of the camera that captured the reference image G2.
(2) The basic matrix E is extracted from the basic matrix F and the extracted focal lengths f1 and f2.
(3) By decomposing the extracted basic matrix E, a camera rotation matrix R and a translation vector T for three-dimensional coordinate transformation from the camera arrangement of the query image G1 to the camera arrangement of the reference image G2 are extracted. (Or similarly, a rotation matrix and a translation vector corresponding to the inverse transformation are extracted.)
(4) to decompose as the extracted rotation matrix R for example, the following equation (2), the rotation angle around the rotation angle theta _Y and Z-axis about the rotation angle theta _X, Y-axis around the X axis to extract the θ _Z. Here, the Z axis is taken as the camera axis (in FIG. 3, the axis passing through the principal point C1 and perpendicular to the image G1 or the axis passing through the principal point C2 and perpendicular to the image G2).

  In addition, it is well-known that extraction like the above (1)-(4) is possible similarly to epipolar geometry. A specific calculation method for the extraction is also disclosed in Non-Patent Document 3 and the like described above.

[Step S5: Evaluation unit 24]
In step S5, the evaluation unit 24 uses the parameters extracted in step S4 by the parameter extraction unit 23 to determine the validity of the temporary model (basic matrix F) generated in step S3 by the geometric model generation unit 22. After evaluating and outputting the evaluation result to the verification unit 25, the process proceeds to step S6.

  It is each part 1-4 of the evaluation part 24 that bears each embodiment of the said evaluation, and it mentions later for details.

  As for the evaluation result, the validity may be given as a numerical value, or by performing a threshold determination on the numerical value, as a binary determination that is valid or invalid (inappropriate as a model) May be given.

[Step S6: Verification unit 25]
In step S6, the verification unit 25 verifies the temporary geometric model (basic matrix F) generated in the geometric model generation unit 22 in step S3 after considering the validity evaluated by the evaluation unit 24 in step S5. Then, the verification result is recorded in the verification unit 25 itself as a result of the loop processing S2 to S7 in the m-th time, and then the process proceeds to step S7.

  The verification unit 25 performs the following first process and second process as the verification process. In the first process, the number of inliers for the generated temporary geometric model (fundamental matrix F) is counted in the same manner as in the normal RANSAC algorithm framework. That is, among the entire local feature correspondence between the query image and the reference image obtained in step S1, the basic matrix F obtained in step S3 (basic matrix F (m) at the mth iteration) is The local feature correspondence that satisfies Equation (3) is determined as an inlier, and the number is counted.

In the above, the vectors x ₁ and x ₂ are the coordinates (u ₁ , v ₁ ) of the feature point of the image G1 and the coordinates of the feature point of the image G2, respectively, in the local feature correspondence between the query image G1 and the reference image G2. (U ₂ , v ₂ ) is expressed in homogeneous coordinates, and TH is a threshold value for determination. As is well known in epipolar geometry, the left side of the above equation “x ₁ ^T F x ₂ ” means the distance between the epipolar line and the points (u ₁ , v ₁ ), (u ₂ , v ₂ ). is there. In other words, the above formula indicates that the distance between the epipolar line of the epipolar plane corresponding to the basic matrix F and one of the two feature point coordinates (u ₁ , v ₁ ), (u ₂ , v ₂ ) is less than the threshold TH. Is determined.

  In the second process, the total score score (m) for the temporary geometric model (basic matrix F (m)) at the mth iteration is calculated, the number of inliers obtained in the first process is the inlier at the mth iteration. It is obtained as a function F (INL (m), EVAL (m)) of the evaluation score EVAL (m) of step S5 in the number INL (m) and the m-th time, and is recorded as the m-th verification result. As the function F, a predetermined function may be prepared as an increase function for both the inlier number INL (m) and the evaluation score EVAL (m).

  Note that, in the embodiment in which the evaluation score EVAL (m) in step S5 is given in a binary format that is valid or not valid by the threshold determination, if the evaluation result is not valid, the overall score F (INL (m), EVAL (m)) may be output as a predetermined minimum value. Alternatively, the above first process itself may be omitted, and the verification result that the geometric model (fundamental matrix F (m)) is inappropriate and should be discarded without counting the number of inliers is provided. Also good. By giving a verification result that the number of inliers is discarded without counting, it is possible to speed up the iterative processing in the RANSAC framework. A specific example of achieving such high speed will be described later with reference to FIG.

[Step S7: Verification unit 25]
In step S7, whether or not the verification unit 25 satisfies a predetermined end condition of the repetitive process in the RANSAC algorithm (conditions such that the repetitive number m has reached a predetermined value) is satisfied by the m-th repetitive process. If it is satisfied, the process proceeds to step S8, and if not satisfied, the selection unit 21 is instructed to continue the next m + 1-th processing, and then the process returns to step S2 and selected in step S2. The unit 21 performs the random sample selection as described above as the m + 1th time.

[Step S8: Verification unit 25]
In step S8, the verification unit 25 obtains and records the total score score (m) = F (INL (m), EVAL (m)) (m = 1, 2,...) Based on the above, an optimal geometric model is determined from a series of temporary geometric models (basic matrix F (m)), and is output to the additional processing unit 31, and the process proceeds to step S9.

  In one embodiment, the determination can be made such that the total score score (m) is the maximum value. In another embodiment, the evaluation value EVAL (m) may be determined to be the largest among the largest inlier numbers INL (m). In this case, in the above-described step S6, the calculation process (second process) of the total score score (m) is omitted and only the first process is performed, and the inlier number INL (m) and the evaluation value EVAL (m) are obtained from the verification unit 25. You may make it record in.

In step S9, based on the verification result by the verification unit 25 in step S8, the additional processing unit 31 performs the additional processing and outputs the result, and then the flow of FIG. 2 ends. Various types of additional processing are possible, and if the geometric model determined to be optimal is at the number of loop processings m = m _max , its total score score (m _max ) or inlier number INL (m _max ) Based on the above, it may be determined whether or not the question image and the reference image represent the same object, or the similarity between the question image and the reference image may be output. Good.

  In addition, when there are a plurality of reference images, the above steps S1 to S8 are performed for each reference image, and in step S9, the additional processing unit 31 determines that the question image (one) is a plurality of reference images. It may be determined based on the total score or the number of inliers for each reference image.

  As described above, by including the processing of the additional processing unit 31 in step S9 (processing in the flow of lines L2 and L3 in FIG. 1), the geometric verification device 10 further performs similarity calculation and the like in addition to the geometric verification. Can take on the function. Further, by omitting step S9 and the additional processing unit 31 (processing in the flow of the line L1 in FIG. 1), the geometric verification apparatus 10 can output only the geometric verification result as the final output. Conversely, when step S9 and the additional processing unit 31 are used, not only similarity calculation and the like, but an arbitrary output based on the optimally determined geometric verification result can be used as the output of the geometric verification device 10. For example, the relationship between the camera coordinates of the query image G1 and the camera coordinates of the reference image G2 (the rotation matrix R and the translation vector T in the parameter extraction unit 23 described above) is output as the result of the geometric verification determined to be optimal. May be.

  The overall flow of FIG. 2 has been described above. Hereinafter, as the details of the evaluation unit 24 in step S5 in the flow of FIG. 2, the units 1 to 4 responsible for the respective embodiments as described above will be described.

[About focal length evaluation unit 1]
In the focal length evaluation unit 1, the focal length f1 of the query image G1 and the focal length f2 of the reference image G2, which are the extracted parameters, are checked to determine whether they are in a realistic range obtained from the following consideration. The evaluation value is given in the form of a numerical value or a binary decision in a form that is appropriate if it is within the range and not appropriate enough to deviate from the range. Hereinafter, the details of the practical range derived from the consideration and the consideration will be described.

  First, the extracted focal lengths f1 and f2 represent the focal length in pixel units, as is apparent from the image coordinates, and are expressed in mm units (the real world distinguished from pixel units). The focal length in millimeters (mm) is used as an example to clearly indicate that the unit is the length unit of the following.

Here, f _pixel is a focal length (= f1, f2) in units of pixels. Since both the focal lengths f1 and f2 satisfy the above in the images G1 and G2 in the respective cameras, f _pixel is used as a representative of both of them. Further, w _pixel and h _pixel are the horizontal width and vertical width of the image (G1 or G2) in pixel units, f _mm is the focal length in mm units, and SS _mm is the sensor size in mm units (the horizontal width or vertical length). The sensor size corresponding to the larger side of the width).

In the pinhole camera model, the relationship shown in FIG. 4 is established when the camera focuses on the object. Based on this relationship, the following equation (4-2) is established, where WD _mm is the working distance in _mm and FoV _mm is the field of view in mm.

In FIG. 4, as a ray reaching from the field of view F to the sensor S through the lens L in the pinhole camera model, it goes straight from the upper end point FE1 of the field of view F to the lower end point SE2 of the sensor S through the center C of the lens L. A ray and a ray traveling straight from the lower end point FE2 of the field of view F through the center C of the lens L to the upper end point SE1 of the sensor S are drawn. Assuming that sensor S, lens L, and field of view F are parallel, two triangles C-FE1-FE2 and triangle C-SE1- are represented by line segments FE1-C-SE2 and FE2-C-SE1 of the two rays. Since SE2 is similar, the above equation (4-2) is established. As shown, the focal length f _mm is the distance between the lens L and the sensor S, and the differential distance WD _mm is the distance between the lens L and the field of view F.

  In the above description, FIG. 4 is described based on the premise of the vertical (vertical direction) of the image. However, the horizontal direction of the image may be used. FIG. 4 assumes the side.

Furthermore, as shown in FIG. 4, the angle of view of the camera is set to θ _AoV (view angle θ _AoV = ∠FE1-C-FE2 or ∠SE1-C-SE2), and the following equation (4-3) holds.

  Therefore, when the camera focuses on the object by the above formulas (4-1) to (4-3), the following formula (4-4) is established.

The value obtained by dividing the extracted f1, f2 (= f _pixel ) by max {w _pixel , h _pixel } using the relationship of the above equation (4-4) (value on the left side) is assumed as the angle of view of the camera. In the predetermined range θ _min <θ _AoV <θ _max , it is only necessary to determine (binary determination) whether the value on the right side of the above formula (the reciprocal of 2 tan (θ _AoV / 2)) is within a possible range. . Further, when obtaining a continuous evaluation value rather than a binary determination, a predetermined function may be used that lowers the evaluation value as the value deviates from the range.

Examples of numerical values for binary determination are as follows. For example, assuming that the range of the angle of view of the camera to be used is a range of θ _min = 10 ° to θ _max = 100 °, the range is set as a predetermined range to which Expression (4-4) is applied. In this case, the value range of the right side (and the left side that is equal to this) in Equation (4-4) should be in the range of about 0.42 to 5.72. Conversely, if the values of the focal lengths f1 and f2 that do not fall within this range are extracted, it can be determined that the current temporary geometric model is not valid. In this way, the validity of the provisional geometric model can be evaluated by determining f1 and f2 assumed in advance using the upper threshold and the lower threshold.

Note that the above evaluation by the focal length evaluation unit 1 is performed on both the focal length f1 extracted from the query image G1 and the focal length f2 extracted from the reference image G2. As the values to be divided, max {w _pixel , h _pixel } also use the respective values in the question image G1 and the reference image G2. Since an evaluation value is obtained in both cases, the final output by the focal length evaluation unit 1 may be output in the form of a total score of the two evaluation values using a predetermined function. If the evaluation is based on binary evaluation, for example, only if it is determined to be appropriate in both cases, the overall evaluation result is evaluated as appropriate, and if at least one of them is determined to be invalid The evaluation may not be valid as a comprehensive evaluation result.

[About Position Evaluation 2]
The positional relationship evaluation unit 2 evaluates whether the extracted rotation matrix R and translation vector T are in a realistic range, and lowers the evaluation value as it deviates from the realistic range. Get an evaluation value. In the case of performing binary determination, it is sufficient to evaluate whether or not it is within a realistic range by threshold determination.

Specifically, determine the angle theta _opt between the optical axis of the rotation matrix R and the translation from the vector T and the optical axis of the query image G1 camera reference image G2 camera, angle theta _opt between the optical axis of the reality It is only necessary to evaluate whether it is within a certain range. The realistic range may be set as the predetermined range based on the following consideration.

When performing geometric verification based on local features, local features cannot be matched in the first place unless the same part of the object is captured by both two cameras. That is, unless the angle θ _opt is small to some extent, the same part of the object cannot be copied, and a correct geometric model cannot be estimated. Therefore, it is possible to evaluate the validity of the temporary geometric model depending on whether the angle θ _opt is within the upper limit threshold assumed in advance. For this reason, the angle θ _opt is defined as 0 ° in the case of the front (when there is no change in the direction of the optical axis of the camera), and the angle θ _opt is determined so as to increase as it deviates from the front.

In addition, as shown separately in [Case 1] and [Case 2] in FIG. 5, the angle θ _opt is the optical axis A1 of the camera of the query image G1 and the camera of the reference image G2 as in [Case 1]. There is a distinction between the case where the optical axis A2 is inward and the case where the optical axis A1 of the question image G1 camera and the optical axis A2 of the reference image G2 are outward as in [Case 2]. Therefore, the range for determining whether the angle θ _opt is a realistic range may be changed between the inward and outward directions.

In FIG. 5, in both [Case 1] and [Case 2], the translation T is applied to the optical axis A2 of the reference image G2, so that the origin of the optical axis A2 is changed from the principal point C2 of the reference image G2 to the question image G1. The object moved to the principal point C1 is drawn as an optical axis A20 by a dotted line. The angle θ _opt is thus the angle between the optical axis A1 and the optical axis A20 whose principal points coincide with each other by the translation T.

Further, in order to perform the above evaluation, the positional relationship evaluation unit 2 performs the following well-known extraction process (5). The process (5) is combined with the processes (1) to (4) described above. The parameter extraction unit 23 may perform this.
(5) Extract the angle θ _opt between the optical axis A1 and the optical axis A2 from the extracted rotation matrix R and translation vector T. Further, based on the rotation matrix R and the translation vector T, it is determined whether the optical axis A1 and the optical axis A2 have an inward or outward relationship as shown in FIG.

[About consistency assessment section 3]
The consistency evaluation unit 3 evaluates whether each extracted parameter is consistent with the local feature in the local feature correspondence, and outputs an evaluation value such that the higher the consistency is, the higher the evaluation value is. It is possible to make a binary determination as to whether or not it is appropriate by determining the threshold value for the height of the. Specifically, the consistency evaluation unit 3 determines whether the consistency is evaluated between which quantity of the extracted parameters and which quantity of the local features. Or a second embodiment is possible.

In the first embodiment, the reciprocal (f1 / f2) ⁻¹ of the ratio f1 / f2 of the extracted focal lengths f1, f2 is the peak value of the scale ratio among the weak geometric parameters obtained by the weak geometric verification unit 13 described above. Whether the value is close to Δs _peak is determined by threshold judgment such as the following formula (5-1) or (5-2), and when it is determined that it is close, an evaluation result is obtained that is appropriate. be able to.
| (f1 / f2) ^-1 -Δs _peak | <THp1… (5-1)
| log (f1 / f2) ^-1 -logΔs _peak | <THp1… (5-2)

In the above, THp1 and THp2 are determination thresholds. In equation (5-1), the difference is taken directly, and in equation (5-2), the logarithm is taken and then the difference is taken to determine whether the value is close to the peak value Δs _peak of the scale ratio. Although it is judged, the way of thinking is the same. Regarding the peak value Δs _peak of the scale ratio, as described above in the description of the weak geometric verification unit 13, the respective scales s1 _i (i = i = _i) in the local features of the question image G1 and the reference image G2 for which the correspondence relationship is set. 1,2, ..., N) and scale s2 _i (i = 1,2, ..., N), the ratio of the scale of image G1 to image G2 Δs _i = s1 _i / s2 _i (i = 1,2,..., N) may be obtained and the peak value Δs _peak as a result that has already been aggregated may be used.

In addition, the above formula (5-1) or (5-2) shows the condition for binary judgment, but when an evaluation value is given as a numerical value, “|| The smaller the value of (f1 / f2) ^-1 -Δs _peak | or the value of | log (f1 / f2) ^-1 -log Δs _peak | The evaluation value may be calculated by using a predetermined function that gives a value.

That is, in the first embodiment, the reciprocal (f1 / f2) ^{−1 of} the ratio f1 / f2 is reflected in the size of the object shown in the question image G1 and the reference image G2 (in the case of a correct image) ( It is based on the consideration that it substantially matches the ratio to the size of the object. This consideration is derived as follows.

In other words, from the first line of Equation (4-4), the ratio of the focal lengths of the two extracted cameras is the WD _mm / at the time of shooting of each camera (assuming that the two image sizes are almost the same). Equal to the ratio of FoV _mm . From FIG. 4 and Formula (4-2), since SS _mm and FoV _mm are in a proportional relationship, it is considered that the ratio of SS _mm of the two cameras matches the ratio of FoV _mm . Since the SS _mm of the camera does not change, the ratio of the focal lengths of the two cameras is considered to be proportional to the ratio of WD _mm . Therefore, it is considered that the reciprocal of the ratio of WD _mm when photographing objects of the same size is consistent with the peak value Δs _peak of the scale ratio in the local feature set (obtained by the weak geometric verification unit 13). Therefore, the above evaluation is possible. When the SS _{mm of the} two cameras is known, the influence of FoV _mm may be canceled by using the ratio.

In the second embodiment, the Z-axis extracted from the rotation matrix R in the process (4), that is, the rotation angle θ _Z around the optical axis of the camera is the weak geometric parameter already determined by the weak geometry verification unit 13 described above. The following formula (5-3) is based on the consideration that the peak value Δθ _{peak of the} difference in direction should substantially match (when the reference image G2 is a correct image with respect to the question image G1). When it is satisfied, an evaluation result that is appropriate can be obtained. Here, THθ3 is a threshold value for determination.
| θ _Z -Δθ _peak | <THθ3… (5-3)

  In the above calculation, conversion is performed so that the absolute value of the difference is minimized. For example, the absolute value of the difference between 5 ° and 355 ° is calculated as 10 °.

In addition, the above formula (5-3) represents the case where the evaluation value is determined as a binary value. However, the smaller the value of “| θ _Z -Δθ _peak |” on the left side of the formula, the more evaluation is performed. The evaluation value may be obtained as a continuous value by using a predetermined function that increases the value. Also in this case, as described above, “converted so that the absolute value of the difference is minimized”.

The rotation angle θ _Z around the optical axis of the above-mentioned camera means the following. That is, after making the principal point C1 of the query image G1 coincide with the principal point C2 of the reference image G2 by applying the translation vector T, two rotations R _Y on the first half side among the three rotations in the above-described equation (2) When (θ _Y ) R _X (θ _X ) is applied, the question image G1 and the reference image G2 are on the same plane. When rotating from the state by the rotation angle θ _Z (that is, applying the last rotation R _Z (θ _Z ) among the three rotations of the expression (2)), the orientation of the question image G1 and the reference image G2 (Vertical and horizontal orientations) match.

[Restorability evaluation unit 4]
The restorability evaluation unit 4 uses the basic matrix F itself as the extracted parameter to evaluate whether or not correct three-dimensional restoration can be performed as follows. That is, it is determined whether the restored point has a positive depth (Z coordinate) in both coordinate systems, and if all the depths are positive, an evaluation result is obtained that is valid, If there is a negative depth, an evaluation result is obtained that is not valid.

FIG. 6 is a diagram for schematically explaining the depth to be calculated. A point P _{[calculation]} is a three-dimensional position of one arbitrary feature point in the calculation specified by the obtained basic matrix F. Represents. Therefore, by converting the point P _{[calculation]} into the respective camera coordinate systems of the question image G1 and the reference image G2, the depths in the respective camera coordinate systems can be obtained. In FIG. 6, the case where the depths in both the images G1 and G2 are both positive as [Case 1], and the case where the depths in both the images G1 and G2 are negative as [Case 2] is the image G1 as [Case 3]. The case where the depth at is positive and the depth at image G2 is negative is shown. The positive / negative distinction of the depth of the three-dimensional position with respect to each point of each image is as described in the figure.

Here, in each of [Case 1] to [Case 3], P1 _[Image] and P2 _[Image] are the points detected by the local feature extraction unit 11 as feature points in the local features from the images G1 and G2, respectively. These are the position coordinates on the images G1, G2, and are the points used to calculate the point P _{[calculation]} . Points C1 and C2 are the main points of the cameras (referred to as imaging systems including lenses and the like) of the images G1 and G2, as described with reference to FIG. As shown in the schematic example of FIG. 6, the positive or negative of the depth of the point P _{[calculation]} is on one straight line (1) the main point (C1 or C2) of the camera, (2) on the image The point (P1 _[Image] or P2 _[Image] ) on the image specified by the position coordinates of, and (3) the three-dimensional position (P _{[Calculation]} restored by the basic matrix F obtained using the point on the image _] ), Which corresponds to the arrangement order of the three points on the one straight line. Specifically, (1) the main point of the camera and (3) the restored three-dimensional position divide the three-dimensional space into two sides (parts) on the plane of the image (G1 or G2). In this case, the depth value of the point P _{[calculation]} is positive if it belongs to different sides, and conversely the depth value of the point P _{[calculation]} is negative if it belongs to the same side.

  In other words, the depth in the Z-axis direction that is a target for determining the positive / negative is the vertical when the 3D restoration point is “seen” from the principal point on the camera in the direction perpendicular to the image plane (optical axis direction). Depth in direction. If it starts from the principal point and passes through the position coordinates on the image plane to reach the three-dimensional position (corresponding to the case of belonging to the above different side), it becomes a positive depth value, and the image plane If it starts from the upper position coordinate and passes through the principal point to reach the three-dimensional position coordinate (corresponding to the case of belonging to the same side as described above), it becomes a negative depth value.

For example, in [Case 1], the point P1 _[image] of the image G1 is arranged in the order of “the main point C1 → the point P1 _[image] on the image G1 → the three-dimensional position P _{[calculation]} ”, and the main point C1 And the restored three-dimensional coordinate P _{[calculation]} belong to different sides in a three-dimensional space partitioned by a plane defined by the image G1. Therefore, with the promise to define the positive direction of the Z axis as the camera optical axis direction from the principal point C1 to the image G1, the depth (Z coordinate ₎ of the restored three-dimensional position P _{[calculation]} of the point P1 _[image] Value) is positive. Conversely, at the point P1 _[image ] of the image G1 in [Case 2], the principal point C1 and the three-dimensional position P _{[calculation]} belong to the same side when the space is divided by the plane of the image G1, The depth (Z coordinate value) of the restored three-dimensional position P _{[calculation]} of the point P1 _[image] is negative. Similarly, regarding other points, the positive and negative depth values are as described above and in the drawings.

Note that the P _[Calculation all points depth calculating the above-mentioned in restoring evaluation unit 4, which is all points P in local feature correspondence selected in step S2 by the selector 21 _{[Calculation.}

  The above-mentioned epipolar equation of equation (1) merely guarantees that the point is located on the epipolar line, and does not necessarily guarantee that the three-dimensional reconstruction can be performed correctly, whereas By the evaluation using the depth, the validity of the temporary geometric model can be more reliably evaluated. This is because if all the selected point correspondences are inliers, they should be able to be restored correctly in 3D. In particular, [Case 2] and [Case 3] in FIG. 6 where even one depth is negative is not feasible as a situation where the query image G1 and the reference image G2 are photographing the same three-dimensional object ( However, according to the restorability evaluation unit 4, such a case is excluded as being an outlier (or has a low relevance). Can be evaluated). Even if the outlier is included in the selected point correspondence, there is a possibility that the three-dimensional restoration is accidentally correct. This is because even an outlier may have an arrangement consistent with the relative posture of the camera.

In the above description, the depth of the restored points is determined for all the points P _{[calculation]} , and the evaluation value is given by binary determination as to whether it is valid or not. In addition, the evaluation value can be obtained as a continuous value. For example, for all points P _{[calculation]} , by counting (counting) those whose depths are positive in both coordinate systems, the evaluation value is set as a predetermined function such that the evaluation value increases as the count increases. It can also be given.

[About the combination of parts 1 to 4 above]
As mentioned above, although each part 1-4 which bears each embodiment of the evaluation part 24 was each demonstrated, the said each part 1-4 may use only any one, and may use it combining arbitrary multiple. When using a plurality of combinations, the evaluation unit 24 may output a final evaluation value / evaluation result as an overall score of each evaluation value of the units 1 to 4 used. In calculating the total score, a predetermined function or a predetermined rule-based method may be used.

  FIG. 7 is a flowchart of the operation of the geometric verification apparatus 10 according to an embodiment. The position embodiment uses all of the parts 1 to 4 in the evaluation unit 24, and obtains an evaluation result by binary determination that each part 1 to 4 is valid or invalid, and at least any of the parts 1 to 4 If one of the results shows that the result is not valid, the evaluation unit 24 determines that the local feature correspondence as the random sample selected in the repetitive processing of the RANSAC is not valid, and discards it. is there. The discard can speed up the RANSAC process.

  One embodiment of FIG. 7 shows one specific example of step S5 in one embodiment of FIG. 2, and a specific example of step S5 of FIG. 2 is a portion from steps S51 to S59 of FIG. It corresponds to.

  Steps S11 to S14 and S16 to S19 in FIG. 7 other than those described above are the same as steps S1 to S4 and S6 to S9 in FIG. Hereinafter, steps S51 to S59, which are steps unique to FIG. 7, will be described.

  When step S14 (similar to step S4 in FIG. 2) is finished and step S51 is reached, the focal length evaluation unit 1 evaluates whether or not the geometric model is valid by the binary evaluation described above, and then proceeds to step S52. In step S52, it is determined whether or not the evaluation result in step S51 is valid. If valid, the process proceeds to step S53, and if not valid, the process proceeds to step S59.

  In step S53, the positional relationship evaluation unit 2 evaluates whether or not the geometric model is valid by the binary evaluation described above, and then proceeds to step S54. In step S54, it is determined whether or not the evaluation result in step S53 is valid. If valid, the process proceeds to step S55, and if not valid, the process proceeds to step S59.

  In step S55, the consistency evaluation unit 3 evaluates whether or not the geometric model is valid by the binary evaluation described above, and then proceeds to step S56. In step S56, it is determined whether or not the evaluation result in step S55 is valid. If valid, the process proceeds to step S57, and if not valid, the process proceeds to step S59.

  In step S57, the restorability evaluating unit 4 evaluates whether or not the geometric model is valid by the binary evaluation described above, and then proceeds to step S58. In step S58, it is determined whether or not the evaluation result in step S57 is valid. If valid, the process proceeds to step S16 (the same step as step S6 in FIG. 2), and if not valid, the process proceeds to step S59.

  In step S59, which is reached when any of the above steps S52, S54, S56, S58 is determined to be invalid, the evaluation unit 24 discards the geometric model (basic matrix F (m)) at the m-th iteration. After obtaining the judgment that it should be, the process proceeds to step S17 (the same step as step S7 in FIG. 2).

  The geometric model (basic matrix F (m)) determined to be discarded at Step S59 is excluded from the objects to be determined as the optimum geometric model by the verification unit 25 at Step S18.

  As described above, in the flow of FIG. 7, the evaluations by the units 1 to 4 are applied in this order, but any other order may be used. Moreover, it can also be made to be the same as that of the flow of FIG. 7 using only a part of each part 1-4.

[Regarding an embodiment in which the parameter extraction unit 23 and the evaluation unit 24 are applied to the result of finishing the normal RANSAC process, not in the middle of the RANSAC process]
In the above embodiment, the parameter extraction unit 23 and the evaluation unit are applied to the temporary geometric model (basic matrix F) generated from the local feature correspondence as the selected random sample in the middle of the iterative process of the RANSAC framework. 24 was applied. As another embodiment, the parameter extraction unit 23 and the evaluation unit 24 are applied in the same manner to the geometric model in the local feature correspondence obtained as the optimum by the normal RANSAC iteration process, and the optimum and It is possible to evaluate the validity of the determined geometric model.

  FIG. 8 is a functional block diagram of the geometric verification apparatus 10 according to the embodiment, and FIG. 9 is a flowchart of the operation of the geometric verification apparatus 10 according to the embodiment.

  In the functional block diagram of FIG. 8, each unit included in the geometric verification device 10 is the same as that in FIG. 1, but in FIG. 1, the parameter extraction unit 23 and the evaluation included in the iterative processing unit 20 that performs the iterative processing in the RANSAC framework. The difference is that the unit 24 is outside the iterative processing unit 20. Therefore, as shown in FIG. 8, the geometric model generated by the geometric model generation unit 22 is directly output to the verification unit 25, and the optimal geometric model (as a result of the iterative processing of the iterative processing unit 20 from the verification unit 25 ( The basic matrix F) is output to the parameter extraction unit 23 and the evaluation unit 24.

  As described above, the individual operation of each part of the geometric verification apparatus 10 is common between the embodiment of FIG. 1 and the embodiment of FIG. 8, and only the data flow between the parts is different. Since this is a common operation, a description of the operation of each unit in FIG. 8 is omitted.

  Next, the flowchart of FIG. 9 showing the operation of the geometric verification apparatus 10 in the embodiment of the configuration of FIG. 8 will be described. The flowchart in FIG. 9 corresponds to a step in which steps S4 and S5 in the flowchart in FIG. 2 are rearranged after the loop processing of the RANSAC framework is completed. The steps S4 and S5 in FIG. 2 that have been rearranged correspond to steps S24 and S25 in FIG. 9 that are assigned corresponding numbers. The other steps are also in correspondence as indicated by the corresponding numbers in FIG. 2 and FIG. 9 and are common as individual operations, and redundant description is omitted. That is, steps S21, S22, S23, and S26 to S29 in FIG. 9 are common to steps S1, S2, S3, and S6 to S9 in FIG.

  Therefore, in the flow of FIG. 9, when step S21 is completed and step S22 is reached, loop processing S22, S23, S26, and S27 are executed in a normal RANSAC processing framework in which the parameter extraction unit 23 and the evaluation unit 24 are not applied. Then, when the end condition is satisfied in step S27, the process proceeds to step S28, and the verification unit 25 outputs an optimal geometric model. Therefore, in step S26 in the loop process, the verification unit 25 may perform verification by performing only the first process described above (unlike step S6 in FIG. 2).

  In steps S24 and S25, the parameter extraction unit 23 and the evaluation unit 24 perform parameter extraction and evaluation of the geometric model, and the extraction and evaluation are geometric models obtained as the optimum results in the RANSAC loop processing S22, S23, S26, and S27. The same processing as that performed on the temporary geometric model in the embodiment of FIGS. 1 and 2 may be performed.

  As described above, the geometric verification by the geometric verification apparatus 10 of the present invention is applicable not only to the inside and outside of the RANSAC process but also to any local feature correspondence. Further, the present invention can be applied to any local feature correspondence without assuming the framework of RANSAC processing.

  Hereinafter, supplementary matters in the present invention will be described.

  (Supplement 1) In both the embodiments of FIGS. 1 and 8, the weak geometric verification unit 13 may be omitted. In this case, all the local feature correspondences of both images acquired by the correspondence acquisition unit 12 may be output to the selection unit 21. Similarly, in step S1 in FIG. 2, step S11 in FIG. 7, and step S21 in FIG. 9, the process of the weak geometric verification unit 13 may be omitted.

  (Supplement 2) In the flow of FIG. 2, FIG. 7 and FIG. 9, it has been described that the framework of the iterative process is in accordance with RANSAC, but other frameworks may be used. For example, a development system of the RANSAC algorithm such as a PROSAC (PROgressive SAmple Consensus) algorithm or an ARRSAC (Adaptive Real-Time Random Sample Consensus) algorithm may be used as a framework, or another robust estimation method may be used. In this case, the selection processing of the selection unit 21 in steps S2, S12, and S22, the end determination processing of the verification unit 25 in steps S7, S17, and S27, etc. may be changed to those according to the other algorithms.

  Note that the RANSAC processing has the following common framework. That is, a random sample is acquired (selected by the selection unit 21 in the present invention) from a given sample (in the present invention, the entire local feature correspondence output from the correspondence acquisition unit 12 or the weak geometry verification unit 13), and the acquisition is performed. A parameter when a given fitting model (in the present invention, an epipolar geometric model) is applied to the random sample obtained is obtained (in the present invention, the geometric model generation unit 22 obtains the basic matrix F), and based on the parameter The above-described model is repeatedly used to determine which of the given samples is inlier and which is outlier (in the present invention, verification is performed by the verification unit 25). From the series of repeated results, the optimum parameter is determined with the maximum number of inliers (in the present invention, the verification unit 25 determines the optimum result).

  (Supplement 3) The parameter extraction unit 23 (step S4 in FIG. 2 and the like) extracts various parameters as described in (1) to (4) and the like, but it is not always necessary to extract all of them. Only the parameters used in each embodiment of the evaluation unit 24 (step S5 in FIG. 2, etc.) need be extracted.

  (Supplement 4) In the present invention, in order to apply the epipolar geometric model, the question image G1 and the reference image G2 (which is a correct image) are taken by the two cameras having the arrangement relationship as described in FIG. However, shooting with the camera is not limited to physical shooting with an actual physical camera. For example, the present invention is applicable even when the images G1 and G2 as three-dimensional computer graphics are generated assuming two camera arrangements as shown in FIG.

  Similarly to the above, the present invention is not limited to the local feature correspondence in the query image G1 and the reference image G2 in applications such as object recognition, and can be applied to the local feature correspondence in the general images G1 and G2.

(Supplement 5) In the present invention, the epipolar geometric model is described as being obtained by the geometric model generation unit 22 assuming that it is the basic matrix F, but instead, an epipolar geometric model is expressed. However, the geometric model generation unit 22 may obtain the basic matrix E. As is well known, the epipolar equation of the above equation (1) is also established in the basic matrix E as in the following equation (6). Here, the points x ₁ and x ₂ are multiplied by the inverse matrix of the matrix K ₁ and K ₂ of the internal parameters of each camera. The threshold value determination of the distance from the epipolar line as in equation (3) is also possible.
x ₁ ^T Ex ₂ = 0… (6)

  The basic matrix E can be calculated from the basic matrix F as described above, and can also be calculated from local feature correspondence by a well-known 5-point algorithm.

  (Supplement 6) The present invention can also be provided as a program that causes a computer to function as all of each part of the geometric verification device 10 or any part thereof. The computer can adopt a known hardware configuration such as a CPU (Central Processing Unit), a memory, and various I / Fs, and the CPU executes instructions corresponding to the functions of the respective units of the geometric verification device 10. It will be.

  10 ... Geometric verification device, 22 ... Geometric model generation unit, 23 ... Parameter extraction unit, 24 ... Evaluation unit

Claims (12)

From the local feature correspondence in which correspondence is given between the local feature extracted from the first image generated from the first camera and the local feature extracted from the second image generated from the second camera, the first A geometric model generation unit that generates a matrix as an epipolar geometric model based on a principal point of the camera and the second camera;
A parameter extraction unit for extracting parameters related to the first camera and the second camera from the matrix;
An evaluation unit that evaluates the validity of the matrix as the epipolar geometric model based on the parameters. The parameter extraction unit extracts the first focal length of the first camera and / or the second focal length of the second camera as the parameter,
The evaluation unit, as a configuration for evaluating the validity,
2. The geometry according to claim 1, further comprising a focal length evaluation unit that evaluates the validity by checking whether the first focal length and / or the second focal length is within a predetermined reasonable range. Verification device. The focal length evaluation unit
The first angle of view is based on a predetermined relationship between a first angle of view when the first camera is in focus in the first image, and the first focal length and the size of the first image. A predetermined reasonable range of the first focal length from a predetermined reasonable range of and / or
Based on a predetermined relationship between the second angle of view when the second camera is in focus in the second image, and the second focal length and the size of the second image, the second angle of view. 3. The geometric verification device according to claim 2, wherein a predetermined reasonable range of the second focal length is determined from a predetermined reasonable range of the second focal length. The parameter extraction unit extracts an angle formed by the first optical axis of the first camera and the second optical axis of the second camera as the parameter,
The evaluation unit, as a configuration for evaluating the validity,
The geometric verification apparatus according to claim 1, further comprising a positional relationship evaluation unit that evaluates the validity by examining whether the corner is in a predetermined reasonable range. The parameter extraction unit extracts the angle, and based on a rotation matrix and a translation vector obtained by decomposing the matrix, the first optical axis and the second optical axis are outside each other at the extracted angle. Distinguish between orientation and inward,
5. The geometric verification device according to claim 4, wherein the positional relationship evaluation unit examines whether the angle is in a predetermined reasonable range by distinguishing between the outward direction and the inward direction. . The local feature correspondence used to generate the matrix in the geometric model generation unit is subjected to weak geometric verification processing in advance, so that the scale ratio of each local feature correspondence is near the maximum number of peak values. And a predetermined number has been selected from those that have been confirmed to generally match in the entire set of local feature correspondences,
The parameter extraction unit extracts a first focal length of the first camera and a second focal length of the second camera as the parameters,
The evaluation unit, as a configuration for evaluating the validity,
A consistency evaluation unit that evaluates the validity by examining whether an inverse of a ratio between the first focal length and the second focal length is close to a peak value of the most numerous scale ratios; The geometric verification device according to claim 1. The local feature correspondence used to generate the matrix by the geometric model generation unit is subjected to weak geometric verification processing in advance, so that the direction difference in each local feature correspondence is near the maximum number of peak values. And a predetermined number has been selected from those that have been confirmed to generally match in the entire set of local feature correspondences,
The parameter extraction unit extracts a rotation angle between a first optical axis of the first camera and a second optical axis of the second camera as the parameter,
The evaluation unit, as a configuration for evaluating the validity,
The geometric verification apparatus according to claim 1, further comprising: a consistency evaluation unit that evaluates the validity by examining whether the rotation angle is close to a peak value of the largest number of direction differences. The parameter extracted by the parameter extraction unit is the generated matrix itself,
The evaluation unit, as a configuration for evaluating the validity,
From the matrix, the first depth in the direction perpendicular to the first image when the three-dimensional restoration point of the same feature point corresponding to the local feature is viewed from the first principal point of the first camera and / or the A second depth in a direction perpendicular to the second image when the three-dimensional restoration point is viewed from the second principal point of the second camera is obtained, and the second depth is obtained based on the first depth and / or the second depth. The geometric verification apparatus according to claim 1, further comprising a restoration property evaluation unit that evaluates validity.   The local feature correspondence used to generate the matrix in the geometric model generation unit is selected as a random sample in the middle of the RANSAC system processing, or a series of random samples after the RANSAC system processing is completed. The geometric verification device according to claim 1, wherein the geometric verification device is determined to be optimal from the above. The local feature correspondence used to generate the matrix in the geometric model generation unit is selected as a random sample in the middle of the RANSAC system processing,
The determination according to any one of claims 1 to 9, wherein when a determination result that is not valid in the validity evaluated by the evaluation unit is obtained, it is determined to discard the local feature correspondence as the random sample. The geometric verification device described in 1.   A geometric verification program for causing a computer to function as the geometric verification device according to claim 1. From the local feature correspondence in which correspondence is given between the local feature extracted from the first image generated from the first camera and the local feature extracted from the second image generated from the second camera, the first A geometric model generation stage for generating a matrix as an epipolar geometric model based on a principal point of the camera and the second camera;
A parameter extraction step of extracting parameters related to the first camera and the second camera from the matrix;
An evaluation stage for evaluating the validity of the matrix as the epipolar geometric model based on the parameter.