CN114332510A - Hierarchical image matching method - Google Patents

Abstract

The invention discloses a hierarchical image matching method, which comprises the steps of obtaining a feature descriptor of a query image and a feature descriptor of a reference image, and searching two most similar candidate matching feature points for each local feature point in the query image from the local feature points contained in the reference image; and screening out the best feature matching results in sequence. The invention can quickly calculate the feature matching points between two images, and then is applied to a series of high-level computer vision tasks based on image matching: the method comprises the steps of three-dimensional reconstruction based on images, simultaneous positioning and map construction, image retrieval, map navigation, digital twinning, image splicing, mixed reality, virtual reality, augmented reality and the like.

Description

Hierarchical image matching method

Technical Field

The invention belongs to the computer image processing technology, and particularly relates to a hierarchical image matching method.

Background

Image matching is a hot research problem in the field of computer vision, and is widely concerned by researchers at home and abroad. A complete image matching procedure comprises the following steps: detecting the feature points in the image, calculating the corresponding feature descriptors, matching the features, and eliminating the wrong feature matching points. In practice, due to the influence of factors such as ambiguity of the feature descriptors, texture of an image, illumination, scale change and the like, the image matching precision under a natural scene is low, and the research process and application of the high-level computer vision technology based on image matching are seriously influenced.

Taking three-dimensional reconstruction based on images as an example, if an incorrect feature matching point is used as input data of a three-dimensional reconstruction system, the completeness and geometric consistency of a three-dimensional model may be damaged, and incorrect camera pose information and a low-precision three-dimensional model are obtained, even resulting in failure of the three-dimensional reconstruction process. Therefore, it is desirable to provide a high-precision image matching method, which can generate a large number of feature matching points and eliminate the wrong feature matching points, so as to improve the performance of the high-level computer vision application system based on image matching. The existing related researches have the following problems:

(1) the time of several seconds is needed when the feature matching between two images is processed, and the requirements of some real-time processing applications are difficult to meet;

(2) either the result of feature matching depends on the selected spatial clustering method, or the feature matching is only suitable for the wide baseline image, i.e. the feature matching has no universality;

(3) all false feature matching points due to many factors such as description ambiguity, image scale, texture and illumination variation cannot be simultaneously eliminated.

Disclosure of Invention

The purpose of the invention is as follows: the invention aims to solve the defects in the prior art, and provides a hierarchical image matching method, which is characterized in that an accurate matching point is searched for each local feature point in a query image from a local feature point set of a reference image, and then a series of high-level computer vision applications based on image matching become possible through the calculated accurate feature matching points.

The technical scheme is as follows: the invention discloses a hierarchical image matching method, which relates to the detection of feature points, the matching of features and the elimination of error matching of images, and specifically comprises the following steps:

step S1: for two images input, i.e. query image I_lAnd a reference picture I_rFor which a local feature point detection method and a local feature descriptor calculation method are respectively used to calculate the query image I_lAnd a reference picture I_rThe local feature points and the corresponding feature descriptors contained in each of the two groups;

wherein the query image I_lDetected local feature point of (2) is K_l＝k(x_i,y_i) The corresponding feature descriptor is D_l＝d(k(x_i,y_i))；i∈[1,M]M denotes a query image I_lThe number of detected local feature points;

from reference picture I_rThe detected local feature point in (1) is K_r＝k(x′_j,y′_j) The corresponding feature descriptor is D_r＝d(k(x′_j,y′_j))；j∈[1,N]N denotes a slave reference picture I_rThe number of detected local feature points;

step S2: from the local feature points contained in the reference image, for each local feature point k (x) in the query image_i,y_i) Finding two most similar candidate matching feature points, namely k (x'_j,y′_j) And k (x'_j+1,y′_j+1)；

Step S3: calculating two candidate matching feature points k (x'_j,y′_j) And k (x'_j+1,y′_j+1) And a feature point k (x)_i,y_i) The distance difference ratio rho between the two, and then determining the initial feature matching result Matches according to the distance difference ratio rho₁；

Step S4: for initial feature matching results Matches₁Using cross-validation to eliminate features due to the fact thatDescribing the error feature matching generated by the ambiguity, and further obtaining the feature matching result as Matches₂；

Step S5: for feature matching results Matches₂Eliminating all error feature matching points caused by not meeting geometric consistency and obtaining accurate feature matching result Matches₃；

Step S6: for feature matching results Matches₃Eliminating error feature matching points caused by the influence of noise data by using a clustering-based method so as to obtain a final feature matching result, namely Matches_Final。

Further, the specific method for calculating the local feature points and the corresponding feature descriptors in step S1 is as follows:

1) pre-training feature points, namely, making three-dimensional objects and carrying out picture interception of a visual angle on the objects to obtain a two-dimensional image; true values of all feature points in the obtained two-dimensional image are known, so that the true values are used for network training;

2) self-labeling the characteristic points, and adopting ImageNet as a training data set and a testing data set of the part; training by using a synthetic scene to obtain a basic characteristic point detection network model, and extracting characteristic points on the ImageNet data set by using the basic characteristic point detection network model to obtain characteristic point self-labeling;

3) and (3) performing joint training, namely performing geometric transformation on the pictures used in the previous step to obtain a plurality of picture pairs, inputting two pictures of the same picture pair into a network, extracting feature points and descriptors, and performing joint training to obtain local features based on deep learning, so as to detect the local feature points and calculate the feature descriptors.

Further, k (x ') is obtained through calculation in the step S2 by using a hierarchical partially perceptual hashing method'_j,y′_j) And k (x'_j+1,y′_j+1) The specific process is as follows:

1) constructing hash indexes

Let G ═ { G: S → U } denote a family of functions, where S is the input, S is the output, and G (v) ═ h₁(v),…,h_k(v)),h_iE H, representing the random independent selection of L hash functions G from the family of functions G₁,…,g_L(ii) a For any point v in the data set, store it in bucket g_i(v) Wherein i is 1, …, L;

2) hash search

For a query point q ═ k (x)_i,y_i) And given a distance threshold r, from the search bucket g₁(q),…,g_L(q) taking out all points v therein₁,…,v_nAs candidate approximate nearest neighbors; for arbitrary v_jIf D (q, v)_j) R is less than or equal to v_jWhere D (-) is a similarity measure function. Through the process, two optimal candidate matching points can be found, and q is a query point.

Further, the feature point k (x) in the step S3_i,y_i) Local feature point k (x ') matched with two candidates'_j,y′_j) And k (x'_j+1,y′_j+1) The calculation method of the difference ratio rho between the two is as follows:

if it is

Then k (x'_j,y′_j) Is a local feature point k (x)_i,y_i) To obtain initial feature matching results from the correct matching points₁：

Matches₁＝{<k(x_i,y_i),k(x′_j,y′_j)>|i∈[1,M-,j∈[1,N-}；

Wherein d (k (x)_i,y_i) Denotes a local feature point k (x)_i,y_i) The feature descriptor of (1); d (k (x'_j,y′_j) Is used for representing candidate matching feature point local feature point k (x'_j,y′_j) The feature descriptor of (1); d (k (x'_j+1,y′_j+1) Denotes local feature point k (x'_j+1,y′_j+1) The feature descriptors of (1).

Further, in the step S4, a cross validation method is adopted to eliminate the wrong feature matching point, and the specific method is as follows:

firstly, the characteristic point k (x) of the query image is searched_i,y_i) And feature point k (x ') of reference image'_j,y′_j) Matching is carried out, and the characteristic points k (x'_j,y′_j) And query feature point k (x)_i,y_i) Matching is carried out, if the results of the two matching are not consistent, the results will be<k(x_i,y_i),k(x′_j,y′_j)>From Matches₁Deleting to obtain more accurate feature matching results of Matches₂：

Matches₂＝{<k(x′_jj,y′_jj),k(x_ii,y_ii)>|ii∈[1,M-,jj∈[1,N]}。

Further, in the step S5, geometry constraint mathes is adopted₂The method for matching the error features of the image data, which do not meet the geometric consistency, comprises the following steps:

first-choice computing query feature point k (x)_i,y_i) And reference characteristic point k (x'_j,y′_j) Homography matrix H between, and then calculates query feature point k (x)_i,y_i) And Hk (x'_j,y′_j) If τ is the distance between<0.5, then it is considered<k(x_i,y_i),k(x′_j,y′_j)>Is a correct pair of specific matching points; otherwise, it is a wrong specific matching point, which should be selected from Matches₂Eliminate to obtain accurate feature matching results Matches₃。

Further, the detailed method of step S6 is:

s6.1, adopting a hierarchical clustering method to match the Matches₃Clustering the reference characteristic points to find out a central point C₁；

S6.2, with C₁Computing circumscribed rectangle Rect of reference feature point for center₁；

S6.3, detecting Rect according to homography transformation₁In querying image I_rOf (1) corresponding region Rect'₁；

S6.4, from Matches₃Delete in region Rect'₁Obtaining final feature matching results from the other matching points_Final。

Has the advantages that: the hierarchical image matching method provided by the invention simultaneously considers a plurality of factors (such as ambiguity of a feature descriptor, illumination of an image, texture and scale change and the like) influencing the local feature matching precision, adopts a hierarchical feature matching error elimination method to gradually eliminate wrong feature matching points, is quick and accurate, and can be used for high-level computer vision application systems such as three-dimensional reconstruction, image retrieval, simultaneous positioning and map construction, digital twinning and the like.

Drawings

FIG. 1 is a schematic overall process flow diagram of the present invention;

FIG. 2 is a schematic diagram of an image processing flow according to an embodiment of the present invention;

FIG. 3 is an input primitive diagram in accordance with an embodiment of the present invention;

FIG. 4 is a diagram illustrating a final output result according to an embodiment of the present invention;

fig. 3(a) is a query image, and fig. 3(b) is a reference image.

Detailed Description

The technical solution of the present invention is described in detail below, but the scope of the present invention is not limited to the embodiments.

As shown in fig. 1, a hierarchical image matching method of the present invention includes the following steps:

step S1, for a given input image, i.e. query image I_lAnd a reference picture I_rRespectively using a local feature detection method to calculate local feature points and corresponding feature descriptors contained in the query image and the reference image;

in which the image I is queried_lHas a local characteristic point of K_l＝k(x_i,y_i)，The corresponding feature descriptor is D_l＝d(k(x_i,y_i))；i∈[1,M]；

Reference picture I_rHas a local feature point of K_r＝k(x′_j,y′_j) The corresponding feature descriptor is D_r＝d(k(x′_j,y′_j))；j∈[1,N]；

M and N respectively represent the number of local feature points in the query image and the reference image;

step S2: from the local feature points contained in the reference image, for each local feature point k (x) in the query image_i,y_i) Finding two most similar candidate matching feature points k (x'_j,y′_j) And k (x'_j+1,y′_j+1)；

Step S3: calculating the distance difference ratio rho between the two most similar candidate matching feature points, and further determining the initial feature matching result Matches according to the ratio₁；

Step S4: for initial feature matching results Matches₁Eliminating all error feature matching generated by the ambiguity of the feature descriptor to obtain accurate feature matching result Matches₂；

Step S5: for feature matching results Matches₂Eliminating all error feature matching points caused by not meeting geometric consistency and obtaining accurate feature matching result Matches₃；

Step S6: for feature matching results Matches₃Eliminating all error feature matching points caused by the influence of noise data on the feature matching method, thereby obtaining the final feature matching result Matches_Final。

Example 1:

as shown in fig. 2, step (a) eliminates false matching points due to ambiguity of descriptors using a scale test method; step (b) eliminating asymmetric feature matching points by using a cross-validation method; using a geometric constraint method to eliminate feature matching points (such as homography matrix constraint, basic matrix constraint, epipolar constraint and the like) which do not meet a specific geometric model; and (d) eliminating the error matching points caused by the influence of the noise data by using a statistical optimization and feature clustering method.

The specific steps of this embodiment are:

step 1: the user entering two images, i.e. query image I_lAnd a reference picture I_r. The input image can contain certain occlusion, blurring, specific shapes and objects, and color difference; and does not require any pre-processing operation by the user on the input image.

Step 2: separately detecting query images I_lAnd a reference picture I_rAnd calculating corresponding feature descriptors.

Recording query image I_lThe query feature point in (1) is K_l＝{k(x_i,y_i)|i∈[1,M]D for a corresponding feature descriptor_l＝{d(k(x_i,y_i))|i∈[1,M]}。

Reference picture I_rThe reference characteristic point in (1) is K_r＝{k(x′_j,y′_j)|j∈[1,N]D for a corresponding feature descriptor_r＝{d(k(x′_j,y′_j))|j∈[1,N]}。

Wherein M and N respectively represent the number of feature points in the query image and the reference image; d (k (x)_i,y_i) Represents a characteristic point k (x)_i,y_i) The feature descriptor of (1); d (k (x'_j,y′_j) Denotes a characteristic point k (x'_j,y′_j) The feature descriptors of (1).

Step 3: query feature point k (x) by using hierarchical local perception hash method_i,y_i) Finding two candidate matching points k (x'_j,y′_j) And k (x'_j+1,y′_j+1) Then, the difference ratio between the two candidate matching points can be calculated:

wherein d (k (x)_i,y_i) Is shown in (a)Local feature point k (x)_i,y_i) The feature descriptor of (1); d (k (x'_j,y′_j) Denotes local feature point k (x'_j,y′_j) The feature descriptor of (1); d (k (x'_j+1,y′_j+1) Denotes local feature point k (x'_j+1,y′_j+1) The feature descriptors of (1). If it is not

Then the characteristic point k (x'_j,y′_j) Is the characteristic point k (x)_i,y_i) The correct matching point of (1); to this end, initial feature matching results Matches may be obtained₁：

Matches₁＝{<k(x_i,y_i),k(x′_j,y′_j)>|i∈[1,M],j∈[1,N]} (2)

Step 4: with reference to picture I_rCharacteristic point k (x)'_jj,y′_jj) As query feature points, from the query image I_lIs a characteristic point k (x'_jj,y′_jj) Find two candidate matching points k (x)_ii,y_ii) And k (x)_ii+1,y_ii+1) (ii) a From equation (1), the feature matching result shown below can be calculated:

Matches₂＝{<k(x′_jj,y′_jj),k(x_ii,y_ii)>|ii∈[1,M],jj∈[1,N]} (3)

merging Matches₁And Matches₂The matching result in (1), when i ═ ii and j ═ jj,<k(x_i,y_i),k(x′_j,y′_j)>the matching points that are considered to be a pair of correct matching points, otherwise the wrong matching points should be deleted, so that the feature matching result shown below can be obtained:

Matches₃＝{<k(x_p,y_p),k(x′_q,y′_q)>|p∈[1,M],q∈[1,N]}； (4)

through the cross validation of the process, the validation result meets the consistency.

Step 5: according to Matches₃Computing queriesImage I_lAnd a reference picture I_rHomography matrix H between_lrAs follows:

note k (x'_q,y′_qAnd 1) is k (x'_q,y′_q) Further, the homogeneous coordinate of (c) can be calculated as the feature point k (x'_q,y′_q) In querying image I_lThe corresponding point in (1):

if characteristic point k (x'_p,y′_p) And a feature point k (x)_p,y_p) Is less than 0.5, then it is considered that<k(x_p,y_p),k(x′_q,y′_q)>Is a correct pair of feature matching points that otherwise should be removed from the Matches₃And then the feature matching result shown below can be obtained.

Matches₄＝{<k(x_g,y_g)，k(x′_s,y′_s)>|g∈[1,M],s∈[1,N]} (7)

Wherein,<k(x_g,y_g)、k(x′_s,y′_s)>is a pair of feature matching results, k (x)_g,y_g) Representing query feature Point, k (x'_s,y′_s) Reference feature points are indicated.

Step 6: in practice, due to the solving of the homography matrix H_lrAffected by noisy data, resulting in feature matching results Matches₄There still exist erroneous feature matching points, and for these erroneous matching points, both statistical-based and clustering-based error elimination methods can be used to remove them from the mathes₄So as to obtain a completely correct feature matching result, as follows:

Matches_Final＝{<k(x_v,y_v),k(x′_c,y′_c)>|v∈[1,M],c∈[1,N]} (8)

wherein M and N are respectively query images I_lAnd a reference picture I_rThe number of feature points in (1).

In fig. 2, the following operations are performed in sequence from step b to step c:

firstly, carrying out initial matching of feature points, and then respectively adopting: and the method comprises the steps of proportional test, cross validation, geometric constraint, statistics and clustering, and the wrong feature matching is eliminated layer by layer respectively, so that a correct feature matching result is obtained.

In the above embodiment, the input image is shown in fig. 3, where fig. 3(a) is a query image and fig. 3(b) is a reference image; in the calculation process of image matching, no preprocessing operation is needed to be carried out on the input image.

As shown in fig. 4, the initial output matching result of the present embodiment is shown in fig. 4, where the green straight line represents the correct feature matching point; both ends of the connecting line represent the positions of the feature points in the image.

Through the embodiment, the invention provides a high-level feature matching method by analyzing various factors causing the feature matching error, eliminates wrong feature matching points layer by layer and lays a foundation for improving the performance of a high-level computer vision application system based on image matching. The corresponding feature matching points can be calculated quickly and efficiently, and then the method is applied to a series of high-level computer vision application systems based on image matching.

Claims (7)

1. A hierarchical image matching method is characterized in that: the method comprises the following steps:

step S1: for two images input, i.e. query image I_lAnd a reference picture I_rFor which a local feature point detection method and a local feature descriptor calculation method are respectively used to calculate the query image I_lAnd a reference picture I_rThe local feature points and the corresponding feature descriptors contained in each of the two groups;

wherein the query image I_lDetected local feature point of (2) is K_l＝k(x_i，y_i) The corresponding feature descriptor is D_l＝d(k(x_i，y_i))；i∈[1，M]M denotes a query image I_lThe number of detected local feature points;

from reference picture I_rThe detected local feature point in (1) is K_r＝k(x′_j，y′_j) The corresponding feature descriptor is D_r＝d(k(x′_j，y′_j))；j∈[1，N]N denotes a slave reference picture I_rThe number of detected local feature points;

step S2: from the local feature points contained in the reference image, for each local feature point k (x) in the query image_i，y_i) Finding two most similar candidate matching feature points, namely k (x'_j，y′_j) And k (x'_j+1，y′_j+1)；

Step S3: calculating two candidate matching feature points k (x'_j，y′_j) And k (x'_j+1，y′_j+1) And a feature point k (x)_i，y_i) The distance difference ratio rho between the two, and then determining the initial feature matching result Matches according to the distance difference ratio rho₁；

Step S4: for initial feature matching results Matches₁Eliminating error feature matching generated by ambiguity of feature descriptor by using cross-validation method, and obtaining feature matching result as Matches₂；

Step S5: for feature matching results Matches₂Eliminating all error feature matching points caused by not meeting geometric consistency to obtain feature matching result Matches₃；

Step S6: for feature matching results Matches₃Eliminating error feature matching points caused by the influence of noise data by using a clustering-based method so as to obtain a final feature matching result, namely Matches_Final。

2. The hierarchical image matching method according to claim 1, wherein: in step S1, the specific method for calculating the local feature points and the corresponding feature descriptors by using the local feature detection method based on deep learning is as follows:

step S1.1, pre-training local feature points: the method comprises the steps of obtaining two-dimensional images by making corresponding three-dimensional objects and carrying out picture interception of a visual angle on the objects, and using all known local feature points in the two-dimensional images for network training;

s1.2, self-labeling of feature points: adopting ImageNet as a training data set and a testing data set; training by using a synthetic scene to obtain a basic characteristic point detection network model, and extracting characteristic points on the ImageNet data set by using the basic characteristic point detection network model to obtain characteristic point self-labeling;

step S1.3, combined training: and performing geometric transformation on the pictures used in the last step, inputting the corresponding picture pairs into a basic feature point detection network, extracting feature points and descriptors, performing joint training to obtain local features based on deep learning, and thus performing detection on the local feature points and calculating the feature descriptors.

3. The hierarchical image matching method according to claim 1, wherein: k (x ') is obtained through calculation in the step S2 by using a hierarchical partial perceptual hashing method'_j，y′_j) And k (x'_j+1，y′_j+1) The specific process is as follows:

s2.1, constructing Hash index

Note G ═ G: s → U represents a family of functions, where S represents input and U represents output; g (v) ═ h₁(v)，…，h_k(v))，h_iE H, representing the random independent selection of L hash functions G from the family of functions G₁，…，g_L(ii) a For any point v in the data set, store it in bucket g_i(v) Wherein i is 1, …, L;

s2.2 Hash search process

For a query point q ═ k (x)_i，y_i) To be provided withAnd given a distance threshold r, from the search bucket g₁(q)，…，g_L(q) taking out all points v therein₁，…，v_nAs candidate approximate nearest neighbors; for arbitrary v_jIf D (q, v)_j) R is less than or equal to v_jWhere D (-) is a similarity measure function and q represents a query point;

and finding two optimal candidate matching points through the process.

4. The hierarchical image matching method according to claim 1, wherein: the feature point k (x) in step S3_i，y_i) Local feature point k (x ') matched with two candidates'_j，y′_j) And k (x'_j+1，y′_j+1) The calculation method of the difference ratio rho between the two is as follows:

if it is

Then k (x'_j，y′_j) Is a local feature point k (x)_i，y_i) To obtain initial feature matching results from the correct matching points₁：

Matches₁＝{＜k(x_i，y_i)，k(x′_j，y′_j)＞|i∈[1，M]，j∈[1，N]}；

Wherein d (k (x)_i，y_i) Denotes a local feature point k (x)_i，y_i) The feature descriptor of (1); d (k (x'_j，y′_j) Denotes local feature point k (x'_j，y′_j) The feature descriptor of (1); d (k (x'_j+1，y′_j+1) Denotes local feature point k (x'_j+1，y′_j+1) The feature descriptors of (1).

5. The hierarchical image matching method according to claim 1, wherein: in the step S4, a cross-validation method is used to eliminate the wrong feature matching points, and the specific method is as follows:

firstly, the characteristic point k (x) of the query image is searched_i，y_i) And feature point k (x ') in reference image'_j，y′_j) Matching is carried out, and the characteristic points k (x'_j，y′_j) And query feature point k (x)_i，y_i) Matching is carried out, if the results of the two matching are not consistent, the results will be<k(x_i，y_i)，k(x′_j，y′_j)>From Matches₁Deleting to obtain more accurate feature matching results of Matches₂。

6. The hierarchical image matching method according to claim 1, wherein: in the step S5, geometry constraint method Matches is adopted₂The method for matching the error features of the image data, which do not meet the geometric consistency, comprises the following steps:

first-choice computing query feature point k (x)_i，y_i) And reference characteristic point k (x'_j，y′_j) Homography matrix H between, and then calculates query feature point k (x)_i，y_i) And Hk (x'_j，y′_j) τ between, and if τ < 0.5, then it is considered to be<k(x_i，y_i)，k(x′_j，y′_j)>Is a correct pair of specific matching points; otherwise, it is a wrong specific matching point, which should be selected from Matches₂Eliminate to obtain accurate feature matching results Matches₃；

Wherein, Hk (x'_j，y′_j) Is a characteristic point k (x'_j，y′_j) Location in the query image.

7. The hierarchical image matching method according to claim 1, wherein: in step S6, a feature matching error elimination method based on clustering is used to eliminate the error feature matching points layer by layer, and the specific content is as follows:

s6.1, layeringClustering method for Matches₃Clustering the reference characteristic points to find out a central point C₁；

S6.2, with C₁Computing circumscribed rectangle Rect of reference feature point for center₁；

S6.3, detecting Rect according to homography transformation₁In querying image I_rOf (1) corresponding region Rect'₁；

S6.4, from Matches₃Delete in region Rect'₁Obtaining final feature matching results from the other matching points_Final。

Images