CN109697692B - Feature matching method based on local structure similarity - Google Patents

Abstract

The invention provides a feature matching method based on local structure similarity, which is used to solve the problem of unsatisfactory matching results caused by noise interference in the process of image registration. The steps include: Step 1, perform feature extraction and initial matching on the two images to be matched; Step 2, establish the neighborhood affine coefficient matrix of the feature points; Step 3, for each match in the initial matching set, calculate the associated The difference of the neighborhood affine coefficient matrix of the feature points; step 4, optimize the neighborhood affine coefficient matrix to obtain the degree of local structure difference; step 5, according to the local structure difference value of each matching associated feature point, Set the comparison threshold and determine the final feature matching pair as the matching relationship result of the image to be matched. The invention technically overcomes the problems of complex optimization process and slow convergence in the prior art, and effectively improves the matching efficiency.

Description

A Feature Matching Method Based on Local Structure Similarity

technical field

The invention belongs to the field of image processing, in particular to image processing when the matching is not accurate enough under the influence of image noise, in particular to a feature matching method based on local structure similarity.

Background technique

With the rapid development of multimedia technology, images have become an important carrier of information transmission, and digital image processing technology has become more and more important. Among them, image matching technology has become the focus of people's attention in recent years. Because other research directions of digital image processing technology, such as image recognition, image retrieval, object recognition, and object tracking, are further developed on the basis of image matching technology. It can be said that the progress of image matching technology can drive the overall development of digital image processing technology. However, image matching technology is not only a research hotspot, but also a research difficulty. The goal of matching is to find the exact correspondence of the same objects in the image, but its implementation process is subject to many limitations. For example, the images to be matched may come from different photographic equipment, different shooting scenes, or even different shooting times. Different storage devices, viewing angles, lighting changes, and background clutter cause object distortion, geometric deformation, noise interference, etc. These effects undoubtedly bring a huge test to image matching technology.

In recent years, many scholars have done a lot of research on image matching technology, and achieved good academic results. The general image matching method is based on image feature points and feature description vectors. An initial matching set is obtained by computing the distance between feature descriptions. Since the commonly used feature extraction algorithm has good discrimination and scale invariance, it can ensure that a large part of the initial matching set obtained through feature description is the correct matching pair. The subsequent algorithm is to remove the wrong matches in the initial matching set by setting geometric constraints or relational constraints, and obtain the final correct matching set. The most common is to use the graph matching algorithm to realize the optimization process of the matching set. The vertices in the graph represent the feature points of the image, and the edges in the graph represent the association of the feature points of the image. The energy function is set to represent the similarity between points and points, and between edges. The matching constraint effect is achieved by minimizing the function. On this basis, carry out secondary optimization, select 3 neighbor points around the key point to linearly represent the point, apply the obtained coefficient matrix to the matching point corresponding to the point, and bring it into the energy function formula to solve it using linear programming to determine the final matching results. However, the selection of neighbor points in this method is not flexible enough, and the subsequent solution process is too complicated and time-consuming, which cannot achieve efficient matching.

Contents of the invention

Aiming at the problems existing in the above matching methods, the present invention provides a feature matching method based on local linear structure similarity. Compared with the existing technology, this method flexibly utilizes the image structure information, simplifies the calculation amount in the optimization process, and greatly improves the matching precision and recall rate.

Purpose of the invention: The present invention aims to solve the problem of insufficient existing matching methods, and proposes a feature matching method based on local structure similarity.

Technical solution: The present invention is a feature matching method based on local structure similarity. The method is characterized in that the image feature description is extracted to obtain an initial matching set, and the key points are linearly represented by adjacent points of key points to obtain a coefficient matrix, and the matching pairs are measured based on the coefficient matrix. The geometric consistency of the local area of the point, the consistency is used as the confidence judgment standard, the matching pair with a small confidence degree is deleted, and the confidence degree is retained to determine the final matching set. Specifically include the following steps:

Step 1, perform feature extraction and initial matching on the two images to be matched, so as to obtain the initial matching corresponding relationship set;

Step 2, for the feature points determined in the initial matching set obtained in step 1, determine the neighbor points of each feature point, and establish the neighborhood affine coefficient matrix of the feature points accordingly;

Step 3. For each match in the initial matching set, calculate the difference of the neighborhood affine coefficient matrix of the associated feature points, and use this difference to represent the local structural similarity between the feature points associated with each match. The smaller the difference value, the higher the local structural similarity, and the larger the difference, the lower the local structural similarity;

Step 4, optimize the neighborhood affine coefficient matrix, define a function formula that uses the neighborhood affine coefficient matrix as a variable to calculate the degree of local structure difference, find the corresponding neighborhood affine coefficient matrix when the function formula takes an extreme value, and set This coefficient matrix is brought into the functional formula to calculate the degree of local structure difference of feature points;

Step 5: Set a comparison threshold according to the local structure difference value of the feature points associated with each match obtained in step 4, retain the matches whose difference value is lower than the threshold, delete the match whose difference value is not lower than the threshold, and determine the final The feature matching pair is used as the matching relationship result of the image to be matched.

Further, the detailed steps of the present invention are as follows:

Step 1, image feature extraction and initial matching: extract local feature points of the image to be matched, that is, find distinguishable key points in the image that are robust to image transformation and have a high detection repetition rate. Gradient statistical calculation is performed on the local area of the detected feature points to complete the feature description process of the feature points. For any feature point, calculate the Euclidean distance value between its descriptor and the description of other feature points, and select the feature point corresponding to the smallest Euclidean distance value as its matching point. Gather all feature points and their matching points to form an initial image matching set;

Step 2, establish the neighborhood affine coefficient matrix of key points in the matching pair: on the basis of obtaining the initial matching set in step 1, for each local feature point in the matching set, find other feature points within a certain range from the local feature point As its neighborhood feature points, and use these neighborhood feature points to linearly represent the key point, and obtain the corresponding affine coefficient matrix;

Step 3, measure the degree of local structure difference based on the neighborhood affine coefficient matrix: for each match in the initial matching set obtained in step 1, obtain the neighborhood affine coefficients of the feature points associated with each match through step 2 Matrix, calculate the difference value of two neighborhood affine coefficient matrices, this difference value represents the degree of local structure difference of the feature points associated with each match. The smaller the difference value, the higher the local structural similarity, and the larger the difference, the lower the local structural similarity;

Step 4, Neighborhood Affine Coefficient Matrix Optimization: Define a function formula that uses the neighborhood affine coefficient matrix as a variable to calculate the degree of local structure difference. The function formula consists of the sum of two data items, and the two data items are the initial The difference between the corresponding feature point in the matching set and the affine combination formed by its neighborhood affine coefficient matrix. The goal of solving the formula is to make the value of the function as close to zero as possible, and obtain the neighborhood analog corresponding to the extreme point of the function formula. Affine coefficient matrix, and bring this neighborhood affine coefficient matrix into the functional formula to calculate the degree of local structure difference of feature points;

Step 5, compare the threshold value of the local structure difference value corresponding to each match in step 4, when the difference value is less than the specified threshold, the corresponding feature point pair belongs to the correct matching pair, and when the difference value is not less than the specified threshold, the corresponding feature point pair The feature point pairs belong to the wrong matching pairs, and the wrong matching pairs are removed from the initial matching correspondence set, and all the remaining matching pairs are taken as the final correct matching results and output.

Furthermore, the implementation process of step 1 is specifically as follows: Step 1.1, image feature extraction: the classic scale-invariant feature transform (SIFT) feature description operator is used to extract image features. A pair of images to be matched I and I' are used as the input of the SIFT algorithm for feature extraction to obtain SIFT feature points and 128-dimensional feature description vectors corresponding to the feature points;

Step 1.2, feature initial matching: According to the feature vector obtained in step 1.1, let the single feature description vectors of the two images I and I' to be matched be X _i and X _j respectively, and calculate the Euclidean distance between the two vectors as d(X _i ,X _j ), if and only if _the ratio _of the distance between Xi and all other feature description vectors to d(X _i ,X _j ) is greater than the set threshold, we consider that Xi and X _j are likely to match, and When the ratio is less than or equal to the set threshold, the corresponding feature points are considered as unmatched. The distance formula between feature description vectors is:

Among them, k represents the dimension number of the 128-dimensional feature description vector, and the range is [1,128]. X _ik represents the k-th dimension component of vector X _i , and X _jk represents the k-th dimension component of vector X _j ;

In other words, this step is: given a pair of images to be matched, measure the distance value between all the feature description vectors in one image and the feature description vector in the other image one by one, and select the corresponding feature point when the distance value is the smallest As its initial matching, each matching pair will correspond to the feature region in the image to be matched. Collect all the initial matching pairs to form the initial matching set of the image to be matched.

Step 2 is as follows: step 2.1, select key neighborhood points: obtain the initial matching set of the image according to step 1.2, let the matching set arbitrarily match the feature point pair (p, p') corresponding to M, where the feature point p belongs to the image I to be matched, And p' corresponds to the point that matches the feature point p in the image I' to be matched; find other matches that match the adjacent area of M, and let the corresponding feature point pair be (q _i , q _i '); then when the feature point p and q _i , and when the spatial distance between feature point p' and q _i ' is less than the specified threshold, we consider that feature point q _i belongs to the neighbor point of feature point p, and feature point p' belongs to the neighbor of feature point q _i ' point, and when its distance value is greater than or equal to the specified threshold, we consider that the feature point p and q _i , p' and q _i ' do not have a neighbor relationship; the neighbor relationship judgment formula is as follows:

||pq _i ||<τ, and||p'-q _i '||<τ,

Among them, τ represents the similarity threshold between the key point and its neighbors. The setting of τ is used to ensure that the key point p of the image to be matched has k nearest neighbors on average, and the value of k can be set according to the local neighborhood information of the key point in order to more accurately represent the structural information of the key point. i represents the number of neighbor points of the feature point p, and the value range is [1, k]. In the experiment, the value of τ can be adjusted to flexibly change the size of k. By calculating the neighbor relationship judgment formula for each feature point and other feature points, the neighbor point set of each feature point will be obtained; that is, the neighbor point of the image point to be matched is determined through this step;

其中k表示特征点p的邻居点的个数，q_i表示p的第i个邻居点；遵循局部线性嵌入(Locally Linear Embedding)理论，图像特征点p附近的几何结构可以用其邻域仿射系数矩阵w＝[w₁,...,w_i,...]^T来刻画，则特征点p用其已确定的邻居点集

的一个仿射组合表示为：p＝∑w_iq_i，其中∑w_i＝1；而对于特征点p在图像I’上对应的匹配特征点p’，则用p’的邻域仿射系数矩阵w'＝[w'₁,...,w'_i,...]^T来刻画，所以得到p'＝∑w'_iq'_i，其中∑w'_i＝1，这样就完成了关键点邻域仿射系数矩阵的建立；Step 2.2, establish the neighborhood affine coefficient matrix of the key points in the matching pair: on the basis of step 2.1, we determine the neighbor points of the image point to be matched, and arbitrarily match the neighbors of the feature point p corresponding to M in image I The point integration is expressed as

Where k represents the number of neighbor points of the feature point p, and q _i represents the i-th neighbor point of p; following the theory of Locally Linear Embedding, the geometric structure near the image feature point p can be affine with its neighborhood Coefficient matrix w=[w ₁ ,..., _wi ,...] ^T to characterize, then feature point p uses its determined neighbor point set

An affine combination of is expressed as: p=∑w _i q _i , where ∑w _i =1; and for the matching feature point p' corresponding to the feature point p on the image I', the neighborhood affine of p' is used Coefficient matrix w'=[w' ₁ ,...,w' _i ,...] ^T to characterize, so get p'=∑w' _i q' _i , where ∑w' _i =1, so complete The establishment of the affine coefficient matrix of the key point neighborhood is established;

The details of step 3 are as follows: Judgment of local structure similarity of matching points: Through the processing of image feature points in step 2, the respective neighborhood affine of any pair of feature points (p, p') associated with any matching M in the initial matching set is obtained The transformation matrix w and w', because the neighborhood affine coefficient matrix describes the local structural geometric properties of the feature points, so the consistency of the neighborhood coefficient matrix of the matching feature point pair means that the corresponding matching pair is the correct matching pair, that is When w=w', keep the corresponding matching pair as the correct matching pair, if the two are not equal, it will be considered as a wrong matching pair and will be deleted;

与特征点p的邻居点集

线性组合，再与特征点p相减，获取两者的差异值并进行范式处理，再用最优仿射系数矩阵系数向量

与特征点p的匹配点p’的邻居点集

线性组合，之后与特征点p’相减并进行范式处理；定义矩阵协误差J由以上两个范式之和组成，用来表示匹配点对的局部结构差异程度；矩阵协误差J的具体形式如下：Step 4 is as follows, optimize the neighborhood affine coefficient matrix of feature points: In order to optimize the neighborhood affine coefficient matrix of feature points, find an optimal neighborhood affine coefficient matrix

Neighbor point set with feature point p

Linear combination, and then subtracted from the feature point p to obtain the difference value between the two and perform paradigm processing, and then use the optimal affine coefficient matrix coefficient vector

Neighbor point set of matching point p' with feature point p

Linear combination, and then subtracted from the feature point p' and processed in a normal form; the definition matrix co-error J is composed of the sum of the above two paradigms, which is used to represent the degree of local structural difference of the matching point pair; the specific form of the matrix co-error J is as follows :

A correct match that maps the same object in the image to be matched has associated feature points in the same image that are usually adjacent in spatial location and share a similar topology between the two images to be matched due to physical constraints, such that The goal of solving the corresponding matrix co-error is to make the value of J as small as possible. Correspondingly, the geometric structure represented by the feature points in the adjacent area of the wrong matching feature point is difficult to keep consistent on different images. Therefore, the matrix co-error J is to judge the local structural similarity of the feature point, that is The accuracy of the matches in the initial set of matches provides a good discriminant.

Step 5 is specifically as follows: Based on the degree of difference in the local structure of each matching feature point obtained in step 4, that is, the value of the matrix co-error. Set a small threshold, and stipulate that when the matrix co-error of the feature point is less than the set threshold, the matching associated with this feature point is considered to be a correct match, and it will be kept in the final matching set; otherwise, when the feature point’s When the matrix covariance is greater than the set threshold, it is considered that the matching associated with this feature point is a wrong matching, and it will be deleted; finally, all the remaining matching sets will be used as the final image matching result set.

Beneficial technical effect

The feature matching method provided by the present invention is used to solve the problem of unsatisfactory matching results caused by noise interference in the process of image registration. A feature matching method based on local structure similarity is adopted. The method comprises the following steps: step 1, performing feature extraction and initial matching on two images to be matched, thereby obtaining an initial matching correspondence set; step 2, determining the neighbors of each feature point for the feature points determined in the obtained initial matching set point, and establish the neighborhood affine coefficient matrix of the feature point accordingly; step 3, for each match in the initial matching set, calculate the difference of the neighborhood affine coefficient matrix of the feature point associated with it, expressed by this difference The local structure similarity between the feature points associated with each match; step 4, optimize the neighborhood affine coefficient matrix, define the function formula that uses the neighborhood affine coefficient matrix as a variable to calculate the degree of local structure difference, and the function The formula takes the extreme value and further obtains the degree of local structure difference; step 5, according to the local structure difference value of the feature point associated with each match, set the comparison threshold, keep the match whose difference value is lower than the threshold, and delete the difference value not less than Threshold matching determines the final feature matching pair as the matching relationship result of the image to be matched.

The invention is based on an image matching method with similar local structures, and designs an effective solution to the problem of noise interference that cannot be avoided in the current image matching technology. The local structure description of feature points is defined by using the affine relationship between image feature points and their neighbors, and the error matching is deleted by comparing the local structure of matching feature points, so as to achieve the purpose of removing noise. The present invention is mainly based on the idea that the matching pairs mapping the same object in the image to be matched are usually located in the adjacent area . It is impossible for the noise points to satisfy the two constraints of the neighborhood point range and the similar local structure at the same time, so it can effectively eliminate noise. On the other hand, in the actual implementation process of the present invention, the structure information of the image itself is flexibly utilized, and an affine relationship constraint is given to accurately realize image matching, and the calculation process is simplified, technically overcoming the prior art optimization The problem of complex process and slow convergence can effectively improve the efficiency of matching.

Description of drawings

Fig. 1 is the basic flowchart of the inventive method

Figure 2 is a representation of the relationship between feature points and their neighbors

Figure 3 is the specific implementation method of the experimental matching effect diagram

Below in conjunction with accompanying drawing, the present invention is further described:

See Figure 1, a matching method based on local structure similarity, by sequentially processing a pair of images to be matched as follows:

Step 1, perform feature extraction and initial matching on the two images to be matched, so as to obtain the initial matching correspondence set; Step 2, determine the neighbor points of each feature point for the feature points determined in the initial matching set obtained in step 1 , and establish the neighborhood affine coefficient matrix of the feature points accordingly; step 3, for each match in the initial matching set, calculate the difference of the neighborhood affine coefficient matrix of the feature points associated with it, and use this difference to represent each The local structural similarity between feature points associated with each match. The smaller the difference value, the higher the local structure similarity, and the larger the difference, the lower the local structure similarity; step 4, optimize the neighborhood affine coefficient matrix, define the neighborhood affine coefficient matrix as a variable to calculate the local structure difference degree of functional formula, find the corresponding neighborhood affine coefficient matrix when the functional formula takes the extreme value, and bring this coefficient matrix into the functional formula to calculate the degree of local structure difference of feature points; step 5, according to each matching correlation obtained in step 4 The local structure difference value of the associated feature points, set a comparison threshold, retain the matching whose difference value is lower than the threshold, delete the matching whose difference value is not lower than the threshold, and determine the final feature matching pair as the matching relationship result of the image to be matched.

Further, the specific steps are: Step 1, image feature extraction and initial matching: extract local feature points of the image to be matched, that is, find distinguishable key points in the image that are robust to image transformation and have a high detection repetition rate. Gradient statistical calculation is performed on the local area of the detected feature points to complete the feature description process of the feature points. For any feature point, calculate the Euclidean distance value between its descriptor and the description of other feature points, and select the feature point corresponding to the smallest Euclidean distance value as its matching point. Gather all feature points and their matching points to form an initial image matching set;

Step 2, establish the neighborhood affine coefficient matrix of key points in the matching pair: on the basis of obtaining the initial matching set in step 1, for each local feature point in the matching set, find other feature points within a certain range from the local feature point As its neighborhood feature points, and use these neighborhood feature points to linearly represent the key point, and obtain the corresponding affine coefficient matrix;

Step 3, measure the degree of local structure difference based on the neighborhood affine coefficient matrix: for each match in the initial matching set obtained in step 1, obtain the neighborhood affine coefficients of the feature points associated with each match through step 2 Matrix, calculate the difference value of two neighborhood affine coefficient matrices, this difference value represents the degree of local structure difference of the feature points associated with each match. The smaller the difference value, the higher the local structural similarity, and the larger the difference, the lower the local structural similarity;

Step 4, Neighborhood Affine Coefficient Matrix Optimization: Define a function formula that uses the neighborhood affine coefficient matrix as a variable to calculate the degree of local structure difference. The function formula consists of the sum of two data items, and the two data items are the initial matching Focus on the difference between the corresponding feature points and the affine combination formed by the domain affine coefficient matrix. The goal of solving the formula is to make the function value as close to zero as possible, and obtain the neighborhood affine coefficient corresponding to the extreme point of the function formula Matrix, and bring this neighborhood affine coefficient matrix into the functional formula to calculate the degree of local structure difference of feature points;

Step 5, compare the threshold value of the local structure difference value corresponding to each match in step 4, when the difference value is less than the specified threshold, the corresponding feature point pair belongs to the correct matching pair, and when the difference value is not less than the specified threshold, the corresponding feature point pair The feature point pairs belong to the wrong matching pairs, and the wrong matching pairs are removed from the initial matching correspondence set, and all the remaining matching pairs are taken as the final correct matching results and output.

As shown in the flow chart in Fig. 1, this method is a serial matching process: first extract image feature points and feature descriptors, and obtain an initial matching set of images according to the similarity of descriptions. Then find the adjacent points of each feature point, and establish the neighborhood affine coefficient matrix of the matching key points in the initial matching set, and define the matrix co-error formula to optimize the neighborhood affine coefficient matrix of the feature point, and finally keep the corresponding The matrix co-error is less than the specified threshold, and the matching not less than the specified threshold is deleted, and the final feature matching set is determined as the matching result of the image to be matched.

Specifically, as shown in FIG. 1 , the present invention discloses a matching method based on local line structure similarity. It mainly includes the following steps:

Step 1, image feature extraction and initial matching: extract the feature points and feature descriptions of the image to be matched, and obtain the initial matching set of images based on the similarity of the description.

Step 1.1, image feature extraction: the classic scale-invariant feature transform (SIFT) feature description operator is used to extract image features. A pair of images to be matched I and I' are used as the input of the SIFT algorithm for feature extraction, and the SIFT feature points and the 128-dimensional feature description vectors corresponding to the feature points are obtained ;

Step 1.2, feature initial matching: According to the feature vector obtained in step 1.1, let the single feature description vectors of the two images I and I' to be matched be X _i and X _j respectively, and calculate the Euclidean distance between the two vectors as d(X _i ,X _j ), if and only if _the ratio _of the distance between Xi and all other feature description vectors to d(X _i ,X _j ) is greater than the set threshold, we consider that Xi and X _j are likely to match, and When the ratio is less than or equal to the set threshold, the corresponding feature points are considered as unmatched. The distance formula between feature description vectors is:

Where X _ik represents the k-th dimension component of vector _Xi , and the threshold is taken as 1.1.

Given an image pair to be matched, measure the distance value between all the feature description vectors in one image and the feature description vector in the other image one by one, and select the corresponding feature point when the distance value is the smallest as the matching point. In this way, each matching pair will correspond to each feature region in the image to be matched. Collect all the initial matching pairs to form the initial matching set of the image to be matched.

Step 2, establish the neighborhood affine coefficient matrix of the key points in the matching pair: further explain in conjunction with Figure 2, step 2.1, select the key neighborhood points: get the initial matching set of the image according to step 1.2, let the matching set match M corresponding feature points arbitrarily pair (p,p'), where the feature point p belongs to the image to be matched I, and p' corresponds to the point in the image to be matched I' that matches the feature point p; find other matches that match the adjacent area of M, and let The corresponding feature point pair is (q _i , q _i '); then when the feature point p and q _i , and the spatial distance between feature point p' and q _i ' is less than the specified threshold, we consider that feature point q _i belongs to the feature The neighbor point of point p, the feature point p' belongs to the neighbor point of the feature point q _i ', and when the distance value is greater than or equal to the specified threshold, we consider that the feature point p and q _i , p' and q _i ' do not have a neighbor relationship ; Neighbor relationship judgment formula is as follows:

||pq _i ||<τ, and||p'-q _i '||<τ,

Among them, τ represents the similarity threshold between the key point and its adjacent points, and the value is 10. The setting of τ is used to ensure that the key point p of the image to be matched has k nearest neighbors on average, and the value of k can be set according to the local neighborhood information of the key point in order to more accurately represent the structural information of the key point. i represents the number of neighbor points of the feature point p, and the value range is [1, k]. In the experiment, the value of τ can be adjusted to flexibly change the size of k, which allows the experiment to show better results.

其中k表示特征点p的邻居点的个数，qi表示p的第i个邻居点；遵循局部线性嵌入(Locally Linear Embedding)理论，图像特征点p附近的几何结构可以用其邻域仿射系数矩阵w＝[w₁,...,w_i,...]^T来刻画，则特征点p用其已确定的邻居点集

的一个仿射组合表示为：p＝∑w_iq_i，其中∑w_i＝1；而对于特征点p在图像I’上对应的匹配特征点p’，则用p’的邻域仿射系数矩阵w'＝[w'₁,...,w'_i,...]^T来刻画，所以得到p'＝∑w'_iq'_i，其中∑w'_i＝1，这样就完成了关键点邻域仿射系数矩阵的建立；Step 2.2, establish the neighborhood affine coefficient matrix of the key point in the matching pair: on the basis of step 2.1, we determine the neighboring points of the image point to be matched, and arbitrarily match the feature point p corresponding to M in image I Neighbor integration is expressed as

Where k represents the number of neighbor points of the feature point p, qi represents the i-th neighbor point of p; following the theory of Locally Linear Embedding, the geometric structure near the image feature point p can be calculated by its neighborhood affine coefficient matrix w=[w ₁ ,..., _wi ,...] ^T to characterize, then feature point p uses its determined neighbor point set

An affine combination of is expressed as: p=∑w _i q _i , where ∑w _i =1; and for the matching feature point p' corresponding to the feature point p on the image I', the neighborhood affine of p' is used Coefficient matrix w'=[w' ₁ ,...,w' _i ,...] ^T to characterize, so get p'=∑w' _i q' _i , where ∑w' _i =1, so complete The establishment of the affine coefficient matrix of the key point neighborhood is established;

Step 3, local structure similarity judgment of matching points: through the processing of image feature points in step 2, the respective neighborhood affine transformation matrices of the feature point pairs (p, p') associated with any matching M in the initial matching set are obtained w and w', because the neighborhood affine coefficient matrix describes the geometric properties of the local structure of the feature point, so the consistency of the neighborhood coefficient matrix of the matching feature point pair means that the corresponding matching pair is the correct matching pair, that is, w= When w', keep the corresponding matching pair as the correct matching pair, if the two are not equal, it will be considered as a wrong matching pair and will be deleted;

与特征点p的邻居点集

线性组合，再与特征点p相减，获取两者的差异值并进行范式处理，再用最优仿射系数矩阵系数向量

与特征点p的匹配点p’的邻居点集

线性组合，之后与特征点p’相减并进行范式处理；定义矩阵协误差J由以上两个范式之和组成，用来表示匹配点对的局部结构差异程度；矩阵协误差J的具体形式如下：Step 4, optimize the neighborhood affine coefficient matrix of feature points: In order to optimize the neighborhood affine coefficient matrix of feature points, find an optimal neighborhood affine coefficient matrix

Neighbor point set with feature point p

Linear combination, and then subtracted from the feature point p to obtain the difference value between the two and perform paradigm processing, and then use the optimal affine coefficient matrix coefficient vector

Neighbor point set of matching point p' with feature point p

Linear combination, and then subtracted from the feature point p' and processed in a normal form; the definition matrix co-error J is composed of the sum of the above two paradigms, which is used to represent the degree of local structural difference of the matching point pair; the specific form of the matrix co-error J is as follows :

The value of J indicates the degree of difference between the local structures of the feature points p and p', and the smaller the value of J, the more similar the local structures of the feature points p and p'. In principle, correct matches mapping the same object in the image to be matched tend to cluster in adjacent regions and share similar topological structures between images due to physical constraints, thus corresponding to small J values. However, the geometric layout of the matching points near the wrong matching is difficult to keep consistent on different images, so the corresponding J value is relatively large. Therefore, the matrix coerror J provides a good way to evaluate the correctness of the matching. Set a small comparison threshold. When the matrix co-error J is less than the set threshold, the corresponding match is considered to be a correct match, and the match is retained in the initial matching set. When the matrix co-error J is not less than the set threshold , the corresponding match is considered to be a wrong match, and the match is deleted from the initial match set.

对图像间的局部几何变化保持不变的假设基础上。在特征匹配的文献中，通常假定对应局部区域是仿射不变的。具体的证明方法如下：对初始匹配集中的任一匹配M对应匹配点对(p,p’)，存在它的邻近匹配M_i对应匹配点对(q,q’)。特征点p’可以由其匹配特征点p经过一个2×2的旋转缩放矩阵A和一个2×1的平移向量t来逼近，同时，邻近区域的特征点具有相似的变换，所以特征点的p的邻居点q也可以由其匹配点q’经过旋转缩放矩阵A和平移向量t逼近。则：The validity of Equation J is based on the neighborhood affine coefficient matrix

It is based on the assumption that the local geometric changes between images remain unchanged. In the literature of feature matching, it is usually assumed that the corresponding local regions are affine invariant. The specific proof method is as follows: For any matching M in the initial matching set, it corresponds to the matching point pair (p, p'), and its adjacent matching M _i corresponds to the matching point pair (q, q'). The feature point p' can be approximated by its matching feature point p through a 2×2 rotation scaling matrix A and a 2×1 translation vector t. At the same time, the feature points in the adjacent area have similar transformations, so the feature point p The neighbor point q of can also be approximated by its matching point q' through the rotation scaling matrix A and translation vector t. but:

这一约束确保了转换的不变性。这样便证明了邻域仿射系数矩阵

对图像间的局部几何变化保持不变，即如果

则

成立。如果较严重的图像失真出现在关键点p或p’附近，则上述公式可能不成立。幸运的是，图像严重失真并不是常见的情况。总之，我们在此证明，当匹配M和其邻近匹配M_i是正确匹配，则局部区域结构是仿射不变的。in

This constraint ensures the invariance of transformations. This proves that the neighborhood affine coefficient matrix

remains invariant to local geometric changes between images, that is, if

but

established. If more severe image distortion occurs near the key point p or p', the above formula may not hold. Fortunately, severely distorted images are not a common occurrence. In summary, we prove here that when a match M and its neighboring matches M _i are correct matches, the local region structure is affine invariant.

Since the initial matching set will contain false matches, jointly constructing matching feature point pairs with linear combinations of their respective neighborhoods in each image can reduce errors caused by noise. And for purely wrong matches, it can also be optimized to provide stronger robustness for matching under noise interference. The specific process of optimization is as follows:

Formula J can be written as:

则公式又可转化为：Where X=[pq ₁ ,...,pq _i ,...], Y=[p'-q' ₁ ,...,p'-q' _i ,...]. Set the set C = X ^T X + Y ^T Y. Then introduce the Lagrangian multiplier λ to perform

Then the formula can be transformed into:

的值可以计算为

Where 1=[1,...,1] ^T is a column vector of |N|×1. By taking the gradient of J and setting it to zero,

The value of can be calculated as

的解需要显式计算矩阵C的逆，按照定义，矩阵C是对称半正定的。然而，因为匹配M的邻接匹配集的数量一般都大于2，所以矩阵C可以是奇异的。为了使公式线性可解，在实际应用中，我们进一步将恒等矩阵I的一个小乘积作为正则化项加入到矩阵C中。formula

The solution of requires explicit computation of the inverse of matrix C, which by definition is symmetric positive semidefinite. However, since the number of contiguous matching sets matching M is generally greater than 2, matrix C may be singular. In order to make the formula linearly solvable, in practical applications, we further add a small product of the identity matrix I to the matrix C as a regularization term.

C _new ＝C+εI,

和调整

来得到

从而使

在处理完

后，可以将计算得出的

带入矩阵协误差J的公式中计算出对应的J值，进而得出每个匹配对应特征点对的局部结构差异值。Compared to the C trace, ε was set to a small value, 10 ⁻³ tr(C) throughout the experiments. We can solve the system of linear equations by

and adjust

come and get

So that

after processing

After that, the calculated

The corresponding J value is calculated by bringing it into the formula of the matrix co-error J, and then the local structural difference value of each matching feature point pair is obtained.

Step 5, based on the degree of difference in the local structure of each matching feature point obtained in step 4, that is, the value of the matrix co-error. Set a small threshold, the threshold value is 8, and stipulate that when the matrix co-error of the feature point is less than the set threshold, the matching associated with this feature point is considered to be a correct match, and it is reserved for the final matching set. Otherwise, when the matrix covariance of the feature point is greater than the set threshold, it is considered that the match associated with this feature point is a wrong match, and it will be deleted; finally, all the remaining matching sets will be used as the final image matching result set.

Example

The experimental hardware environment of the present invention is: Intel (R) Core (TM) i5-4590CPU@3.30GHz 3.30GHz, 8G memory, Microsoft Windows7 flagship edition, programming environment is Visual Studio 2015, MATLAB (R2016a) 64, test figure ( For details, see the experimental matching effect diagram shown in Figure 3) from the standard image set of multi-target object matching published on the Internet by Seoul National University (SNU).

All the images in the SNU image set are selected. There are 6 groups of images to be matched. The image names are Books, Bulletins, Jigsaws, Mickeys, Minnies, and Toys. Each image contains several objects, and there are differences between each object to be matched. The viewing angle transformation, different lighting and different self-variation all make the matching technology using these images face great challenges.

The present invention compares the accuracy and recall rate with other matching methods on the basis of the realization of the matching technology. The matching methods used for comparison include HV (Feature Matching with Alternate Hough and Inverted Hough Transforms), discrete topology Search matching method DTS (Discrete TabuSearch For Graph Matching), weighted random walk graph matching method RRWM (Reweighted Random Walks for Graph Matching) and spatial matching method EWGR (Spatial Matching asensemble of weak geometric relations) under weak geometric relations. In the matching process, the SIFT description operator is used for feature description, and all are based on obtaining the initial matching set, with the goal of removing the wrong match in the initial matching set. The parameters in the comparison method are all selected when the experimental effect is the best. The performance of a method is expressed using precision and recall. The comparison results are as follows:

HV RRWM DTS EWGR Ours Precision(%) 71.36 77.68 78.22 85.19 95.45 Recall (%) 94.04 90.05 94.17 71.49 97.50

It can be seen from the data in the table that the matching method of the present invention performs better than other matching methods in terms of accuracy and recall data performance, thus proving the significance of the present invention.

Claims (7)

1. A feature matching method based on local structure similarity is characterized in that: the method comprises the following steps of carrying out matching processing on a pair of images to be matched through a computer to obtain a feature point matching pair set with a similar local structure, wherein the matching pair set comprises the following steps:

step 1, performing feature extraction and initial matching on two images to be matched to obtain a relation set corresponding to the initial matching; when feature extraction is carried out, feature points of the initial matching corresponding relation set are obtained;

step 2, determining neighbor points of each feature point for the feature points determined in the initial matching corresponding relation set obtained in the step 1, and establishing a neighborhood affine coefficient matrix of the feature points according to the neighbor points;

step 3, for each match in the initial matching set, calculating the difference of the neighborhood affine coefficient matrixes of the feature points associated with the match, and expressing the local structural similarity between the feature points associated with each match by using the difference: the smaller the difference value is, the higher the local structure similarity is, the larger the difference is, and the lower the local structure similarity is;

step 4, optimizing the neighborhood affine coefficient matrix, calculating a function of the local structure difference degree by taking the neighborhood affine coefficient matrix as a variable, solving the corresponding neighborhood affine coefficient matrix when the function takes an extreme value, and substituting the coefficient matrix into the function to calculate the local structure difference degree of the characteristic point;

and 5, setting a comparison threshold according to the local structure difference value of each matched and associated feature point obtained in the step 4, keeping the matching with the difference value lower than the threshold, deleting the matching with the difference value not lower than the threshold, and determining the final feature matching pair as the matching relation result of the image to be matched.

2. The local structure similarity-based feature matching method according to claim 1, wherein: the method comprises the following specific steps in sequence:

step 1, image feature extraction and initial matching: extracting local characteristic points of the image to be matched, namely finding distinguishable key points which have strong robustness on image transformation and high detection repetition rate in the image; performing gradient statistical calculation on the local area of the detected feature points to complete the feature description process of the feature points; calculating Euclidean distance values between the descriptor of any one feature point and the descriptions of other feature points, and selecting the feature point with the minimum Euclidean distance value as a matching point of the feature point; collecting all the characteristic points and the matching points thereof to form an initial image matching set;

step 2, establishing a neighborhood affine coefficient matrix of key points in the matching pairs: on the basis of obtaining an initial matching set in the step 1, for each local feature point in the matching set, finding other feature points within a certain range from the local feature point as neighborhood feature points of the local feature point, linearly representing the key point by using the neighborhood feature points, and obtaining a corresponding affine coefficient matrix;

step 3, measuring the local structure difference degree based on the neighborhood affine coefficient matrix: for each match in the initial matching set obtained in the step 1, obtaining a neighborhood affine coefficient matrix of the feature point associated with each match through the step 2, and calculating a difference value of the two neighborhood affine coefficient matrices, wherein the difference value represents the local structure difference degree of the feature point associated with each match; the smaller the difference value is, the higher the local structural similarity is, the larger the difference is, the lower the local structural similarity is;

step 4, neighborhood affine coefficient matrix optimization: defining a function formula for calculating the local structure difference degree by taking a neighborhood affine coefficient matrix as a variable, wherein the function formula consists of the sum of two data items, the two data items are respectively the difference between affine combinations consisting of characteristic points corresponding to initial matching set matching and the neighborhood affine coefficient matrix, the solving objective of the formula is to make the value of the function as close to zero as possible, obtain the neighborhood affine coefficient matrix corresponding to the function formula at an extreme point, and bring the neighborhood affine coefficient matrix into the function formula to calculate the local structure difference degree of the characteristic points;

step 5, comparing the threshold value of each local structure difference value corresponding to the matching in the step 4:

when the difference value is smaller than a specified threshold value, the corresponding characteristic point pair belongs to a correct matching pair;

and when the difference value is not less than the specified threshold value, the corresponding characteristic point pairs belong to wrong matching pairs, the wrong matching pairs are removed from the initial matching corresponding relation set, and all the remaining matching pairs are used as final correct matching results and output.

3. A feature matching method based on local structure similarity according to claim 2, characterized in that: the step 1 is as follows:

step 1.1, image feature extraction: extracting the characteristics of the image to be matched by adopting a Scale Invariant Feature Transform (SIFT) characteristic description operator; taking a pair of images I and I' to be matched as input of an SIFT algorithm to perform feature extraction, and acquiring SIFT feature points and feature description vectors corresponding to the feature points; the feature description vector has 128 dimensions;

step 1.2, initial feature matching: according to the feature description vectors obtained in step 1.1, the single feature description vectors of the two images I and I' to be matched in step 1.1 are respectively X _i 、X _j Calculating X _i Vector and X _j The Euclidean distance of the vector is d (X) _i ,X _j ) If and only if X _i Distance to d (X) of all other feature description vectors in image I _i ,X _j ) When the ratio of (A) is greater than a set threshold value, X is judged _i And X _j The corresponding feature points are matched; when the ratio is less than or equal to the set threshold, the corresponding characteristic points are considered as unmatched; the distance between feature description vectors is formulated as:

where k denotes the 128-dimensional eigenvector dimension index number, the range is [1,128 ]]；X _ik Representing a vector X _i The k-th dimension component, X _jk Representing vector X _j The k-th dimension component of (a);

and finally outputting an initial image matching relationship pair set by extracting the characteristics of the images I and I' to be matched in the step 1.1 and calculating the distance value between the characteristic descriptions in the step 1.2.

4. A feature matching method based on local structure similarity according to claim 2, characterized in that: the step 2 is specifically as follows:

step 2.1, selecting key neighborhood points: obtaining an initial matching set of the image according to the step 1.2, and randomly matching M corresponding feature point pairs (p, p ') in the matching set, wherein the feature point p belongs to the image I to be matched, and the point p ' corresponds to the point matched with the feature point p in the image I ' to be matched; find other matches that match M neighbors and let the corresponding pairs of feature points be (q) _i ,q _i ') to a host; when the feature points p and q are _i Simultaneous feature points p' and q _i When the spatial distance value of' is smaller than a prescribed threshold value, we consider the feature point q _i Neighbor points belonging to feature point p, feature point p' belonging to feature point q _i ' and when the distance value is greater than or equal to a specified threshold, we consider the feature points p and q _i P' and q _i ' having no neighborsA relationship; the neighbor relation judgment formula is as follows:

||p-q _i ||＜τ,and||p'-q _i '||＜τ,

wherein tau represents the similarity threshold value of the key point and the adjacent points; tau is set to ensure that key points p of the image to be matched have k nearest neighbors on average, and the setting of k value can be specifically set according to local neighborhood information of the key points so as to more accurately represent the structural information of the key points; i represents the number of neighbor points of the characteristic point p, and the value range is [1,k ]](ii) a Calculating a neighbor relation judgment formula of each feature point and other feature points to obtain a neighbor point set of each feature point; namely, the adjacent local points of the image feature points to be matched are determined through the step; step 2.2, establishing a neighborhood affine coefficient matrix of key points in the matching pair: on the basis of the step 2.1, the adjacent local points of the characteristic points of the image to be matched are determined, and the adjacent points of the corresponding characteristic points p of any matching M in the image I are integrally expressed as

Wherein k represents the number of neighbor points of the characteristic point p, and qi represents the ith neighbor point of p; following local Linear Embedding (local Linear Embedding) theory, the geometric structure near the image feature point p can be defined by its neighborhood affine coefficient matrix w = [ w ] ₁ ,...,w _i ,...] ^T To depict, the feature point p uses its determined neighbor point set

Is expressed as: p = ∑ w _i q _i Where Σ w _i =1; and for the matched feature point p 'corresponding to the feature point p on the image I', using a neighborhood affine coefficient matrix w '= [ w' ₁ ,...,w' _i ,...] ^T To obtain p '= sigma w' _i q' _i Wherein Σ w' _i =1; thus, the establishment of the key point neighborhood affine coefficient matrix is completed.

5. A local structure similarity-based feature matching method as claimed in any one of claims 2 or 4, characterized in that: the step 3 is as follows:

and (3) judging the local structural similarity of the matching points: through the processing of the image feature points in the step 2, neighborhood affine transformation matrixes w and w ' of each feature point pair (p, p ') associated with any matching M in the initial matching set are obtained, because the neighborhood affine coefficient matrixes describe the local structural geometric properties of the feature points, the corresponding matching pairs are retained as correct matching pairs if the corresponding matching pairs are represented in a consistent way by the neighborhood coefficient matrixes of the matching feature point pairs, namely w = w ', and the matching pairs are determined as incorrect matching pairs and deleted if the matching pairs are not equal to each other.

6. A local structure similarity-based feature matching method as claimed in any one of claims 2 or 4, characterized in that: the step 4 is specifically as follows:

optimizing a feature point neighborhood affine coefficient matrix: in order to optimize the neighborhood affine coefficient matrix of the feature points, an optimal neighborhood affine coefficient matrix is found

Neighbor point set with feature point p

Linear combination, subtracting the characteristic point p to obtain the difference value of the characteristic point p and carrying out normal form processing, and then using the coefficient vector of the optimal affine coefficient matrix

Set of neighbor points of matching point p' with feature point p

Linear combination, and then subtraction is carried out on the linear combination and the characteristic point p' and normal form processing is carried out; defining matrix covariance J composed of the sum of the above two normal forms, and representing the local structure difference degree of the matching point pair; the matrix covariance J is specified as follows:

7. the feature matching method based on local structure similarity as claimed in claim 2, wherein: the step 5 is as follows:

acquiring a matrix covariance value corresponding to each feature point in the initial matching set according to the step 4, setting a threshold value, and stipulating:

when the matrix covariance of the feature points is smaller than the set threshold, the matching associated with the feature points is considered to be correct matching, and the final matching set is reserved;

otherwise, when the covariance of the matrix of the feature point is larger than the set threshold value, the matching associated with the feature point is considered to cause the wrong matching, and the wrong matching is deleted;

and finally, taking all the reserved matching sets as a final image matching result set and outputting the final image matching result set.

Images