CN111783834A - Heterogeneous image matching method based on joint graph spectrum feature analysis - Google Patents

Heterogeneous image matching method based on joint graph spectrum feature analysis Download PDF

Info

Publication number
CN111783834A
CN111783834A CN202010493597.3A CN202010493597A CN111783834A CN 111783834 A CN111783834 A CN 111783834A CN 202010493597 A CN202010493597 A CN 202010493597A CN 111783834 A CN111783834 A CN 111783834A
Authority
CN
China
Prior art keywords
matching
feature
point
points
region
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010493597.3A
Other languages
Chinese (zh)
Other versions
CN111783834B (en
Inventor
王鑫
张丽荷
张之露
严勤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hohai University HHU
Original Assignee
Hohai University HHU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hohai University HHU filed Critical Hohai University HHU
Priority to CN202010493597.3A priority Critical patent/CN111783834B/en
Publication of CN111783834A publication Critical patent/CN111783834A/en
Application granted granted Critical
Publication of CN111783834B publication Critical patent/CN111783834B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a heterogeneous image matching method based on joint graph spectrum feature analysis. Firstly, extracting angular points of a visible light image and an infrared image, and constructing an adjacent matrix by using the angular point relation of the two images; secondly, obtaining a characteristic vector of the joint graph by defining the conventional Laplace and decomposing the characteristic value of the adjacent matrix, and then constructing a characteristic function pair by three-dimensional reconstruction; thirdly, providing a SUSAN-MSER-SURF algorithm to detect the extreme value region of each group of feature function pairs, and matching the region features by using the provided Euclidean-Hamming algorithm; and finally, matching the extreme value regions of the feature spectrum pairs reconstructed by the minimum K feature vectors according to the steps to obtain a final matching result.

Description

Heterogeneous image matching method based on joint graph spectrum feature analysis
Technical Field
The invention belongs to the field of image processing, and particularly relates to a heterogeneous image matching method based on joint graph spectrum characteristic analysis.
Background
In the heterogeneous image matching, the traditional matching mode based on gray scale is difficult to accurately match, and the heterogeneous image matching based on characteristics is a common heterogeneous image matching mode. With the characteristic-based algorithms with excellent performance proposed by numerous scholars at home and abroad, the heterogeneous image matching technology has a quite remarkable breakthrough, but the problem of insufficient matching precision exists for images shot under special conditions. Visible light images taken during high-light day or low-light night tend to be blurred, and in this case, the use of the feature-based matching method is often affected by blurring of local feature descriptors, resulting in a decrease in matching accuracy.
Aiming at the scenes, a method based on the frequency spectrum characteristic analysis of the joint image is provided, the structural relation of corners in the visible light image and the infrared image is utilized to construct the joint image by applying the K neighbor rule, the characteristic vector of the joint image is obtained by defining the conventional Laplace and decomposing the characteristic value of the Laplace, and the three-dimensional reconstruction is carried out to construct the characteristic function pair. The extreme value of the characteristic function pair represents the image persistent region, so the invention provides a SUSAN-MSER-SURF maximum stable extreme value region detector for detecting the position of the extreme value and matching the maximum stable extreme value region. Experimental results show that the scheme has excellent matching rate when being applied to the scenes.
Disclosure of Invention
The purpose of the invention is as follows: aiming at the problems, the invention provides a heterogeneous image matching method based on joint graph spectrum feature analysis to solve the problem of matching visible light and infrared images shot under the condition of over-high or over-low light intensity.
The technical scheme is as follows: in order to realize the purpose of the invention, the technical scheme adopted by the invention is as follows: a heterogeneous image matching method based on combined graph spectral feature analysis comprises the following steps:
(1) extracting angular points of the visible light and the infrared image, and constructing an adjacent matrix by using the angular point relation of the two images;
(2) obtaining a feature vector of the joint graph by defining the conventional Laplace and decomposing the feature values of the adjacent matrixes, and then constructing a feature function pair by three-dimensional reconstruction;
(3) providing a SUSAN-MSER-SURF algorithm to detect an extreme value region of each group of feature function pairs, and matching the region features by using the provided Euclidean-Hamming algorithm;
(4) and matching the extreme value regions of the feature spectrum pairs reconstructed by the minimum K feature vectors according to the steps to obtain a final matching result.
Further, in the step (1), corner points of the visible light image and the infrared image are extracted, and an adjacent matrix is constructed by using the relationship between the corner points of the two images, wherein the specific method comprises the following steps:
(1.1) giving a visible light image and an infrared image to be matched, extracting angular points of the visible light image and the infrared image to be matched by using an SIFT algorithm respectively, and collecting the angular points as an angular point set V1And V2Where V is the set of all corner points, called the set of nodes { V }1,v2,...,vnN refers to the number of nodes;
(1.2) corner Point set V1And V2Their correlation diagrams are G1(V1,E1,W1) And G2(V2,E2,W2). Wherein E is1Is V1Set of edges between any two nodes in (E)2Is V2Set of edges between any two nodes in (W)1Is an n × n adjacency matrix containing weight values between all points, defined as follows:
Figure BDA0002521992350000021
n1reference to a set of nodes V1Number of middle nodes, W1Set of reference connections V1N composed of the weight values of the edges of any two nodes1×n1The neighboring matrix of (a); w is aijIs a weight value which explains the angular point viAnd vjThe correlation between the represented pixels.
The joint graph G (V, E, W) is defined as V ═ V1∪V2，E＝E1∪E2∪E12，E12Denotes a connection V1Any one of the nodes and V2A set of edges for any of the nodes. The adjacency matrix W of the join graph G is defined as follows:
Figure BDA0002521992350000031
c means a connection set V1Any one node and set V2N composed of weighted values of edges of any node1×n2An adjacent matrix; cTIs the transposed matrix of C.
(1.3) assigning the adjacent matrix: the basic rule of the adjacent matrix construction is that if the distance between two points is short, the weight value of the edge connecting the two points is larger; the smaller the opposite. Traversing all points in the corner point set by using a KNN algorithm, taking k points nearest to each sample point as the neighbors of the points, and only taking the weight values w between the k points nearest to the sample pointij>0, and the weight values between other points take 0. To ensure symmetry, s is retained as long as one point is in the K neighbor of another pointij. The formula for measuring the edge weight value connecting two corners is as follows:
Figure BDA0002521992350000032
further, in the step (2), a specific implementation manner of constructing a feature function pair by three-dimensional reconstruction is as follows, by defining a feature vector of a joint graph obtained by a conventional laplacian and decomposing feature values of adjacent matrices:
(2.1) using the adjacency matrix W and the degree matrix D thereof, we can calculate the laplacian matrix as L ═ W-D and normalize it;
(2.2) decomposition of formula Using eigenvalues
Figure BDA0002521992350000033
Its feature vector is calculated. Wherein the feature vector U1,U2,...,UkCorresponding to the minimum K characteristic values, and the dimension of each characteristic vector is n1+n2These feature vectors have a close relationship with the structure of the image.
(2.3) for each of the K feature vectors, weSplit it into two vectors, whose dimensions are n respectively1And n2。
(2.4) extracting n from each feature vector1The dimensional vector is reconstructed to the size of the visible image, and the reconstructed feature function is called the feature spectrum by assigning its component values to sampling positions (the sampling positions are the positions where the feature points are extracted), and linearly interpolating the values between the sampling positions
Figure BDA0002521992350000041
N extracted from each feature vector is similarly2The vector of dimensions is reconstructed to the size of the infrared image, resulting in a characteristic spectrum
Figure BDA0002521992350000042
(2.5) according to the above method, we derive from the eigenvector U corresponding to the smallest K eigenvalues1,U2,...,UkRespectively obtaining K groups of characteristic frequency spectrum pairs
Figure BDA0002521992350000043
Further, in the step (3), a specific implementation manner of the SUSAN-MSER-SURF algorithm for detecting the extremum region of each group of feature function pairs is provided as follows:
(3.1) checking all pixel points in the image, if the gray value of the pixel point is larger than or equal to the pixel of the threshold value, saving the pixel point, otherwise, ignoring the pixel point. In the stored points, the adjacent points form an extreme value area;
(3.2) the stability of these along with the components is checked: starting from 0 to 255, the threshold value is gradually increased to the size of delta, the steps are repeated again, and the areas with the size changed by a plurality of threshold values are kept as the maximum stable extremum areas without changing.
(3.3) extracting corner points in the characteristic function by adopting an SUSAN operator, and reserving the MSER region when the number of the corner points in the MSER region is more than or equal to m;
(3.4) performing affine normalization on each detected MSER ellipse, taking the center point of the normalized MSER ellipse as an interest point, taking a circular region around the interest point, and calculating a SURF descriptor for the region by segmenting the region and calculating gradients on each sub-region;
(3.5) Using this method to detect each pair of feature function pairs
Figure BDA0002521992350000044
All MSER regions of (a) normalize it to a point of interest and compute SURF descriptors for it.
Further, in the step (3), the region features are matched by using the proposed Euclidean-Hamming algorithm. The specific implementation mode is as follows:
(4.1) for one feature point in the feature point set of the infrared image, calculating the improved Euclidean distance from the feature point to all the feature points on the visible light image to obtain a set of distances, easily obtaining the minimum distance by comparison, wherein the visible light feature point with the minimum distance from the infrared feature point is the matching point thereof, and if the minimum distance exceeds the threshold value MatchThreshold, excluding the matching point pair. Finding out all matching point pairs of the infrared image and the visible light image according to the rule;
(4.2) setting a comparison parameter MaxRadio, and if the ratio of the minimum distance and the next minimum distance between a certain characteristic point on the visible light and the characteristic point on the infrared image is smaller than the comparison parameter, reserving the matching point pair. Wherein, the improved Euclidean distance adds one more addition and subtraction operation on the original Euclidean distance. For vector a ═ x1,x2,...,xn) Sum vector b ═ y1,y2,...,yn) The formula of the improved Euclidean distance between the two is as follows:
Figure BDA0002521992350000051
and (4.3) screening the coarse matching points matched by the nearest neighbor rule. Let the minimum Euclidean distance of all matching point pairs be denoted as dominRespectively calculate hasThe hamming distance of the matched pairs of points. Comparing the Hamming distances of all matching point pairs if they are less than dominTwice, the matching pair is considered as a mismatch, otherwise, if the value is larger than the value, the matching pair is considered as a correct matching pair.
Has the advantages that: compared with the prior art, the technical scheme of the invention has the following beneficial technical effects:
(1) the invention is different from the prior matching method based on geometric features, and is a matching method for image structure analysis instead. The method does not rely on matching features detected in the two images on their descriptors, but on analyzing the feature spectrum of a combined image formed by the structural relationship of the corner points of the two images.
(2) The union graph is used for combining the structural relation of corner points in two images as an adjacent matrix. The method relies on the structural calculation characteristics of two images at the same time, obtains characteristic vectors through Laplace decomposition, and obtains corresponding vector pairs by splitting the characteristic vectors. The complicated search strategy for searching the corresponding relation between the feature vectors is omitted. The matching method is high in fault tolerance and excellent in time performance.
(3) Since the extreme value of the characteristic function frequency spectrum pair represents the persistent region of the image, the invention extracts and matches the extreme value of the characteristic function pair. Not only the structural information of the graph is considered, but also the stable region characteristics of the image are fully utilized.
(4) We need to study the MSER regions with a dense number of corners, so we propose a SUSAN-MSER-SURF algorithm to detect the extremum regions of each set of feature function pairs. Different from the traditional detection method of the maximum stable extremum region, the detection region is screened after the region is extracted. The MSER algorithm, while avoiding unusually small and large regions, returns only a few general smooth regions. When these regions surround a corner point, they are not completely smooth and therefore their regions are more characteristic.
Drawings
FIG. 1 is an overall flow chart of the present invention;
FIG. 2 is a graph of the matching comparison of visible light and infrared images for low exposure in accordance with the present invention;
FIG. 3 is a graph showing the comparison result of matching between visible light and infrared images under high exposure conditions in accordance with the present invention;
FIG. 4 is a graph of accuracy polyline comparison results for various heterogeneous image matching methods;
FIG. 5 is a comparison graph of RMSE statistics;
FIG. 6 is a graph comparing matching times.
Detailed Description
The technical solution of the present invention is further described below with reference to the accompanying drawings and examples.
As shown in fig. 1, the present invention provides a heterogeneous image matching method based on a composite operator, which comprises the following steps:
(1) respectively acquiring edge images of a visible light image and an infrared image by adopting an improved Harris edge extraction operator;
(2) extracting feature points of the edges of the visible light image and the infrared image by using a SURF operator, and establishing a feature description vector;
(3) an improved nearest neighbor principle is provided, matching pairs are extracted by utilizing the similarity between Euclidean distance measurement feature points, the average Euclidean distance and the variance of all the matching point pairs are calculated, a threshold value is set according to the variance size, the wrong matching point pairs are removed, and coarse matching is realized;
(4) and finally, fitting all rough matching point pairs by adopting a gradient descent method based on a neural network to obtain a function model, calculating the errors of all the rough matching point pairs and the function model, and removing the matching point pairs with large errors to obtain an accurate matching result.
Further, in the step (1), corner points of the visible light image and the infrared image are extracted, and an adjacent matrix is constructed by using the relationship between the corner points of the two images, wherein the specific method comprises the following steps:
(1.1) giving a visible light image and an infrared image to be matched, extracting angular points of the visible light image and the infrared image to be matched by using an SIFT algorithm respectively, and collecting the angular points as an angular point set V1And V2Where V is the set of all corner points, called the set of nodes { V }1,v2,...,vnN refers to the number of nodes;
(1.2) corner Point set V1And V2Their correlation diagrams are G1(V1,E1,W1) And G2(V2,E2,W2). Wherein E is1Is V1Set of edges between any two nodes in (E)2Is V2Set of edges between any two nodes in (W)1Is an n × n adjacency matrix containing weight values between all points, defined as follows:
Figure BDA0002521992350000071
n1reference to a set of nodes V1Number of middle nodes, W1Set of reference connections V1N composed of the weight values of the edges of any two nodes1×n1The neighboring matrix of (a); w is aijIs a weight value which explains the angular point viAnd vjThe correlation between the represented pixels.
The joint graph G (V, E, W) is defined as V ═ V1∪V2，E＝E1∪E2∪E12，E12Denotes a connection V1Any one of the nodes and V2A set of edges for any of the nodes. The adjacency matrix W of the join graph G is defined as follows:
Figure BDA0002521992350000072
c means a connection set V1Any one node and set V2N composed of weighted values of edges of any node1×n2An adjacent matrix; cTIs the transposed matrix of C.
(1.3) assigning the adjacent matrix: the basic rule of the adjacent matrix construction is that if the distance between two points is short, the weight value of the edge connecting the two points is larger; otherwise, the smaller the. Traversing all points in the corner point set by using a KNN algorithm, taking k points nearest to each sample point as the neighbors of the points, and only taking the weight values w between the k points nearest to the sample pointij>0, and the weight values between other points take 0. To ensure symmetry, s is retained as long as one point is in the K neighbor of another pointij. The formula for measuring the edge weight value connecting two corners is as follows:
Figure BDA0002521992350000081
further, in the step (2), a specific implementation manner of constructing a feature function pair by three-dimensional reconstruction is as follows, by defining a feature vector of a joint graph obtained by a conventional laplacian and decomposing feature values of adjacent matrices:
(2.1) using the adjacency matrix W and its degree matrix D, we can calculate the laplacian matrix as L ═ W-D and normalize it. For the definition of its degree matrix D, it is an n × n diagonal matrix, with only the main diagonal having a value corresponding to the sum of each column. It is defined as follows:
Figure BDA0002521992350000082
Figure BDA0002521992350000083
using the adjacency matrix W and the degree matrix D, we can compute the Laplace matrix (Graph Laplacians). Its definition is simple, and the laplace matrix L ═ W-D. The normalization operation is carried out to obtain:
Figure BDA0002521992350000084
(2.2) decomposition of formula Using eigenvalues
Figure BDA0002521992350000085
Its feature vector is calculated. Wherein the characteristicsVector U1,U2,...,UkCorresponding to the minimum K characteristic values, and the dimension of each characteristic vector is n1+n2These feature vectors have a close relationship with the structure of the image.
(2.3) for each of the K feature vectors, we split it into two vectors, whose dimensions are n, respectively1And n2。
(2.4) extracting n from each feature vector1The dimensional vector is reconstructed to the size of the visible image, and the reconstructed feature function is called the feature spectrum by assigning its component values to sampling positions (the sampling positions are the positions where the feature points are extracted), and linearly interpolating the values between the sampling positions
Figure BDA0002521992350000091
N extracted from each feature vector is similarly2The vector of dimensions is reconstructed to the size of the infrared image, resulting in a characteristic spectrum
Figure BDA0002521992350000092
(2.5) according to the above method, we derive from the eigenvector U corresponding to the smallest K eigenvalues1,U2,...,UkRespectively obtaining K groups of characteristic frequency spectrum pairs
Figure BDA0002521992350000093
Further, in the step (3), a specific implementation manner of the SUSAN-MSER-SURF algorithm for detecting the extremum region of each group of feature function pairs is provided as follows:
(3.1) checking all pixel points in the image, if the gray value of the pixel point is larger than or equal to the pixel of the threshold value, saving the pixel point, otherwise, ignoring the pixel point. In the stored points, the adjacent points form an extreme value area;
(3.2) the stability of these along with the components is checked: starting from 0 to 255, the threshold value is gradually increased to the size of delta, the steps are repeated again, and the areas with the size changed by a plurality of threshold values are kept as the maximum stable extremum areas without changing.
(3.3) extracting corner points in the characteristic function by adopting an SUSAN operator, and reserving the MSER region when the number of the corner points in the MSER region is more than or equal to m;
(3.4) performing affine normalization on each detected MSER ellipse, taking the center point of the normalized MSER ellipse as an interest point, taking a circular region around the interest point, and calculating a SURF descriptor for the region by segmenting the region and calculating gradients on each sub-region;
(3.5) Using this method to detect each pair of feature function pairs
Figure BDA0002521992350000101
All MSER regions of (a) normalize it to a point of interest and compute SURF descriptors for it.
Further, in the step (3), the region features are matched by using the proposed Euclidean-Hamming algorithm. The specific implementation mode is as follows:
(4.1) for one feature point in the feature point set of the infrared image, calculating the improved Euclidean distance from the feature point to all the feature points on the visible light image to obtain a set of distances, easily obtaining the minimum distance by comparison, wherein the visible light feature point with the minimum distance from the infrared feature point is the matching point thereof, and if the minimum distance exceeds the threshold value MatchThreshold, excluding the matching point pair. Finding out all matching point pairs of the infrared image and the visible light image according to the rule;
(4.2) setting a comparison parameter MaxRadio, and if the ratio of the minimum distance and the next minimum distance between a certain characteristic point on the visible light and the characteristic point on the infrared image is smaller than the comparison parameter, reserving the matching point pair. Wherein, the improved Euclidean distance adds one more addition and subtraction operation on the original Euclidean distance. For vector a ═ x1,x2,...,xn) Sum vector b ═ y1,y2,...,yn) The formula of the improved Euclidean distance between the two is as follows:
Figure BDA0002521992350000102
and (4.3) screening the coarse matching points matched by the nearest neighbor rule. Let the minimum Euclidean distance of all matching point pairs be denoted as dominAnd respectively calculating hamming distances of the matched point pairs. Comparing hamming distances of all matching point pairs if they are less than dominTwice, the matching pair is considered as a mis-match, otherwise, if the value is larger than the value, the matching pair is considered as a correct matching pair.
To validate the algorithm proposed by the present invention, we performed experimental analysis.
First, in order to verify the Matching effect of the visible light and the infrared Image under exposure or over exposure in this method, we selected SURF + PIID + RPM algorithm (Gang Wang et al, "Robust Point Matching method for Multi-modal Image Registration", biological Signal Processing and control,2015, Vol.19, pp.68-76.), classic fuzzy shape context algorithm, dense + ORB operator algorithm (Reed strength, Schopper, Livicle, etc.. Image feature Point Matching algorithm based on dense-ORB features,% Imageventure matched base on dense-ORB feature [ J. Nature university of technology, 2019,035(001):13-20.) and the heterogeneous Image Matching method based on spectrum feature analysis proposed by the present invention for comparison experiments. Fig. 2-3 are graphs comparing the results of several sets of experiments.
Fig. 2 shows a matching graph of visible and infrared images of a road in dark light. Wherein the graph (a) uses the SURF + PIID-RPM algorithm, there are 10 matching pairs in total, of which 3 are correct matching pairs, the matching rate is 0.3, and it takes 2.5484 seconds; graph (b) uses the fuzzy shape context algorithm, and a total of 13 matching pairs, of which 2 are correct matching pairs, the matching rate is 0.153, and it takes 3.5108 seconds; the graph (c) uses the dense + ORB matching algorithm, and has a total of 7 matching pairs, wherein 1 is the correct matching pair, the matching rate is 0.142, and the time is 1.5612 seconds; the graph (d) uses the heterogeneous image matching method based on the joint graph spectral feature analysis, which is proposed by the invention, and has 8 matching pairs in total, wherein 6 matching pairs are correct matching pairs, the matching rate is 0.75, and the time is 2.7061 seconds.
From the matching rate, the method based on the combined graph spectrum feature analysis has obvious advantages. Because the application scene is shot under the condition of low light intensity, the image lacks of local intensity and a gray mode, the algorithm of the invention carries out matching by analyzing the characteristic spectrum of the combined image formed by the structural relationship of the corner points of the two images, and does not depend on the description of the characteristic local neighborhood information, thereby having better matching effect. The dense + ORB has obvious advantages in the matching speed, and because the ORB is a fast key point detector and descriptor, the defects of large calculation amount and low speed of SIFT and SURF algorithms are overcome.
Fig. 3 shows a matching graph of visible light and infrared images of the bulletin board in the case of bright light. Wherein, the graph (a) uses SURF + PIID + RPM algorithm, there are 10 matching pairs in total, 4 of which are correct matching pairs, the matching rate is 0.4, and it takes 2.7302 seconds; the fuzzy shape context algorithm is used in the graph (b), 8 matching pairs are in total, 2 of the matching pairs are correct, the matching rate is 0.25, and the time is 4.09 seconds; graph (c) uses the dense + ORB algorithm, with a total of 8 matching pairs, of which 1 is the correct matching pair, with a matching rate of 0.125, which takes 2.1835 seconds; the graph (d) uses the heterogeneous image matching method based on the joint graph spectral feature analysis, which is proposed by the invention, and has 12 matching pairs in total, wherein 11 matching pairs are correct matching pairs, the matching rate is 0.916, and the time is 2.8669 seconds.
The application scene is shot under the condition of high light intensity, and similarly, the method based on the combined graph spectrum characteristic analysis has better matching rate and shorter time consumption in the scene. The fuzzy shape context utilizes the unequal coordinate space to extract the histogram of the local shape statistical information, so that the time consumption is long, and the matching effect is poor due to single matching characteristic. And the SURF + PIID + RPM algorithm is applied to a plurality of complex multimodal retinal images with poor quality, is not suitable for the scene, and therefore the matching rate is low. The similarity + ORB performs matching analysis by utilizing the similarity of pixel densities in a certain feature point neighborhood space of the same region in two images, and cannot solve the problem that the images lack local intensity and gray level modes.
Through the comparison experiment, the different-source image matching method based on the combined image spectral feature analysis, which is proposed in the text, can be clearly used for matching the visible light image with the infrared image with low definition caused by underexposure or overexposure.
Secondly, in order to verify the reliability of the algorithm, the invention respectively uses SURF + PIID + RPM, fuzzy shape context, dense + ORB operator and the algorithm of the invention to carry out 10 times of experiments on 9 groups of images. Line graphs of the average accuracy contrast of the experiments are taken. Nine groups of visible light and infrared images are used as an experimental data set in the experiment, and the sample set is a visible light and infrared image with too high or low light. The matching-comparison line graph is shown in fig. 4, in which the horizontal axis of the image represents the group number of nine groups of images, respectively, and the vertical axis represents the average matching rate obtained after ten times of experiments on the group of images. Through the line graph, the judgment can be clearly made, in most experiments, the algorithm provided by the invention is better than other methods when being applied to the scene, and the matching performance of repeated experiments is stable.
Thirdly, the invention respectively counts the RMSE of 9 groups of images by using 4 methods, as shown in FIG. 5, and it can be seen that the RMSE value of the method of the invention is relatively smooth and lower, and the matching is more accurate. The fuzzy shape context utilizes the characteristic that two images are only similar in shape in a fuzzy mode, has a certain matching effect, but is poor in stability due to single characteristics. The dense + ORB performs matching by utilizing the similarity of pixel density in a certain feature point neighborhood space of the same region in two images, and cannot solve the problem that the images lack local intensity and gray level modes. The SURF + PIID + RPM is mostly applied to complex multi-modal images, poor quality, non-vessel images and the like, and the scene matching is poor.
Finally, the time lengths consumed in matching the four algorithms are compared. As shown in fig. 6, it can be seen that the matching algorithm of the present invention does not provide much advantage when used. The algorithm of the invention needs to detect and match the combined image frequency spectrum of two images and the three-dimensional reconstruction characteristic function to the maximum value area of each group of characteristic functions, thereby consuming more time and improving the matching rate. The fuzzy shape context extracts the histogram of the local shape statistics using the unequal coordinate space, resulting in a long time consumption. The similarity + ORB algorithm is the shortest in matching and is more suitable for tracking and matching of images.

Claims (5)

1. A heterogeneous image matching method based on combined graph spectral feature analysis is characterized by comprising the following steps:
(1) extracting angular points of the visible light image and the infrared image with too high or low light rays, and constructing an adjacent matrix by using the angular point relation of the two images;
(2) obtaining a feature vector of a joint graph by defining the conventional Laplace and decomposing the feature values of adjacent matrixes, and then constructing a feature function pair by three-dimensional reconstruction;
(3) detecting an extreme value region of each group of feature function pairs, and matching region features by using a proposed Euclidean-Hamming algorithm;
(4) and matching the extreme value regions of the feature spectrum pairs reconstructed by the minimum K feature vectors according to the steps to obtain a final matching result.
2. The method for matching the heterogeneous image based on the joint image spectral feature analysis according to claim 1, wherein in the step (1), the corners of the visible light image and the infrared image are extracted, and the adjacent matrix is constructed by using the relationship between the corners of the two images, and the specific method is as follows:
(1.1) giving a visible light image and an infrared image to be matched, extracting angular points of the visible light image and the infrared image to be matched by using an SIFT algorithm respectively, and collecting the angular points into an angular point set V1And V2Wherein V is1Set of all corner points in the visible light image, called node set
Figure FDA0002521992340000011
n1The number of the designated nodes; v2Set of all corner points in the infrared image, called node set
Figure FDA0002521992340000012
n2The number of the designated nodes;
(1.2) corner Point set V1And V2Their correlation diagrams are G1(V1,E1,W1) And G2(V2,E2,W2) Wherein E is1Is V1Set of edges between any two nodes in (E)2Is V2Set of edges between any two nodes in (W)1Is a weight value of n between all corner points of the visible light image1×n1The adjacency matrix of (a), which is defined as follows:
Figure FDA0002521992340000013
wherein n is1Reference to a set of nodes V1Number of middle nodes, W1Set of reference connections V1N composed of weighted values of edges of any two nodes1×n1Of adjacent matrices, wijIs a weight value, which represents the weight of the corner viAnd vjThe correlation between the represented pixels;
W2is a weighted value n between all corner points of the infrared image2×n2The adjacency matrix of (a), which is defined as follows:
Figure FDA0002521992340000021
wherein n is2Reference to a set of nodes V2Number of middle nodes, W2Set of reference connections V2N composed of weighted values of edges of any two nodes2×n2Of adjacent matrices, wijIs a weight value, which represents the weight of the corner viAnd vjIs represented byA correlation between pixels of (a);
the joint graph G (V, E, W) is defined as V ═ V1∪V2，E＝E1∪E2∪E12，E12Denotes a connection V1Any one of the nodes and V2The definition of the adjacency matrix W of the joint graph G for the set of edges of any node is as follows:
Figure FDA0002521992340000022
wherein C refers to the connection set V1Any one node and set V2N composed of weighted values of edges of any node1×n2Adjacent matrix, CTIs the transposed matrix of C;
(1.3) traversing all points in the corner point set by using a KNN algorithm, taking k points nearest to each sample point as the neighbors of the points, and only taking the weight values w between the k points nearest to the sample pointij>0, and the weight values between the points are 0, and to ensure symmetry, s is retained as long as one point is in the K neighborhood of another pointijThe formula for measuring the edge weight value connecting two corners is as follows:
Figure FDA0002521992340000023
wherein x isi,xjIs any two angular points vi,vjDescriptor of (1), xi∈KNN(xj) Refer to the corner point viPertaining to a distance corner vjOne of the nearest k points, xj∈KNN(xi) Refer to the corner point vjPertaining to a distance corner viOne of the nearest k points.
3. The method for matching a heterogeneous image based on the spectral feature analysis of a joint graph according to claim 2, wherein in the step (2), feature vectors of the joint graph are obtained by defining feature values of a conventional Laplace and a decomposed adjacent matrix, and feature function pairs are constructed by three-dimensional reconstruction, and the specific method is as follows:
(2.1) using the adjacency matrix W and the degree matrix D thereof, we can calculate the laplacian matrix as L ═ W-D and normalize it;
(2.2) decomposition of formula Using eigenvalues
Figure FDA0002521992340000031
Calculating its feature vector, wherein the feature vector U1,U2,...,UkCorresponding to the minimum K characteristic values, and the dimension of each characteristic vector is n1+n2；
(2.3) for each of the K feature vectors, splitting it into two vectors, the dimensions of which are n, respectively1And n2；
(2.4) extracting n from each feature vector1The dimensional vector is reconstructed as the size of the visible image, and the reconstructed feature function is called the feature spectrum by assigning it to the sampling positions, i.e. where the feature points are extracted, and linearly inserting a self-defined value between these sampling positions
Figure FDA0002521992340000032
Extracting n from each feature vector2The vector of dimensions is reconstructed to the size of the infrared image, thus obtaining a characteristic frequency spectrum
Figure FDA0002521992340000033
(2.5) according to the above method, from the eigenvector U corresponding to the smallest K eigenvalues1,U2,...,UkRespectively obtaining K groups of characteristic frequency spectrum pairs
Figure FDA0002521992340000034
4. The method for matching a heterogeneous image based on the joint graph spectrum feature analysis according to claim 3, wherein in the step (3), a SUSAN-MSER-SURF algorithm is provided to detect the extremum regions of each group of feature spectrum pairs, and the specific method is as follows:
(3.1) examining the first set of characteristic spectral pairs
Figure FDA0002521992340000035
If the gray value of the pixel point is larger than or equal to the pixel of the threshold value, storing the pixel point, otherwise, neglecting, and forming an extreme value area by adjacent points in the stored points;
(3.2) detecting the stability of the extremum regions, wherein the speed of the area change of the extremum regions can be expressed by the following formula:
Figure FDA0002521992340000036
wherein △ denotes the change in threshold, QiReferring to the area of the ith extremal region, starting from 0 to 255, gradually increasing to a threshold of △, for regions of varying threshold but not varying region size, they are retained as the most stable extremal region, i.e. the region Q is considered to be when v (i) indicating how fast the area of the extremal region varies is less than a given thresholdiIs the region of maximum stable extremum;
(3.3) adopting an SUSAN operator to extract angular points in the characteristic function, and reserving the maximum stable extremum region when the number of the angular points in the maximum stable extremum region is more than or equal to m;
(3.4) performing affine normalization on each detected MSER ellipse, taking the center point of the normalized MSER ellipse as an interest point, taking a circular region around the interest point, and calculating a SURF descriptor for the region by segmenting the region and calculating gradients on each sub-region;
(3.5) detecting each pair of feature function pairs using steps (3.1) - (3.4)
Figure FDA0002521992340000041
All MSER regions of (a) normalize it to a point of interest and compute SURF descriptors for it.
5. The method for matching heterogeneous images based on the joint graph spectral feature analysis according to claim 3, wherein in the step (3), the region features are matched by using a proposed Euclidean-Hamming algorithm, which is implemented as follows:
(4.1) for the characteristic function
Figure FDA0002521992340000042
To calculate it to a feature function
Figure FDA0002521992340000043
Obtaining a set of distances by Euclidean distances of all the characteristic points, and obtaining the minimum distance by comparison
Figure FDA0002521992340000044
The upper feature point is its matching point, if the minimum distance exceeds the threshold value, the matching point pair is excluded, so that the feature function pair can be found out according to the rule
Figure FDA0002521992340000045
All matching point pairs; wherein, the Euclidean distance is added and subtracted once more on the original Euclidean distance, and the vector a is equal to (x)1,x2,...,xn) Sum vector b ═ y1,y2,...,yn) The formula of the improved Euclidean distance between the two is as follows:
Figure FDA0002521992340000046
(4.2) setting a comparison parameter MaxRadio if
Figure FDA0002521992340000047
Point of certain characteristic
Figure FDA0002521992340000048
If the ratio of the minimum distance of the upper characteristic point to the next minimum distance is smaller than the comparison parameter, the matching point pair is reserved;
(4.3) recording the minimum Euclidean distance of all the matching point pairs as dominRespectively calculating the Hamming distances of the matched point pairs, comparing the Hamming distances of all the matched point pairs, and if they are less than dominTwice, the matching pair is considered as a mis-match, otherwise, if the value is larger than the value, the matching pair is considered as a correct matching pair.
CN202010493597.3A 2020-06-03 2020-06-03 Heterogeneous image matching method based on joint graph spectrum feature analysis Active CN111783834B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010493597.3A CN111783834B (en) 2020-06-03 2020-06-03 Heterogeneous image matching method based on joint graph spectrum feature analysis

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010493597.3A CN111783834B (en) 2020-06-03 2020-06-03 Heterogeneous image matching method based on joint graph spectrum feature analysis

Publications (2)

Publication Number Publication Date
CN111783834A true CN111783834A (en) 2020-10-16
CN111783834B CN111783834B (en) 2022-08-19

Family

ID=72753662

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010493597.3A Active CN111783834B (en) 2020-06-03 2020-06-03 Heterogeneous image matching method based on joint graph spectrum feature analysis

Country Status (1)

Country Link
CN (1) CN111783834B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112396112A (en) * 2020-11-20 2021-02-23 北京百度网讯科技有限公司 Clustering method, clustering device, electronic equipment and storage medium
CN113516184A (en) * 2021-07-09 2021-10-19 北京航空航天大学 Mismatching elimination method and system for image feature point matching
CN117596487A (en) * 2024-01-18 2024-02-23 深圳市城市公共安全技术研究院有限公司 Camera disturbance self-correction method, device, equipment and storage medium

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20140004942A (en) * 2012-07-03 2014-01-14 삼성테크윈 주식회사 Image matching method
CN110097093A (en) * 2019-04-15 2019-08-06 河海大学 A kind of heterologous accurate matching of image method

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20140004942A (en) * 2012-07-03 2014-01-14 삼성테크윈 주식회사 Image matching method
CN110097093A (en) * 2019-04-15 2019-08-06 河海大学 A kind of heterologous accurate matching of image method

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112396112A (en) * 2020-11-20 2021-02-23 北京百度网讯科技有限公司 Clustering method, clustering device, electronic equipment and storage medium
CN112396112B (en) * 2020-11-20 2024-05-14 北京百度网讯科技有限公司 Clustering method, clustering device, electronic equipment and storage medium
CN113516184A (en) * 2021-07-09 2021-10-19 北京航空航天大学 Mismatching elimination method and system for image feature point matching
CN113516184B (en) * 2021-07-09 2022-04-12 北京航空航天大学 Mismatching elimination method and system for image feature point matching
CN117596487A (en) * 2024-01-18 2024-02-23 深圳市城市公共安全技术研究院有限公司 Camera disturbance self-correction method, device, equipment and storage medium
CN117596487B (en) * 2024-01-18 2024-04-26 深圳市城市公共安全技术研究院有限公司 Camera disturbance self-correction method, device, equipment and storage medium

Also Published As

Publication number Publication date
CN111783834B (en) 2022-08-19

Similar Documents

Publication Publication Date Title
CN111783834B (en) Heterogeneous image matching method based on joint graph spectrum feature analysis
Strom et al. Graph-based segmentation for colored 3D laser point clouds
Kadam et al. R-pointhop: A green, accurate, and unsupervised point cloud registration method
EP3101594A1 (en) Saliency information acquisition device and saliency information acquisition method
CN109215053B (en) Method for detecting moving vehicle with pause state in aerial video shot by unmanned aerial vehicle
CN111369495B (en) Panoramic image change detection method based on video
CN109993800A (en) A kind of detection method of workpiece size, device and storage medium
CN108399627B (en) Video inter-frame target motion estimation method and device and implementation device
CN108960115B (en) Multidirectional text detection method based on angular points
CN108109163A (en) A kind of moving target detecting method for video of taking photo by plane
CN106485740A (en) A kind of combination point of safes and the multidate SAR image registration method of characteristic point
CN109063549B (en) High-resolution aerial video moving target detection method based on deep neural network
CN108550166B (en) Spatial target image matching method
CN101630407B (en) Method for positioning forged region based on two view geometry and image division
JP6611681B2 (en) Method and system for segmenting images
CN112287906B (en) Template matching tracking method and system based on depth feature fusion
CN113689331B (en) Panoramic image stitching method under complex background
Hu et al. Texture-aware dense image matching using ternary census transform
CN115601574A (en) Unmanned aerial vehicle image matching method for improving AKAZE characteristics
CN112946679A (en) Unmanned aerial vehicle surveying and mapping jelly effect detection method and system based on artificial intelligence
CN116664892A (en) Multi-temporal remote sensing image registration method based on cross attention and deformable convolution
CN112734822A (en) Stereo matching algorithm based on infrared and visible light images
CN112818905A (en) Finite pixel vehicle target detection method based on attention and spatio-temporal information
Feng et al. A novel saliency detection method for wild animal monitoring images with WMSN
CN107609562A (en) A kind of metric space characteristic detection method based on SIFT algorithms

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant