CN109508674A - Airborne lower view isomery image matching method based on region division - Google Patents

Abstract

The invention provides an airborne down-view heterogeneous image matching method based on area division, which belongs to the technical field of image matching. The present invention first uses the standard deviation STD of the direction histogram to determine the texture feature of the target image; if it is a rich texture image, the target image and the real-time image are respectively segmented using the Meanshift image segmentation algorithm, divided into several regions, and corresponding masks are generated. image; if it is a non-textured image, divide the real-time image to form several area blocks; then use the SIFT feature matching method to match each real-time image area with all target image areas respectively; finally, use the direction-based histogram The evaluation function evaluates each matching result, and selects the optimal matching area as the matching result. The invention solves the problem of low matching accuracy for complex and heterogeneous airborne images in the prior art. The present invention can be used for heterogeneous image matching of unmanned aerial vehicles.

Description

Airborne Down-View Heterogeneous Image Matching Method Based on Region Division

technical field

The invention relates to an airborne down-view heterogeneous image matching method, and belongs to the technical field of image matching.

Background technique

With the improvement of drone technology, drone technology has attracted a large number of producers and users, which makes drone-related technologies have a wide market. UAV-based heterogeneous image matching is the process of aligning real-time images from UAVs and target images from satellites. This technology is used in UAV navigation, UAV landing, UAV attack and space vehicle Navigation and other aspects have a wide range of applications. However, the airborne down-view images have complex structural characteristics, and the accurate matching of the air-borne down-view heterogeneous images has become a key technology in the application of UAVs, which is of great significance.

In the UAV target positioning, the target image is non-real-time, and the positioning image is real-time, which makes the target image and the positioning image have different image structures. Real-time images will be very complex, usually including scale-change images, rotation-change images, seasonal-change images, blurred images, occlusion images, and SAR (Synthetic Aperture Radar) images. This makes the airborne down-view heterogeneous image matching a challenging problem, and the classical image matching methods are difficult to meet the application requirements of UAV heterogeneous image matching.

At present, the most classic image matching method is based on SIFT (Scale-invariant feature transform) feature image matching method. Wu Gang et al. invented a building image matching and fusion method based on contour extraction. The method first extracts the contour of the object, extracts the straight line on the contour map, and uses the straight line matching algorithm to match the two images according to the straight line feature. The optimal matching calculates the angle between the straight lines in the set, obtains the angle matrix, and calculates the similarity of the angle matrix. Zhang Haopeng et al. invented a spatial target image matching method, using the GMS (Grid-based Motion Statistics grid-based motion estimation) matching algorithm to roughly match the three-view images of the spatial target, and introducing the error threshold algorithm NFA to eliminate mismatched points Yes, it makes the algorithm more adaptive. Wang Shuang et al. invented a heterogeneous image matching method based on deep learning. The method first creates a heterogeneous image block dataset, performs image preprocessing, obtains image block feature maps, obtains feature vectors from the feature maps, and fuses the feature maps. And normalize, train the image matching network, predict the matching probability, and effectively solve the problem of image matching overfitting. The image matching methods mentioned above are relatively mature, and usually can achieve a relatively high image matching accuracy. However, in actual situations, the airborne down-view images are often more complicated, as shown in Figure 1, Figure 2, Figure 3, and Figure 4. , Figure 5, Figure 6, Figure 7; for complex and heterogeneous airborne images, these methods have not achieved good results, and it is still necessary to seek technologies that can achieve higher matching accuracy.

SUMMARY OF THE INVENTION

In order to solve the problem of low matching accuracy of complex and heterogeneous airborne images in the prior art, the present invention provides an airborne down-view heterogeneous image matching method based on area division.

The airborne down-view heterogeneous image matching method based on area division according to the present invention is realized by the following technical solutions:

(1) Use the standard deviation STD of the orientation histogram as a parameter to determine the texture features of the target image: if the standard deviation STD is greater than the threshold S, the target image is determined to be a rich texture image; if the standard deviation STD is less than or equal to the threshold S, the target image is determined is a non-rich texture image;

(2) If the target image is a rich texture image, use the Meanshift mean shift image segmentation algorithm to segment the target image and the real-time image respectively, and divide them into several regions, and then layer the segmented target image region and real-time image region to generate corresponding the mask image;

If the target image is a non-textured image, the real-time image is divided to form several area blocks, and the whole target image is regarded as one area;

(3) Using the SIFT feature matching method, each real-time image area is matched with all target image areas for consistency;

(4) Use the evaluation function based on the direction histogram to evaluate the results of each matching, and select the optimal matching area as the matching result.

The most prominent feature and significant beneficial effect of the present invention are:

1. Classify the target image, and perform different processing for different types of images, so that the matching accuracy of the images is improved; in the simulation experiment, compared with the traditional SIFT algorithm, the method of the present invention (based on the airborne down-view heterogeneous image based on area division) matching method), the image matching accuracy is improved by about 15%;

2. It integrates a variety of image processing technologies and uses the information of different types of real-time images to robustly and accurately complete the image matching process in the airborne down-view image with a lot of noise and interference;

3. The invention is of great significance to the target positioning system of the airborne down-view image, and greatly expands the application scope of the UAV target positioning system based on image matching;

4. The present invention uses the standard deviation of the direction histogram as an evaluation parameter to determine the texture features of the target image, and classifies the target, including rich texture images and non-rich texture images, and processes these two images respectively, which improves the robustness of the system. stickiness;

5. Aiming at rich texture images, the present invention first uses the Meanshift image segmentation method to obtain mask images of different segmentation regions, and then matches different regions by introducing mask images, and takes the result of the optimal matching region matching as the final matching result. , this idea makes the intra-class region similarity stronger, the inter-class region similarity weaker, and the target matching accuracy rate is higher;

6. For the non-textured image, the present invention divides the real-time wide image into blocks, matches these area blocks with the target image respectively, and uses the result of the optimal matching area as the final matching result. For non-textured images, after segmentation is performed using the segmentation method, the segmented areas are scattered, and accurate matching cannot be completed in the segmented area. To some extent, dividing the area into blocks can also make the intra-class area more similar and the inter-class area similar. weaker, which improves the accuracy of matching non-texture-rich target images;

7. The present invention proposes to use the evaluation function based on the orientation histogram to obtain the matching result of the optimal matching area. In order to ensure the accuracy of the matching results, the Bhattacharyya distance (BD) is used to compare the similarity of the two matching images. If the BD value is larger, the matching is considered to be more accurate, and the invention makes the final matching result more accurate.

Description of drawings

Figure 1 is a real-time image of scale change;

Fig. 2 is the real-time image of rotation change;

Figure 3 is a real-time image of spring;

Figure 4 is a real-time image in winter;

Figure 5 is a blurred real-time image;

Fig. 6 is the real-time image of occlusion;

Figure 7 is a real-time image of synthetic aperture radar SAR;

8 is a schematic flow chart of the method of the present invention;

Fig. 9 is a ground image (also the target image in the embodiment);

Figure 10 is a histogram of the direction of a; Pixel number represents the number of pixels, and Angle represents the angle;

Figure 11 is the image of b;

Figure 12 is a histogram of the direction of b;

Figure 13 is an image of ground c;

Figure 14 is a histogram of the direction of c;

Figure 15 is an image of d;

Figure 16 is a histogram of the direction of d;

Figure 17 is a schematic diagram of the execution time and matching rate corresponding to different h in the SIFT algorithm;

Figure 18 is a schematic diagram of the detection of the maximum and minimum extreme points in the DOG image;

19 is a schematic diagram of the calculation of the description operator;

FIG. 20 is a real-time image used to match the image of c in the embodiment.

Detailed ways

Embodiment 1: This embodiment will be described with reference to FIG. 8 . The airborne down-view heterogeneous image matching method based on area division given in this embodiment is used for the target image shown in FIG. 1 and the complex and heterogeneous image For airborne real-time images, the method first uses the standard deviation STD of the orientation histogram as a parameter to determine the texture features of the target image; if the target image is a rich texture image, an image matching method based on image segmentation is used to complete the airborne down-view image localization process , the process uses image segmentation to generate mask images of different areas of the image, in these areas, the improved SIFT image matching method is used to match each area, and the evaluation function based on the orientation histogram is used to obtain the optimal matching area as the matching result ; For non-rich texture images, use the image matching method based on regional block division to complete the image positioning process. This process divides the real-time image into regions, and uses the improved SIFT image matching method to match each block area with the target image. The evaluation function of the direction histogram obtains the optimal matching block area as the matching result. Specifically include the following steps:

(1) Use the standard deviation STD of the orientation histogram as a parameter to determine the texture features of the target image: if the standard deviation STD is greater than the threshold S, the target image is determined to be a rich texture image; if the standard deviation STD is less than or equal to the threshold S, the target image is determined It is a non-rich texture image; as shown in Figure 9 to Figure 16 are four different texture images and their direction histograms of a, b, c, and d, where a image STD=0.152, b image STD=0.157, and c image STD=0.157 Image STD=0.129, d image STD=0.0747.

(2) If the target image is a rich texture image, use the Meanshift mean shift image segmentation algorithm to segment the target image and the real-time image respectively, and divide them into several regions, and then layer the segmented target image region and real-time image region to generate corresponding the mask image;

If the target image is a non-textured image, the features detected by the traditional feature detection algorithm have no obvious difference, and image matching on a large real-time image will result in a matching failure. This embodiment divides the real-time image to form several regions. The target image as a whole is regarded as an area (when the step of consistency matching is performed later, the target image is transformed to the corresponding position of the real-time image, and finally the position of the target is demarcated according to the position of the sub-image in the large image. The area is divided, each area is matched with the target image using the SIFT image matching method, and the method of obtaining the optimal matching area using the evaluation function based on the orientation histogram is named as the image matching algorithm based on block division).

MeanShift is a feature space analysis method, which for image segmentation maps the problem to the color feature space. The image segmentation problem is to find its class center for each pixel. MeanShift believes that the center is the maximum value point of the probability density. Kernel probability density estimation (which can be seen as a Parzen window technique in pattern recognition) is the most popular probability density estimation method _. For a point set x _i , i=1,...,n of n data, in the d-dimensional space R ^d , at the point x position, given a kernel K(x) and a d × d size symmetric positive definite matrix H, The multivariate kernel density estimate is defined as:

Here are the conditions to be met:

K _H (x)＝|H| ^-1/2 K(H ^-1/2 x)

At the same time, K(x) satisfies:

Here c _K is a constant, x ^T represents the transpose of x, and I represents the grayscale value of the image. In practical applications, in order to reduce the complexity of the algorithm, H uses a diagonal matrix Get the most famous expression:

here, is the diagonal element of the diagonal matrix H; h is the width of the kernel K(x), where K(x) usually takes a special kind of symmetric kernel:

K(x)=c _K,d k(||x|| ² )

Among them, c _{K, d} is the standardization coefficient; k(||x|| ² ) represents a projection function;

The final probability density estimate can be written as:

MeanShift is OK (Probability Density Estimation The effective method of the extreme value), let g(x)=-k'(x), k'(x) is the derivative of k(x); define the kernel G(x)=c _g,d g(||x || ² ), c _g,d are normalization constants. According to the linear expression of the above formula, the MeanShift vector can be obtained:

Therefore, the implementation of the MeanShift algorithm consists of two parts: 1) calculating the MeanShift vector m _h,G (x); 2) convolving the kernel window G(x) by m _h,G (x). This method can guarantee the final probability density estimation expression Converge around the point where the gradient is 0 and find the cluster center.

The target image segmentation effect is very good; however, the real-time image is usually relatively large. Figure 17 is a schematic diagram of the execution time and matching rate corresponding to different h in the SIFT algorithm; in order to meet the real-time requirements, the resolution of the image can be reduced and then the target segmentation.

(3) Using the SIFT feature matching method, each real-time image area is matched with all target image areas for consistency.

(4) Use the evaluation function based on the direction histogram to evaluate the results of each matching, and select the optimal matching area as the matching result.

Embodiment 2: The difference between this embodiment and Embodiment 1 is that the SIFT feature matching method is used in step (3), and in the process of matching each real-time image area with all target image areas for consistency, adding Corner points are used as feature key points.

The traditional SIFT algorithm includes four parts: scale space extremum detection, key point location, direction setting and key point description operator determination. The improved SIFT image matching method combines the corner key points and SIFT key points to form the key point set of the algorithm, uses the corner area as a mask to generate the key points of the corner area, and adds the corner points as the SIFT image of the feature key points. The matching algorithm is performed in the segmented image region, which improves the performance of image matching.

The SIFT image matching method that adds corner points as feature key points specifically includes the following steps:

(3.1) Peak selection in scale space

The peak selection in the scale space is to find peaks on the convoluted image D(x, y, σ) of the DOG (difference-of-Gaussian) function of different scales determined by a constant factor k, D(x, y, σ) is defined as:

D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)=L(x,y,kσ)-L(x,y ,σ)

G(x, y, σ) represents a Gaussian filter, L(x, y, σ) represents a Gaussian smoothed image, and the maximum and minimum values of the scale space are obtained by comparing the current pixel on different scale spaces with the 3×3×3 neighborhood. The 26 adjacent pixels are compared, and if the current value is the largest or the smallest, it becomes the largest or smallest candidate pixel, as shown in Figure 18.

(3.2) Determination of key points

In 2002, Lowe et al. proposed a method to determine the sampling point. The Taylor expansion of D(x, y, σ) is as follows:

D represents the function value of D(x) at point 0 or D represents the function value of D(x) at the sampling point x=(0,0,0), and the extreme value of the sampling point is calculated by calculating the gradient of the function to zero time obtained, there are:

choose The position of the peak point less than 0.03 is used as the key point, and the edge effect is also considered.

(3.3) Direction setting

The amplitude m and θ of each point on the image are calculated according to the pixel difference, so there are:

Through the above two formulas, the direction histogram of all sampling points is calculated, and the peak value of the histogram can be regarded as the main direction of the local gradient, as shown on the right side of Figure 19.

(3.4) Local description operator determination and key point matching

The calculation of the SIFT description operator is shown in Figure 19. At each key point, the image gradient and direction are calculated, and the direction histogram is created in the 4×4 area where the key point is located, and each histogram contains 8 directions. , each arrow represents each direction in the histogram, and the length represents the magnitude of each direction. One gradient sampling can select four 4×4 regions, and each 4×4 region can calculate a direction histogram. These four histograms are arranged in the middle diagram of Figure 19 as the description operator of SIFT. In this embodiment, 4×4 small regions are selected, and a 4×4×8=128-dimensional vector is obtained, and the 128-dimensional vector is used for similarity measurement of image matching.

After the keypoint description of the image is generated, for each feature point in the target, the corresponding consistent keypoint is found in the real-time image, and this process is called keypoint matching. The problem can be described as: given a set T and unknown samples x, find the most probable sample c corresponding to x. Using a Bayesian approach, one can choose Here P(c _j |x, T) represents the posterior probability of class c _j where x is located, and c _j belongs to an element in the training data set T. The k-nearest neighbor algorithm used in this implementation is as follows:

Step 1: Find the k nearest neighbors of x in the set T, that is, find a set satisfy both

Step 2: In Y, put frequently occurring keypoint pairs into the T set to break the randomly generated association.

(3.5) Determination of conversion function

The transformation function in this embodiment uses perspective transformation. The essence of perspective transformation is to project the image to a new viewing plane. The general transformation formula is:

Here (u,v) is the original image position and (x,y) is the position of the transformed image. Therefore, x=x ^* /w ^* , y=x ^* /w ^* , the transformation matrix can be divided into four parts: Represents a linear transformation, [a ₃₁ a ₃₂ ] is the offset, and [a ₁₃ a ₂₃ ] ^T produces a perspective transformation. So you can get

The RANSAC method is a method of model fitting on the data, which can be used to match the target image with the airborne down-view real-time image. The RANSAC method is compared with the classical least squares fitting to determine the projection function, which has the inherent property of detecting and rejecting erroneous pixels. The execution process of RANSAC is as follows:

Given a set P of corresponding keypoint pairs of known target images and live images, where (|P|>n), n is the minimum number of keypoint pairs required to instantiate the projection parameters. A subset S1 of n data points is randomly selected in the set P, the model is instantiated, and the instantiated model M1 is used to determine the subset S1 ^* in the set P, the data in this set has the smallest error for M1. If the size of S1 ^* is greater than a certain threshold, a new model M1 ^* is calculated using the least squares fitting method. If the size of S1 ^* is less than the threshold t, a new subset S1 is randomly selected and the data is re-instanced If the number of cycles is greater than the threshold ln, it is considered that the set of consistency points cannot be found, and the algorithm is terminated. The matching corresponding projection function is obtained through the RANSAC algorithm, and the target is calibrated in the real-time image at the same time.

Other steps and parameters are the same as in the first embodiment.

Embodiment 3: The difference between this embodiment and Embodiment 1 is that in step (4), the evaluation function based on the direction histogram is used to evaluate the matching result, and the optimal matching area is selected as the specific matching result. The process includes:

(4.1) use the Babbitt distance BD as the matching similarity measurement coefficient of the two region histograms for matching;

(4.2) If BD is greater than the threshold T, it is determined that the matching is successful; otherwise, the BD is less than or equal to the threshold T, the matching fails;

(4.3) Among the successfully matched regions, the region with the largest corresponding BD value is selected as the optimal matching region.

Other steps and parameters are the same as in the first or second embodiment.

Embodiment 4: The difference between this embodiment and Embodiment 3 is that the Bavarian distance BD described in step (4.1) is specifically:

Among them, th(j) represents the orientation histogram of the target image, rh(j) represents the orientation histogram of the real-time image; j=1,...,N; N represents the number of categories of image gray levels.

Other steps and parameters are the same as in the first, second or third embodiment.

Embodiment 5: The difference between this embodiment and Embodiment 3 is that the definition of the standard deviation STD in step (1) is specifically:

Among them, μ represents the mean value of the orientation histogram; h(j) represents the orientation histogram of the image; j=1,...,N; N represents the number of categories of image gray levels.

Other steps and parameters are the same as in the first, second, third or fourth embodiment.

Embodiment 6: This embodiment is different from Embodiment 2 in that the corner points are obtained by using the Harris corner point detection method.

The Harris corner detection method determines the corners in the image by determining the size of the corner response function R. I represents the gray value of the image, and (x, y) represents the pixel coordinate position in the image. For a small area at the (x, y) position, the variation E _x , _y of the area is defined as follows:

E _x,y =∑ _u,v w _u,v |I _x+u,y+v -I _u,v | ²

Here w _{u, v} represents an image window, (u, v) is the position of the coordinates in the window, (x, y) can be translated in 4 directions {(-1,1),(1,1),(-1, -1),(1,-1)}. After analysis and expansion, another expression form of _{Ex, y} can be obtained:

E _x,y =(x,y)M(x,y) ^T

M is a 2×2 symmetric matrix:

in,

The following formulas are defined as:

Tr(M)=A+B

Det(M)=AB-C ²

Then the corner response function R can be obtained:

R=Det(M)-qTr ² (M)

Here q is an empirical value, and corner detection is to find a point whose response function is greater than a certain threshold as a corner.

Other steps and parameters are the same as in the first, second, third, fourth or fifth embodiment.

Embodiment 7: This embodiment differs from Embodiments 1 to 6 in that the threshold value S is 0.13-0.15. For images with less texture, the features detected by traditional feature detection algorithms are not distinct, and image matching on large real-time images will lead to matching failure. By calculating STD statistics from a large number of images, it is obtained that when the STD is less than 0.13, it can be judged as a non-textured image in the usual sense; when the STD is greater than 0.15, it can be judged as an image with rich texture in the usual sense, so the threshold S is taken as 0.13 When ~0.15, it is used to distinguish between rich and non-rich image textures, which is more reasonable and can achieve better results.

Other steps and parameters are the same as in the first, second, third, fourth, fifth or sixth embodiment.

Example

Adopt the following examples to verify the beneficial effects of the present invention:

Select the image of ground a as the target image (as shown in Figure 9), and the real-time images as shown in Figure 1, Figure 2, and Figure 6; select the image of ground c (as shown in Figure 13) as the target image, and the real-time image as shown in Figure 20.

The specific steps of using the airborne down-view heterogeneous image matching method based on area division are as follows:

1. Set the threshold value S to 0.14, using the formula The standard deviation of the histogram in the direction of a is STD=0.152 (as shown in Figure 10), and it is determined as a rich texture image; the standard deviation of the histogram in the direction of c is calculated as STD=0.129 (as shown in Figure 14), and it is determined as a non-textured image.

2. As shown in Figure 8, the target image and the real-time image are segmented using the Meanshift mean shift image segmentation algorithm, divided into several regions, and the segmented target image region and the real-time image region are layered to generate corresponding mask images. . Keypoints will be detected in different regions and consistent keypoints will be found in different region pairs, then a transformation function will be used to generate labeled images, and finally, an orientation histogram-based evaluation function is used to select the optimal matching region.

In order to check the effect of the method of the present invention, the above-mentioned target image and the real-time image are used to carry out a matching test with other existing algorithms, and the comparison of the effect of the method of the present invention and other algorithms is obtained as follows:

Table 1 Comparison of matching effects of different methods

It can be seen from the above table that the method of the present invention can achieve higher image matching accuracy. The combination of MeanShift, SIFT and corner points used in the present invention can greatly improve the image matching accuracy of rotational changes. When other methods are ineffective for non-rich texture images, the block-based image matching algorithm used in the present invention can effectively perform image matching. match. Compared with the traditional SIFT algorithm, the image matching accuracy of the method of the present invention (airborne down-view heterogeneous image matching method based on area division) is improved by about 15%;

The present invention can also have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and deformations according to the present invention, but these corresponding changes and deformations are all It should belong to the protection scope of the appended claims of the present invention.

Claims (7)

1. The airborne down-view heterogeneous image matching method based on area division is characterized in that, the method specifically comprises the following steps: (1) Use the standard deviation STD of the orientation histogram as a parameter to determine the texture features of the target image: if the standard deviation STD is greater than the threshold S, the target image is determined to be a rich texture image; if the standard deviation STD is less than or equal to the threshold S, the target image is determined is a non-rich texture image; (2) If the target image is a rich texture image, use the Meanshift mean shift image segmentation algorithm to segment the target image and the real-time image respectively, and divide them into several regions, and then layer the segmented target image region and real-time image region to generate corresponding the mask image; If the target image is a non-textured image, the real-time image is divided to form several area blocks, and the whole target image is regarded as one area; (3) Using the SIFT feature matching method, each real-time image area is matched with all target image areas for consistency; (4) Use the evaluation function based on the direction histogram to evaluate the results of each matching, and select the optimal matching area as the matching result. 2. the airborne down-view heterogeneous image matching method based on regional division according to claim 1, is characterized in that, in described step (3), use SIFT feature matching method, each real-time image area and all target images are respectively In the process of consistent matching of regions, corner points are added as feature key points. 3. The airborne down-view heterogeneous image matching method based on area division according to claim 1, is characterized in that, described in step (4), utilizes the evaluation function based on direction histogram to evaluate the result of matching, and selects the best result. The specific process of using the optimal matching area as the matching result includes: (4.1) Use the Babbitt distance BD as the matching similarity measurement coefficient of the two region histograms for matching; (4.2) If BD is greater than the threshold T, it is determined that the matching is successful; otherwise, the BD is less than or equal to the threshold T, the matching fails; (4.3) Among the successfully matched regions, the region with the largest corresponding BD value is selected as the optimal matching region. 4. The airborne down-view heterogeneous image matching method based on area division according to claim 3, wherein the Bavarian distance BD in step (4.1) is specifically: Among them, th(j) represents the orientation histogram of the target image, rh(j) represents the orientation histogram of the real-time image; j=1,...,N; N represents the number of categories of image gray levels. 5. The airborne down-view heterogeneous image matching method based on area division according to claim 1, is characterized in that, the definition of described standard deviation STD in step (1) is specifically: Among them, μ represents the mean value of the orientation histogram; h(j) represents the orientation histogram of the image; j=1,...,N; N represents the number of categories of image gray levels. 6 . The airborne down-view heterogeneous image matching method based on area division according to claim 2 , wherein the corner points are obtained by using the Harris corner point detection method. 7 . 7 . The airborne down-view heterogeneous image matching method based on area division according to any one of claims 1 to 6 , wherein the threshold value S ranges from 0.13 to 0.15. 8 .