CN111008932A - A Panoramic Image Stitching Method Based on Image Screening - Google Patents

Abstract

The invention discloses a panorama image stitching method based on image screening. The method includes the following steps: an image screening algorithm is proposed according to the similarity matrix between images, and redundant images are removed from the original image group; The optimal reference image is obtained, the splicing sequence is determined, and the screened image groups are grouped into multiple small image groups; similarity transformation is performed between the small image groups and the small image groups to obtain the initialized matching Then, according to the stitching sequence, the initialized registration parameters are refined by the homography model between adjacent images under the perspective constraints. The invention screens multiple images with overlapping areas collected in the target area by the drone, removes redundant images from the image group, and finally splices a panoramic image of the target area through the screened image group.

Description

A Panoramic Image Stitching Method Based on Image Screening

technical field

The invention relates to the field of panoramic image stitching, in particular to a panoramic image stitching technology based on image screening by collecting multiple images with overlapping regions from a target area based on an unmanned aerial vehicle.

Background technique

UAV remote sensing is a new remote sensing method, which has shown a good development momentum in recent years because of its high efficiency, flexibility, speed, low cost and high resolution. However, because the UAV remote sensing platform is limited by the flight height and the focal length of the camera in the process of aerial photography, the obtained single image has a small range and often cannot cover the entire required area. Image stitching into a panoramic image with a large viewing angle has become an important technology.

Panoramic image stitching can stitch multiple adjacent small-view images into a large-view panorama image. Image stitching is the mapping of all images onto a common coordinate system through a projective distortion (eg cylindrical, spherical, or perspective). Due to the irregular movement of the lens, there is a certain parallax between the images, and the panorama images after stitching almost inevitably have problems such as insufficient local stitching accuracy (Fig. 1(a)) and serious global deformation accumulation (Fig. 1(b)).

In order to improve the stitching quality, Konolige et al ^[1] proposed to use the beam adjustment method for global optimization, which can minimize the global reprojection error. To avoid nonlinear optimization, Kekec et al ^[2] employ an affine model for initial alignment and a homography model for global optimization. With the increase of stitched images, the accumulation of perspective deformation will become more and more. In order to avoid this problem, Caballero et al ^[3] proposed to use a layered model to register images according to the registration quality of the images. Image registration has less degrees of freedom, and the essence of the algorithm is to obtain a trade-off between improving registration accuracy and reducing deformation accumulation.

For the large-view image stitching problem, it is also a very efficient solution to use the topological relationship between images to improve the quality of image stitching. In order to efficiently estimate the topological relationship between images, Elibol et al ^[4] used rough feature point matching combined with a minimum spanning tree algorithm to detect the overlapping relationship between images. Regarding the choice of baseline images, Richard et al ^[5] show that the most appropriate choice is the image closest to the center of the panoramic image, since the center image has the shortest average shortest path to all other images, which minimizes deformation accumulation. In order to realize this algorithm, Choe et al ^[6] selected the optimal reference image through the algorithm of graph theory, but the premise is that the deformation error between each pair of images needs to be calculated in advance. M.Xia et al ^[7] proposed a registration model. First, the affine model is used to initialize the registration, and then the parameters between adjacent images are refined through the homography model.

The above method can stitch multiple images into a complete panoramic image, but as the number of stitching increases, the data will become redundant, resulting in many unnecessary calculations.

SUMMARY OF THE INVENTION

The present invention provides a panorama image stitching method based on image screening. The present invention screens multiple images with overlapping areas collected from a target area by a drone, removes redundant images from the image group, and finally passes the screening. A panoramic image of the target area is stitched from the image group, as described in the following description:

A panorama image stitching method based on image screening, the method comprises the following steps:

According to the similarity matrix between images, an image screening algorithm is proposed to remove redundant images from the original image group;

Based on the weight matrix of the screened image groups, the optimal reference image is obtained, the splicing sequence is determined, and the screened image groups are grouped into multiple small image groups;

Similar transformation ^[9] is performed between the small image group and the small image group to obtain the initial registration parameters, and then according to the splicing sequence, the homography model ^[10] is used to initialize the registration between adjacent images under the perspective constraint. parameters are refined.

Further, the removal of redundant images from the original image group according to image screening and similarity matrix is specifically:

1) Based on the similarity matrix of the original image group, set the similarity threshold;

2) Determine whether the maximum value in the similarity matrix of the current image group is greater than the similarity threshold, if so, select the two images with the highest similarity in the current image group, determine one of the images as a redundant image, and remove the image group;

3) Repeat the operation of step 2) until the maximum value in the similarity matrix of the current image group is smaller than the similarity threshold.

Wherein, determining one of the images as a redundant image is specifically: calculating the sum of the similarities between all the images in the remaining image groups after deleting each image, and then calculating the sum of the similarities that minimizes the sum of the similarities after deletion. Images are designated as redundant images.

Described based on the weight matrix of the screened image group, to find the optimal reference image is specifically:

Based on the similarity matrix of the images, establish a weight matrix between all the filtered images, run the shortest path algorithm based on the weight matrix, and calculate the sum of the weights of the shortest paths from each point to all other points; The image represented by the point is used as the optimal reference map.

The determined splicing sequence is specifically:

Based on the weight matrix of the image group, starting from the point representing the optimal reference map, using the breadth-first traversal algorithm, the obtained point sequence is used as the image stitching sequence.

The method also includes:

The similarity transformation is performed between the small image group and the small image group to obtain the initialized registration parameters, and then according to the splicing sequence, the initialized registration parameters are refined by the homography model between adjacent images under the perspective constraint; Minimize the total energy function to obtain the optimal solution.

The beneficial effects of the technical scheme provided by the present invention are:

1. Under the premise of ensuring the integrity and quality of the final splicing result, this method filters the redundant images in the collected small-view image group, which greatly improves the splicing rate;

2. The present invention overcomes the accumulation of perspective deformation under large data sets, groups images, and performs initialization registration between image groups through a similarity transformation model to obtain good global consistency;

3. Based on the topology between images, this method performs homography registration between adjacent images, and obtains a good stitching effect locally;

4. The experimental results show that this method can effectively remove redundant images from the image group, and the quality and integrity of the generated large-view panorama are not much different from the effect obtained by stitching all images.

Description of drawings

Figure 1 shows the problems encountered in the process of panoramic image stitching;

Among them, (a) the local stitching accuracy is not high, ghosting or dislocation occurs; (b) the deformation accumulation is serious, and the final stitching effect lacks global consistency.

Fig. 2 is the flow chart of the panoramic image stitching method based on image screening;

Figure 3 shows the spatial representation of the splicing sequence obtained by selecting the optimal reference map for 61 images, and starting from the optimal reference map, running breadth-first traversal;

Fig. 4 is the topological structure analysis schematic diagram of 61 images and the images after screening;

Among them, (a) is the topological structure of the original image group; (b) is the topological structure of the filtered image group (including 31 images).

Fig. 5 is a schematic diagram of the result of stitching 61 images and images after screening;

Among them, (a) is the panoramic image spliced into the original image group; (b) is the panoramic image spliced into the filtered image group (including 31 images).

Fig. 6 is the topological structure analysis schematic diagram to 744 images and the image after screening;

Among them, (a) is the topological structure of the original image group; (b) is the topological structure of the filtered image group (including 375 images).

FIG. 7 is a schematic diagram of the results of stitching 744 images and filtered images.

Among them, (a) is the panoramic image spliced into the original image group; (b) is the panoramic image spliced into the filtered image group (including 375 images).

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the embodiments of the present invention are further described in detail below.

Example 1

Referring to Figure 1, in the process of panoramic image stitching, as the number of image stitching increases, data redundancy will appear, resulting in unnecessary calculations and reducing the stitching rate. In order to solve the above problems, an embodiment of the present invention proposes a panorama image stitching method based on image screening. Referring to FIG. 2 , the method includes the following steps:

101: Quickly obtain the similarity matrix of the image group, and propose an image screening algorithm to remove redundant images from the original image group;

102: Based on the weight matrix of the screened image groups, obtain the optimal reference image, determine the splicing sequence, and group the screened image groups into multiple small image groups;

103: Perform similarity transformation between the small image group and the small image group to obtain the initialized registration parameters, and then refine the initialized registration parameters through the homography model between adjacent images according to the splicing sequence under the inverse perspective constraint. change.

Example 2

The scheme in embodiment 1 is further introduced below in conjunction with specific examples, and is described in detail below:

201 : photographing the target area based on the drone, and obtaining multiple consecutive images with overlapping areas as the original image group;

The image information of the target area can be acquired through the operation of this step.

202: Perform rough feature point matching on the original image group to obtain a similarity matrix between images;

Wherein, the step 202 is specifically:

According to the number of matching feature points of the two images, establish a similarity matrix M between all images (I _i i=1,...,N, N represents the number of original images), where the ith image and the ith image The similarity M(i,j) between j images can be expressed as:

Among them, p is the number of feature points in image i, and q represents the number of feature points in image j. m and n represent the feature point numbers in image i and image j. If the feature points m and n match, v _mn is recorded as 1, otherwise it is recorded as 0.

The above method of calculating feature point matching is to extract feature points through the SURF (Speeded up robust features) ^[8-10] algorithm, and then filter the feature points extracted from each image, and each image only retains the first one from the Gaussian pyramid. The feature points extracted in the layer, and finally each image uses the filtered feature points and the remaining other images to match the feature points one by one.

203: Based on the similarity matrix between images, an image screening algorithm is proposed to remove redundant images from the original image group;

Wherein, this step 203 includes:

Based on the similarity matrix M of the original image group obtained in the previous step, (1) set the similarity threshold; (2) judge whether the maximum value in the similarity matrix of the current image group is greater than the similarity threshold, if it is greater than the similarity threshold, select The two images with the highest similarity in the current image group are determined, one of the images is determined as a redundant image, and the image group is removed; (3) the operation of step (2) is repeated until the similarity matrix of the current image group is in the The maximum value of is less than the similarity threshold.

Further, the formula for determining whether one of the images is a redundant image is:

Among them, a, b represent the numbers of the two images with the highest similarity, n represents the number of images in the current image group, calculate the sum of the similarity between all the images in the remaining image group after deleting each image, and then The image with the smallest sum of similarity after deletion is regarded as redundant image.

For example, there are 5 images A-E, where image A and image B are the two most similar images. Delete image A, calculate the sum of the similarity between the remaining four images B-E, delete image B, and calculate the sum of the similarity between the four images A, C-E. If the sum of the similarities obtained by deleting the image A is less than the sum of the similarities obtained by deleting the image B, then the image A is determined as a redundant image, it is removed from the image group, and no operation is performed on the image B. Conversely, image B is set as a redundant image, and it is removed from the image group, and no operation is performed on image A.

204: Based on the weight matrix of the filtered image group, find the optimal reference image, determine the splicing sequence, and group the images;

Wherein, this step 204 includes:

The method to find the optimal benchmark map is as follows: First, based on the similarity matrix M of the images, establish a weight matrix W between all images after screening, where the weight between image i and image j is defined as follows:

Among them, ε is the balance weight, inf is infinity, and In is the natural logarithm.

Then, based on the weight matrix W, run the shortest path algorithm, calculate the sum of the weights of the shortest paths from each point to all other points, and take the image represented by the point with the smallest sum of the shortest path weights as the optimal reference image, which is represented by the letter O. express.

For example, there are 5 images A-E and their weight matrices, run the shortest path algorithm, calculate the shortest path from image A to the four images B-E, and then calculate the sum of the weights of the four shortest paths, and so on, calculate each image The image reaches the shortest path of the remaining four images, and the sum of the weights of the four shortest paths corresponding to each image is obtained. If the sum of the weights of the shortest paths from image A to the remaining four images is the smallest, then image A is set as the optimal reference image.

The method of determining the splicing order is as follows: Based on the weight matrix W of the image group, starting from the point O representing the optimal reference map, using the breadth-first traversal algorithm, the sequence of obtained points is used as the image splicing sequence. As shown in Figure 3, Figure 7 is the spatial relationship diagram of 61 images, each point represents an image, the connection between the points indicates that the images have an overlapping relationship, and the numbers on the points represent the stitching of the images Order, point 0 represents the optimal benchmark map, and the numbers on other points represent the weight matrix W, starting from point 0, and running the breadth-first traversal algorithm to obtain the order of points in turn, and use this as the order of panorama image stitching.

The method of grouping images is as follows: set the number s of images contained in each group of images (s is less than the number of images in the image group n), and set the first to s images in the image sequence as the first group G ₁ ={I _i i=1,2,...,s}, the s+1 to 2s-th images are set as the second group G ₂ ={I _i i=s+1,s+2,. ..,2s},... until the number of remaining images is less than s, set the remaining images as the last group of images G _m ={I _i i=(m-1)s+1,i=(m-1) s+2,...,n}.

205: Image registration method combining image global similarity transformation and local perspective transformation;

Wherein, the step 205 is specifically:

In order to obtain good global consistency and prevent the accumulation of deformation errors, the initial registration between image groups is performed through a similarity transformation model.

表示图像组G_m中第i幅图像变换到最优基准图O的相似变换矩阵，n₁为图像组G_m中图像的个数；与图像组G_m相邻的图像组G_m+的相似变换模型的参数集合设为

表示图像组G_m+1中第j幅图像变换到最优基准图O的相似变换矩阵，n₂为图像组G_m+1中图像的个数，其中初始化配准的能量函数是：Set the parameter set of the similarity transformation model of the image group G _m as

Represents the similarity transformation matrix of the i-th image in the image group G _m transformed to the optimal reference image O, n ₁ is the number of images in the image group G _m ; the similarity transformation of the image group G _{m +} adjacent to the image group G m The parameter set of the model is set to

Represents the similarity transformation matrix of the jth image in the image group G _m+1 transformed to the optimal reference image O, n ₂ is the number of images in the image group G _m+1 , and the energy function of the initialization registration is:

E(S)=E ₁ (S|G _m ,G _m+1 )+E ₂ (S _m |G _m ) (4)

Among them, E ₁ (S|G _m , G _m+1 ) represents the sum of the registration errors of the image group G _m and its adjacent image group G _m+1 , S=S _m ∪S _m+1 , representing the image group G The union of the parameter sets of the similarity transformation models of _m and image group G _m+1 .

E ₂ (S _m |G _m ) represents the sum of the registration errors between images with overlapping regions within the image group G _m , and it is defined as:

t(x)代表对非齐次坐标x的变换，

代表图像i变换到最优基准图O的相似变换矩阵，

为图像i与图像j的第k个匹配点在图像i上的二维坐标，M_i,j代表图像i和图像j匹配点的个数。in,

t(x) represents the transformation of the inhomogeneous coordinate x,

The similarity transformation matrix representing the transformation of image i to the optimal reference image O,

is the two-dimensional coordinate of the kth matching point between image i and image j on image i, and M _i,j represents the number of matching points between image i and image j.

In order to obtain the best global consistency effect, the optimal solution is obtained by minimizing the total energy function. Through the above operations, the set of similar transformation matrices of all images transformed to the optimal reference map O is obtained:

其中

为图像i变换到最优基准图O的相似变换矩阵，n为图像组的图像总个数。为了提高全景图像局部重叠区域的配准精度，接下来通过单应性模型进行优化。

in

is the similarity transformation matrix for transforming the image i to the optimal reference graph O, and n is the total number of images in the image group. In order to improve the registration accuracy of the partial overlapping area of the panoramic image, the homography model is used for optimization.

的参数设置为单应性变换矩阵

的初始化参数，对变换矩阵

的优化公式如下：The optimization process of the homography transformation matrix is as follows:

The parameters are set to the homography transformation matrix

The initialization parameters for the transformation matrix

The optimization formula is as follows:

E(H)=E ₁ (H)+λE ₂ (H) (7)

Among them, λ is the weight coefficient for balancing E ₁ (H) and E ₂ (H); the purpose of E ₁ (H) is to minimize the sum of squares of registration errors of feature points between images and obtain a good local registration effect, the definition as follows:

代表图像i变换到最优基准图O的单应性变换矩阵，

为图像j变换到最优基准图O的单应性变换矩阵，

为图像i和图像j的第k对匹配点在图像i上的二维点坐标，

为图像j和图像i的第k对匹配点在图像i上的二维点坐标。in,

represents the homography transformation matrix of the image i transformed to the optimal reference graph O,

is the homography transformation matrix that transforms the image j to the optimal reference graph O,

is the two-dimensional point coordinates of the kth pair of matching points of image i and image j on image i,

is the two-dimensional point coordinates of the kth pair of matching points in image j and image i on image i.

The purpose of E ₂ (H) is to maintain global consistency and prevent severe accumulation of perspective distortion. Therefore, when using the homography model for optimization, the parameters of the homography model should be close to the parameters of the initialized similarity transformation model to prevent the displacement of the points from being too large during the transformation of the feature points. The definition is as follows:

的参数进行细化，将优化的结果作为每幅图像最后进行拼接的变换矩阵。By transforming the matrix homography for each image

The parameters are refined, and the optimized result is used as the transformation matrix for the final stitching of each image.

Example 3

To verify the effectiveness of the method, experiments are conducted on two sets of images collected by UAVs in this section, and the panoramic images generated with the original image set are compared with the panoramic images generated by the filtered image set. Figure 4 is a comparison of the topological structures of the 61 images before and after screening, Figure 4(a) is the topological structure of the original image group, Figure 4(b) is the topological structure of the filtered image group, and Figure 5 is the topological structure of the 61 images Comparison of splicing results before and after image screening, Figure 5(a) is the splicing result of the original image group, Figure 5(b) is the splicing result of the image group after screening; Figure 6 is the splicing result of 744 images before and after screening Figure 6(a) is the topological structure of the original image group, Figure 6(b) is the topological structure of the filtered image group, and Figure 7 is the comparison of the splicing results of 744 images before and after screening, Figure 7(a) is the stitching result of the original image group, and Figure 7(b) is the stitching result of the filtered image group.

The experimental results show the effectiveness of the method, which can remove redundant images from the image group on the premise of ensuring the integrity and quality of the final stitching result, and improve the stitching rate.

references

[1] K. Konolige, Sparse sparse bundle adjustment, in: British MachineVision Conference, 2010, pp.1–10.

[2] A.Y.Taygun Kekec, M.Unel, A new approach to real-time mosaicing ofaerialimages, Robot.Auton.Syst.62(12)(2014) 1755–1767.

[3]F.Caballero,L.Merino,J.Ferruz,A.Ollero,Homography based Kalmanfilter for mosaic building.applications to UAV position estimation,in:Proceedings of the IEEE International Conference on Robotics and Automation,2007,pp.2004 –2009

[4] A. Elibol, N. Gracias, R. Garcia, Fast topology estimation for imagemosaicing using adaptive information thresholding, Robot.Auton.Syst.61(2)(2013) 125–136.

[5] R. Szeliski, Image alignment and stitching: a tutorial, Found. TrendsComput.Graph.Vis.2(1)

(2006) 1–104.

[6] T.E.Choe, I.Cohen, M.Lee, G.Medioni, Optimal global mosaic generation from retinal images, in: Proceedings of the IEEE International Conference on Pattern Recognition, Vol.3, 2006, pp.681–684

[7] M. Xia, J. Yao, R. Xie, L. Li, and W. Zhang. Globally consistent alignment for planar mosaicking via topology analysis. Pattern Recognition, 66: 239–252, 2017.

[8] Bay, Herbert, Tinne Tuytelaars, and Luc Van Gool. "Surf: Speeded uprobust features." European conference on computer vision. Springer, Berlin, Heidelberg, 2006.

[9]Taussky O,Zassenhaus H.On the similarity transformation between amatirx and its transpose[J].Pacific Journal of Mathematics,1959,9(3):893-896.

[10]Dubrofsky E.Homography Estimation[D].UNIVERSITY OF BRITISHCOLUMBIA(Vancouver,2009.

Those skilled in the art can understand that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (6)

1. a panorama image stitching method based on image screening, is characterized in that, described method comprises the following steps: According to the similarity matrix between images, an image screening algorithm is proposed to remove redundant images from the original image group; Based on the weight matrix of the screened image groups, the optimal reference image is obtained, the splicing sequence is determined, and the screened image groups are grouped into multiple small image groups; Similar transformation is performed between the small image group and the small image group to obtain the initialized registration parameters, and then according to the splicing sequence, the initialized registration parameters are refined by the homography model between adjacent images under the constraint of inverse perspective. 2. a kind of panorama image stitching method based on image screening according to claim 1, is characterized in that, the described image screening algorithm is proposed according to the similarity matrix according to the image group, and the redundant image is removed from the original image group, specifically: : 1) Based on the similarity matrix of the original image group, set the similarity threshold; 2) Determine whether the maximum value in the similarity matrix of the current image group is greater than the similarity threshold, if so, select the two images with the highest similarity in the current image group, determine one of the images as a redundant image, and remove the image group; 3) Repeat the operation of step 2) until the maximum value in the similarity matrix of the current image group is smaller than the similarity threshold. 3. a kind of panorama image stitching method based on image screening according to claim 2, is characterized in that, the described one image is judged as redundant image is specifically: after calculating and deleting each image respectively, remaining image group The sum of the similarities between all the images in the image, and then the image with the smallest sum of similarities after deletion is designated as the redundant image. 4. a kind of panorama image stitching method based on image screening according to claim 1, is characterized in that, described based on the weight matrix of the screened image group, to find the optimal reference image is specifically: Based on the similarity matrix of the images, establish a weight matrix between all the filtered images, run the shortest path algorithm based on the weight matrix, and calculate the sum of the weights of the shortest paths from each point to all other points; The image represented by the point is used as the optimal reference map. 5. a kind of panorama image stitching method based on image screening according to claim 1, is characterized in that, described determining stitching sequence is specifically: Based on the weight matrix of the image group, starting from the point representing the optimal reference map, using the breadth-first traversal algorithm, the obtained point sequence is used as the image stitching sequence. 6. A panorama image stitching method based on image screening according to claim 1, wherein the method further comprises: The similarity transformation is performed between the small image group and the small image group to obtain the initialized registration parameters, and then according to the splicing sequence, the initialized registration parameters are refined by the homography model between adjacent images under the perspective constraint; Minimize the total energy function to obtain the optimal solution.

Images