CN112435163B - Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization - Google Patents
Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization Download PDFInfo
- Publication number
- CN112435163B CN112435163B CN202011292721.6A CN202011292721A CN112435163B CN 112435163 B CN112435163 B CN 112435163B CN 202011292721 A CN202011292721 A CN 202011292721A CN 112435163 B CN112435163 B CN 112435163B
- Authority
- CN
- China
- Prior art keywords
- grid
- constraint
- image
- vertex
- linear
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000005457 optimization Methods 0.000 title claims abstract description 27
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 48
- 239000011159 matrix material Substances 0.000 claims abstract description 15
- 238000013507 mapping Methods 0.000 claims abstract description 13
- 230000004927 fusion Effects 0.000 claims abstract description 9
- RZVAJINKPMORJF-UHFFFAOYSA-N Acetaminophen Chemical compound CC(=O)NC1=CC=C(O)C=C1 RZVAJINKPMORJF-UHFFFAOYSA-N 0.000 claims abstract 5
- 208000029618 autoimmune pulmonary alveolar proteinosis Diseases 0.000 claims abstract 5
- 230000009466 transformation Effects 0.000 claims description 22
- 239000013598 vector Substances 0.000 claims description 14
- 238000005070 sampling Methods 0.000 claims description 13
- 238000012216 screening Methods 0.000 claims description 4
- 230000002708 enhancing effect Effects 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 4
- 238000012545 processing Methods 0.000 abstract description 2
- 238000005452 bending Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000011426 transformation method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
-
- G06T3/18—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
- G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/35—Determination of transform parameters for the alignment of images, i.e. image registration using statistical methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
An unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization belongs to the technical field of image processing. The method comprises the following steps: s1, inputting two unmanned aerial vehicle aerial images with parallax, pre-configuring the aerial images by using an APAP algorithm which is as consistent as possible, and obtaining a global homography matrix. And S2, carrying out grid division on the aerial image to obtain a local homography constraint item. And S3, constructing a total energy function of the grid optimization model, and adding a global similarity term constraint term, a global homography constraint term and a linear constraint term into the APAP algorithm to optimize the grid model. And S4, solving the total energy function in the sparse linear system by adopting an iterative optimization method, and guiding grid deformation. And S5, mapping the deformed image onto a canvas, and carrying out pixel fusion with the reference image to finish the aerial image splicing work. The method has good splicing effect and certain parallax tolerance, not only solves the problems of ghost and distortion of aerial image splicing, but also can protect the linear structure in the original scene and meet the actual requirements of aerial image splicing.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a method for splicing aerial images of an unmanned aerial vehicle by linear feature protection and grid optimization.
Background
The unmanned aerial vehicle aerial photography technology has the advantages of being light, flexible, free of space-time region limitation and the like, information can be obtained in severe and dangerous environments, and therefore the unmanned aerial vehicle aerial photography technology is widely applied to various fields. However, considering that a scene covered by a single aerial image cannot meet the requirement of a researcher on information acquisition, the scene needs to be post-processed, and two or more images with overlapped areas are spliced to obtain a high-resolution panoramic image with a wider range and more information.
At present, the splicing of aerial images is mainly based on a homography transformation method, an image to be spliced and a reference image are transformed to the same plane through a simple global transformation, the algorithm speed is high, but the method is only suitable for the situation that the images are at the same visual angle, parallax exists between the aerial images, and the spliced images can generate ghost and obvious distortion, so that the problem of homography transformation is solved by using a grid optimization method. The aerial image based on grid optimization is to divide an image to be spliced into a plurality of grids, calculate a transformation matrix of each grid respectively to guide grid deformation, enable the image to be aligned better and reduce distortion of non-overlapping areas, and the main calculation methods comprise APAP, SPHP, AANAP, NISwGSP and the like. Despite the advantages of each of these algorithms, they are not suitable for the actual aerial image stitching task. For example, the APAP algorithm originally proposed the idea of a grid, and the image alignment is good, but serious distortion is generated; the splicing speed of the SPHP algorithm and the AANAP algorithm is high, the problem of image distortion is solved, and the bending of a linear set structure of the original scene is caused; the niswgp algorithm also does not solve the problem of good image warping. An aerial image sequence obtained by using an unmanned aerial vehicle often has a certain parallax, and the problems of distortion and image naturalness improvement are solved and the performance of an aerial image splicing algorithm is improved on the premise of ensuring the alignment of spliced images.
Disclosure of Invention
The invention aims to solve the problems of ghost, distortion, structural deformation and the like of the current mainstream unmanned aerial vehicle aerial image splicing method, two constraint items of global similarity and linear feature protection are added in an APAP algorithm and are improved, and the unmanned aerial image splicing method based on linear feature protection and grid optimization is provided.
In order to achieve the purpose, the invention adopts the technical scheme that:
an unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization comprises the following steps:
s1: inputting two aerial images I with certain parallax 1 、I 2 ，I 1 Is a reference picture, I 2 For the images to be spliced, an As-project-As-Possible algorithm APAP (As-project-As-Possible) is adopted to carry out aerial image shootingAnd (4) pre-registering. The pre-registration specifically comprises the steps of firstly extracting and matching feature points by using an SIFT algorithm, then screening out error feature points by using an RANSAC algorithm, and calculating to obtain a global transformation matrix.
S2: and (4) carrying out grid division on the aerial image in the step (S1) to obtain a grid vertex set.
S3: and (3) constructing a total energy function of the grid optimization model through the grid vertex set obtained in the step (S2), and adding three constraint items of global similarity, local similarity and linear protection to optimize the grid model on the basis of the local homography constraint item provided by the APAP algorithm.
S4: in a sparse linear system, an iterative optimization method is adopted to solve a total energy function and guide grid deformation.
S5: and mapping the deformed image onto a canvas, and performing pixel fusion with the reference image to finish the aerial image splicing work.
Further, the S2 carries out grid division on the aerial photography image in the S1 to obtain a grid vertex set. Specifically, reference image I 1 And an image I to be stitched 2 After meshing, a 2 n-dimensional vector is used:
V 2 ＝[x 1 y 1 x 2 y 2 … x n y n ] T
to represent I 2 And (3) a grid vertex coordinate set to be optimized, wherein n is the number of the divided vertexes, and x and y are horizontal and vertical coordinates of the grid vertexes. The vectors are used simultaneously:
V 2 ′＝[x′ 1 y′ 1 x′ 2 y′ 2 … x′ n y′ n ] T
to express the coordinate set of the mesh vertex after deformation, where x 'and y' are the horizontal and vertical coordinates of the mesh vertex to be optimized. V 2 The solution process for' will be given in the following steps.
Further, the grid vertex set obtained in the step S3 through the step S2 is used to construct a total energy function of a grid optimization model, and three constraint terms, namely global similarity, local similarity and linear protection, are added to optimize the grid model on the basis of a local homography constraint term provided by an original APAP algorithm, so as to protect the problems of distortion of non-overlapping regions of images and bending of linear geometric structures. The method comprises the following specific steps:
s3.1: a grid energy function is defined. For any sample point p (x, y) in the reference image, it can be represented by bilinear interpolation of the coordinates of the grid vertex where it is located, that is:
v(p)＝w 1 v 1 +w 2 v 2 +w 3 v 3 +w 4 v 4
in the formula, w is the distance weight from a sampling point to a grid vertex v; thus, the constraint applied to the sampling point is equivalent to the constraint applied to the vertex of the mesh where the sampling point is located. The total energy function of the grid optimization may be defined as:
E′(V′ 2 )＝λ lh E lh (V′ 2 )+λ ls E ls (V′ 2 )+λ gs E gs (V′ 2 )+λ l E l (V′ 2 )
in the formula, E lh (V′ 2 ) As local homography constraints, E ls (V′ 2 ) For local similarity constraint, E gs (V′ 2 ) For global similarity constraint, E l (V′ 2 ) For the linear protection constraint, lambda is the adjusting weight of each constraint term, and the larger lambda is, the stronger the corresponding constraint is. The specific form of the four constraint terms is as in steps 3.2-3.5.
S3.2: local similarity constraints are defined. The local similarity constraint term helps to maintain the continuity of the transformation between grids, and the specific expression is as follows:
wherein E' (j, k) is the image I to be spliced 2 Set of edges of v j And v k Are the two endpoints of the grid edge, v' j And v k ' is a mesh edge vertex mapped onto canvas, S jk Representing grid edge experiencesSimilarity transformation of (2).
S3.3: a global similarity constraint is defined. The global similarity item can reduce the distortion problem of image registration, the full alignment is ensured mainly by local homography and local similarity in an overlapping area, the retention is enhanced mainly by the global similarity in a non-overlapping area by taking the thought of the NISwGSP algorithm as reference, and the image distortion of a large parallax scene caused in the alignment process of the Moving DLT method is reduced. The specific expression is as follows:
w jk ＝d jk +β
in the formula, d jk Is the distance between the grid edge and the overlap region, beta is the adjustment quantity, s I′ And theta I′ Globally similar scale factor of I' and 2D rotation angle, v, respectively j And v k Are the two endpoints of the mesh edge, v' j And v k ' is the mesh edge vertex mapped onto the canvas.
S3.4: a local homography constraint term is defined. The local homography item corresponds to solve the alignment problem through the enhancement point, and the specific expression is as follows:
wherein M (p, q) is I and I obtained by APAP algorithm 2 The characteristic point sets are p and q are I and I respectively 2 Characteristic point of (3), W I And W I2 The weight matrix of the grid where p and q are respectively:
where w' is the weight of the aforementioned grid vertices.
S3.5: a straight-line protection constraint term is defined. Although the image distortion is weakened to a certain extent by the similarity transformation, for the case that the image has parallax, after the similarity constraint is added, the line direction of the scene linear structure is obviously bent, so that the splicing result is unnatural. The reason is that during image mapping, the scene in each grid is independently transformed along with grid transformation, and if the scene spans two or more grids, it cannot be guaranteed that the straight line geometric structure in the original scene is still on one straight line after transformation. The linear protection constraint term can enable the homography transformation to be closer to the motion of a camera, and protect the linear structure in a scene, and the specific expression is as follows:
wherein L is the collected line segment set, m 0 And m 1 For sampling the coordinates of the two end points of the line segment, S i Is the current point and m 1 The area of the quadrangle formed.
Further, the S4 is used for solving the total energy function obtained by the S3 in the sparse system to guide grid deformation. Specifically, the constraint term is substituted into the sparse matrix to obtain:
in the formula, J lj As a constraint term E lh (V′ 2 )、J ls As a constraint term E ls (V′ 2 )、J gs As a constraint term E gs (V′ 2 )、J l As a constraint term E l (V′ 2 ) Set of grid vertices V 2 ' corresponding Jacob matrix, a gs Residual vector, a, being a global similarity term l The residual vector of the straight-line constraint term. The optimal vertex set V can be obtained by solving the equation set 2 ′。
Further, the aerial photography in S5And carrying out image mapping and pixel fusion on the image. Specifically, firstly, a reference image I is mapped to a canvas, and then an end point of the reference image I is used as a coordinate origin to obtain an image I to be spliced 2 And (4) the grid vertex coordinates are also mapped to the canvas to complete image mapping, and finally, the panoramic aerial image is obtained through simple linear weighting fusion.
The beneficial effects of the invention are as follows:
the invention provides an unmanned aerial vehicle aerial image splicing algorithm with linear structure protection and grid optimization, aiming at the problems of ghosting, distortion and unnaturalness in the process of splicing aerial images with parallax by the current mainstream algorithm. Firstly, pre-registering by using an APAP algorithm, and calculating local homography constraint through grid division; then, constructing a grid optimization model, defining and solving a total energy function containing a global similarity term, a local homography term and a linear protection term to obtain a deformed grid vertex set; and finally, mapping the image into a canvas according to the obtained grid vertex, and completing aerial image splicing work by a weighted fusion method. Experiments prove that the method has good splicing effect and certain parallax tolerance, not only solves the problems of ghost and distortion of aerial image splicing, but also can protect the linear structure in the original scene, and can meet the actual requirements of aerial image splicing.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
FIG. 2 is a schematic diagram of a reference image and an image to be stitched, in which (a), (b), and (c) are three sets of aerial images with overlapping regions, respectively;
fig. 3 is an image obtained by extracting features of two images by using a SIFT algorithm in the embodiment of the present invention, where (a) feature points are extracted for reference images, and (b) feature points are extracted for images to be stitched;
FIG. 4 is an image obtained after adding a linear feature constraint in an APAP algorithm modified in an embodiment of the present invention, where (a) a linear feature is extracted for a reference image, and (b) a linear feature is extracted for an image to be stitched;
fig. 5 is a panoramic image comparison obtained by splicing the algorithm provided by the invention and the APAP algorithm. Wherein (a), (b) and (c) are the stitching results of the three groups of aerial images in fig. 2.
Detailed Description
For a better understanding of the present invention, those skilled in the art will now make a more detailed description of the present invention with reference to the accompanying drawings and the following examples.
Referring to fig. 1, the embodiment provides an unmanned aerial vehicle aerial image stitching method for linear feature protection and grid optimization, which includes the following steps:
s1: inputting two aerial images I with certain parallax 1 、I 2 As shown in fig. 2. The As-Projective-As-Possible (APAP) algorithm is used for pre-registration. The pre-registration specifically includes firstly extracting and matching feature points by using an SIFT algorithm, then screening out error feature points by using an RANSAC algorithm, and calculating to obtain a global transformation matrix, as shown in FIG. 3.
S2: and (5) carrying out mesh division on the aerial image in the step (S1) to obtain a mesh vertex set.
S3: and (3) constructing a total energy function of a grid optimization model through the grid vertex set obtained in the step (S2), and adding three constraint items of global similarity, local similarity and linear protection to optimize the grid model on the basis of the local homography constraint item provided by the APAP algorithm, as shown in FIG. 4.
S4: in a sparse linear system, an iterative optimization method is adopted to solve a total energy function and guide grid deformation.
S5: and mapping the deformed image onto a canvas, and performing pixel fusion with the reference image to finish the aerial image splicing work, as shown in fig. 5.
Specifically, in S1, an As-project-As-Possible (APAP) algorithm is used for pre-configuring aerial images. The APAP algorithm pre-registration specifically comprises the steps of firstly extracting and matching feature points by using a SIFT algorithm, then screening error feature points by using a RANSAC algorithm, and calculating to obtain a global transformation matrix.
Further, in S2, the aerial image in S1 is subjected to mesh division, in this embodiment, the mesh size is set to 80 × 80, and a local homography constraint term is calculated. In particular, the method comprises the following steps of,
specifically, reference picture I 1 And images I to be stitched 2 After meshing, a 2 n-dimensional vector is used:
V 2 ＝[x 1 y 1 x 2 y 2 … x n y n ] T
to represent I 2 And (3) a grid vertex coordinate set to be optimized, wherein n is the number of the divided vertexes, and x and y are horizontal and vertical coordinates of the grid vertexes. The vectors are used simultaneously:
V 2 ′＝[x′ 1 y′ 1 x′ 2 y′ 2 … x′ n y′ n ] T
to express the coordinate set of the mesh vertex after deformation, where x 'and y' are the horizontal and vertical coordinates of the mesh vertex to be optimized. V 2 The solution process of' will be given in the following steps.
Further, the total energy function of the grid optimization model is constructed through the grid vertex set obtained in the step S3 through the step S2, and global and local similarities and straight line protection constraint items are added on the basis of the original APAP algorithm local homography constraint item so as to protect the problems of image non-overlapping area distortion and straight line geometric structure bending. Comprises the following specific steps
S3.1: a mesh energy function is defined. For any sample point p (x, y) in the reference image, it can be represented by bilinear interpolation of the coordinates of the grid vertex where it is located, that is:
v(p)＝w 1 v 1 +w 2 v 2 +w 3 v 3 +w 4 v 4
in the formula, w is the distance weight from a sampling point to a grid vertex v; thus, the constraint applied to the sampling point is equivalent to the constraint applied to the vertex of the mesh where the sampling point is located. The total energy function of the grid optimization may be defined as:
E′(V′ 2 )＝λ lh E lh (V′ 2 )+λ ls E ls (V′ 2 )+λ gs E gs (V′ 2 )+λ l E l (V′ 2 )
in the formula, E lh (V′ 2 ) As local homography constraints, E ls (V′ 2 ) For local similarity constraint, E gs (V′ 2 ) For global similarity constraint, E l (V′ 2 ) For the linear protection constraint, lambda is the adjusting weight of each constraint term, and the larger lambda is, the stronger the corresponding constraint is. The specific form of the four constraint terms is as in steps 3.2-3.5. In the present embodiment, the parameter is set to λ lh ＝5、λ ls ＝5、λ gs ＝100、λ l ＝5。
S3.2: local similarity constraints are defined. The local similarity constraint term helps to maintain the continuity of the transformation between grids, and the specific expression is as follows:
wherein E' (j, k) is an image I to be spliced 2 Set of edges of v j And v k Are the two endpoints of the mesh edge, v' j And v k ' is a mesh edge vertex mapped onto canvas, S jk Representing the similarity transformation experienced by the edges of the mesh.
S3.3: a global similarity constraint is defined. The global similarity item can reduce the distortion problem of image registration, the full alignment is ensured mainly by local homography and local similarity in an overlapping area, and the image distortion of a large parallax scene caused in the alignment process of the Moving DLT method is reduced in a non-overlapping area by taking the global similarity as the main enhancement and the retentivity as the reference of the idea of the NISwGSP algorithm. The specific expression is as follows:
w jk ＝d jk +β
in the formula (d) jk Is the distance between the grid edge and the overlap region, beta is the adjustment quantity, s I′ And theta I′ Globally similar scale factor of I' and 2D rotation angle, v, respectively j And v k Are the two endpoints of the mesh edge, v' j And v k ' is the mesh edge vertex mapped onto the canvas.
S3.4: a local homography constraint term is defined. The local homography item corresponds to solve the alignment problem through the enhancement point, and the specific expression is as follows:
wherein M (p, q) is I and I obtained by APAP algorithm 2 A characteristic point set, p and q are I and I respectively 2 Characteristic point of (3), W I And W I2 The weight matrix of the grid where p and q are respectively:
where w' is the weight of the aforementioned grid vertices.
S3.5: a straight-line protection constraint term is defined. Although the image distortion is weakened to a certain extent by the similarity transformation, for the case that the image has parallax, after the similarity constraint is added, the line direction of the scene linear structure is obviously bent, so that the splicing result is unnatural. The reason is that during image mapping, the scene in each grid is independently transformed along with grid transformation, and if the scene spans two or more grids, it cannot be guaranteed that the straight line geometric structure in the original scene is still on one straight line after transformation. The linear protection constraint term can enable the homography transformation to be closer to the motion of a camera, and protect the linear structure in a scene, the linear matching length threshold value in the embodiment is 100, the extracted linear features are shown in fig. 4, and the specific expression is as follows:
wherein L is the collected line segment set, m 0 And m 1 For sampling the coordinates of the two end points of the line segment, S i Is the current point and m 1 The area of the quadrangle formed.
Further, S4, the grid deformation is guided by the total energy function obtained by solving S3 in the sparse system. Specifically, the constraint term is substituted into the sparse matrix to obtain:
in the formula, J lj As a constraint term E lh (V′ 2 )、J ls As a constraint term E ls (V′ 2 )、J gs As a constraint term E gs (V′ 2 )、J l As a constraint term E l (V′ 2 ) Set V 'at grid vertices' 2 Corresponding Jacob matrix, a gs Residual vector, a, as global similarity term l The residual vector of the straight-line constraint term. Solving the equation set can obtain an optimal vertex set V' 2 。
Further, in the step S5, image mapping and pixel fusion are performed on the aerial image. Specifically, firstly, a reference image I is mapped to a canvas, and then an end point of the reference image I is used as a coordinate origin to obtain an image I to be spliced 2 The grid vertex coordinates minus the offset are also mapped to the canvas to complete image mapping, and finally the panoramic aerial image is obtained through simple linear weighted fusion, as shown in fig. 5.
The pictures used in the simulation experiment are all actual images shot by the unmanned aerial vehicle, the resolution ratio is 4000 × 3000, and as shown in fig. 2, an overlapped scene and a certain parallax exist between two images in each group. Fig. 5 is a comparison of the panoramic image obtained by using the algorithm of the present invention and the original APAP algorithm, which shows that the present invention has better completed the splicing task, not only ensures the alignment effect of the original APAP algorithm, but also improves the problems of distortion and linear geometry bending of the overlapping area, and the obtained panoramic image is more natural.
The comparison table of the time complexity of the invention and the aerial image splicing based on the APAP algorithm is shown in Table 1, and it can be seen that the invention can greatly improve the image splicing effect and enhance the comprehensive performance of the algorithm on the premise that the time complexity is basically equal to the original algorithm.
TABLE 1 Algorithm time complexity contrast(s)
The above description is only a preferred embodiment of the present invention, and not intended to limit the present invention, the scope of the present invention is defined by the appended claims, and all equivalent structural changes made by using the contents of the specification and the drawings of the present invention should be covered by the scope of the present invention.
Claims (4)
1. A method for splicing aerial images of an unmanned aerial vehicle based on linear feature protection and grid optimization is characterized by comprising the following steps:
s1: inputting two aerial images I with certain parallax 1 、I 2 ，I 1 Is a reference picture, I 2 Pre-registering the images to be spliced by adopting an APAP algorithm as consistent as possible;
s2: carrying out mesh division on the aerial image in the step S1 to obtain a mesh vertex set;
s3: and (3) constructing a total energy function of the grid optimization model through the grid vertex set obtained in the step (S2), and adding three constraint items of global similarity, local similarity and linear protection to optimize the grid model on the basis of the local homography constraint item provided by the APAP algorithm, wherein the specific steps are as follows:
s3.1: defining a grid energy function; for any sampling point p (x, y) in the reference image, it is represented by bilinear interpolation of the coordinates of the grid vertex where it is located, that is:
v(p)＝w 1 v 1 +w 2 v 2 +w 3 v 3 +w 4 v 4
in the formula, w is the distance weight from a sampling point to a grid vertex v;
the constraint applied to the sampling points is equivalent to the constraint applied to the grid vertexes where the sampling points are located; the total energy function of the grid optimization is defined as:
E′(V′ 2 )＝λ lh E lh (V′ 2 )+λ ls E ls (V′ 2 )+λ gs E gs (V′ 2 )+λ l E l (V′ 2 )
in the formula, E lh (V′ 2 ) As local homography constraints, E ls (V′ 2 ) For local similarity constraint, E gs (V′ 2 ) For global similarity constraints, E l (V′ 2 ) For linear protection constraint, lambda is the adjusting weight of each constraint term, and the larger lambda is, the stronger the corresponding constraint is;
s3.2: defining a local similarity constraint; the local similarity constraint term helps to maintain the continuity of the transformation between grids, and the expression is as follows:
wherein E' (j, k) is an image I to be spliced 2 Set of edges of v j And v k Are the two endpoints of the mesh edge, v' j And v k ' is a mesh edge vertex mapped onto canvas, S jk Representing the similarity transformation undergone by the mesh edges;
s3.3: defining a global similarity constraint; in the overlapping area, the local homography and the local similarity are mainly used for ensuring full alignment, in the non-overlapping area, the concept of the NISwGSP algorithm is used for reference, the global similarity is mainly used for enhancing the retentivity, and the image distortion of a large parallax scene caused in the alignment process of the Moving DLT method is reduced; the specific expression is as follows:
w jk ＝d jk +β
in the formula (d) jk Is the distance between the grid edge and the overlap region, beta is the adjustment quantity, s I′ And theta I′ Globally similar scale factor and 2D rotation angle, v, respectively j And v k Are the two endpoints of the mesh edge, v' j And v k ' is a mesh edge vertex mapped onto the canvas;
s3.4: defining a local homography constraint term; the local homography item corresponds to solve the alignment problem through the enhancement point, and the specific expression is as follows:
wherein M (p, q) is I and I obtained by APAP algorithm 2 A characteristic point set, p and q are I and I respectively 2 Characteristic point of (3), W I And
the weight matrix of the grid where p and q are respectively:
where w' is the weight of the aforementioned grid vertices;
s3.5: defining a straight line protection constraint item; the linear protection constraint term can enable the homography transformation to be closer to the motion of a camera, and protect the linear structure in a scene, and the specific expression is as follows:
wherein L is the collected line segment set, m 0 And m 1 For sampling the coordinates of the two end points of the line segment, S i Is the current point and m 1 The area of the formed quadrangle;
s4: the total energy function obtained by the S3 is solved in a sparse system to guide grid deformation;
s5: carrying out image mapping and pixel fusion on the aerial image; specifically, firstly, a reference image I is mapped to a canvas, and then an end point of the reference image I is used as a coordinate origin to obtain an image I to be spliced 2 And (4) subtracting the offset from the grid vertex coordinates, mapping the grid vertex coordinates onto canvas to complete image mapping, and finally obtaining the panoramic aerial image through simple linear weighting fusion.
2. The unmanned aerial vehicle aerial image splicing method for linear feature protection and grid optimization according to claim 1, wherein the pre-registration manner in the step S1 is as follows: firstly, extracting and matching feature points by using an SIFT algorithm, then screening out error feature points by using an RANSAC algorithm, and calculating to obtain a global transformation matrix.
3. The unmanned aerial vehicle aerial image splicing method for linear feature protection and grid optimization according to claim 1, wherein in S2, the aerial image in S1 is subjected to grid division to obtain a grid vertex set; specifically, reference image I 1 And an image I to be stitched 2 After grid division, a 2 n-dimensional vector is obtained:
V 2 ＝[x 1 y 1 x 2 y 2 …x n y n ] T
representing I by 2 n-dimensional vectors 2 A grid vertex coordinate set to be optimized, wherein n is the number of divided vertexes, and x and y are horizontal and vertical coordinates of the grid vertexes;
the method adopts a mesh vertex coordinate set after deformation represented by using vectors, wherein x 'and y' are horizontal and vertical coordinates of a mesh vertex to be optimized, and the vectors are used as follows:
V 2 ′＝[x′ 1 y′ 1 x′ 2 y′ 2 …x′ n y′ n ] T 。
4. the unmanned aerial vehicle aerial image splicing method for linear feature protection and grid optimization according to claim 1, wherein S4 is used for solving a total energy function obtained in S3 in a sparse system to guide grid deformation; specifically, the constraint term is substituted into the sparse matrix to obtain:
in the formula, J lj As a constraint term E lh (V′ 2 )、J ls As a constraint term E ls (V′ 2 )、J gs As a constraint term E gs (V′ 2 )、J l As a constraint term E l (V′ 2 ) Set V 'at grid vertices' 2 Corresponding Jacob matrix, a gs Residual vector, a, being a global similarity term l A residual vector that is a straight line constraint term; solving the equation set can obtain an optimal vertex set V' 2 。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011292721.6A CN112435163B (en) | 2020-11-18 | 2020-11-18 | Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011292721.6A CN112435163B (en) | 2020-11-18 | 2020-11-18 | Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112435163A CN112435163A (en) | 2021-03-02 |
| CN112435163B true CN112435163B (en) | 2022-10-18 |
Family
ID=74693183
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011292721.6A Active CN112435163B (en) | 2020-11-18 | 2020-11-18 | Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112435163B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112967217B (en) * | 2021-03-11 | 2024-02-06 | 大连理工大学 | Image stitching method based on straight line feature matching and constraint |
| CN113313659B (en) * | 2021-04-25 | 2024-01-26 | 中国人民解放军火箭军工程大学 | High-precision image stitching method under multi-machine cooperative constraint |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109961398A (en) * | 2019-02-18 | 2019-07-02 | 鲁能新能源(集团)有限公司 | Fan blade image segmentation and grid optimization joining method |
| CN110211043A (en) * | 2019-05-11 | 2019-09-06 | 复旦大学 | A kind of method for registering based on grid optimization for Panorama Mosaic |
| CN110781903A (en) * | 2019-10-12 | 2020-02-11 | 中国地质大学（武汉） | Unmanned aerial vehicle image splicing method based on grid optimization and global similarity constraint |
-
2020
- 2020-11-18 CN CN202011292721.6A patent/CN112435163B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109961398A (en) * | 2019-02-18 | 2019-07-02 | 鲁能新能源(集团)有限公司 | Fan blade image segmentation and grid optimization joining method |
| CN110211043A (en) * | 2019-05-11 | 2019-09-06 | 复旦大学 | A kind of method for registering based on grid optimization for Panorama Mosaic |
| CN110781903A (en) * | 2019-10-12 | 2020-02-11 | 中国地质大学（武汉） | Unmanned aerial vehicle image splicing method based on grid optimization and global similarity constraint |
Non-Patent Citations (2)
| Title |
|---|
| 具有直线结构保护的网格化图像拼接;何川等;《中国图象图形学报》;20180716(第07期);全文 * |
| 基于控制点网格优化的航拍图像拼接算法;石祥滨等;《计算机应用与软件》;20200512(第05期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112435163A (en) | 2021-03-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ji et al. | Deep view morphing | |
| CN112435163B (en) | Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization | |
| CN107274337B (en) | Image splicing method based on improved optical flow | |
| CN110246161B (en) | Method for seamless splicing of 360-degree panoramic images | |
| CN107767339B (en) | Binocular stereo image splicing method | |
| CN107451952B (en) | Splicing and fusing method, equipment and system for panoramic video | |
| CN111553841B (en) | Real-time video splicing method based on optimal suture line updating | |
| CN110111250A (en) | A kind of automatic panorama unmanned plane image split-joint method and device of robust | |
| CN111242848B (en) | Binocular camera image suture line splicing method and system based on regional feature registration | |
| Li et al. | A unified framework for street-view panorama stitching | |
| CN112862683B (en) | Adjacent image splicing method based on elastic registration and grid optimization | |
| CN110781903A (en) | Unmanned aerial vehicle image splicing method based on grid optimization and global similarity constraint | |
| CN108280805B (en) | Image splicing method based on manifold optimization | |
| CN116958437A (en) | Multi-view reconstruction method and system integrating attention mechanism | |
| CN110580715A (en) | Image alignment method based on illumination constraint and grid deformation | |
| Lu et al. | Multiperspective image stitching and regularization via hybrid structure warping | |
| Meng et al. | High-quality panorama stitching based on asymmetric bidirectional optical flow | |
| Rui et al. | Research on fast natural aerial image mosaic | |
| CN109087250B (en) | Image splicing method based on regular boundary constraint | |
| Shibayama et al. | Reconstruction of 3D surface and restoration of flat document image from monocular image sequence | |
| Islam et al. | Overview and challenges of different image morphing algorithms | |
| Sun et al. | Image stitching with weighted elastic registration | |
| Xu et al. | Document Image Rectification in Complex Scene Using Stacked Siamese Networks | |
| Zhou et al. | Rectangular-Output Image Stitching | |
| CN113469880A (en) | Image splicing method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |