CN112435163A - Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization - Google Patents

Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization Download PDF

Info

Publication number
CN112435163A
CN112435163A CN202011292721.6A CN202011292721A CN112435163A CN 112435163 A CN112435163 A CN 112435163A CN 202011292721 A CN202011292721 A CN 202011292721A CN 112435163 A CN112435163 A CN 112435163A
Authority
CN
China
Prior art keywords
grid
constraint
image
vertex
linear
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011292721.6A
Other languages
Chinese (zh)
Other versions
CN112435163B (en
Inventor
刘强
王钧
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dalian University of Technology
Original Assignee
Dalian University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dalian University of Technology filed Critical Dalian University of Technology
Priority to CN202011292721.6A priority Critical patent/CN112435163B/en
Publication of CN112435163A publication Critical patent/CN112435163A/en
Application granted granted Critical
Publication of CN112435163B publication Critical patent/CN112435163B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/18—Image warping, e.g. rearranging pixels individually
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/35—Determination of transform parameters for the alignment of images, i.e. image registration using statistical methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10032—Satellite or aerial image; Remote sensing
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Processing (AREA)

Abstract

An unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization belongs to the technical field of image processing. The method comprises the following steps: s1, inputting two unmanned aerial vehicle aerial images with parallax, pre-configuring the aerial images by using an APAP algorithm which is as consistent as possible, and obtaining a global homography matrix. S2, meshing the aerial image to obtain a local homography constraint item. S3, a total energy function of the grid optimization model is constructed, and a global similarity term constraint term, a global homography constraint term and a straight line constraint term are added into an APAP algorithm to optimize the grid model. S4, solving the total energy function in the sparse linear system by adopting an iterative optimization method, and guiding the grid deformation. And S5, mapping the deformed image onto a canvas, performing pixel fusion with the reference image, and completing aerial image splicing. The method has good splicing effect and certain parallax tolerance, not only solves the problems of ghost and distortion of aerial image splicing, but also can protect the linear structure in the original scene and meet the actual requirements of aerial image splicing.

Description

Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization
Technical Field
The invention relates to the technical field of image processing, in particular to an unmanned aerial vehicle aerial image splicing method for linear feature protection and grid optimization.
Background
The unmanned aerial vehicle aerial photography technology has the advantages of being light, flexible, free of space-time region limitation and the like, information can be obtained in severe and dangerous environments, and therefore the unmanned aerial vehicle aerial photography technology is widely applied to various fields. However, considering that a scene covered by a single aerial image cannot meet the requirement of a researcher on information acquisition, the scene needs to be post-processed, and two or more images with overlapped areas are spliced to obtain a high-resolution panoramic image with a wider range and more information.
At present, the splicing of aerial images is mainly based on a homography transformation method, an image to be spliced and a reference image are transformed to the same plane through a simple global transformation, the algorithm speed is high, but the method is only suitable for the situation that the images are at the same visual angle, parallax exists between the aerial images, and the spliced images can generate ghost and obvious distortion, so that the problem of homography transformation is solved by using a grid optimization method. The aerial image based on grid optimization is to divide an image to be spliced into a plurality of grids, calculate a transformation matrix of each grid respectively to guide grid deformation, enable the image to be aligned better and reduce distortion of non-overlapping areas, and the main calculation methods comprise APAP, SPHP, AANAP, NISwGSP and the like. Despite the advantages of each of these algorithms, they are not suitable for the actual aerial image stitching task. For example, the APAP algorithm originally proposed the idea of grid, the image alignment is good, but serious distortion is generated; the splicing speed of the SPHP algorithm and the AANAP algorithm is high, the problem of image distortion is solved, and the bending of a linear set structure of the original scene is caused; the nissgsp algorithm also does not solve the problem of good image warping. An aerial image sequence obtained by using an unmanned aerial vehicle often has a certain parallax, and the problems of distortion and image naturalness improvement are solved and the performance of an aerial image splicing algorithm is improved on the premise of ensuring the alignment of spliced images.
Disclosure of Invention
The invention aims to solve the problems of ghost, distortion, structural deformation and the like of the current mainstream unmanned aerial vehicle aerial image splicing method, two constraint items of global similarity and linear feature protection are added into an APAP algorithm to improve the APAP algorithm, and the unmanned aerial image splicing method of linear feature protection and grid optimization is provided.
In order to achieve the purpose, the invention adopts the technical scheme that:
an unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization comprises the following steps:
s1: inputting two aerial images I with certain parallax1、I2，I1Is a reference picture, I2The aerial images are pre-registered for the images to be stitched and by using an As-project-As-Possible algorithm APAP (As-project-As-Possible) which is As uniform As Possible. The pre-registration specifically comprises the steps of firstly extracting and matching feature points by using an SIFT algorithm, then screening out error feature points by using an RANSAC algorithm, and calculating to obtain a global transformation matrix.
S2: and (4) performing grid division on the aerial image in the step (S1) to obtain a grid vertex set.
S3: and (4) constructing a total energy function of the grid optimization model through the grid vertex set obtained in the step (S2), and adding three constraint items of global similarity, local similarity and linear protection to optimize the grid model on the basis of the local homography constraint item provided by the APAP algorithm.
S4: in a sparse linear system, an iterative optimization method is adopted to solve a total energy function and guide grid deformation.
S5: and mapping the deformed image onto a canvas, and performing pixel fusion with the reference image to finish the aerial image splicing work.
Further, in the step S2, the aerial photography image in the step S1 is subjected to grid division to obtain a grid vertex set. Specifically, reference picture I1And an image I to be stitched2After meshing, a 2 n-dimensional vector is used:
V2＝[x1 y1 x2 y2 … xn yn]T
to represent I2And (3) a grid vertex coordinate set to be optimized, wherein n is the number of the divided vertexes, and x and y are the horizontal and vertical coordinates of the grid vertex. The vectors are used simultaneously:
V2′＝[x′1 y′1 x′2 y′2 … x′n y′n]T
to express the coordinate set of the mesh vertex after deformation, where x 'and y' are the horizontal and vertical coordinates of the mesh vertex to be optimized. V2The solution process for' will be given in the following steps.
Further, the total energy function of the mesh optimization model is constructed through the mesh vertex set obtained in S2 in S3, and the three constraint terms, namely global similarity, local similarity and linear protection, are added to optimize the mesh model on the basis of the local homography constraint term provided by the original APAP algorithm, so as to protect the problems of distortion of non-overlapping regions of images and bending of linear geometric structures. The method comprises the following specific steps:
s3.1: a grid energy function is defined. For any sampling point p (x, y) in the reference image, it can be represented by bilinear interpolation of the coordinates of the grid vertex where it is located, that is:
v(p)＝w1v1+w2v2+w3v3+w4v4
in the formula, w is the distance weight from a sampling point to a grid vertex v; thus, the constraint applied to the sampling point is equivalent to the constraint applied to the vertex of the mesh where the sampling point is located. The total energy function of the grid optimization may be defined as:
E′(V′2)＝λlhElh(V′2)+λlsEls(V′2)+λgsEgs(V′2)+λlEl(V′2)
in the formula, Elh(V′2) As local homography constraints, Els(V′2) For local similarity constraint, Egs(V′2) For global similarity constraint, El(V′2) For the linear protection constraint, lambda is the adjusting weight of each constraint term, and the larger lambda is, the stronger the corresponding constraint is. The specific form of the four constraint terms is as in steps 3.2-3.5.
S3.2: local similarity constraints are defined. The local similarity constraint term helps to maintain the continuity of the transformation between grids, and the specific expression is as follows:
Figure BDA0002784278360000031
wherein E' (j, k) is the image I to be spliced2Set of edges of vjAnd vkAre the two endpoints of the mesh edge, v'jAnd vk' is a mesh edge vertex mapped onto canvas, SjkRepresenting the similarity transformation experienced by the edges of the mesh.
S3.3: a global similarity constraint is defined. The global similarity item can reduce the distortion problem of image registration, the full alignment is ensured mainly by local homography and local similarity in an overlapping area, the retention is enhanced mainly by the global similarity in a non-overlapping area by taking the thought of the NISwGSP algorithm as reference, and the image distortion of a large parallax scene caused in the alignment process of the Moving DLT method is reduced. The specific expression is as follows:
Figure BDA0002784278360000032
wjk＝djk+β
Figure BDA0002784278360000033
in the formula (d)jkIs the distance between the grid edge and the overlap region, beta is the adjustment quantity, sI′And thetaI′Globally similar scale factor of I' and 2D rotation angle, v, respectivelyjAnd vkAre the two endpoints of the mesh edge, v'jAnd vk' is the mesh edge vertex mapped onto the canvas.
S3.4: a local homography constraint term is defined. The local homography item corresponds to solve the alignment problem through the enhancement point, and the specific expression is as follows:
Figure BDA0002784278360000034
wherein M (p, q) is I and I obtained by APAP algorithm2A characteristic point set, p and q are I and I respectively2Characteristic point of (3), WIAnd WI2The weight matrix of the grid where p and q are respectively:
Figure BDA0002784278360000035
where w' is the weight of the aforementioned grid vertices.
S3.5: a straight-line protection constraint term is defined. Although the image distortion is weakened to a certain extent by the similarity transformation, for the case that the image has parallax, after the similarity constraint is added, the line direction of the scene linear structure is obviously bent, so that the splicing result is unnatural. The reason is that during image mapping, scenes in each grid are independently transformed along with grid transformation, and if a scene spans two or more grids, it cannot be guaranteed that a straight line geometric structure in an original scene is still on a straight line after transformation. The linear protection constraint term can enable the homography transformation to be closer to the motion of a camera, and protect the linear structure in a scene, and the specific expression is as follows:
Figure BDA0002784278360000041
wherein L is the collected line segment set, m0And m1For sampling the coordinates of the two end points of the line segment, SiIs the current point and m1The area of the quadrangle formed.
Further, the S4 is used for solving the total energy function obtained in the S3 in the sparse system to guide grid deformation. Specifically, the constraint term is substituted into the sparse matrix to obtain:
Figure BDA0002784278360000042
in the formula, JljAs a constraint term Elh(V′2)、JlsAs a constraint term Els(V′2)、JgsAs a constraint term Egs(V′2)、JlAs a constraint term El(V′2) Set of grid vertices V2' corresponding Jacob matrix, agsResidual vector, a, as global similarity termlThe residual vector of the straight-line constraint term. Solving the equation set can obtain the optimal vertex set V2′。
Further, in S5, image mapping and pixel fusion are performed on the aerial image. Specifically, firstly, a reference image I is mapped to a canvas, and then an end point of the reference image I is used as a coordinate origin to obtain an image I to be spliced2And (4) the grid vertex coordinates are also mapped to the canvas to complete image mapping, and finally, the panoramic aerial image is obtained through simple linear weighting fusion.
The invention has the beneficial effects that:
the invention provides an unmanned aerial vehicle aerial image splicing algorithm with linear structure protection and grid optimization, aiming at the problems of ghosting, distortion and unnaturalness in the process of splicing aerial images with parallax by the current mainstream algorithm. Firstly, using an APAP algorithm to perform pre-registration, and calculating local homography constraint through grid division; then, constructing a grid optimization model, defining and solving a total energy function containing a global similarity term, a local homography term and a linear protection term to obtain a deformed grid vertex set; and finally, mapping the image into a canvas according to the obtained grid vertex, and completing aerial image splicing work by a weighted fusion method. Experiments prove that the method has good splicing effect and certain parallax tolerance, not only solves the problems of ghost and distortion of aerial image splicing, but also can protect the linear structure in the original scene, and can meet the actual requirements of aerial image splicing.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
FIG. 2 is a schematic diagram of a reference image and an image to be stitched, in which (a), (b), and (c) are three sets of aerial images with overlapping regions, respectively;
fig. 3 is an image obtained by extracting features of two images by using a SIFT algorithm in the embodiment of the present invention, where (a) feature points are extracted for reference images, and (b) feature points are extracted for images to be stitched;
FIG. 4 is an image obtained after adding a linear feature constraint in an APAP algorithm modified in an embodiment of the present invention, where (a) a linear feature is extracted for a reference image, and (b) a linear feature is extracted for an image to be stitched;
fig. 5 is a panoramic image comparison obtained by splicing the algorithm provided by the invention and the APAP algorithm. Wherein (a), (b) and (c) are the stitching results of the three groups of aerial images in fig. 2.
Detailed Description
For a better understanding of the present invention, those skilled in the art will now make a more detailed description of the present invention with reference to the accompanying drawings and the following examples.
Referring to fig. 1, the embodiment provides an unmanned aerial vehicle aerial image stitching method for linear feature protection and grid optimization, which includes the following steps:
s1: inputting two aerial images I with certain parallax1、I2As shown in fig. 2. The As-Projective-As-Possible (APAP) algorithm is used for pre-registration. The pre-registration specifically includes firstly extracting and matching feature points by using an SIFT algorithm, then screening out error feature points by using an RANSAC algorithm, and calculating to obtain a global transformation matrix, as shown in FIG. 3.
S2: and (4) performing grid division on the aerial image in the step (S1) to obtain a grid vertex set.
S3: and (4) constructing a total energy function of the grid optimization model through the grid vertex set obtained in the step (S2), and adding three constraint items of global similarity, local similarity and linear protection to optimize the grid model on the basis of the local homography constraint item provided by the APAP algorithm, as shown in FIG. 4.
S4: in a sparse linear system, an iterative optimization method is adopted to solve a total energy function and guide grid deformation.
S5: and mapping the deformed image onto a canvas, and performing pixel fusion with the reference image to finish the aerial image splicing work, as shown in fig. 5.
Specifically, in S1, the aerial image is pre-registered using an As-project-As-missible (apap) algorithm. The APAP algorithm pre-registration specifically comprises the steps of firstly extracting and matching feature points by using an SIFT algorithm, then screening error feature points by using an RANSAC algorithm, and calculating to obtain a global transformation matrix.
Further, the S2 meshes the aerial image in S1, and in this embodiment, the mesh size is set to 80 × 80, and a local homography constraint term is calculated. In particular, the method comprises the following steps of,
specifically, reference picture I1And an image I to be stitched2After meshing, a 2 n-dimensional vector is used:
V2＝[x1 y1 x2 y2 … xn yn]T
to represent I2And (3) a grid vertex coordinate set to be optimized, wherein n is the number of the divided vertexes, and x and y are the horizontal and vertical coordinates of the grid vertex. The vectors are used simultaneously:
V2′＝[x′1 y′1 x′2 y′2 … x′n y′n]T
to express the coordinate set of the mesh vertex after deformation, where x 'and y' are the horizontal and vertical coordinates of the mesh vertex to be optimized. V2The solving process of' will be given in the following steps.
Further, the grid vertex set obtained in S2 in S3 is used to construct a total energy function of the grid optimization model, and global and local similarities and linear protection constraint terms are added on the basis of the local homography constraint term of the original APAP algorithm, so as to protect the problems of distortion of non-overlapping regions of images and bending of linear geometric structures. Comprises the following specific steps
S3.1: a grid energy function is defined. For any sampling point p (x, y) in the reference image, it can be represented by bilinear interpolation of the coordinates of the grid vertex where it is located, that is:
v(p)＝w1v1+w2v2+w3v3+w4v4
in the formula, w is the distance weight from a sampling point to a grid vertex v; thus, the constraint applied to the sampling point is equivalent to the constraint applied to the vertex of the mesh where the sampling point is located. The total energy function of the grid optimization may be defined as:
E′(V′2)＝λlhElh(V′2)+λlsEls(V′2)+λgsEgs(V′2)+λlEl(V′2)
in the formula, Elh(V′2) As local homography constraints, Els(V′2) For local similarity constraint, Egs(V′2) For global similarity constraint, El(V′2) For the linear protection constraint, lambda is the adjusting weight of each constraint term, and the larger lambda is, the stronger the corresponding constraint is. The specific form of the four constraint terms is as in steps 3.2-3.5. In the present embodiment, the parameter is set to λlh＝5、λls＝5、λgs＝100、λl＝5。
S3.2: local similarity constraints are defined. The local similarity constraint term helps to maintain the continuity of the transformation between grids, and the specific expression is as follows:
Figure BDA0002784278360000061
wherein E' (j, k) is the image I to be spliced2Set of edges of vjAnd vkAre the two endpoints of the mesh edge, v'jAnd vk' is a mesh edge vertex mapped onto canvas, SjkRepresenting the similarity transformation experienced by the edges of the mesh.
S3.3: a global similarity constraint is defined. The global similarity item can reduce the distortion problem of image registration, the full alignment is ensured mainly by local homography and local similarity in an overlapping area, the retention is enhanced mainly by the global similarity in a non-overlapping area by taking the thought of the NISwGSP algorithm as reference, and the image distortion of a large parallax scene caused in the alignment process of the Moving DLT method is reduced. The specific expression is as follows:
Figure BDA0002784278360000071
wjk＝djk+β
Figure BDA0002784278360000072
in the formula (d)jkIs the distance between the grid edge and the overlap region, beta is the adjustment quantity, sI′And thetaI′Globally similar scale factor of I' and 2D rotation angle, v, respectivelyjAnd vkAre the two endpoints of the mesh edge, v'jAnd vk' is the mesh edge vertex mapped onto the canvas.
S3.4: a local homography constraint term is defined. The local homography item corresponds to solve the alignment problem through the enhancement point, and the specific expression is as follows:
Figure BDA0002784278360000073
wherein M (p, q) is I and I obtained by APAP algorithm2A characteristic point set, p and q are I and I respectively2Characteristic point of (3), WIAnd WI2The weight matrix of the grid where p and q are respectively:
Figure BDA0002784278360000074
where w' is the weight of the aforementioned grid vertices.
S3.5: a straight-line protection constraint term is defined. Although the image distortion is weakened to a certain extent by the similarity transformation, for the case that the image has parallax, after the similarity constraint is added, the line direction of the scene linear structure is obviously bent, so that the splicing result is unnatural. The reason is that during image mapping, scenes in each grid are independently transformed along with grid transformation, and if a scene spans two or more grids, it cannot be guaranteed that a straight line geometric structure in an original scene is still on a straight line after transformation. The linear protection constraint term can enable the homography transformation to be closer to the motion of a camera, and protect the linear structure in a scene, the linear matching length threshold value in the embodiment is 100, the extracted linear features are shown in fig. 4, and the specific expression is as follows:
Figure BDA0002784278360000075
wherein L is the collected line segment set, m0And m1For sampling the coordinates of the two end points of the line segment, SiIs the current point and m1The area of the quadrangle formed.
Further, the S4 is used for solving the total energy function obtained in the S3 in the sparse system to guide grid deformation. Specifically, the constraint term is substituted into the sparse matrix to obtain:
Figure BDA0002784278360000081
in the formula, JljAs a constraint term Elh(V′2)、JlsAs a constraint term Els(V′2)、JgsAs a constraint term Egs(V′2)、JlAs a constraint term El(V′2) Set V 'at grid vertices'2Corresponding Jacob matrix, agsResidual vector, a, as global similarity termlThe residual vector of the straight-line constraint term. Solving the equation set can obtain an optimal vertex set V'2。
Further, in S5, image mapping and pixel fusion are performed on the aerial image. Specifically, firstly, a reference image I is mapped to a canvas, and then an end point of the reference image I is used as a coordinate origin to obtain an image I to be spliced2The grid vertex coordinates minus the offset are also mapped to the canvas to complete image mapping, and finally the panoramic aerial image is obtained through simple linear weighting fusion, as shown in fig. 5.
The pictures used in the simulation experiment are all actual images shot by the unmanned aerial vehicle, the resolution ratio is 4000 x 3000, and as shown in fig. 2, an overlapped scene and a certain parallax exist between two images in each group. Fig. 5 is a comparison of the panoramic image obtained by using the algorithm of the present invention and the original APAP algorithm, which shows that the present invention has better completed the splicing task, not only ensures the alignment effect of the original APAP algorithm, but also improves the problems of distortion and linear geometry bending of the overlapping area, and the obtained panoramic image is more natural.
The comparison table of the time complexity of the invention and the aerial image splicing based on the APAP algorithm is shown in Table 1, and it can be seen that the invention can greatly improve the image splicing effect and enhance the comprehensive performance of the algorithm on the premise that the time complexity is basically equal to the original algorithm.
TABLE 1 Algorithm time complexity contrast(s)
Figure RE-GDA0002899253470000082
The above description is only a preferred embodiment of the present invention, and not intended to limit the present invention, the scope of the present invention is defined by the appended claims, and all structural changes that can be made by using the contents of the description and the drawings of the present invention are intended to be embraced therein.

Claims (4)

1. An unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization is characterized by comprising the following steps:
s1: inputting two aerial images I with certain parallax1、I2，I1Is a reference picture, I2Pre-registering the images to be spliced by adopting an APAP algorithm as consistent as possible;
s2: performing mesh division on the aerial image in the step S1 to obtain a mesh vertex set;
s3: and (4) constructing a total energy function of the grid optimization model through the grid vertex set obtained in the step (S2), and adding three constraint items of global similarity, local similarity and linear protection to optimize the grid model on the basis of the local homography constraint item provided by the APAP algorithm, wherein the specific steps are as follows:
s3.1: defining a grid energy function; for any sampling point p (x, y) in the reference image, the sampling point is represented by bilinear interpolation of the coordinates of the vertex of the grid where the sampling point is located, that is:
v(p)＝w1v1+w2v2+w3v3+w4v4
in the formula, w is the distance weight from a sampling point to a grid vertex v;
the constraint applied to the sampling points is equivalent to the constraint applied to the grid vertexes where the sampling points are located; the total energy function of the grid optimization is defined as:
E′(V′2)＝λlhElh(V′2)+λlsEls(V′2)+λgsEgs(V′2)+λlEl(V′2)
in the formula, Elh(V′2) As local homography constraints, Els(V′2) For local similarity constraint, Egs(V′2) For global similarity constraints, El(V′2) For linear protection constraint, lambda is the adjusting weight of each constraint term, and the larger lambda is, the stronger the corresponding constraint is;
s3.2: defining a local similarity constraint; the local similarity constraint term helps to maintain the continuity of the transformation between grids, and the expression is as follows:
Figure FDA0002784278350000011
wherein E' (j, k) is the image I to be spliced2Set of edges of vjAnd vkAre the two endpoints of the mesh edge, v'jAnd vk' is a mesh edge vertex mapped onto canvas, SjkRepresenting the similarity transformation undergone by the mesh edges;
s3.3: defining a global similarity constraint; in the overlapping area, the local homography and the local similarity are mainly used for ensuring full alignment, in the non-overlapping area, the concept of the NISwGSP algorithm is used for reference, the global similarity is mainly used for enhancing the retentivity, and the image distortion of a large parallax scene caused in the alignment process of the Moving DLT method is reduced; the specific expression is as follows:
Figure FDA0002784278350000021
wjk＝djk+β
Figure FDA0002784278350000022
in the formula (d)jkIs the distance between the grid edge and the overlap region, beta is the adjustment quantity, sI′And thetaI′Globally similar scale factor and 2D rotation angle, v, respectivelyjAnd vkAre the two endpoints of the mesh edge, v'jAnd vk' is a mesh edge vertex mapped onto the canvas;
s3.4: defining a local homography constraint term; the local homography item corresponds to solve the alignment problem through the enhancement point, and the specific expression is as follows:
Figure FDA0002784278350000023
wherein M (p, q) is I and I obtained by APAP algorithm2A characteristic point set, p and q are I and I respectively2Characteristic point of (3), WIAnd
Figure FDA0002784278350000024
the weight matrix of the grid where p and q are respectively:
Figure FDA0002784278350000025
where w' is the weight of the aforementioned grid vertices;
s3.5: defining a linear protection constraint term; the linear protection constraint term can enable the homography transformation to be closer to the motion of a camera, and protect the linear structure in a scene, and the specific expression is as follows:
Figure FDA0002784278350000026
wherein L is the collected line segment set, m0And m1For sampling the coordinates of the two end points of the line segment, SiIs the current point and m1The area of the formed quadrangle;
s4: the grid deformation is guided by solving the total energy function obtained in S3 in a sparse system;
s5: carrying out image mapping and pixel fusion on the aerial image; specifically, firstly, a reference image I is mapped to a canvas, and then an end point of the reference image I is used as a coordinate origin to obtain an image I to be spliced2And (4) subtracting the offset from the grid vertex coordinates, mapping the grid vertex coordinates onto canvas to complete image mapping, and finally obtaining the panoramic aerial image through simple linear weighting fusion.
2. The unmanned aerial vehicle aerial image stitching method based on linear feature protection and grid optimization according to claim 1, wherein the pre-registration manner in step S1 is as follows: firstly, extracting and matching feature points by using an SIFT algorithm, then screening out error feature points by using an RANSAC algorithm, and calculating to obtain a global transformation matrix.
3. The unmanned aerial vehicle aerial image stitching method for linear feature protection and grid optimization according to claim 1, wherein the aerial image in S1 is subjected to grid division in S2 to obtain a grid vertex set; specifically, reference picture I1And an image I to be stitched2After grid division, a 2 n-dimensional vector is obtained:
V2＝[x1 y1 x2 y2…xn yn]T
representing I by 2 n-dimensional vectors2A grid vertex coordinate set to be optimized, wherein n is the number of divided vertexes, and x and y are horizontal and vertical coordinates of the grid vertexes;
the method adopts a mesh vertex coordinate set after deformation represented by using vectors, wherein x 'and y' are horizontal and vertical coordinates of a mesh vertex to be optimized, and the vectors are used as follows:
V2′＝[x′1 y′1 x′2 y′2…x′n y′n]T。
4. the unmanned aerial vehicle aerial image stitching method for linear feature protection and grid optimization according to claim 1, wherein the S4 is used for solving the total energy function obtained in S3 in a sparse system to guide grid deformation; specifically, the constraint term is substituted into the sparse matrix to obtain:
Figure FDA0002784278350000031
in the formula, JljAs a constraint term Elh(V′2)、JlsAs a constraint term Els(V′2)、JgsAs a constraint term Egs(V′2)、JlAs a constraint term El(V′2) Set V 'at grid vertices'2Corresponding Jacob matrix, agsResidual vector, a, as global similarity termlA residual vector that is a straight line constraint term; solving the equation set can obtain an optimal vertex set V'2。
CN202011292721.6A 2020-11-18 2020-11-18 Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization Active CN112435163B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011292721.6A CN112435163B (en) 2020-11-18 2020-11-18 Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011292721.6A CN112435163B (en) 2020-11-18 2020-11-18 Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization

Publications (2)

Publication Number Publication Date
CN112435163A true CN112435163A (en) 2021-03-02
CN112435163B CN112435163B (en) 2022-10-18

Family

ID=74693183

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011292721.6A Active CN112435163B (en) 2020-11-18 2020-11-18 Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization

Country Status (1)

Country Link
CN (1) CN112435163B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112967217A (en) * 2021-03-11 2021-06-15 大连理工大学 Image splicing method based on linear feature matching and constraint
CN113313659A (en) * 2021-04-25 2021-08-27 中国人民解放军火箭军工程大学 High-precision image splicing method under multi-machine cooperative constraint

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109961398A (en) * 2019-02-18 2019-07-02 鲁能新能源(集团)有限公司 Fan blade image segmentation and grid optimization joining method
CN110211043A (en) * 2019-05-11 2019-09-06 复旦大学 A kind of method for registering based on grid optimization for Panorama Mosaic
CN110781903A (en) * 2019-10-12 2020-02-11 中国地质大学（武汉） Unmanned aerial vehicle image splicing method based on grid optimization and global similarity constraint

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109961398A (en) * 2019-02-18 2019-07-02 鲁能新能源(集团)有限公司 Fan blade image segmentation and grid optimization joining method
CN110211043A (en) * 2019-05-11 2019-09-06 复旦大学 A kind of method for registering based on grid optimization for Panorama Mosaic
CN110781903A (en) * 2019-10-12 2020-02-11 中国地质大学（武汉） Unmanned aerial vehicle image splicing method based on grid optimization and global similarity constraint

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
何川等: "具有直线结构保护的网格化图像拼接", 《中国图象图形学报》 *
石祥滨等: "基于控制点网格优化的航拍图像拼接算法", 《计算机应用与软件》 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112967217A (en) * 2021-03-11 2021-06-15 大连理工大学 Image splicing method based on linear feature matching and constraint
CN112967217B (en) * 2021-03-11 2024-02-06 大连理工大学 Image stitching method based on straight line feature matching and constraint
CN113313659A (en) * 2021-04-25 2021-08-27 中国人民解放军火箭军工程大学 High-precision image splicing method under multi-machine cooperative constraint
CN113313659B (en) * 2021-04-25 2024-01-26 中国人民解放军火箭军工程大学 High-precision image stitching method under multi-machine cooperative constraint

Also Published As

Publication number Publication date
CN112435163B (en) 2022-10-18

Similar Documents

Publication Publication Date Title
CN112435163B (en) Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization
CN107274337B (en) Image splicing method based on improved optical flow
CN109934772B (en) Image fusion method and device and portable terminal
CN110246161B (en) Method for seamless splicing of 360-degree panoramic images
CN107767339B (en) Binocular stereo image splicing method
CN111553841B (en) Real-time video splicing method based on optimal suture line updating
CN111242848B (en) Binocular camera image suture line splicing method and system based on regional feature registration
CN112862683B (en) Adjacent image splicing method based on elastic registration and grid optimization
Li et al. A unified framework for street-view panorama stitching
CN108280805B (en) Image splicing method based on manifold optimization
CN110717936A (en) Image stitching method based on camera attitude estimation
CN110580715B (en) Image alignment method based on illumination constraint and grid deformation
CN116958437A (en) Multi-view reconstruction method and system integrating attention mechanism
Wan et al. Drone image stitching using local mesh-based bundle adjustment and shape-preserving transform
Lu et al. Multiperspective image stitching and regularization via hybrid structure warping
CN112365589B (en) Virtual three-dimensional scene display method, device and system
Rui et al. Research on fast natural aerial image mosaic
Meng et al. High-quality panorama stitching based on asymmetric bidirectional optical flow
CN109087250B (en) Image splicing method based on regular boundary constraint
CN116883565A (en) Digital twin scene implicit and explicit model fusion rendering method and application
Shibayama et al. Reconstruction of 3D surface and restoration of flat document image from monocular image sequence
CN114118367B (en) Method and equipment for constructing incremental nerve radiation field
Sun et al. Image stitching with weighted elastic registration
Zhou et al. Rectangular-Output Image Stitching
Kawasaki et al. Super‐resolution omnidirectional camera images using spatio‐temporal analysis

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant