CN109978760B

CN109978760B - Image stitching method and device

Info

Publication number: CN109978760B
Application number: CN201711445096.2A
Authority: CN
Inventors: 姚佳宝; 王莉
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2017-12-27
Filing date: 2017-12-27
Publication date: 2023-05-02
Anticipated expiration: 2037-12-27
Also published as: CN109978760A

Abstract

The embodiment of the application provides an image stitching method and device, wherein the method comprises the following steps: the method comprises the steps of obtaining a basic spliced image and an image to be spliced, extracting characteristic points of the basic spliced image and the image to be spliced, determining a plurality of matching characteristic point pairs, calculating a homography matrix of each image to be spliced according to the determined plurality of matching characteristic point pairs, determining an overlapping area of each image to be spliced according to the homography matrix of each image to be spliced, building a first grid for the overlapping area of each image to be spliced according to a reprojection error of the matching characteristic point pairs, adjusting the homography matrix of each image to be spliced according to the weight of the first grid for each first grid of each image to be spliced, obtaining the homography matrix of the first grid, and splicing the basic spliced image and at least one image to be spliced according to the homography matrix of each image to be spliced and the homography matrix of each first grid. By applying the embodiment of the application, the problems of dislocation and double image of image stitching are solved.

Description

Image stitching method and device

Technical Field

The present disclosure relates to the field of image processing technologies, and in particular, to an image stitching method and apparatus.

Background

In order to solve the problem of narrow field angle of single-lens cameras, multi-lens panoramic cameras have been developed. The multi-lens panoramic camera can collect multiple paths of images in different directions at the same time and splice the images to obtain a panoramic image with a large visual angle.

Currently, the image stitching method includes: acquiring at least one image to be stitched and a basic stitched image, wherein the image to be stitched is based on the basic stitched image, for example, the image to be stitched is converted into a coordinate system of the basic stitched image; determining matching characteristic point pairs of every two images in the image to be spliced and the basic spliced image; determining a homography matrix of each image to be spliced aiming at the basic spliced image according to the coordinates of the matched characteristic point pairs; and splicing the images to be spliced and the basic spliced image according to the homography matrix.

In practical application, an image has depth information, different objects in the image are located in different planes, the depth information is different, for example, a person in the image stands in front of a building, the depth information of the person in the image and the depth information of the building are different, and thus, the characteristic distribution in the image is uneven. Under the condition of uneven feature distribution in the image, if the image is spliced by adopting a single homography matrix, the problems of dislocation and double image of the spliced image are likely to be caused.

Disclosure of Invention

An embodiment of the application aims to provide an image stitching method and device for solving the problems of dislocation and double image of image stitching. The specific technical scheme is as follows:

in one aspect, an embodiment of the present application provides an image stitching method, where the method includes:

acquiring a basic spliced image and at least one image to be spliced;

extracting characteristic points of the basic spliced image and the at least one image to be spliced;

matching the feature points of each two images in the at least one image to be spliced and the basic spliced image, and determining a plurality of matched feature point pairs;

calculating a homography matrix of each image to be spliced aiming at the basic spliced image according to the coordinates of the characteristic points in the determined plurality of matched characteristic point pairs;

determining an overlapping area of each image to be spliced according to the homography matrix of each image to be spliced;

according to the re-projection error of the matched feature point pairs, a first grid is established for the overlapped area of each image to be spliced;

for each first grid of each image to be spliced, adjusting a homography matrix of the image to be spliced according to the weight of the first grid to obtain the homography matrix of the first grid;

And splicing the basic spliced image and the at least one image to be spliced according to the homography matrix of each image to be spliced and the homography matrix of each first grid.

In one embodiment of the present application, the step of extracting the feature points of the base stitched image and the at least one image to be stitched includes:

and extracting the feature points in the at least one image to be spliced and the basic spliced image according to a Scale-invariant feature transform (Scale-invariant feature transform) algorithm.

In one embodiment of the present application, the step of matching feature points of each two images in the at least one image to be stitched and the base stitched image, and determining a plurality of matching feature point pairs includes:

for each characteristic point of each image in the at least one image to be spliced and the basic spliced image, calculating Euclidean distance between a description operator of the characteristic point of other images and the description operator of the characteristic point; the other images are the at least one image to be spliced and the images except the image in the basic spliced image; and forming a matching characteristic point pair by the characteristic point and the characteristic point corresponding to the description operator with the minimum Euclidean distance.

In one embodiment of the present application, before matching the feature points of each two images in the at least one image to be stitched and the base stitched image, the method further includes:

for each extracted feature point, establishing a 360-degree gradient histogram by utilizing gradient information of a neighborhood pixel of the feature point to obtain a description operator of the feature point in space; the description operator of the feature point is normalized.

In one embodiment of the present application, the step of determining the overlapping area of each image to be stitched according to the homography matrix of each image to be stitched includes:

for each image to be stitched, determining an overlapping region of the image to be stitched by adopting the following steps:

determining the abscissa and the ordinate of four vertexes of the image to be spliced on the plane of the basic spliced image;

determining the abscissa and the ordinate of four vertexes of the matched image on the plane of the basic spliced image; the matching image is an image in which feature points forming a matching feature point pair with the feature points of the image to be spliced are located;

determining a first abscissa, a second abscissa, a first ordinate and a second ordinate; the first abscissa is the maximum value of the minimum abscissa of the four vertexes of the image to be spliced and the minimum abscissa of the four vertexes of the matched image; the second abscissa is the minimum value of the maximum abscissa of the four vertexes of the image to be spliced and the maximum abscissa of the four vertexes of the matched image; the first ordinate is the maximum value of the minimum ordinate of the four vertexes of the image to be spliced and the minimum ordinate of the four vertexes of the matched image; the second ordinate is the minimum value of the maximum ordinate of the four vertexes of the image to be spliced and the maximum ordinate of the four vertexes of the matched image;

And determining the overlapping area of the image to be spliced and the matching image by the first abscissa, the second abscissa, the first ordinate and the second ordinate.

In an embodiment of the present application, the step of establishing a first grid for the overlapping area of each image to be stitched according to the reprojection error of the matched pair of feature points includes:

obtaining a maximum reprojection error and a minimum reprojection error from reprojection errors of all the matched characteristic point pairs corresponding to each image to be spliced;

and according to the maximum re-projection error and the minimum re-projection error, establishing a first grid for the overlapped area of each image to be spliced.

In an embodiment of the present application, the step of establishing a first grid for the overlapping area of each image to be stitched according to the maximum re-projection error and the minimum re-projection error includes:

for each image to be spliced, a first grid is established for the overlapped area of the image to be spliced according to the following formula:

wherein Deltah is _i As the re-projection error of the matching characteristic point pair where the characteristic point in the ith first grid of the image to be spliced is, deltah _min For the minimum re-projection error corresponding to the acquired image to be spliced, deltah _max N is a positive integer for obtaining the maximum re-projection error corresponding to the image to be spliced; the value range of i is 0-N-1.

In an embodiment of the present application, the step of adjusting, for each first grid of each image to be stitched, a homography matrix of the image to be stitched according to a weight of the first grid, to obtain the homography matrix of the first grid includes:

for each first grid of each image to be spliced, determining the weight of a matched feature point pair where the feature point in the first grid is located according to the weight of the first grid;

for each first grid of each image to be spliced, determining the weight of a matched feature point pair where the feature point outside the first grid is located according to the weight of the first grid;

and for each first grid of each image to be spliced, adjusting a homography matrix of the image to be spliced according to the weight of the matched feature point pair of the feature point in the first grid and the weight of the matched feature point pair of the feature point outside the first grid, and taking the adjusted homography matrix as the homography matrix of the first grid.

In one embodiment of the present application, the step of determining the weight of the matching feature point pair where the feature point in the first grid is located according to the weight of the first grid includes:

For each feature point in the first grid, determining the weight of the matching feature point pair where the feature point is located by the following formula:

wherein w is _＊ And for the weight of the matched feature point pair where the feature point is located, gamma is the weight of the first grid, the value range of gamma is 0-1, and M is the number of the feature points included in the matched feature point pair in the first grid.

In one embodiment of the present application, the step of determining the weight of the matching feature point pair where the feature point outside the first grid is located according to the weight of the first grid includes:

for each feature point outside the first grid, determining the weight of the matched feature point pair where the feature point is located by the following formula:

wherein w' _＊ For the weight value t of the matching feature point pair where the feature point is located _＊ T is the coordinates of the feature point _i Coordinates of feature points included in the matching feature point pair are the ith within the first grid.

In one embodiment of the present application, the step of stitching the base stitched image and the at least one image to be stitched according to the homography matrix of each image to be stitched and the homography matrix of each first grid includes:

establishing a second grid for the non-overlapping area of each image to be spliced according to preset parameters; the non-overlapping area is an area except for the overlapping area in each image to be spliced;

For each second grid of each image to be spliced, adjusting a homography matrix of the image to be spliced according to weight ratio of other second grids adjacent to the second grid to obtain the homography matrix of the second grid;

and splicing the basic spliced image and the at least one image to be spliced according to the homography matrix of each first grid and the homography matrix of each second grid.

In an embodiment of the present application, for each second grid of each image to be stitched, the step of adjusting a homography matrix of the image to be stitched according to a weight ratio of other second grids adjacent to the second grid to obtain the homography matrix of the second grid includes:

for each second grid of each image to be spliced, determining the weight of other adjacent second grids of the second grid according to the weight proportion of other adjacent second grids of the second grid;

and for each second grid of each image to be spliced, adjusting the homography matrix of the image to be spliced according to the weight of other second grids adjacent to the second grid, and taking the homography matrix after adjustment as the homography matrix of the second grid.

In one embodiment of the present application, the step of determining the weight of the other second grid adjacent to the second grid according to the weight ratio of the other second grid adjacent to the second grid includes:

The weights of other second grids adjacent to the second grid are determined according to the following formula:

wherein w is _j Weight, Q, of the j-th other second grid adjacent to the second grid _j For the weight ratio of the j other second grids adjacent to the second grid, Q _i And (3) the weight ratio of the ith other second grids adjacent to the second grid is given, and S is the total number of other second grids adjacent to the second grid.

In one embodiment of the present application, the size of each second network is equal.

In two aspects, an embodiment of the present application provides an image stitching apparatus, including:

the acquisition unit is used for acquiring a basic spliced image and at least one image to be spliced;

the extraction unit is used for extracting the characteristic points of the basic spliced image and the at least one image to be spliced;

the matching unit is used for matching the characteristic points of each two images in the at least one image to be spliced and the basic spliced image, and determining a plurality of matching characteristic point pairs;

the computing unit is used for computing a homography matrix of each image to be spliced aiming at the basic spliced image according to the determined coordinates of the characteristic points in the plurality of matched characteristic point pairs;

The determining unit is used for determining the overlapping area of each image to be spliced according to the homography matrix of each image to be spliced;

the establishing unit is used for establishing a first grid for the overlapped area of each image to be spliced according to the re-projection error of the matched characteristic point pairs;

the adjusting unit is used for adjusting the homography matrix of each first grid of each image to be spliced according to the weight of the first grid to obtain the homography matrix of the first grid;

and the splicing unit is used for splicing the basic spliced image and the at least one image to be spliced according to the homography matrix of each image to be spliced and the homography matrix of each first grid.

In one embodiment of the present application, the extraction unit is specifically configured to:

and extracting characteristic points in the at least one image to be spliced and the basic spliced image according to a SIFT algorithm.

In one embodiment of the present application, the matching unit is specifically configured to:

In one embodiment of the present application, the apparatus further comprises:

the obtaining unit is used for establishing a 360-degree gradient histogram for each extracted characteristic point by utilizing gradient information of the neighborhood pixels of the characteristic point to obtain a description operator of the characteristic point in space;

and the normalization unit is used for normalizing the description operator of each extracted characteristic point.

In an embodiment of the present application, the determining unit is specifically configured to determine, for each image to be stitched, an overlapping area of the image to be stitched, including:

the first determining subunit is used for determining the abscissa and the ordinate of the four vertexes of the image to be spliced on the plane where the basic spliced image is located;

the second determining subunit is used for determining the abscissa and the ordinate of the four vertexes of the matched image on the plane where the base spliced image is located; the matching image is an image in which feature points forming a matching feature point pair with the feature points of the image to be spliced are located;

a third determination subunit configured to determine a first abscissa, a second abscissa, a first ordinate, and a second ordinate; the first abscissa is the maximum value of the minimum abscissa of the four vertexes of the image to be spliced and the minimum abscissa of the four vertexes of the matched image; the second abscissa is the minimum value of the maximum abscissa of the four vertexes of the image to be spliced and the maximum abscissa of the four vertexes of the matched image; the first ordinate is the maximum value of the minimum ordinate of the four vertexes of the image to be spliced and the minimum ordinate of the four vertexes of the matched image; the second ordinate is the minimum value of the maximum ordinate of the four vertexes of the image to be spliced and the maximum ordinate of the four vertexes of the matched image;

And the fourth determination subunit is used for determining the overlapping area of the image to be spliced and the matching image according to the first abscissa, the second abscissa, the first ordinate and the second ordinate.

In one embodiment of the present application, the establishing unit includes:

the acquisition subunit is used for acquiring a maximum reprojection error and a minimum reprojection error from reprojection errors of all the matched characteristic point pairs corresponding to each image to be spliced;

and the first building subunit is used for building a first grid for the overlapped area of each image to be spliced according to the maximum re-projection error and the minimum re-projection error.

In one embodiment of the present application, the first establishing subunit is specifically configured to:

In one embodiment of the present application, the adjusting unit includes:

a fifth determining subunit, configured to determine, for each first grid of each image to be spliced, a weight of a matching feature point pair where a feature point in the first grid is located according to the weight of the first grid;

a sixth determining subunit, configured to determine, for each first grid of each image to be spliced, a weight of a matching feature point pair where a feature point outside the first grid is located according to the weight of the first grid;

the first adjustment subunit is configured to adjust, for each first grid of each image to be spliced, a homography matrix of the image to be spliced according to a weight of a matching feature point pair where a feature point in the first grid is located and a weight of a matching feature point pair where a feature point outside the first grid is located, and take the adjusted homography matrix as the homography matrix of the first grid.

In one embodiment of the present application, the fifth determining subunit is specifically configured to:

for each feature point in each first grid of each image to be spliced, determining the weight of the matched feature point pair where the feature point is located by the following formula:

In one embodiment of the present application, the sixth determining subunit is specifically configured to:

for each feature point outside each first grid of each image to be spliced, determining the weight of the matched feature point pair where the feature point is located by the following formula:

In one embodiment of the present application, the splicing unit includes:

the second building subunit is used for building a second grid for the non-overlapping area of each image to be spliced according to preset parameters; the non-overlapping area is an area except for the overlapping area in each image to be spliced;

the second adjusting subunit is used for adjusting the homography matrix of each second grid of each image to be spliced according to the weight ratio of other second grids adjacent to the second grid to obtain the homography matrix of the second grid;

and the splicing subunit is used for splicing the basic spliced image and the at least one image to be spliced according to the homography matrix of each first grid and the homography matrix of each second grid.

In one embodiment of the present application, the second adjusting subunit is specifically configured to:

for each second grid of each image to be spliced, determining the weight of other second grids adjacent to the second grid according to the following formula:

In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;

the memory is used for storing a computer program;

the processor is configured to execute the computer program stored in the memory, and implement any one of the image stitching method steps.

In a fourth aspect, embodiments of the present application provide a machine-readable storage medium having stored thereon a computer program which, when executed by a processor, performs any of the above-described image stitching method steps.

In the embodiment of the application, after determining the homography matrix of each image to be spliced aiming at the basic spliced image, according to the reprojection error between the matched characteristic point pairs, a first grid is established for the overlapped area of each image to be spliced, the reprojection error is an important parameter for embodying the characteristics, and according to the reprojection error, the characteristic points with similar characteristics can be established in a first grid, further, for each first grid of each image to be spliced, the homography matrix of the image to be spliced is adjusted according to the weight of the first grid, the homography matrix of the first grid is obtained, the problem that the image with uneven characteristic distribution adopts a single homography matrix is avoided, and according to the homography matrix of each first grid, the basic spliced image and the image to be spliced are spliced, and under the condition that the characteristic distribution is uneven in the image, the problems of misplacement and ghost of image splicing are effectively solved. Of course, it is not necessary for any of the products or methods of the present application to be practiced with all of the advantages described above.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related art, the drawings that are required to be used in the embodiments or the related technical descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to the drawings without inventive effort for a person having ordinary skill in the art.

Fig. 1 is a first flowchart of an image stitching method according to an embodiment of the present application;

fig. 2 is a first schematic diagram of an image to be stitched provided in an embodiment of the present application;

fig. 3 is a second flowchart of an image stitching method according to an embodiment of the present application;

FIG. 4 is a distribution diagram of weight ratios provided in the embodiments of the present application;

fig. 5 is a second schematic diagram of an image to be stitched provided in an embodiment of the present application;

fig. 6 is a schematic diagram of a first structure of an image stitching device according to an embodiment of the present application;

fig. 7 is a schematic diagram of a second structure of an image stitching device according to an embodiment of the present application;

fig. 8 is a schematic diagram of a third structure of an image stitching device according to an embodiment of the present application;

Fig. 9 is a schematic diagram of a fourth structure of an image stitching device according to an embodiment of the present application;

fig. 10 is a schematic view of a fifth structure of an image stitching device according to an embodiment of the present application;

fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.

At present, when a single homography matrix is adopted for image stitching, the problem that the stitched image is misplaced and ghost is likely to occur under the condition that the characteristic distribution in the image is uneven.

In order to solve the problems of dislocation and ghost image of image stitching, the embodiment of the application provides an image stitching method and device. Referring to fig. 1, fig. 1 is a first flowchart of an image stitching method according to an embodiment of the present application, where the method includes:

S101: and acquiring a basic spliced image and at least one image to be spliced.

In one embodiment of the present application, each time an image is acquired by the image acquisition device, the acquired image may be acquired from the image acquisition device, after a plurality of images are acquired, one image may be randomly selected from the plurality of images to be used as a base stitching image, and other images are used as images to be stitched.

In one embodiment of the present application, each time an image is acquired by the image acquisition device, the image may be stored in the database, when the image to be stitched and the basic stitched image are acquired, a plurality of images may be acquired from the database, and one image is randomly selected from the plurality of images to be used as the basic stitched image, and other images are used as the images to be stitched.

In the embodiment of the present application, the image capturing device may be a multi-lens panoramic camera, a single-lens camera, or the like.

S102: and extracting the characteristic points of the basic spliced image and at least one image to be spliced.

In one embodiment of the application, in order to improve the rationality of feature point extraction, feature points in an image to be spliced and a basic spliced image are extracted according to a SIFT algorithm.

For example, a scale space can be constructed by adopting a SIFT algorithm, the feature points of the image to be spliced and the basic spliced image are extracted, the SIFT algorithm has good robustness, and the feature points extracted by the SIFT algorithm have scaling invariance and certain noise resistance.

In one embodiment of the application, for each extracted feature point, a 360-degree gradient histogram can be established by using gradient information of a neighborhood pixel of the feature point, and a high-dimensional space descriptor is generated, namely, a description operator of the feature point in space is obtained, so that the feature point has rotation invariance; and carrying out normalization processing on the obtained high-dimensional space description operator to obtain a description vector of the feature point, which leads the feature point to have contrast invariance. The homography matrix is calculated through the feature points with rotation invariance and contrast invariance, and the accuracy of the homography matrix obtained through calculation is effectively improved.

S103: and matching the characteristic points of each two images in at least one image to be spliced and the basic spliced image, and determining a plurality of matched characteristic point pairs.

In one embodiment of the application, for each feature point of each image in the acquired image to be stitched and the basic stitched image, calculating Euclidean distance between a description operator of the feature point of other images and the description operator of the feature point, wherein the other images are the acquired images except the image in the image to be stitched and the basic stitched image; and forming a matching characteristic point pair by the characteristic point and the characteristic point corresponding to the description operator with the minimum Euclidean distance.

For example, a base stitched image a and an image B, C to be stitched are acquired, wherein the feature points extracted in the image a have p01 and p02, the feature points extracted in the image B have p03 and p04, and the feature point extracted in the image C has p05.

The feature points of the image A are matched with the feature points of the image B: calculating the Euclidean distance d1 between the description operator of p01 and the description operator of p 03; calculating the Euclidean distance d2 between the description operator of p01 and the description operator of p 04; calculating the Euclidean distance d3 between the description operator of p02 and the description operator of p 03; the Euclidean distance d4 between the description operator of p02 and the description operator of p04 is calculated.

The feature points of image a are matched with the feature points of image C: calculating the Euclidean distance d5 between the description operator of the p01 and the description operator of the p05 of the image C; the euclidean distance d6 between the description operator of p02 and the description operator of p05 of image C is calculated.

The feature points of image B are matched with the feature points of image C: calculating the Euclidean distance d7 between the description operator of the p03 and the description operator of the p05 of the image C; the Euclidean distance d8 between the description operator of p04 and the description operator of p05 of image C is calculated.

The determined euclidean distance for the feature point p01 in the image a is: d1, d2, d5. If d1< d2< d5, d1 corresponds to p03, then p01 and p03 constitute a matching feature point pair.

The determined euclidean distance for the feature point p02 in the image a is: d3, d4, d6. If d4< d3< d6, d4 corresponds to p04, then p02 and p04 constitute a matching feature point pair.

The determined euclidean distance for the feature point p03 in the image B is: d1, d3, d5. If d1< d3< d5, d1 corresponds to p01, then p03 and p01 constitute a matching feature point pair.

The euclidean distance determined for the feature point p04 in the image B is: d2, d4, d6. If d4< d2< d6, d2 corresponds to p02, then p04 and p02 constitute a matching feature point pair.

The determined euclidean distance for the feature point p05 in the image C is: d5, d6, d7, d8. If d7< d5< d6< d8, d7 corresponds to p03, then p05 and p03 form a matching feature point pair.

In one embodiment of the present application, for a feature point, after determining a description operator with a minimum euclidean distance between description operators of the feature point, it may be determined whether the minimum euclidean distance is less than a distance threshold; if the feature point is smaller than the minimum Euclidean distance, the feature point corresponding to the description operator of the feature point and the minimum Euclidean distance is formed into a matched feature point pair; otherwise, determining that the feature point cannot form a matched feature point pair with other feature points, and calculating a homography matrix later, wherein the feature point can be not considered when the image is spliced.

In one embodiment of the present application, a k-dtree (k-dimensional tree) may be employed to accelerate the algorithm for calculating the Euclidean distance; the algorithm can also be optimized and screened by adopting a RANSAC (Random Sample Consensus, random sample matching) algorithm and a double-choice matching principle so as to improve the accuracy of determining the matched characteristic point pairs.

S104: and calculating a homography matrix of each image to be spliced aiming at the basic spliced image according to the determined coordinates of the characteristic points in the plurality of matched characteristic point pairs.

In the embodiment of the present application, the homography matrix is:

if the coordinates of two feature points in a matching feature point pair are [ x, y ]]And [ x ', y ]']，

An ith column representing a homography matrix;

can be according to the formula

And->

The positive equation is constructed as follows

According to the matrix A, constructing a solving equation of an optimal homography matrix:

in the formula (6), h _i Representing a homography matrix corresponding to the ith matching feature point pair; SVD (Singular Value Decomposition ) of matrix A can be performed to obtain:

the eigenvector v corresponding to the Cmin eigenvalue λ in equation (7) _min The optimal homography matrix H is obtained.

In the embodiments of the present application, assume h ₃₃ In 1, through 4 matched characteristic point pairs, namely 8 positive definite equations are constructed through 8 characteristic points, the homography matrix of two images where the characteristic points in the matched characteristic point pairs are located can be determined.

In the embodiment of the application, if the characteristic points in one image to be spliced and the characteristic points in the basic spliced image form a matched characteristic point pair, the homography matrix of the image to be spliced aiming at the basic spliced image can be directly determined;

if the characteristic points in one image to be spliced and the characteristic points in the other image to be spliced form a matched characteristic point pair, the characteristic points in the other image to be spliced and the characteristic points in the basic spliced image form a matched characteristic point pair. For example, X, Y for the images to be stitched, Z for the basic stitched image, where the feature points in the image to be stitched X and the feature points in the image to be stitched Y form a matching feature point pair, and the feature points in the image to be stitched Y and the feature points in the basic stitched image Z form a matching feature point pair, then it is necessary to calculate a homography matrix 1 from the image to be stitched X for the image to be stitched Y, and calculate a homography matrix 2 from the image to be stitched Y for the basic stitched image Z; and combining the homography matrix 1 and the homography matrix 2, and calculating a homography matrix 3 of the image X to be spliced aiming at the basic spliced image Z.

S105: and determining the overlapping area of each image to be spliced according to the homography matrix of each image to be spliced.

In one embodiment of the present application, for each image to be stitched, the following steps are used to determine the overlapping area of the image to be stitched:

s1051: and determining the abscissa and the ordinate of the four vertexes of the image to be spliced on the plane of the basic spliced image.

S1052: and determining the abscissa and the ordinate of the four vertexes of the matched image on the plane of the basic spliced image.

Here, the matching image is an image in which feature points that form a matching feature point pair with the feature points of the image to be stitched are located. For example, if the feature points in the image to be stitched B and the feature points in the base stitched image a form a matching feature point pair, the base stitched image a is a matching image of the image to be stitched B, and the image to be stitched B is a matching image of the base stitched image a.

S1053: a first abscissa, a second abscissa, a first ordinate, and a second ordinate are determined.

The first abscissa is the maximum value of the minimum abscissa of the four vertexes of the image to be spliced and the minimum abscissa of the four vertexes of the matched image; the second abscissa is the minimum value of the maximum abscissa of the four vertexes of the image to be spliced and the maximum abscissa of the four vertexes of the matched image; the first ordinate is the maximum value of the minimum ordinate of the four vertexes of the image to be spliced and the minimum ordinate of the four vertexes of the matched image; the second ordinate is the minimum value of the maximum ordinate of the four vertexes of the image to be spliced and the maximum ordinate of the four vertexes of the matched image.

S1054: and determining the overlapping area of the image to be spliced and the matching image by the first abscissa, the second abscissa, the first ordinate and the second ordinate.

The overlapping area of the image to be spliced and the matching image is the overlapping area of the image to be spliced.

For example, acquiring A, B, C images, stitching images based on image A, wherein B and C are images to be stitched, image A is a matching image of image B, and image B is a matching image of image C;

the width of the image A is W ₁ Height is L ₁ The coordinates of the four vertices of image A are t ₁₁ (0，0)、t ₁₂ (0，L ₁ )、t ₁₃ (W ₁ ，0)、t ₁₄ (W ₁ ，L ₁ ) The method comprises the steps of carrying out a first treatment on the surface of the The minimum abscissa of image A is 0 and the maximum abscissa is W ₁ The minimum ordinate is 0, and the maximum ordinate is L ₁ ；

For the image B, when the image B is projected onto the plane of the image A according to the obtained homography matrix of the image B, the coordinates of four vertexes of the image B are t ₂₁ (x ₂₁ ，y ₂₁ )、t ₂₂ (x ₂₂ ，y ₂₂ )、t ₂₃ (x ₂₃ ，y ₂₃ )、t ₂₄ (x ₂₄ ，y ₂₄ )；

x _{min 1} ＝min(x ₂₁ ，x ₂₂ ，x ₂₃ ，x ₂₄ )；

x _{max 1} ＝max(x ₂₁ ，x ₂₂ ，x ₂₃ ，x ₂₄ )；

y _{min 1} ＝min(y ₂₁ ，y ₂₂ ，y ₂₃ ，y ₂₄ )；

y _{max 1} ＝max(y ₂₁ ，y ₂₂ ，y ₂₃ ，y ₂₄ )；

According to x _{min 1} 、x _{max 1} 、y _{min 1} 、y _{max 1} And image A has a minimum abscissa of 0 and a maximum abscissa of W ₁ The minimum ordinate is 0, and the maximum ordinate is L ₁ Determining the lower left corner coordinates (u) _{min 1} ，v _{min 1} ) And upper right corner coordinates (u) _{max 1} ，v _{max 1} )：

u _{min 1} ＝max(x _{min 1} ，0)；

v _{min 1} ＝max(y _{min 1} ，0)；

u _{max 1} ＝max(x _{max 1} ，W ₁ )；

v _{max 1} ＝max(y _{max 1} ，L ₁ )；

From the lower left corner coordinates (u _{min 1} ，v _{min 1} ) And upper right corner coordinates (u) _{max 1} ，v _{max 1} ) The determined rectangular area may be regarded as the overlapping area of the image B.

For the image C, when the image C is projected onto the plane of the image A according to the homography matrix of the image C, the coordinates of four vertexes of the image C are t ₃₁ (x ₃₁ ，y ₃₁ )、t ₃₂ (x ₃₂ ，y ₃₂ )、t ₃₃ (x ₃₃ ，y ₃₃ )、t ₃₄ (x ₃₄ ，y ₃₄ )；

x _{min 2} ＝min(x ₃₁ ，x ₃₂ ，x ₃₃ ，x ₃₄ )；

x _{max 2} ＝max(x ₃₁ ，x ₃₂ ，x ₃₃ ，x ₃₄ )；

y _{min 2} ＝min(y ₃₁ ，y ₃₂ ，y ₃₃ ，y ₃₄ )；

y _{max 2} ＝max(y ₃₁ ，y ₃₂ ，y ₃₃ ，y ₃₄ )；

According to x _{min 2} 、x _{max 2} 、y _{min 2} 、y _{max 2} And x is as described above _{min 1} 、x _{max 1} 、y _{min 1} 、y _{max 1} Determining the lower left corner coordinates (u) _{min 2} ，v _{min 2} ) And upper right corner coordinates (u) _{max 2} ，v _{max 2} )：

u _{min 2} ＝max(x _{min 2} ，x _{min 1} )；

v _{min 2} ＝max(y _{min 2} ，y _{min 1} )；

u _{max 2} ＝max(x _{max 2} ，x _{max 1} )；

v _{max 2} ＝max(y _{max 2} ，y _{max 1} )；

From the lower left corner coordinateu _{min 2} ，v _{min 2} ) And upper right corner coordinates (u) _{max 2} ，v _{max 2} ) The determined rectangular area may be regarded as an overlapping area of the image C.

In the embodiment of the application, other manners may also be used to determine the overlapping area of the images to be spliced. For example, the maximum abscissa, the minimum abscissa, the maximum ordinate and the minimum ordinate of the matching feature points in the image to be stitched on the plane where the base stitched image is located are obtained, and the rectangular area determined by the obtained 4 coordinates can be used as the overlapping area of the image to be stitched. Here, the matching feature points are feature points that constitute a matching feature point pair with feature points in the matching image.

After determining the overlapping area of the image to be stitched, the area except the overlapping area in the image to be stitched may be determined as a non-overlapping area of the image to be stitched.

S106: and according to the reprojection errors of the matched feature point pairs, carrying out first grid establishment on the overlapped area of each image to be spliced.

In one embodiment of the present application, the re-projection error of the matching feature point pairs may be determined by the following formula:

In the formula (8), Δh is a homography matrix error of the matching feature point pairs, which may be referred to herein as a reprojection error between the matching feature point pairs;

to match the homogeneous coordinates of two feature points in the feature point pair, respectively

For H in formula (8), if no grid is established for the image, the H is the homography matrix obtained by initial calculation, and the determined reprojection error is: the reprojection error of the matched feature point pair where the feature point in the image is located; if a grid is established for the image, the H is a homography matrix of the grid,the determined re-projection error is: the feature points in the grid are matched with the re-projection errors of the feature point pairs.

The reprojection errors among all the matched characteristic point pairs can be calculated through the formula (8), and the norm delta h I is obtained ₁ . Here, the norm Δh ₁ Is signed:

for each image to be stitched, determining the re-projection error of the matched feature point pair where the feature point is located in the image to be stitched, that is, the re-projection error of the matched feature point pair corresponding to the image to be stitched, for convenience of description, hereinafter referred to as the re-projection error corresponding to the image to be stitched, and obtaining the maximum re-projection error Δh from all the re-projection errors corresponding to the image to be stitched _max And a minimum re-projection error Δh _min At this time, the re-projection error region is Δh _min -△h _max According to Deltah _max And Deltah _min And establishing a first grid for the overlapped area of the images to be spliced.

In one implementation, the method can be based on Δh _max And Deltah _min And establishing a first grid for the overlapped area of the images to be spliced through the following formula:

wherein Deltah is _i The method comprises the steps of (1) obtaining a re-projection error of a matched characteristic point pair where characteristic points in an ith first grid of an image to be spliced are located, wherein N is a positive integer; the value range of i is 0-N-1.

The problem that the image stitching effect is distorted is usually caused when the feature distribution is uneven, and especially when the angle of view of an image acquisition device is large, the distortion phenomenon is caused when the distance between the edge image area of the non-overlapping area of the image to be stitched and the feature point in the matched feature point is too large.

According to the formula (10), the re-projection error area is divided into N sections, characteristic points in each section are built in the same first grid, so that as shown in fig. 2, the white area is an overlapping area, the grid built in the white area is a first grid, the gray area is a non-overlapping area, the first grid is built in the overlapping area according to the formula (10), the size of the first grid is possibly unequal, but the re-projection error of the matched characteristic point pairs of the characteristic points in each first grid is in a certain range, namely, the characteristics of the characteristic points in one first grid are similar, the uniformity of the characteristics in the first grid is ensured for each first grid, and the problems of misplacement and double images of image splicing in the first grid can be effectively solved according to the single-stress matrix spliced image of the first grid.

S107: and for each first grid of each image to be spliced, adjusting the homography matrix of the image to be spliced according to the weight of the characteristic points in the first grid to obtain the homography matrix of the first grid.

In one embodiment of the present application, for each first grid of each image to be spliced, determining a weight of a matching feature point pair where a feature point in the first grid is located according to the weight of the first grid, and determining a weight of a matching feature point pair where a feature point outside the first grid is located; and for each first grid of each image to be spliced, adjusting a homography matrix of the image to be spliced according to the weight of the matched feature point pair of the feature point in the first grid and the weight of the matched feature point pair of the feature point outside the first grid, and taking the adjusted homography matrix as the homography matrix of the first grid.

In one implementation manner, after the first grid is established, for each feature point in each first grid of each image to be spliced, the weight of the matching feature point pair where the feature point is located can be determined according to the weight of the first grid by the following formula:

wherein w is _＊ M is the weight of the matching feature point pair where the feature point is located And for the number of the matched characteristic points in the first grid, gamma is the weight of the first grid, and the value range of gamma is 0-1.

Here, according to actual needs, γ may be controlled by manual adjustment to reduce the influence of the matching feature points with lower confidence on the homography matrix calculation. The matching feature points are feature points in the matching feature point pair.

For each feature point outside each first grid of each image to be spliced, determining the weight of the matching feature point pair where the feature point is located according to the weight of the first grid by the following formula:

wherein w' _＊ For the weight of the matching feature point pair where the feature point is located, M is the number of the matching feature points in the first grid, t _＊ T is the coordinates of the feature point _i Coordinates of the i-th matching feature point in the first grid. The matching feature points are feature points in the matching feature point pair.

For each first grid of each image to be spliced, after determining the weight of the matching feature point pair of the feature point in the first grid and the weight of the matching feature point pair of the feature point outside the first grid, adjusting the homography matrix of the image to be spliced according to the weight of the matching feature point pair of the feature point in the first grid and the weight of the matching feature point pair of the feature point outside the first grid, and taking the homography matrix after adjustment as the homography matrix of the first grid.

As above, for a first grid of an image to be stitched, the matrix a in equation (5) is iterated according to equations (11) (12):

based on equation (13) and equation (6), the homography matrix of the first grid can be determined as:

SVD is performed on the matrix WA in equation (14) to obtain a homography matrix for the first grid.

S108: and splicing the basic spliced image and at least one image to be spliced according to the homography matrix of each image to be spliced and the homography matrix of each first grid.

In one embodiment of the application, for each image to be spliced, projecting an overlapping area of the image to be spliced to a plane where a basic spliced image is located according to a homography matrix of each first grid, projecting a non-overlapping area of the image to be spliced to the plane where the basic spliced image is located according to the homography matrix of the image to be spliced, searching an optimal splice seam of the image by using a graph cut algorithm, then expanding edges on two sides of the optimal splice seam, building a Gaussian convolution image for the spliced image, downsampling convolution images of different convolution kernels, performing difference building to build a Laplace image, building a corresponding weight table for each layer, fusing the images according to the weight table, and finally recovering the fused image to generate a fused image under the original resolution.

In the embodiment of the application, the feature distribution in each first grid is uniform, the basic spliced image and at least one image to be spliced are spliced according to the homography matrix of each first grid, the problem that the single homography matrix is adopted for the image with nonuniform feature distribution in the overlapped area is avoided, the basic spliced image and the image to be spliced are spliced according to the homography matrix of each first grid, and the problems of dislocation and double images of image splicing are effectively solved under the condition that the feature distribution is nonuniform in the overlapped area.

In practical application, if the non-overlapping region of the image to be spliced is far from the matching feature points in the overlapping region, serious distortion and deformation occur at the position of the non-overlapping region, for the following reasons:

1) The distance between the non-overlapping area and the matching characteristic points is far, so that the weight is too large to cause crease and distortion of the spliced image;

2) The characteristic points are unevenly distributed, the characteristic points of the near view are more, the characteristic points of the far view are less, and the weight of the non-overlapping area is excessively uneven, so that crease phenomenon can occur at the position far away from the matched characteristic points, particularly at the edge, and black holes can also occur in certain specific scenes.

In an embodiment of the present application, in order to reduce the phenomena of distortion and deformation occurring in the non-overlapping area, referring to fig. 3, fig. 3 is a second flowchart of an image stitching method provided in the embodiment of the present application, where the method includes:

S301: and acquiring a basic spliced image and at least one image to be spliced.

S302: and extracting the characteristic points of the basic spliced image and at least one image to be spliced.

S303: and matching the characteristic points of each two images in at least one image to be spliced and the basic spliced image, and determining a plurality of matched characteristic point pairs.

S304: and calculating a homography matrix of each image to be spliced aiming at the basic spliced image according to the determined coordinates of the characteristic points in the plurality of matched characteristic point pairs.

S305: and determining the overlapping area of each image to be spliced according to the homography matrix of each image to be spliced.

S306: and according to the reprojection errors of the matched feature point pairs, carrying out first grid establishment on the overlapped area of each image to be spliced.

S307: and for each first grid of each image to be spliced, adjusting the homography matrix of the image to be spliced according to the weight of the first grid to obtain the homography matrix of the first grid.

Steps S301 to S307 are identical to steps S101 to S107, and will not be described here again.

S308: and establishing a second grid for the non-overlapping area of each image to be spliced according to preset parameters.

Here, the non-overlapping region is a region other than the overlapping region in each image to be stitched. Illustratively, in the embodiment of the present invention, the sizes of each of the second grids may be different or equal.

For one image to be spliced, if the sizes of the second grids are the same, the basic spliced image and at least one image to be spliced are spliced according to the homography matrix of each second grid, so that smooth transition of a non-overlapping area of image splicing can be ensured, and the phenomena of distortion and deformation in the non-overlapping area are reduced.

The preset parameters may be the number of the established grids, or may be the size of the grids, which is not limited in the embodiment of the present application.

S309: and for each second grid of each image to be spliced, adjusting the homography matrix of the image to be spliced according to the weight ratio of other second grids adjacent to the second grid to obtain the homography matrix of the second grid.

In an embodiment of the present application, for each second grid of each image to be stitched, the weight of the other second grids adjacent to the second grid may be determined according to the weight ratio of the other second grids adjacent to the second grid. And for each second grid of each image to be spliced, adjusting the homography matrix of the image to be spliced according to the weight of other second grids adjacent to the second grid, and taking the homography matrix after adjustment as the homography matrix of the second grid.

In one implementation manner, for each second grid of each image to be stitched, the weight of other second grids adjacent to the second grid may be determined according to the weight proportion of other second grids adjacent to the second grid by the following formula:

wherein w is _j Weight, Q, of the j-th other second grid adjacent to the second grid _i And (3) the weight ratio of the ith other second grids adjacent to the second grid is given, and S is the total number of other second grids adjacent to the second grid.

In the embodiment of the application, the weight proportion of the second grid can be controlled by artificial adjustment. For example, the weight ratio of the second grid in the vertical direction is set to be larger than that of the second grid in the horizontal direction, as shown in a weight ratio distribution diagram shown in fig. 4, different weight ratios are respectively arranged in different directions in fig. 4, wherein the weight ratio in the vertical direction is the largest, and the weight ratio in the horizontal direction is the smallest, so that the requirement that the human eyes are sensitive to the dislocation in the horizontal direction for the dislocation in the vertical direction can be effectively met.

For each second grid of each image to be spliced, after the weights of other second grids adjacent to the second grid are determined, adjusting the homography matrix of the image to be spliced according to the weights of other second grids adjacent to the second grid, and taking the homography matrix after adjustment as the homography matrix of the second grid:

In the formula (16), H _＊ Is the homography matrix of the second grid, w _j Weighting the jth second grid adjacent to the second grid, h _j And the homography matrix of the j second grids adjacent to the second grid.

For example, as shown in fig. 5, the

grids

1, 2, 3 in the image to be spliced are grids of overlapping areas, and the

grids

4, 5, 6, 7, 8, 9 are grids of non-overlapping areas; the homography matrix h of grid 1 is currently determined according to formulas (2) - (14) ₁ Homography matrix h of grid 2 ₂ Homography matrix h of grid 3 ₃ Homography matrix of (a).

As for the grid 4, grids of the adjacent known homography matrixes are provided with

grids

1 and 2, and the weight ratio distribution diagram shown in fig. 4 is combined, it can be known that for the grid 4, the weight ratio of the grid 1 is 1, the weight ratio of the grid 2 is 2, and at the moment, the weight w of the grid 1 can be determined ₄₁ The method comprises the following steps: 1/(1+2) =1/3; the weight w of grid 2 may be determined ₄₂ The method comprises the following steps: 2/(1+2) =2/3; homography matrix h of grid 4 ₄ The method comprises the following steps:

for grid 5, there is a grid of adjacent known homography matricesGrid 1, grid 2 and grid 4, in combination with the weight ratio distribution diagram shown in fig. 4, it can be known that, for grid 5, the weight ratio of grid 1 is 2, the weight ratio of grid 2 is 1, the weight ratio of grid 4 is 4, and at this time, the weight w of grid 1 can be determined ₅₁ The method comprises the following steps: 2/(2+1+4) =2/7; the weight w of grid 2 may be determined ₅₂ The method comprises the following steps: 1/(2+1+4) =1/7; the weight w of the grid 4 can be determined ₅₃ The method comprises the following steps: 4/(2+1+4) =4/7; homography matrix h of grid 5 ₅ The method comprises the following steps:

by analogy, homography matrices of grids 6, 7, 8, and 9 can be obtained respectively.

S310: and splicing the basic spliced image and at least one image to be spliced according to the homography matrix of each first grid and the homography matrix of each second grid.

In one embodiment of the present application, for each image to be stitched, an overlapping area of the image to be stitched may be projected onto a plane where a base stitched image is located according to a homography matrix of each first grid, a non-overlapping area of the image to be stitched is projected onto the plane where the base stitched image is located according to a homography matrix of each second grid, an image segmentation algorithm is used to find an optimal stitching seam of the image, then edges are spread on two sides of the optimal stitching seam, a gaussian convolution image is built for the stitched image, downsampling is performed by using convolution images of different convolution kernels, a laplace image is built by making difference, a corresponding weight table is built for each layer, image fusion is performed according to the weight table, and finally, the fused image is recovered, so as to generate a fused image under the original resolution.

After determining the homography matrix of each image to be spliced aiming at the basic spliced image, the embodiment of the application establishes a first grid according to the reprojection error between the matched characteristic point pairs, wherein the reprojection error is an important parameter for reflecting the characteristics, and the characteristic points with similar characteristics can be established in one first grid according to the reprojection error.

Corresponding to the embodiment of the image stitching method, the embodiment of the application also provides an image stitching device. Referring to fig. 6, fig. 6 is a schematic structural diagram of an image stitching device according to an embodiment of the present application, where the device includes:

an acquiring unit 601, configured to acquire a base stitched image and at least one image to be stitched;

An extracting unit 602, configured to extract feature points of the base stitched image and at least one image to be stitched;

a matching unit 603, configured to match feature points of each two images in the at least one image to be stitched and the base stitched image, and determine a plurality of matching feature point pairs;

a calculating unit 604, configured to calculate a homography matrix of each image to be stitched for the base stitched image according to the determined coordinates of the feature points in the plurality of matching feature point pairs;

a determining unit 605, configured to determine an overlapping area of each image to be stitched according to the homography matrix of each image to be stitched;

the establishing unit 606 is configured to perform a first grid establishment on the overlapping area of each image to be spliced according to the re-projection error of the matched feature point pair;

the adjusting unit 607 is configured to adjust, for each first grid of each image to be stitched, a homography matrix of the image to be stitched according to a weight of the first grid, to obtain a homography matrix of the first grid;

and the stitching unit 608 is configured to stitch the base stitched image and at least one image to be stitched according to the homography matrix of each image to be stitched and the homography matrix of each first grid.

In one embodiment of the present application, the extracting unit 602 may specifically be configured to:

And extracting characteristic points in at least one image to be spliced and the basic spliced image according to the SIFT algorithm.

In one embodiment of the present application, the matching unit 603 may specifically be configured to:

for each characteristic point of each image in at least one image to be spliced and the basic spliced image, calculating Euclidean distance between a description operator of the characteristic point of other images and the description operator of the characteristic point; the other images are at least one image to be spliced and images except the image in the basic spliced image; and forming a matching characteristic point pair by the characteristic point and the characteristic point corresponding to the description operator with the minimum Euclidean distance.

In an embodiment of the present application, the image stitching device may further include:

In one embodiment of the present application, the determining unit 605 may specifically be configured to determine, for each image to be stitched, an overlapping area of the image to be stitched, with reference to the second structural schematic diagram of the image stitching apparatus shown in fig. 7, based on fig. 6, in this apparatus, the determining unit 605 may include:

A first determining subunit 6051, configured to determine an abscissa and an ordinate of four vertices of the image to be stitched on a plane on which the base stitched image is located;

a second determining subunit 6052, configured to determine an abscissa and an ordinate of four vertices of the matched image on a plane on which the base stitched image is located; the matched image is an image in which the feature points forming a matched feature point pair with the feature points of the image to be spliced are located;

a third determining subunit 6053 for determining a first abscissa, a second abscissa, a first ordinate, and a second ordinate; the first abscissa is the maximum value of the minimum abscissa of the four vertexes of the image to be spliced and the minimum abscissa of the four vertexes of the matched image; the second abscissa is the minimum value of the maximum abscissa of the four vertexes of the image to be spliced and the maximum abscissa of the four vertexes of the matched image; the first ordinate is the maximum value of the minimum ordinate of the four vertexes of the image to be spliced and the minimum ordinate of the four vertexes of the matched image; the second ordinate is the minimum value of the maximum ordinate of the four vertexes of the image to be spliced and the maximum ordinate of the four vertexes of the matched image;

a fourth determining subunit 6054 is configured to determine, according to the first abscissa, the second abscissa, the first ordinate, and the second ordinate, an overlapping area of the image to be stitched and the matching image.

In one embodiment of the present application, referring to a third structural schematic diagram of the image stitching apparatus shown in fig. 8, based on fig. 6, in the apparatus, the establishing unit 606 may include:

an obtaining subunit 6061, configured to obtain a maximum reprojection error and a minimum reprojection error from the reprojection errors of all the matching feature point pairs corresponding to each image to be stitched;

the first establishing subunit 6062 is configured to perform a first grid establishment on the overlapping area of each image to be stitched according to the maximum re-projection error and the minimum re-projection error.

In one embodiment of the present application, the first establishing subunit may specifically be configured to:

wherein Deltah is _i As the re-projection error of the matching characteristic point pair where the characteristic point in the ith first grid of the image to be spliced is, deltah _min For the minimum re-projection error corresponding to the acquired image to be spliced, deltah _max To obtainThe maximum re-projection error corresponding to the image to be spliced is taken, and N is a positive integer; the value range of i is 0-N-1.

In one embodiment of the present application, referring to a fourth structural schematic diagram of the image stitching apparatus shown in fig. 9, based on fig. 6, the adjusting unit 607 may include:

A fifth determining subunit 6071, configured to determine, for each first grid of each image to be stitched, a weight of a matching feature point pair where a feature point in the first grid is located according to the weight of the first grid;

a sixth determining subunit 6072, configured to determine, for each first grid of each image to be stitched, a weight of a matching feature point pair where a feature point outside the first grid is located according to the weight of the first grid;

the first adjustment subunit 6073 is configured to, for each first grid of each image to be stitched, adjust a homography matrix of the image to be stitched according to a weight of a matching feature point pair where a feature point in the first grid is located and a weight of a matching feature point pair where a feature point outside the first grid is located, and use the adjusted homography matrix as a homography matrix of the first grid.

In one embodiment of the present application, the fifth determining subunit 6071 may specifically be configured to:

In one embodiment of the present application, the sixth determining subunit 6072 may specifically be configured to:

In an embodiment of the present application, referring to a fifth structural schematic diagram of the image stitching apparatus shown in fig. 10, based on fig. 6, the stitching unit 608 may include:

a second establishing subunit 6081, configured to perform a second grid establishment on the non-overlapping area of each image to be spliced according to the preset parameters; the non-overlapping area is an area except the overlapping area in each image to be spliced;

a second adjustment subunit 6082, configured to, for each second grid of each image to be stitched, adjust a homography matrix of the image to be stitched according to a weight ratio of other second grids adjacent to the second grid, and obtain a homography matrix of the second grid;

and a stitching subunit 6083, configured to stitch the base stitched image and at least one image to be stitched according to the homography matrix of each first grid and the homography matrix of each second grid.

In one embodiment of the present application, the second adjustment subunit 6082 may specifically be configured to:

Corresponding to the embodiment of the image stitching method, the embodiment of the application also provides electronic equipment. The electronic device includes a processor and a memory; a memory for storing a computer program; and the processor is used for executing the computer program stored in the memory to realize the image stitching method.

The electronic device as shown in fig. 11 includes a processor 1101 and a memory 1103. In addition, the electronic device may further include a communication interface 1102 and a communication bus 1104, where the processor 1101, the communication interface 1102, and the memory 1103 perform communication with each other through the communication bus 1104;

a memory 1103 for storing a computer program;

the processor 1101 is configured to implement an image stitching method when executing the computer program stored in the memory. The image stitching method comprises the following steps:

acquiring a basic spliced image and at least one image to be spliced;

extracting characteristic points of a basic spliced image and at least one image to be spliced;

matching the characteristic points of each two images in at least one image to be spliced and the basic spliced image, and determining a plurality of matched characteristic point pairs;

And splicing the basic spliced image and at least one image to be spliced according to the homography matrix of each image to be spliced and the homography matrix of each first grid.

In one embodiment of the present application, the step of extracting the feature points of the base stitched image and the at least one image to be stitched may include:

In one embodiment of the present application, the step of matching feature points of each two images in at least one image to be stitched and the base stitched image, and determining a plurality of matching feature point pairs may include:

In an embodiment of the present application, before matching the feature points of each two images in the at least one image to be stitched and the base stitched image, the image stitching method may further include:

In one embodiment of the present application, the step of determining the overlapping area of each image to be stitched according to the homography matrix of each image to be stitched may include:

determining the abscissa and the ordinate of four vertexes of the matched image on a plane where the basic spliced image is located; the matched image is an image in which the feature points forming a matched feature point pair with the feature points of the image to be spliced are located;

In an embodiment of the present application, the step of establishing a first grid for the overlapping area of each image to be stitched according to the re-projection error of the matched feature point pair may include:

In an embodiment of the present application, the step of performing the first grid establishment on the overlapping area of each image to be stitched according to the maximum re-projection error and the minimum re-projection error may include:

In an embodiment of the present application, for each first grid of each image to be stitched, the step of adjusting a homography matrix of the image to be stitched according to a weight of the first grid to obtain the homography matrix of the first grid may include:

In one embodiment of the present application, the step of determining, according to the weight of the first grid, the weight of the matching feature point pair where the feature point in the first grid is located may include:

In one embodiment of the present application, the step of determining, according to the weight of the first grid, the weight of the matching feature point pair where the feature point outside the first grid is located may include:

wherein the method comprises the steps of，w' _＊ For the weight value t of the matching feature point pair where the feature point is located _＊ T is the coordinates of the feature point _i Coordinates of feature points included in the matching feature point pair are the ith within the first grid.

In one embodiment of the present application, the step of stitching the base stitched image and the at least one image to be stitched according to the homography matrix of each image to be stitched and the homography matrix of each first grid may include:

establishing a second grid for the non-overlapping area of each image to be spliced according to preset parameters; the non-overlapping area is an area except the overlapping area in each image to be spliced;

and splicing the basic spliced image and at least one image to be spliced according to the homography matrix of each first grid and the homography matrix of each second grid.

In an embodiment of the present application, for each second grid of each image to be stitched, the step of adjusting the homography matrix of the image to be stitched according to the weight ratio of other second grids adjacent to the second grid to obtain the homography matrix of the second grid may include:

In one embodiment of the present application, the step of determining the weight of the other second grid adjacent to the second grid according to the weight ratio of the other second grid adjacent to the second grid may include:

The communication bus 1104 may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or an EISA (Extended Industry Standard Architecture ) bus, or the like. The communication bus 1104 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 11, but not only one bus or one type of bus.

The communication interface 1102 is used for communication between the electronic device and other devices.

The Memory 1103 may include RAM (Random Access Memory ) or NVM (Non-Volatile Memory), such as at least one magnetic disk Memory. Optionally, the memory 1103 may also be at least one storage device located remotely from the aforementioned processor.

The processor 1101 may be a general-purpose processor including a CPU (Central Processing Unit ), NP (Network Processor, network processor), and the like; but also DSP (Digital Signal Processing, digital signal processor), ASIC (Application Specific Integrated Circuit ), FPGA (Field-Programmable Gate Array, field programmable gate array) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components.

The embodiment of the application also provides a machine-readable storage medium, wherein the machine-readable storage medium stores a computer program, and the computer program realizes the image stitching method when being executed by a processor. The image stitching method comprises the following steps:

acquiring a basic spliced image and at least one image to be spliced;

wherein w is _＊ For the weight of the matching feature point pair where the feature point is located, γ is the first gridThe value range of gamma is 0-1, and M is the number of the feature points included in the matching feature point pair in the first grid.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the image stitching apparatus, electronic device, machine readable storage medium embodiments, since they are substantially similar to the image stitching method embodiments, the description is relatively simple, and reference should be made to the partial description of the image stitching method embodiments shown in fig. 1-5.

The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the scope of the present application. Any modifications, equivalent substitutions, improvements, etc. that are within the spirit and principles of the present application are intended to be included within the scope of the present application.

Claims

1. A method of image stitching, the method comprising:

acquiring a basic spliced image and at least one image to be spliced;

Splicing the basic spliced image and the at least one image to be spliced according to the homography matrix of each image to be spliced and the homography matrix of each first grid;

and the step of establishing a first grid for the overlapped area of each image to be spliced according to the re-projection error of the matched characteristic point pairs comprises the following steps:

according to the maximum re-projection error and the minimum re-projection error, a first grid is established for the overlapped area of each image to be spliced;

the step of adjusting the homography matrix of each first grid of each image to be spliced according to the weight of the first grid to obtain the homography matrix of the first grid comprises the following steps:

2. The method according to claim 1, wherein the step of extracting feature points of the base stitched image and the at least one image to be stitched comprises:

and extracting feature points in the at least one image to be spliced and the basic spliced image according to a scale-invariant feature transform (SIFT) algorithm.

3. The method of claim 1, wherein the step of matching the feature points of each two of the at least one image to be stitched and the base stitched image to determine a plurality of matched feature point pairs comprises:

4. A method according to claim 3, wherein prior to matching the feature points of each two of the at least one image to be stitched and the base stitched image, the method further comprises:

5. The method of claim 1, wherein the step of determining the overlapping area of each image to be stitched from the homography matrix of each image to be stitched comprises:

6. The method according to claim 1, wherein the step of performing first grid establishment for the overlapping area of each image to be stitched according to the maximum re-projection error and the minimum re-projection error comprises:

7. The method of claim 1, wherein the step of determining weights of pairs of matching feature points where feature points in the first grid are located according to the weights of the first grid comprises:

8. The method according to claim 1, wherein the step of stitching the base stitched image and the at least one image to be stitched according to the homography matrix of each image to be stitched and the homography matrix of each first grid comprises:

9. The method according to claim 8, wherein the step of adjusting the homography matrix of each second grid of each image to be stitched according to the weight ratio of other second grids adjacent to the second grid to obtain the homography matrix of the second grid includes:

10. The method of claim 9, wherein the step of determining the weights of the other second grids adjacent to the second grid according to the weight ratio of the other second grids adjacent to the second grid comprises:

wherein w is _j Weights for the j-th other second grid adjacent to the second gridValue, Q _j For the weight ratio of the j other second grids adjacent to the second grid, Q _i And (3) the weight ratio of the ith other second grids adjacent to the second grid is given, and S is the total number of other second grids adjacent to the second grid.

11. The method according to any of claims 8-10, wherein each second network is of equal size.

12. An image stitching device, the device comprising:

the splicing unit is used for splicing the basic spliced image and the at least one image to be spliced according to the homography matrix of each image to be spliced and the homography matrix of each first grid;

The establishing unit includes:

the first establishing subunit is used for establishing a first grid for the overlapped area of each image to be spliced according to the maximum re-projection error and the minimum re-projection error;

the adjusting unit includes:

13. The device according to claim 12, characterized in that said extraction unit is specifically configured to:

14. The apparatus according to claim 12, wherein the matching unit is specifically configured to:

15. The apparatus of claim 14, wherein the apparatus further comprises:

16. The apparatus according to claim 12, wherein the determining unit, in particular for determining, for each image to be stitched, an overlapping region of the image to be stitched, comprises:

17. The apparatus of claim 12, wherein the first setup subunit is specifically configured to:

18. The apparatus according to claim 12, wherein the fifth determination subunit is specifically configured to:

19. The apparatus of claim 12, wherein the splicing unit comprises:

20. The apparatus according to claim 19, wherein the second adjustment subunit is specifically configured to:

21. The apparatus according to claim 20, wherein the second adjustment subunit is specifically configured to:

22. The apparatus of any of claims 19-21, wherein each second network is equal in size.

23. An electronic device comprising a processor and a memory;

the memory is used for storing a computer program;

the processor being adapted to execute a computer program stored on the memory for implementing the method steps of any of claims 1-11.