CN114511448A - Method, device, equipment and medium for splicing images - Google Patents

Method, device, equipment and medium for splicing images Download PDF

Info

Publication number
CN114511448A
CN114511448A CN202210407577.9A CN202210407577A CN114511448A CN 114511448 A CN114511448 A CN 114511448A CN 202210407577 A CN202210407577 A CN 202210407577A CN 114511448 A CN114511448 A CN 114511448A
Authority
CN
China
Prior art keywords
image
spliced
images
vertical direction
splicing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210407577.9A
Other languages
Chinese (zh)
Other versions
CN114511448B (en
Inventor
陈一航
胡思行
蒋念娟
沈小勇
吕江波
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Simou Intelligent Technology Co ltd
Shenzhen Smartmore Technology Co Ltd
Original Assignee
Beijing Simou Intelligent Technology Co ltd
Shenzhen Smartmore Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Simou Intelligent Technology Co ltd, Shenzhen Smartmore Technology Co Ltd filed Critical Beijing Simou Intelligent Technology Co ltd
Priority to CN202210407577.9A priority Critical patent/CN114511448B/en
Publication of CN114511448A publication Critical patent/CN114511448A/en
Application granted granted Critical
Publication of CN114511448B publication Critical patent/CN114511448B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Abstract

The application relates to the technical field of image processing, and provides a method, a device, computer equipment, a storage medium and a computer program product for splicing panoramic images. The method and the device can improve the speed of splicing the panoramic image. The method comprises the following steps: acquiring a plurality of images to be spliced, extracting an image feature point set of each image to be spliced, acquiring gyroscope information corresponding to each image to be spliced, performing image feature point matching on adjacent images to be spliced according to the gyroscope information, optimizing the gyroscope information corresponding to each image to be spliced according to an image feature point matching result, determining a vertical seam line of each image pair to be spliced adjacent to the same vertical direction according to the optimized gyroscope information, splicing the images to be spliced to the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent to the same vertical direction, and splicing image blocks to be spliced obtained by splicing in each vertical direction according to a transverse seam line.

Description

Method, device, equipment and medium for splicing images
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a method, an apparatus, a computer device, a storage medium, and a computer program product for stitching panoramic images.
Background
With the development of image processing technology, the technology of stitching panoramic images becomes an important image processing technology. At present, unmanned aerial vehicle is applied to in more and more fields as important shooting instrument, carries out panoramic image's concatenation to many high resolution ratio images like unmanned aerial vehicle shoots, can provide directly perceived and comprehensive image information for people.
The conventional technology is generally to splice panoramic images through a global optimization algorithm, but the speed of splicing the panoramic images through the technology is low.
Disclosure of Invention
In view of the above, it is necessary to provide a method, an apparatus, a computer device, a computer readable storage medium and a computer program product for stitching panoramic images.
In a first aspect, the present application provides a method for stitching panoramic images. The method comprises the following steps:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced;
acquiring gyroscope information corresponding to each image to be spliced;
matching image feature points of adjacent images to be spliced according to gyroscope information;
optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result;
determining a vertical joint line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction;
splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction;
and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse seam lines.
In one of the embodiments, the first and second electrodes are,
the method further comprises the following steps:
acquiring corresponding camera internal reference information when shooting each image to be spliced;
optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result, wherein the gyroscope information comprises the following steps:
and optimizing gyroscope information corresponding to each image to be spliced based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, optimizing gyroscope information corresponding to each image to be stitched based on the image feature point matching result and camera internal reference information corresponding to each image to be stitched comprises:
and optimizing gyroscope information corresponding to each image to be spliced by a least square method based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, determining a vertical seam line of each pair of images to be stitched adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be stitched in the same vertical direction includes:
mapping each image to be spliced in the same vertical direction to a spherical coordinate system according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction;
and determining the vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the overlapping area of each image pair to be spliced adjacent in the same vertical direction in the spherical coordinate system.
In one embodiment, determining a vertical seam line of each vertically adjacent pair of images to be stitched comprises:
and taking the seam line with the minimum energy accumulation in each vertically adjacent image pair to be spliced as a vertical seam line based on a preset energy function through a dynamic programming algorithm.
In one of the embodiments, the first and second electrodes are,
the method further comprises the following steps:
obtaining an Antarctic image to be spliced;
splicing the image blocks to be spliced obtained by splicing in each vertical direction according to the transverse joint lines, and the splicing method comprises the following steps:
splicing image blocks to be spliced obtained by splicing in the vertical directions according to a transverse joint line to obtain a first panoramic image without a south pole;
and obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
In one embodiment, extracting an image feature point set of each image to be stitched comprises:
and extracting an image feature point set of each image to be spliced by an SIFT algorithm.
In one embodiment, before the image blocks to be spliced obtained by splicing in the vertical directions are spliced according to the transverse seam lines, the method further includes:
mapping the image blocks to be spliced to a spherical coordinate system according to the gyroscope information of the image blocks to be spliced;
and determining a transverse joint line of each transversely adjacent image block pair to be spliced according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system.
In one embodiment, determining the transverse seam line of each transversely adjacent image block pair to be spliced comprises:
and taking the seam line with the minimum energy accumulation in each transversely adjacent image block pair to be spliced as a transverse seam line on the basis of a preset energy function through a dynamic programming algorithm.
In a second aspect, the application further provides a device for splicing panoramic images. The device comprises:
the image acquisition module is used for acquiring a plurality of images to be spliced and extracting an image feature point set of each image to be spliced;
the gyroscope information acquisition module is used for acquiring gyroscope information corresponding to the images to be spliced;
the characteristic point matching module is used for matching the image characteristic points of the adjacent images to be spliced according to the gyroscope information;
the gyroscope information optimization module is used for optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result;
the vertical joint line determining module is used for determining a vertical joint line of each image pair to be spliced adjacent to the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction;
the image splicing module is used for splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction;
and the image block splicing module is used for splicing the image blocks to be spliced which are obtained by splicing in the vertical directions according to the transverse seam lines.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the following steps when executing the computer program:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced; acquiring gyroscope information corresponding to each image to be spliced; image feature point matching is carried out on adjacent images to be spliced according to gyroscope information; optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result; determining a vertical seam line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction; and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse joint lines.
In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced; acquiring gyroscope information corresponding to each image to be spliced; image feature point matching is carried out on adjacent images to be spliced according to gyroscope information; optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result; determining a vertical joint line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction; and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse seam lines.
In a fifth aspect, the present application further provides a computer program product. The computer program product comprising a computer program which when executed by a processor performs the steps of:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced; acquiring gyroscope information corresponding to each image to be spliced; image feature point matching is carried out on adjacent images to be spliced according to gyroscope information; optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result; determining a vertical joint line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction; and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse joint lines.
The method, the device, the computer equipment, the storage medium and the computer program product for splicing the panoramic images acquire a plurality of images to be spliced, extract an image feature point set of each image to be spliced, acquire gyroscope information corresponding to each image to be spliced, perform image feature point matching on adjacent images to be spliced according to the gyroscope information, optimize the gyroscope information corresponding to each image to be spliced according to an image feature point matching result, determine a vertical joint line of each image pair to be spliced adjacent in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction, splice each image to be spliced in the same vertical direction according to the vertical joint line of each image pair to be spliced adjacent in the same vertical direction, and splice image blocks to be spliced obtained in each vertical direction according to a horizontal joint line. The scheme acquires a plurality of images to be spliced, extracts an image feature point set of each image to be spliced, acquires gyroscope information corresponding to each image to be spliced when an unmanned aerial vehicle shoots each image to be spliced, organizes the position relation of each image to be spliced according to the gyroscope information, matches image feature points of each image to be spliced which is vertically adjacent or horizontally adjacent aiming at each image to be spliced which is vertically adjacent or horizontally adjacent, optimizes the gyroscope information of each image to be spliced according to the matching result, determines a vertical seam line of each image to be spliced which is vertically adjacent according to the optimized gyroscope information, splices each image to be spliced according to the vertical seam line to obtain each vertical image block, splices each vertical image block according to the horizontal seam line to obtain a panoramic image, thereby realizing the improvement of the speed of splicing the panoramic image, furthermore, the position relation of the images to be spliced is determined in advance according to the gyroscope information, images do not need to be matched pairwise and two pairwise joint lines do not need to be searched, the calculation complexity is greatly reduced, the algorithm precision can be improved by optimizing the gyroscope information instead of optimizing a homography matrix in the traditional technology, and a better panoramic image splicing effect is achieved.
Drawings
FIG. 1 is a schematic flow chart diagram of a method for stitching panoramic images in one embodiment;
FIG. 2 is a schematic flow chart of a method for stitching panoramic images in another embodiment;
FIG. 3 is a flowchart illustrating the steps of optimizing gyroscope information corresponding to each image to be stitched in one embodiment;
FIG. 4 is a flow chart illustrating the steps of determining vertical seam lines for each vertically adjacent pair of images to be stitched in one embodiment;
FIG. 5 is a flowchart illustrating the steps of obtaining a second panoramic image, in accordance with one embodiment;
FIG. 6 is a block diagram of an apparatus for stitching panoramic images in one embodiment;
FIG. 7 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In an embodiment, as shown in fig. 1, a method for stitching panoramic images is provided, and this embodiment is exemplified by applying the method to a terminal, and includes the following steps:
step S101, acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced.
In this step, the image to be stitched may be an image photographed by the unmanned aerial vehicle.
Specifically, the terminal acquires a plurality of images to be spliced and extracts an image feature point set of each image to be spliced.
Exemplarily, as shown in fig. 2, the terminal reads a plurality of images to be stitched according to the shooting sequence of the unmanned aerial vehicle, and performs image feature point set and descriptor extraction on each image to be stitched.
And S102, acquiring gyroscope information corresponding to each image to be spliced.
The gyroscope information can be the gyroscope information corresponding to the unmanned aerial vehicle when shooting each image to be spliced.
Specifically, the terminal acquires gyroscope information corresponding to shooting of each image to be spliced.
For example, as shown in fig. 2, the terminal may acquire gyroscope information corresponding to each image to be stitched when the unmanned aerial vehicle shoots the images to be stitched in real time, where the gyroscope information may be shown in the following table:
Figure 851784DEST_PATH_IMAGE002
wherein, unmanned aerial vehicle can shoot according to the shooting order of table, and with each image of waiting to splice divide into 8 groups and erect and shoot picture and 2 south pole pictures, the first digit in the table is the shooting order, and the content of bracket is the shooting angle (roll, pitch, yaw) of presetting, gyroscope information promptly.
And S103, matching image feature points of adjacent images to be spliced according to gyroscope information.
In this step, the adjacent images to be stitched may be adjacent images to be stitched shot by the unmanned aerial vehicle, the adjacent images to be stitched may refer to vertical adjacent or horizontal adjacent, and the adjacent images to be stitched may refer to images to be stitched having an overlapping region between two images, for example, for image 1, the adjacent images of image 1 are image 2 and image 7, and for image 2, the adjacent images of image 2 are image 1, image 3, and image 6.
Specifically, the terminal performs image feature point matching on adjacent images to be spliced according to gyroscope information. Illustratively, the terminal performs image feature point matching on the images to be evaluated which are horizontally adjacent or vertically adjacent according to the gyroscope information, for example, for the image 2, the image 2 performs image feature point matching with the images 1, 3 and 6. Optionally, after the acquisition of a certain group of vertically shot images is completed, image feature point matching can be performed group by group, and when the acquisition of the group of vertically shot images 1 (images 1, 2 and 3) is completed, for image 1, image 1 is image feature point matched with image 2, for image 2, image 2 is image feature point matched with images 1, 3, for the image 3, the image 3 and the image 2 are subjected to image feature point matching, so that the image feature point matching is completed by vertically shooting 1 group of images, and then when the images (images 5, 6 and 7) of the vertically shooting 2 groups are acquired, for image 5, image 5 is image feature point matched to images 3, 6, for image 6, image 6 is image feature point matched to images 2, 5, 7, for the image 7, the image 7 is matched with the images 1 and 6 by image feature points, and the matching of the image feature points of all the adjacent images to be spliced can be finished by the same way.
And step S104, optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result.
In this step, for example, a pair of images to be stitched (e.g., image A and image B are adjacent images to be stitched)Image a and image B constitute an image pair), the image feature point matching result may be a matching result of the feature point composition of the feature point of image a corresponding to that of image B, for example, piFor a certain point in image A homogeneous coordinate (x)i，yi，1），pjIs piHomogeneous coordinates (x) of matching points in image Bj，yj，1）。
Specifically, the terminal optimizes gyroscope information corresponding to each image to be spliced according to the image feature point matching result. Illustratively, the terminal can optimize gyroscope information corresponding to each image to be stitched group by group (vertical shooting image group) according to the image feature point matching result, for example, the gyroscope information of the vertical shooting 1 group (images 1, 2 and 3) is optimized according to the image feature point matching result of the vertical shooting 1 group (images 1, 2 and 3) and the corresponding adjacent image to be stitched, the gyroscope information of the vertical shooting 2 group (images 5, 6 and 7) is optimized according to the image feature point matching result of the vertical shooting 2 group (images 5, 6 and 7) and the corresponding adjacent image to be stitched, and the optimization of the gyroscope information corresponding to all the images to be stitched can be finished by analogy.
And S105, determining a vertical joint line of each image pair to be spliced adjacent to the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction.
In this step, for each vertically adjacent pair of images to be stitched, the vertical seam line may be a vertical seam line between image 1 and image 2 of the group of vertical shots 1 in the above table, and a vertical seam line between image 2 and image 3.
Specifically, the terminal determines a vertical seam line of each image pair to be spliced adjacent to the same vertical direction according to optimized gyroscope information corresponding to each image to be spliced in the same vertical direction.
And S106, splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction.
Specifically, as shown in fig. 2, the terminal splices the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction, so as to obtain the image blocks to be spliced in each vertical direction.
And S107, splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse seam lines.
In this step, the horizontal seam line may be a horizontal seam line between the image blocks to be spliced obtained by vertically splicing adjacent image blocks, for example, a certain horizontal seam line may be a seam line between the vertically spliced vertical racket 1 group and the vertically spliced vertical racket 2 group in the table.
Specifically, the terminal splices the image blocks to be spliced obtained by splicing in each vertical direction according to the transverse seam lines.
The method for splicing the panoramic images comprises the steps of obtaining a plurality of images to be spliced, extracting an image feature point set of each image to be spliced, obtaining gyroscope information corresponding to each image to be spliced, carrying out image feature point matching on adjacent images to be spliced according to the gyroscope information, optimizing the gyroscope information corresponding to each image to be spliced according to an image feature point matching result, determining a vertical seam line of each image pair to be spliced adjacent to the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced on the same vertical direction, splicing the images to be spliced on the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent to the same vertical direction, and splicing image blocks to be spliced obtained by splicing in each vertical direction according to the horizontal seam lines. The scheme acquires a plurality of images to be spliced, extracts an image feature point set of each image to be spliced, acquires gyroscope information corresponding to each image to be spliced when an unmanned aerial vehicle shoots each image to be spliced, organizes the position relation of each image to be spliced according to the gyroscope information, matches image feature points of each image to be spliced which is vertically adjacent or horizontally adjacent aiming at each image to be spliced which is vertically adjacent or horizontally adjacent, optimizes the gyroscope information of each image to be spliced according to the matching result, determines a vertical seam line of each image to be spliced which is vertically adjacent according to the optimized gyroscope information, splices each image to be spliced according to the vertical seam line to obtain each vertical image block, splices each vertical image block according to the horizontal seam line to obtain a panoramic image, thereby realizing the improvement of the speed of splicing the panoramic image, furthermore, the position relation of the images to be spliced is determined in advance according to the gyroscope information, images do not need to be matched pairwise and two pairwise joint lines do not need to be searched, the calculation complexity is greatly reduced, the algorithm precision can be improved by optimizing the gyroscope information instead of optimizing a homography matrix in the traditional technology, and a better panoramic image splicing effect is achieved.
In an embodiment, as shown in fig. 3, the method may further optimize gyroscope information corresponding to each image to be stitched in the same vertical direction by the following steps, specifically including: s301, acquiring corresponding camera internal reference information when shooting each image to be spliced; and S302, optimizing gyroscope information corresponding to each image to be spliced based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In this embodiment, the camera internal reference information may be a camera internal reference matrix.
Specifically, the terminal acquires camera internal reference information corresponding to shooting each image to be spliced, and gyroscope information corresponding to each image to be spliced can be optimized through a least square method based on an image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
For example, as shown in fig. 2, when the terminal acquires the images to be stitched, the terminal acquires the camera internal reference information and the gyroscope information corresponding to the images to be stitched, and the camera internal reference information and the gyroscope information are acquired according to formula 1
Figure 771198DEST_PATH_IMAGE004
Optimizing gyroscope information corresponding to each image to be spliced, wherein n is the number of matching points of an image pair (the image pair is an image pair formed by a certain image to be spliced and another image to be spliced adjacent to the image to be spliced), a is the serial number of the matching points of the image pair, K is a camera internal reference matrix corresponding to the image to be spliced, and R is a rotation matrix of the gyroscope information, for example, the image A and the image B are adjacent (the adjacent images can be the same)Vertically adjacent or laterally adjacent) of a certain pair of images to be stitched, piFor a certain point in image A homogeneous coordinate (x)i，yi，1），pjIs piHomogeneous coordinates (x) of matching points in image Bj，yj，1），KiAnd RiK and R, K for corresponding image AjAnd RjFor K and R corresponding to image B, L2 is the L2 norm, also known as the least squares method, where the rotation matrix R of the gyroscope information is based on the gyroscope information as given by equation 2
Figure 465485DEST_PATH_IMAGE006
Where α is gyroscope yaw, β is gyroscope pitch, γ is gyroscope roll, for example, the gyroscope information (roll, pitch, yaw) of an image to be stitched is (0, 15, 45), then R isz（α）Ry（β）Rx(γ) a rotation matrix rotated by an angle of 0 degrees around the z-axis, a rotation matrix rotated by an angle of 15 degrees around the y-axis, and a rotation matrix rotated by an angle of 45 degrees around the x-axis, respectively; the terminal calculates what value the rotation matrix R is when it is by adjusting the rotation matrix R according to equation 1,
Figure 461254DEST_PATH_IMAGE008
and finally, the rotation matrix obtained through calculation is used as the rotation matrix corresponding to the optimized gyroscope information, so that the optimized gyroscope information is obtained through calculation according to the rotation matrix corresponding to the optimized gyroscope information.
According to the technical scheme, the gyroscope information corresponding to the images to be spliced in the same vertical direction is optimized based on the image feature point matching result and the camera internal reference information, so that the optimized gyroscope information corresponding to the images to be spliced in the same vertical direction can be obtained more accurately, and a better panoramic image splicing effect can be realized subsequently.
In an embodiment, as shown in fig. 4, the determining, according to the optimized gyroscope information corresponding to the images to be stitched in the same vertical direction in step S105, a vertical seam line of each pair of images to be stitched adjacent in the same vertical direction specifically includes: step S401, mapping each image to be spliced in the same vertical direction to a spherical coordinate system according to optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; step S402, determining a vertical joint line of each pair of images to be spliced adjacent in the same vertical direction according to the overlapping area of each pair of images to be spliced adjacent in the same vertical direction in the spherical coordinate system.
Specifically, the terminal maps each image to be spliced in the same vertical direction to a spherical coordinate system according to optimized gyroscope information corresponding to each image to be spliced in the same vertical direction, and determines a vertical joint line of each image to be spliced adjacent in the same vertical direction according to an overlapping region of each image to be spliced adjacent in the same vertical direction in the spherical coordinate system.
According to the technical scheme, the vertical seam line of each image pair to be spliced adjacent to the same vertical direction in the spherical coordinate system is determined according to the overlapping area of each image pair to be spliced adjacent to the same vertical direction in the spherical coordinate system, so that the vertical seam line of each image pair to be spliced adjacent to the same vertical direction can be determined more quickly and accurately, and the panoramic image can be spliced more quickly and accurately.
In one embodiment, the method may further determine a vertical seam line of each vertically adjacent pair of images to be stitched, specifically including: and taking the seam line with the minimum energy accumulation in each vertically adjacent image pair to be spliced as a vertical seam line based on a preset energy function through a dynamic programming algorithm.
In this embodiment, the preset energy function may be energy = energy func (i 1, i 2), where energy is a defined function and may be a gradient, energy is an energy function of an overlapping portion of the image pairs to be stitched, i1 and i2 represent the overlapping portions of the image pairs to be stitched, for example,
Figure 975412DEST_PATH_IMAGE010
，
wherein G isx1=Gx（GRAY1），Gx2=Gx（GRAY2），Gy1=Gy（GRAY1），Gy2=Gy(GRAY 2), wherein GRAY1= i1Blue*a1+i1green*b1+i1red*c1，a1+ b1+ c1=1，
GRAY2=i2Blue* a2+i2green* b2+i2red* c2，a2+ b2+ c2=1, wherein,
Figure 964096DEST_PATH_IMAGE012
，
wherein, A in the formula is GRAY1 or GRAY 2; minimum energy accumulation can be formulated
Costi，j=energyi，j+min（Costi-1，j，Costi，j-1，Costi-1，j-1) The calculation results in that,
wherein cost represents the minimum energy accumulation in all paths to pixel point (i, j), which may be an image coordinate, energyi，jIs the energy value of pixel (i, j).
Specifically, the terminal uses the seam line with the minimum energy accumulation in each vertically adjacent image pair to be spliced as a vertical seam line based on a preset energy function through a dynamic programming algorithm.
According to the technical scheme, the seam line with the minimum energy accumulation in each pair of images to be spliced adjacent to the same vertical direction is used as the vertical seam line, so that the optimal vertical seam line of each pair of images to be spliced adjacent to the same vertical direction can be determined more accurately, and the panoramic images can be spliced more accurately.
In an embodiment, as shown in fig. 5, the method may further obtain the second panoramic image by the following steps, specifically including: step S501, obtaining a south pole image to be spliced; step S502, splicing image blocks to be spliced obtained by splicing in the vertical directions according to a transverse seam line to obtain a first panoramic image without a south pole; and S503, obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
In this embodiment, the south pole images to be stitched may be the south pole 1 (0, 75, 0) image and the south pole 2 (0, 75, 180) image in the above table.
Specifically, the terminal obtains a south pole image to be spliced, splicing image blocks to be spliced obtained by splicing in the vertical directions according to a transverse seam line to obtain a first panoramic image without a south pole, and obtaining a complete second panoramic image according to the south pole image to be spliced and the first panoramic image.
Exemplarily, as shown in fig. 2, the terminal obtains a south pole image to be spliced, splices image blocks to be spliced obtained by splicing in each vertical direction according to a transverse seam line to obtain a first panoramic image without a south pole, and obtains a complete second panoramic image through a seam finding algorithm and a fusion algorithm according to the south pole image to be spliced and the first panoramic image.
According to the technical scheme of the embodiment, the complete second panoramic image is obtained according to the south pole image to be spliced and the first panoramic image, so that the splicing is facilitated to obtain a more complete panoramic image.
In an embodiment, the extracting of the image feature point set of each image to be stitched in step S101 specifically includes: and extracting an image feature point set of each image to be spliced by an SIFT algorithm.
SIFT is Scale-invariant feature transform (SIFT).
Specifically, the terminal extracts an image feature point set of each image to be spliced through an SIFT algorithm.
Exemplarily, as shown in fig. 2, the terminal performs image feature point set and descriptor extraction on each image to be stitched through a SIFT algorithm.
According to the technical scheme, the image feature point set of each image to be spliced is extracted through the SIFT algorithm, so that the accuracy of extracting the image feature points is improved, and the follow-up realization of a better panoramic image splicing effect is facilitated.
In an embodiment, the method may further determine a transverse seam line of each transversely adjacent pair of image blocks to be stitched by: mapping the image blocks to be spliced to a spherical coordinate system according to the gyroscope information of the image blocks to be spliced; and determining a transverse joint line of each transversely adjacent image block pair to be spliced according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system.
In this embodiment, each image block pair to be spliced which is horizontally adjacent may refer to each image block pair to be spliced which has an overlapping region between image blocks to be spliced obtained by vertically splicing, for example, the image block pair to be spliced is a group of vertically-spliced 1 (images 1, 2, and 3 after vertically splicing) and a group of vertically-spliced 2 (images 5, 6, and 7 after vertically splicing).
According to the technical scheme of the embodiment, the transverse joint line of each transversely adjacent image block pair to be spliced is determined according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system, so that the transverse joint line can be determined more quickly and accurately, and the panoramic image can be spliced more quickly and accurately subsequently.
In one embodiment, the method may further include determining a transverse seam line by: and taking the seam line with the minimum energy accumulation in each transversely adjacent image block pair to be spliced as a transverse seam line on the basis of a preset energy function through a dynamic programming algorithm.
In this embodiment, the preset energy function may be energy = energy func (i 1, i 2).
According to the technical scheme of the embodiment, the dynamic programming algorithm and the preset energy function are used for taking the seam line with the minimum energy accumulation in each transversely adjacent image block pair to be spliced as the transverse seam line, so that the optimal transverse seam line of each image block pair to be spliced can be determined more accurately, and the panoramic image can be spliced more accurately subsequently.
The following application example illustrates that the method provided by the present application can be applied to panoramic image stitching on an image shot by an unmanned aerial vehicle, as shown in fig. 2, the main steps include:
the method comprises the following steps: the terminal reads images to be spliced sent by the unmanned aerial vehicle and corresponding camera internal reference information and gyroscope information when shooting the images to be spliced in real time; the images to be spliced are divided into 8 groups of vertical shooting images (for example, a group of vertical shooting images comprises three images to be spliced) and two south pole images to be spliced in advance.
Step two: and the terminal extracts the image feature point set of each image to be spliced through an SIFT algorithm.
Step three: the terminal judges whether all the current vertical shooting pictures are obtained (if yes, the vertical shooting pictures are obtained for 1 group), if not, the first step is executed, and if yes, the next step is executed.
Step four: and the terminal performs image feature point matching on the adjacent images to be spliced according to the image feature point set obtained by the SIFT algorithm.
Step five: and the terminal optimizes gyroscope information corresponding to each image to be spliced according to the image feature point matching result.
Step six: and the terminal maps the images to be spliced in the same vertical direction to the panoramic image of the spherical coordinate system according to the camera internal reference information corresponding to the images to be spliced in the same vertical direction and the optimized gyroscope information.
Step seven: and the terminal determines the vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the optimized gyroscope information and the overlapped area of each image pair to be spliced adjacent in the same vertical direction in the spherical coordinate system.
Step eight: and the terminal performs fusion splicing on the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction (like a group of vertical shot images) to obtain image blocks Imagel to be spliced obtained by splicing in each vertical direction.
Step nine: and the terminal calculates the transverse joint lines among the image blocks to be spliced obtained by splicing in the vertical direction.
Step ten: and the terminal judges whether all the images to be spliced are read or not, if not, the step one is executed, and if so, the next step is executed.
Step eleven: and splicing the image blocks to be spliced obtained by splicing in the vertical directions by the terminal according to the transverse seam lines to obtain a first panoramic image without the south pole.
Step twelve: and the terminal obtains a second panoramic image through a seam finding algorithm and a fusion algorithm according to the south pole image to be spliced and the first panoramic image.
According to the application example, the shooting sequence of the unmanned aerial vehicle is preset, a mode of shooting while splicing can be adopted, the unmanned aerial vehicle only needs to splice the south pole images after shooting the images, splicing is not needed after loading of all the images is completed, and the effect of quickly outputting a panoramic image after shooting the images can be achieved; furthermore, the position relation of the images can be determined in advance through gyroscope information, so that the image position is determined without pairwise image matching; moreover, by optimizing gyroscope information instead of a homography matrix, algorithm precision can be improved, and a better panoramic image splicing effect is obtained.
It should be understood that, although the steps in the flowcharts related to the embodiments as described above are sequentially displayed as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be rotated or alternated with other steps or at least a part of the steps or stages in other steps.
Based on the same inventive concept, the embodiment of the application also provides a device for splicing the panoramic images, which is used for realizing the method for splicing the panoramic images. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the above method, so specific limitations in one or more embodiments of the apparatus for stitching panoramic images provided below may refer to the limitations in the above method for stitching panoramic images, and details are not repeated here.
In one embodiment, as shown in fig. 6, there is provided an apparatus for stitching panoramic images, the apparatus 600 may include:
the image acquisition module 601 is configured to acquire a plurality of images to be stitched and extract an image feature point set of each image to be stitched;
a gyroscope information obtaining module 602, configured to obtain gyroscope information corresponding to each image to be stitched;
the feature point matching module 603 is configured to perform image feature point matching on adjacent images to be stitched according to the gyroscope information;
the gyroscope information optimizing module 604 is configured to optimize gyroscope information corresponding to each image to be stitched according to the image feature point matching result;
a vertical seam line determining module 605, configured to determine a vertical seam line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction;
the image splicing module 606 is configured to splice the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction;
and the image block splicing module 607 is configured to splice the image blocks to be spliced obtained by splicing in the vertical directions according to the horizontal seam line.
In one embodiment, the apparatus 600 further comprises: the camera internal reference information acquisition module is used for acquiring the corresponding camera internal reference information when the images to be spliced are shot; and the gyroscope information optimizing module 604 is further configured to optimize gyroscope information corresponding to each image to be stitched based on the image feature point matching result and the camera internal reference information corresponding to each image to be stitched.
In an embodiment, the gyroscope information optimizing module 604 is further configured to optimize, based on the image feature point matching result and the camera internal reference information corresponding to each image to be stitched, gyroscope information corresponding to each image to be stitched by using a least square method.
In an embodiment, the vertical seam line determining module 605 is further configured to map each image to be stitched in the same vertical direction to a spherical coordinate system according to the optimized gyroscope information corresponding to each image to be stitched in the same vertical direction; and determining a vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the overlapping area of each image pair to be spliced adjacent in the same vertical direction in the spherical coordinate system.
In one embodiment, the vertical seam line determining module 605 is further configured to use, by a dynamic planning algorithm, the seam line with the smallest energy accumulation in each vertically adjacent pair of images to be stitched based on a preset energy function as the vertical seam line.
In one embodiment, the apparatus 600 further comprises: the south pole image acquisition module is used for acquiring a south pole image to be spliced; the image block splicing module 607 is further configured to splice the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse seam line to obtain a first panoramic image without a south pole; and obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
In an embodiment, the image obtaining module 601 is further configured to extract an image feature point set of each image to be stitched through a SIFT algorithm.
In one embodiment, the apparatus 600 further comprises: the transverse seam line determining module is used for mapping the image blocks to be spliced to a spherical coordinate system according to the gyroscope information of the image blocks to be spliced; and determining a transverse joint line of each transversely adjacent image block pair to be spliced according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system.
In one embodiment, the transverse seam line determining module is further configured to use, as the transverse seam line, a seam line with the smallest energy accumulation in each pair of laterally adjacent image blocks to be stitched based on a preset energy function through a dynamic planning algorithm.
The modules in the device for splicing panoramic images can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 7. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer equipment also comprises an input/output interface, wherein the input/output interface is a connecting circuit for exchanging information between the processor and external equipment, and is connected with the processor through a bus, and the input/output interface is called an I/O interface for short. The computer program is executed by a processor to implement a method of stitching panoramic images. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced; acquiring gyroscope information corresponding to each image to be spliced; image feature point matching is carried out on adjacent images to be spliced according to gyroscope information; optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result; determining a vertical joint line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction; and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse joint lines.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
acquiring corresponding camera internal reference information when shooting each image to be spliced;
and optimizing gyroscope information corresponding to each image to be spliced based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
and optimizing gyroscope information corresponding to each image to be spliced by a least square method based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
mapping each image to be spliced in the same vertical direction to a spherical coordinate system according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; and determining the vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the overlapping area of each image pair to be spliced adjacent in the same vertical direction in the spherical coordinate system.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
and taking the seam line with the minimum energy accumulation in each vertically adjacent image pair to be spliced as a vertical seam line based on a preset energy function through a dynamic programming algorithm.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
obtaining a south pole image to be spliced;
splicing image blocks to be spliced obtained by splicing in the vertical directions according to a transverse seam line to obtain a first panoramic image without a south pole; and obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
and extracting an image feature point set of each image to be spliced by an SIFT algorithm.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
mapping the image blocks to be spliced to a spherical coordinate system according to the gyroscope information of the image blocks to be spliced; and determining a transverse joint line of each transversely adjacent image block pair to be spliced according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
and taking the seam line with the minimum energy accumulation in each transversely adjacent image block pair to be spliced as a transverse seam line on the basis of a preset energy function through a dynamic programming algorithm.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced; acquiring gyroscope information corresponding to each image to be spliced; image feature point matching is carried out on adjacent images to be spliced according to gyroscope information; optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result; determining a vertical joint line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction; and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse joint lines.
In one embodiment, the computer program when executed by the processor further performs the steps of:
acquiring corresponding camera internal reference information when shooting each image to be spliced;
and optimizing gyroscope information corresponding to each image to be spliced based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and optimizing gyroscope information corresponding to each image to be spliced by a least square method based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, the computer program when executed by the processor further performs the steps of:
mapping each image to be spliced in the same vertical direction to a spherical coordinate system according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; and determining the vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the overlapping area of each image pair to be spliced adjacent in the same vertical direction in the spherical coordinate system.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and taking the seam line with the minimum energy accumulation in each vertically adjacent image pair to be spliced as a vertical seam line based on a preset energy function through a dynamic programming algorithm.
In one embodiment, the computer program when executed by the processor further performs the steps of:
obtaining a south pole image to be spliced;
splicing image blocks to be spliced obtained by splicing in the vertical directions according to a transverse joint line to obtain a first panoramic image without a south pole; and obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and extracting an image feature point set of each image to be spliced through an SIFT algorithm.
In one embodiment, the computer program when executed by the processor further performs the steps of:
mapping the image blocks to be spliced to a spherical coordinate system according to the gyroscope information of the image blocks to be spliced; and determining a transverse joint line of each transversely adjacent image block pair to be spliced according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and taking the seam line with the minimum energy accumulation in each transversely adjacent image block pair to be spliced as a transverse seam line on the basis of a preset energy function through a dynamic programming algorithm.
In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, performs the steps of:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced; acquiring gyroscope information corresponding to each image to be spliced; image feature point matching is carried out on adjacent images to be spliced according to gyroscope information; optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result; determining a vertical joint line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction; and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse joint lines.
In one embodiment, the computer program when executed by the processor further performs the steps of:
acquiring corresponding camera internal reference information when shooting each image to be spliced;
and optimizing gyroscope information corresponding to each image to be spliced based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and optimizing gyroscope information corresponding to each image to be spliced by a least square method based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, the computer program when executed by the processor further performs the steps of:
mapping each image to be spliced in the same vertical direction to a spherical coordinate system according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; and determining the vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the overlapping area of each image pair to be spliced adjacent in the same vertical direction in the spherical coordinate system.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and taking the seam line with the minimum energy accumulation in each vertically adjacent image pair to be spliced as a vertical seam line based on a preset energy function through a dynamic programming algorithm.
In one embodiment, the computer program when executed by the processor further performs the steps of:
obtaining a south pole image to be spliced;
splicing image blocks to be spliced obtained by splicing in the vertical directions according to a transverse joint line to obtain a first panoramic image without a south pole; and obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and extracting an image feature point set of each image to be spliced by an SIFT algorithm.
In one embodiment, the computer program when executed by the processor further performs the steps of:
mapping the image blocks to be spliced to a spherical coordinate system according to the gyroscope information of the image blocks to be spliced; and determining a transverse joint line of each transversely adjacent image block pair to be spliced according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and taking the seam line with the minimum energy accumulation in each transversely adjacent image block pair to be spliced as a transverse seam line on the basis of a preset energy function through a dynamic programming algorithm.
It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), Magnetic Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims (13)

1. A method of stitching panoramic images, the method comprising:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced;
acquiring gyroscope information corresponding to the images to be spliced;
performing image feature point matching on adjacent images to be spliced according to the gyroscope information;
optimizing gyroscope information corresponding to the images to be spliced according to the image feature point matching result;
determining a vertical joint line of each image pair to be spliced adjacent to the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction;
splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction;
and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse joint lines.
2. The method of claim 1,
the method further comprises the following steps:
acquiring corresponding camera internal reference information when the images to be spliced are shot;
and optimizing gyroscope information corresponding to the images to be spliced according to the image feature point matching result, wherein the gyroscope information comprises:
and optimizing gyroscope information corresponding to the images to be spliced based on the image feature point matching result and the camera internal reference information corresponding to the images to be spliced.
3. The method according to claim 2, wherein the optimizing gyroscope information corresponding to each image to be stitched based on the image feature point matching result and the camera internal reference information corresponding to each image to be stitched comprises:
and optimizing gyroscope information corresponding to the images to be spliced by a least square method based on the image feature point matching result and the camera internal reference information corresponding to the images to be spliced.
4. The method of claim 1, wherein the determining the vertical seam line of each vertically adjacent pair of images to be stitched according to the optimized gyroscope information corresponding to the images to be stitched in the same vertical direction comprises:
mapping each image to be spliced in the same vertical direction to a spherical coordinate system according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction;
and determining a vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the overlapping area of each image pair to be spliced adjacent in the same vertical direction in the spherical coordinate system.
5. The method of claim 4, wherein said determining a vertical seam line for each vertically adjacent pair of images to be stitched comprises:
and taking the seam line with the minimum energy accumulation in each vertically adjacent image pair to be spliced as the vertical seam line based on a preset energy function through a dynamic programming algorithm.
6. The method of claim 1,
the method further comprises the following steps:
obtaining a south pole image to be spliced;
splicing the image blocks to be spliced obtained by splicing in each vertical direction according to the transverse joint lines comprises the following steps:
splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse seam lines to obtain a first panoramic image without a south pole;
and obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
7. The method according to claim 1, wherein the extracting the image feature point set of each image to be stitched comprises:
and extracting the image feature point set of each image to be spliced through an SIFT algorithm.
8. The method according to claim 1, wherein before the image blocks to be spliced obtained by splicing in the vertical directions are spliced according to the transverse seam lines, the method further comprises:
mapping the image blocks to be spliced to a spherical coordinate system according to the gyroscope information of the image blocks to be spliced;
and determining a transverse joint line of each transversely adjacent image block pair to be spliced according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system.
9. The method of claim 8, wherein the determining the transverse seam line of each transversely adjacent pair of image blocks to be stitched comprises:
and taking the seam line with the minimum energy accumulation in each transversely adjacent image block pair to be spliced as the transverse seam line based on a preset energy function through a dynamic programming algorithm.
10. An apparatus for stitching panoramic images, the apparatus comprising:
the image acquisition module is used for acquiring a plurality of images to be spliced and extracting an image feature point set of each image to be spliced;
the gyroscope information acquisition module is used for acquiring gyroscope information corresponding to the images to be spliced;
the characteristic point matching module is used for matching the image characteristic points of the adjacent images to be spliced according to the gyroscope information;
the gyroscope information optimization module is used for optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result;
the vertical joint line determining module is used for determining a vertical joint line of each image pair to be spliced adjacent to the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction;
the image splicing module is used for splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction;
and the image block splicing module is used for splicing the image blocks to be spliced which are obtained by splicing in the vertical directions according to the transverse seam lines.
11. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor realizes the steps of the method of any one of claims 1 to 9 when executing the computer program.
12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 9.
13. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1 to 9 when executed by a processor.
CN202210407577.9A 2022-04-19 2022-04-19 Method, device, equipment and medium for splicing images Active CN114511448B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210407577.9A CN114511448B (en) 2022-04-19 2022-04-19 Method, device, equipment and medium for splicing images

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210407577.9A CN114511448B (en) 2022-04-19 2022-04-19 Method, device, equipment and medium for splicing images

Publications (2)

Publication Number Publication Date
CN114511448A true CN114511448A (en) 2022-05-17
CN114511448B CN114511448B (en) 2022-07-26

Family

ID=81554726

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210407577.9A Active CN114511448B (en) 2022-04-19 2022-04-19 Method, device, equipment and medium for splicing images

Country Status (1)

Country Link
CN (1) CN114511448B (en)

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102968777A (en) * 2012-11-20 2013-03-13 河海大学 Image stitching method based on overlapping region scale-invariant feather transform (SIFT) feature points
CN103685955A (en) * 2013-12-10 2014-03-26 福州瑞芯微电子有限公司 Panoramic shooting method of handheld device with double cameras and device
US20150154776A1 (en) * 2013-12-03 2015-06-04 Huawei Technologies Co., Ltd. Image splicing method and apparatus
US20150215532A1 (en) * 2014-01-24 2015-07-30 Amazon Technologies, Inc. Panoramic image capture
CN107111371A (en) * 2015-09-30 2017-08-29 华为技术有限公司 A kind of method, device and terminal for showing panoramic vision content
US10021295B1 (en) * 2013-06-03 2018-07-10 Amazon Technologies, Inc. Visual cues for managing image capture
CN110390640A (en) * 2019-07-29 2019-10-29 齐鲁工业大学 Graph cut image split-joint method, system, equipment and medium based on template
CN110969145A (en) * 2019-12-19 2020-04-07 珠海大横琴科技发展有限公司 Remote sensing image matching optimization method and device, electronic equipment and storage medium
CN111260550A (en) * 2018-12-03 2020-06-09 微鲸科技有限公司 Splicing line optimization method and equipment for panoramic video
US20200250429A1 (en) * 2017-10-26 2020-08-06 SZ DJI Technology Co., Ltd. Attitude calibration method and device, and unmanned aerial vehicle
CN112866556A (en) * 2019-11-28 2021-05-28 南京理工大学 Image stabilization method and system based on gyroscope and feature point matching
CN113112518A (en) * 2021-04-19 2021-07-13 深圳思谋信息科技有限公司 Feature extractor generation method and device based on spliced image and computer equipment

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102968777A (en) * 2012-11-20 2013-03-13 河海大学 Image stitching method based on overlapping region scale-invariant feather transform (SIFT) feature points
US10021295B1 (en) * 2013-06-03 2018-07-10 Amazon Technologies, Inc. Visual cues for managing image capture
US20150154776A1 (en) * 2013-12-03 2015-06-04 Huawei Technologies Co., Ltd. Image splicing method and apparatus
CN103685955A (en) * 2013-12-10 2014-03-26 福州瑞芯微电子有限公司 Panoramic shooting method of handheld device with double cameras and device
US20150215532A1 (en) * 2014-01-24 2015-07-30 Amazon Technologies, Inc. Panoramic image capture
CN107111371A (en) * 2015-09-30 2017-08-29 华为技术有限公司 A kind of method, device and terminal for showing panoramic vision content
US20200250429A1 (en) * 2017-10-26 2020-08-06 SZ DJI Technology Co., Ltd. Attitude calibration method and device, and unmanned aerial vehicle
CN111260550A (en) * 2018-12-03 2020-06-09 微鲸科技有限公司 Splicing line optimization method and equipment for panoramic video
CN110390640A (en) * 2019-07-29 2019-10-29 齐鲁工业大学 Graph cut image split-joint method, system, equipment and medium based on template
CN112866556A (en) * 2019-11-28 2021-05-28 南京理工大学 Image stabilization method and system based on gyroscope and feature point matching
CN110969145A (en) * 2019-12-19 2020-04-07 珠海大横琴科技发展有限公司 Remote sensing image matching optimization method and device, electronic equipment and storage medium
CN113112518A (en) * 2021-04-19 2021-07-13 深圳思谋信息科技有限公司 Feature extractor generation method and device based on spliced image and computer equipment

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
张勇: "实战无人机360全景摄影制作", 《影视制作》 *

Also Published As

Publication number Publication date
CN114511448B (en) 2022-07-26

Similar Documents

Publication Publication Date Title
JP5116416B2 (en) Panorama video generation apparatus and method
US10726580B2 (en) Method and device for calibration
CN109308678B (en) Method, device and equipment for repositioning by using panoramic image
CN113240769B (en) Spatial link relation identification method and device and storage medium
CN111754579A (en) Method and device for determining external parameters of multi-view camera
CN109448088B (en) Method and device for rendering three-dimensional graphic wire frame, computer equipment and storage medium
JP7220785B2 (en) Survey sampling point planning method, device, control terminal and storage medium
CN113034347B (en) Oblique photography image processing method, device, processing equipment and storage medium
CN114742703A (en) Method, device and equipment for generating binocular stereoscopic panoramic image and storage medium
CN114511448B (en) Method, device, equipment and medium for splicing images
CN113963072B (en) Binocular camera calibration method and device, computer equipment and storage medium
CN116758206A (en) Vector data fusion rendering method and device, computer equipment and storage medium
CN114022518A (en) Method, device, equipment and medium for acquiring optical flow information of image
CN114663449A (en) Road defect area calculation method and device, computer equipment and storage medium
CN114626991B (en) Image splicing method, device, equipment and medium
Wang et al. Comparison of the camera calibration between photogrammetry and computer vision
CN115601451B (en) External parameter data calibration method, device, computer equipment and storage medium
CN111970504A (en) Display method, device and system for reversely simulating three-dimensional sphere by utilizing virtual projection
CN114185429B (en) Gesture key point positioning or gesture estimating method, electronic device and storage medium
CN116612067B (en) Method, apparatus, device and computer readable storage medium for checking aviation quality
CN117392317A (en) Live three-dimensional modeling method, device, computer equipment and storage medium
CN113012302A (en) Three-dimensional panorama generation method and device, computer equipment and storage medium
CN116295031A (en) Sag measurement method, sag measurement device, computer equipment and storage medium
CN115272470A (en) Camera positioning method and device, computer equipment and storage medium
CN116977386A (en) House point cloud data processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant