CN114511448B - Method, device, equipment and medium for splicing images - Google Patents

Method, device, equipment and medium for splicing images Download PDF

Info

Publication number
CN114511448B
CN114511448B CN202210407577.9A CN202210407577A CN114511448B CN 114511448 B CN114511448 B CN 114511448B CN 202210407577 A CN202210407577 A CN 202210407577A CN 114511448 B CN114511448 B CN 114511448B
Authority
CN
China
Prior art keywords
image
spliced
images
vertical direction
adjacent
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210407577.9A
Other languages
Chinese (zh)
Other versions
CN114511448A (en
Inventor
陈一航
胡思行
蒋念娟
沈小勇
吕江波
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Simou Intelligent Technology Co ltd
Shenzhen Smartmore Technology Co Ltd
Original Assignee
Beijing Simou Intelligent Technology Co ltd
Shenzhen Smartmore Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Simou Intelligent Technology Co ltd, Shenzhen Smartmore Technology Co Ltd filed Critical Beijing Simou Intelligent Technology Co ltd
Priority to CN202210407577.9A priority Critical patent/CN114511448B/en
Publication of CN114511448A publication Critical patent/CN114511448A/en
Application granted granted Critical
Publication of CN114511448B publication Critical patent/CN114511448B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)

Abstract

The application relates to the technical field of image processing, and provides a method, a device, computer equipment, a storage medium and a computer program product for splicing panoramic images. The method and the device can improve the speed of splicing the panoramic image. The method comprises the following steps: acquiring a plurality of images to be spliced, extracting an image feature point set of each image to be spliced, acquiring gyroscope information corresponding to each image to be spliced, performing image feature point matching on adjacent images to be spliced according to the gyroscope information, optimizing the gyroscope information corresponding to each image to be spliced according to an image feature point matching result, determining a vertical seam line of each image pair to be spliced adjacent to the same vertical direction according to the optimized gyroscope information, splicing the images to be spliced to the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent to the same vertical direction, and splicing image blocks to be spliced obtained by splicing in each vertical direction according to the horizontal seam lines.

Description

Method, device, equipment and medium for splicing images
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a method, an apparatus, a computer device, a storage medium, and a computer program product for stitching panoramic images.
Background
With the development of image processing technology, the technology of stitching panoramic images becomes an important image processing technology. At present, unmanned aerial vehicle is applied to in more and more fields as important shooting instrument, carries out panorama image's concatenation like many high resolution ratio images that unmanned aerial vehicle shot, can provide directly perceived and comprehensive image information for people.
The traditional technology is generally used for splicing panoramic images through a global optimization algorithm, but the speed of splicing the panoramic images through the technology is low.
Disclosure of Invention
In view of the above, it is necessary to provide a method, an apparatus, a computer device, a computer readable storage medium and a computer program product for stitching panoramic images.
In a first aspect, the present application provides a method for stitching panoramic images. The method comprises the following steps:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced;
acquiring gyroscope information corresponding to each image to be spliced;
matching image feature points of adjacent images to be spliced according to gyroscope information;
optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result;
determining a vertical joint line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction;
splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction;
and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse seam lines.
In one of the embodiments, the first and second electrodes are,
the method further comprises the following steps:
acquiring corresponding camera internal reference information when shooting each image to be spliced;
optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result, wherein the gyroscope information comprises the following steps:
and optimizing gyroscope information corresponding to each image to be spliced based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, optimizing gyroscope information corresponding to each image to be stitched based on the image feature point matching result and camera internal reference information corresponding to each image to be stitched comprises:
and optimizing gyroscope information corresponding to each image to be spliced by a least square method based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, determining a vertical seam line of each pair of images to be stitched adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be stitched in the same vertical direction includes:
mapping each image to be spliced in the same vertical direction to a spherical coordinate system according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction;
and determining a vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the overlapping area of each image pair to be spliced adjacent in the same vertical direction in the spherical coordinate system.
In one embodiment, determining a vertical seam line of each vertically adjacent pair of images to be stitched comprises:
and taking the seam line with the minimum energy accumulation in each vertically adjacent image pair to be spliced as a vertical seam line based on a preset energy function through a dynamic programming algorithm.
In one of the embodiments, the first and second parts of the device,
the method further comprises the following steps:
obtaining an Antarctic image to be spliced;
splicing the image blocks to be spliced obtained by splicing in each vertical direction according to the transverse joint lines, and the splicing method comprises the following steps:
splicing image blocks to be spliced obtained by splicing in the vertical directions according to a transverse joint line to obtain a first panoramic image without a south pole;
and obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
In one embodiment, extracting an image feature point set of each image to be stitched comprises:
and extracting an image feature point set of each image to be spliced by an SIFT algorithm.
In one embodiment, before the image blocks to be spliced obtained by splicing in the vertical directions are spliced according to the transverse seam lines, the method further comprises the following steps:
mapping the image blocks to be spliced to a spherical coordinate system according to the gyroscope information of the image blocks to be spliced;
and determining a transverse joint line of each transversely adjacent image block pair to be spliced according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system.
In one embodiment, determining a transverse seam line of each transversely adjacent pair of image blocks to be stitched comprises:
and taking the seam line with the minimum energy accumulation in each transversely adjacent image block pair to be spliced as a transverse seam line based on a preset energy function through a dynamic programming algorithm.
In a second aspect, the application further provides a device for splicing panoramic images. The device comprises:
the image acquisition module is used for acquiring a plurality of images to be spliced and extracting an image feature point set of each image to be spliced;
the gyroscope information acquisition module is used for acquiring gyroscope information corresponding to the images to be spliced;
the characteristic point matching module is used for matching the image characteristic points of the adjacent images to be spliced according to the gyroscope information;
the gyroscope information optimization module is used for optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result;
the vertical seam line determining module is used for determining a vertical seam line of each image pair to be spliced adjacent to the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction;
the image splicing module is used for splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction;
and the image block splicing module is used for splicing the image blocks to be spliced which are obtained by splicing in the vertical directions according to the transverse seam lines.
In a third aspect, the application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the following steps when executing the computer program:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced; acquiring gyroscope information corresponding to each image to be spliced; matching image feature points of adjacent images to be spliced according to gyroscope information; optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result; determining a vertical joint line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction; and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse seam lines.
In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced; acquiring gyroscope information corresponding to each image to be spliced; matching image feature points of adjacent images to be spliced according to gyroscope information; optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result; determining a vertical joint line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction; and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse joint lines.
In a fifth aspect, the present application further provides a computer program product. The computer program product comprising a computer program which when executed by a processor performs the steps of:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced; acquiring gyroscope information corresponding to each image to be spliced; matching image feature points of adjacent images to be spliced according to gyroscope information; optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result; determining a vertical joint line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction; and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse seam lines.
The method, the device, the computer equipment, the storage medium and the computer program product for splicing panoramic images are used for acquiring a plurality of images to be spliced, extracting an image feature point set of each image to be spliced, acquiring gyroscope information corresponding to each image to be spliced, performing image feature point matching on adjacent images to be spliced according to the gyroscope information, optimizing the gyroscope information corresponding to each image to be spliced according to an image feature point matching result, determining a vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction, splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction, and splicing image blocks to be spliced obtained by splicing in each vertical direction according to the horizontal seam lines. The scheme acquires a plurality of images to be spliced, extracts an image feature point set of each image to be spliced, acquires gyroscope information corresponding to the images to be spliced when an unmanned aerial vehicle shoots each image to be spliced, organizes the position relationship of each image to be spliced according to the gyroscope information, matches the image feature points of each image to be spliced which is vertically adjacent or horizontally adjacent aiming at the images to be spliced which are vertically adjacent or horizontally adjacent, optimizes the gyroscope information of each image to be spliced according to the matching result, determines the vertical seam line of each image to be spliced which is vertically adjacent according to the optimized gyroscope information, splices each image to be spliced which is vertically adjacent according to the vertical seam lines to obtain each vertical image block, splices each vertical image block according to the horizontal seam lines to obtain a panoramic image, thereby realizing the improvement of the speed of splicing the panoramic image, furthermore, the position relation of the images to be spliced is determined in advance according to the gyroscope information, images do not need to be matched pairwise and two pairwise joint lines do not need to be searched, the calculation complexity is greatly reduced, the algorithm precision can be improved by optimizing the gyroscope information instead of optimizing a homography matrix in the traditional technology, and a better panoramic image splicing effect is achieved.
Drawings
FIG. 1 is a schematic flow chart diagram of a method for stitching panoramic images in one embodiment;
FIG. 2 is a schematic flow chart of a method for stitching panoramic images in another embodiment;
FIG. 3 is a flowchart illustrating the steps of optimizing gyroscope information corresponding to each image to be stitched in one embodiment;
FIG. 4 is a flowchart illustrating the steps for determining a vertical seam line for each vertically adjacent pair of images to be stitched in one embodiment;
FIG. 5 is a flowchart illustrating the steps for obtaining a second panoramic image, in accordance with one embodiment;
FIG. 6 is a block diagram of an apparatus for stitching panoramic images in one embodiment;
FIG. 7 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.
In an embodiment, as shown in fig. 1, a method for stitching panoramic images is provided, and this embodiment is exemplified by applying the method to a terminal, and includes the following steps:
and S101, acquiring a plurality of images to be spliced and extracting an image characteristic point set of each image to be spliced.
In this step, the image to be stitched may be an image shot by an unmanned aerial vehicle.
Specifically, the terminal acquires a plurality of images to be spliced and extracts an image feature point set of each image to be spliced.
Exemplarily, as shown in fig. 2, the terminal reads a plurality of images to be stitched according to the shooting sequence of the unmanned aerial vehicle, and performs image feature point set and descriptor extraction on each image to be stitched.
And S102, acquiring gyroscope information corresponding to each image to be spliced.
The gyroscope information can be the gyroscope information corresponding to the unmanned aerial vehicle when shooting each image to be spliced.
Specifically, the terminal acquires gyroscope information corresponding to shooting of each image to be spliced.
For example, as shown in fig. 2, the terminal may acquire gyroscope information corresponding to each image to be stitched when the unmanned aerial vehicle shoots the images to be stitched in real time, where the gyroscope information may be shown in the following table:
Figure 851784DEST_PATH_IMAGE002
the unmanned aerial vehicle can shoot according to the shooting sequence of the upper table, each image to be spliced is divided into 8 groups of vertical shooting images and 2 south pole images, the beginning number in the table is the shooting sequence, and the content of brackets is a preset shooting angle (roll, pitch, yaw), namely gyroscope information.
And S103, matching image feature points of adjacent images to be spliced according to gyroscope information.
In this step, the adjacent images to be stitched may be adjacent images to be stitched that are shot by an unmanned aerial vehicle, the adjacent images to be stitched may refer to vertical adjacent or horizontal adjacent, and the adjacent images to be stitched may refer to images to be stitched that have an overlapping region between two images, for example, for image 1, the adjacent images of image 1 are image 2 and image 7, and for image 2, the adjacent images of image 2 are image 1, image 3, and image 6.
Specifically, the terminal performs image feature point matching on adjacent images to be spliced according to gyroscope information. Illustratively, the terminal performs image feature point matching on horizontally adjacent or vertically adjacent images to be evaluated according to gyroscope information, for example, for an image 2, the image 2 performs image feature point matching with the images 1, 3 and 6. Optionally, after the acquisition of a certain group of vertically shot images is completed, image feature point matching can be performed group by group, and when the acquisition of the group of vertically shot images 1 (images 1, 2 and 3) is completed, for image 1, image 1 is image feature point matched with image 2, for image 2, image 2 is image feature point matched with images 1, 3, for the image 3, the image 3 and the image 2 are subjected to image feature point matching, so that the image feature point matching is completed by vertically shooting 1 group of images, and then when the images (images 5, 6 and 7) of the vertically shooting 2 groups are acquired, for image 5, image 5 is image feature point matched to images 3, 6, for image 6, image 6 is image feature point matched to images 2, 5, 7, for the image 7, the image 7 is matched with the images 1 and 6 by image feature points, and the matching of the image feature points of all the adjacent images to be spliced can be finished by the same way.
And S104, optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result.
In this step, for example, a certain image pair to be stitched (for example, the image a and the image B are adjacent images to be stitched, and the image a and the image B form an image pair), the image feature point matching result may be a matching result formed by corresponding feature points of the image a in the image B, for example, p i For a certain point in image A homogeneous coordinate (x) i ，y i ，1），p j Is p i Homogeneous coordinates (x) of matching points in image B j ，y j ，1）。
Specifically, the terminal optimizes gyroscope information corresponding to each image to be spliced according to the image feature point matching result. Illustratively, the terminal can optimize gyroscope information corresponding to each image to be stitched group by group (vertical shooting image group) according to the image feature point matching result, for example, the gyroscope information of the vertical shooting 1 group (images 1, 2 and 3) is optimized according to the image feature point matching result of the vertical shooting 1 group (images 1, 2 and 3) and the corresponding adjacent image to be stitched, the gyroscope information of the vertical shooting 2 group (images 5, 6 and 7) is optimized according to the image feature point matching result of the vertical shooting 2 group (images 5, 6 and 7) and the corresponding adjacent image to be stitched, and the optimization of the gyroscope information corresponding to all the images to be stitched can be finished by analogy.
And S105, determining a vertical joint line of each image pair to be spliced adjacent to the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction.
In this step, for each pair of images to be stitched that are vertically adjacent to each other, the vertical seam line may be a vertical seam line between image 1 and image 2 of the group of vertically shot 1 in the above table, and a vertical seam line between image 2 and image 3.
Specifically, the terminal determines a vertical seam line of each image pair to be spliced adjacent to the same vertical direction according to optimized gyroscope information corresponding to each image to be spliced in the same vertical direction.
And S106, splicing the images to be spliced in the same vertical direction according to the vertical seam line of each adjacent image pair to be spliced in the same vertical direction.
Specifically, as shown in fig. 2, the terminal splices the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction, so as to obtain the image blocks to be spliced in each vertical direction.
And S107, splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse seam lines.
In this step, the horizontal seam line may be a horizontal seam line between the image blocks to be spliced obtained by vertically splicing each adjacent image block, for example, a certain horizontal seam line may be a seam line between the vertically spliced vertical racket 1 group and the vertically spliced vertical racket 2 group in the table.
Specifically, the terminal splices the image blocks to be spliced obtained by splicing in each vertical direction according to the transverse seam lines.
The method for splicing the panoramic images comprises the steps of obtaining a plurality of images to be spliced, extracting an image feature point set of each image to be spliced, obtaining gyroscope information corresponding to each image to be spliced, carrying out image feature point matching on adjacent images to be spliced according to the gyroscope information, optimizing the gyroscope information corresponding to each image to be spliced according to an image feature point matching result, determining a vertical seam line of each image pair to be spliced adjacent to the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced on the same vertical direction, splicing the images to be spliced on the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent to the same vertical direction, and splicing image blocks to be spliced obtained by splicing in each vertical direction according to the horizontal seam lines. The scheme acquires a plurality of images to be spliced, extracts an image feature point set of each image to be spliced, acquires gyroscope information corresponding to each image to be spliced when an unmanned aerial vehicle shoots each image to be spliced, organizes the position relation of each image to be spliced according to the gyroscope information, matches image feature points of each image to be spliced which is vertically adjacent or horizontally adjacent aiming at each image to be spliced which is vertically adjacent or horizontally adjacent, optimizes the gyroscope information of each image to be spliced according to the matching result, determines a vertical seam line of each image to be spliced which is vertically adjacent according to the optimized gyroscope information, splices each image to be spliced according to the vertical seam line to obtain each vertical image block, splices each vertical image block according to the horizontal seam line to obtain a panoramic image, thereby realizing the improvement of the speed of splicing the panoramic image, furthermore, the position relation of each image to be spliced is determined in advance according to gyroscope information, images do not need to be matched pairwise and joint lines do not need to be searched pairwise, the calculation complexity is greatly reduced, algorithm precision can be improved through optimizing the gyroscope information instead of optimizing a homography matrix of the traditional technology, and a better panoramic image splicing effect is achieved.
In an embodiment, as shown in fig. 3, the method may further optimize gyroscope information corresponding to each image to be stitched in the same vertical direction by the following steps, specifically including: s301, acquiring corresponding camera internal reference information when shooting each image to be spliced; and S302, optimizing gyroscope information corresponding to each image to be spliced based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In this embodiment, the camera internal reference information may be a camera internal reference matrix.
Specifically, the terminal acquires camera internal reference information corresponding to shooting each image to be spliced, and gyroscope information corresponding to each image to be spliced can be optimized through a least square method based on an image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
For example, as shown in fig. 2, when the terminal acquires the images to be stitched, the terminal acquires the camera internal reference information and the gyroscope information corresponding to the images to be stitched, and the camera internal reference information and the gyroscope information are acquired according to formula 1
Figure 771198DEST_PATH_IMAGE004
Optimizing gyroscope information corresponding to each image to be spliced, wherein n is the number of matching points of an image pair (the image pair is an image pair formed by a certain image to be spliced and another image to be spliced adjacent to the image to be spliced), a is the serial number of the matching points of the image pair, K is a camera internal reference matrix corresponding to the image to be spliced, R is a rotation matrix of the gyroscope information, for example, the image A and the image B form an adjacent (adjacent can be vertically adjacent or transversely adjacent) certain image pair to be spliced, and p is i For a certain point in image A homogeneous coordinate (x) i ，y i ，1），p j Is p i Homogeneous coordinates (x) of matching points in image B j ，y j ，1），K i And R i To correspond to K and R, K of image A j And R j For K and R corresponding to image B, L2 is the L2 norm, also known as the least squares method, where the rotation matrix R of the gyroscope information is based on the gyroscope information as given by equation 2
Figure 465485DEST_PATH_IMAGE006
Where α is gyroscope yaw, β is gyroscope pitch, γ is gyroscope roll, for example, the gyroscope information (roll, pitch, yaw) of an image to be stitched is (0, 15, 45), then R is z （α）R y （β）R x (γ) a rotation matrix rotated by an angle of 0 degrees around the z-axis, a rotation matrix rotated by an angle of 15 degrees around the y-axis, and a rotation matrix rotated by an angle of 45 degrees around the x-axis, respectively; the terminal calculates the value of the rotation matrix R according to formula 1 by adjusting the rotation matrix R,
Figure 461254DEST_PATH_IMAGE008
and finally, the rotation matrix obtained through calculation is used as the rotation matrix corresponding to the optimized gyroscope information, so that the optimized gyroscope information is obtained through calculation according to the rotation matrix corresponding to the optimized gyroscope information.
According to the technical scheme, the gyroscope information corresponding to the images to be spliced in the same vertical direction is optimized based on the image feature point matching result and the camera internal reference information, so that the optimized gyroscope information corresponding to the images to be spliced in the same vertical direction is obtained more accurately, and the follow-up better panoramic image splicing effect is achieved.
In an embodiment, as shown in fig. 4, the determining, according to the optimized gyroscope information corresponding to the images to be stitched in the same vertical direction in step S105, a vertical seam line of each pair of images to be stitched that are adjacent in the same vertical direction specifically includes: step S401, mapping each image to be spliced in the same vertical direction to a spherical coordinate system according to optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; step S402, determining a vertical joint line of each pair of images to be spliced adjacent in the same vertical direction according to the overlapping area of each pair of images to be spliced adjacent in the same vertical direction in the spherical coordinate system.
Specifically, the terminal maps each image to be spliced in the same vertical direction to a spherical coordinate system according to optimized gyroscope information corresponding to each image to be spliced in the same vertical direction, and determines a vertical joint line of each image to be spliced adjacent in the same vertical direction according to an overlapping region of each image to be spliced adjacent in the same vertical direction in the spherical coordinate system.
According to the technical scheme, the vertical seam line of each image pair to be spliced adjacent to the same vertical direction in the spherical coordinate system is determined according to the overlapping area of each image pair to be spliced adjacent to the same vertical direction in the spherical coordinate system, so that the vertical seam line of each image pair to be spliced adjacent to the same vertical direction can be determined more quickly and accurately, and the panoramic image can be spliced more quickly and accurately.
In one embodiment, the method may further determine a vertical seam line of each vertically adjacent pair of images to be spliced by: and taking the seam line with the minimum energy accumulation in each vertically adjacent image pair to be spliced as a vertical seam line based on a preset energy function through a dynamic programming algorithm.
In this embodiment, the preset energy function may be energy = energy func (i 1, i 2), where energy is a defined function and may be a gradient, energy is an energy function of an overlapping portion of the image pairs to be stitched, i1 and i2 represent the overlapping portions of the image pairs to be stitched, for example,
Figure 975412DEST_PATH_IMAGE010
，
wherein, G x1 =G x （GRAY1），G x2 =G x （GRAY2），G y1 =G y （GRAY1），G y2 =G y (GRAY 2) wherein GR isAY1=i1 Blue *a 1 +i1 green *b 1 +i1 red *c 1 ，a 1 + b 1 + c 1 =1，
GRAY2=i2 Blue * a 2 +i2 green * b 2 +i2 red * c 2 ，a 2 + b 2 + c 2 =1, wherein,
Figure 964096DEST_PATH_IMAGE012
，
wherein, A in the formula is GRAY1 or GRAY 2; minimum energy accumulation can be expressed by
Cost i，j =energy i，j +min（Cost i-1，j ，Cost i，j-1 ，Cost i-1，j-1 ) The result of the calculation is that,
wherein cost represents the minimum energy accumulation in all paths to pixel point (i, j), which may be an image coordinate, energy i，j Is the energy value of pixel (i, j).
Specifically, the terminal uses a seam line with the minimum energy accumulation in each vertically adjacent pair of images to be spliced as a vertical seam line based on a preset energy function through a dynamic programming algorithm.
According to the technical scheme, the seam line with the minimum energy accumulation in each pair of images to be spliced adjacent to the same vertical direction is used as the vertical seam line, so that the optimal vertical seam line of each pair of images to be spliced adjacent to the same vertical direction can be determined more accurately, and the panoramic images can be spliced more accurately.
In an embodiment, as shown in fig. 5, the method may further obtain the second panoramic image by the following steps, specifically including: step S501, obtaining an Antarctic image to be spliced; step S502, splicing image blocks to be spliced obtained by splicing in the vertical directions according to a transverse seam line to obtain a first panoramic image without a south pole; and S503, obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
In this embodiment, the south pole images to be stitched may be the south pole 1 (0, 75, 0) image and the south pole 2 (0, 75, 180) image in the above table.
Specifically, the terminal acquires the south pole images to be spliced, splicing image blocks to be spliced obtained by splicing in each vertical direction according to the transverse seam lines to obtain a first panoramic image without the south pole, and obtaining a complete second panoramic image according to the south pole images to be spliced and the first panoramic image.
Exemplarily, as shown in fig. 2, the terminal obtains a south pole image to be spliced, splices image blocks to be spliced obtained by splicing in each vertical direction according to a transverse seam line to obtain a first panoramic image without a south pole, and obtains a complete second panoramic image through a seam finding algorithm and a fusion algorithm according to the south pole image to be spliced and the first panoramic image.
According to the technical scheme of the embodiment, the complete second panoramic image is obtained according to the south pole image to be spliced and the first panoramic image, so that the splicing is facilitated to obtain a more complete panoramic image.
In an embodiment, the extracting of the image feature point set of each image to be stitched in step S101 specifically includes: and extracting an image feature point set of each image to be spliced through an SIFT algorithm.
SIFT is Scale-invariant feature transform (SIFT).
Specifically, the terminal extracts an image feature point set of each image to be spliced through an SIFT algorithm.
Exemplarily, as shown in fig. 2, the terminal performs image feature point set and descriptor extraction on each image to be stitched through a SIFT algorithm.
According to the technical scheme, the image feature point sets of the images to be spliced are extracted through the SIFT algorithm, so that the accuracy of extracting the image feature points is improved, and the follow-up realization of a better panoramic image splicing effect is facilitated.
In an embodiment, the method may further determine a transverse seam line of each transversely adjacent pair of image blocks to be stitched by: mapping the image blocks to be spliced to a spherical coordinate system according to gyroscope information of the image blocks to be spliced; and determining a transverse seam line of each transversely adjacent image block pair to be spliced according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system.
In this embodiment, each image block pair to be spliced that is horizontally adjacent may refer to each image block pair to be spliced that has an overlapping region between image blocks to be spliced that are obtained by vertically splicing, for example, the image block pair to be spliced is a group of vertically-shot 1 (images 1, 2, and 3 after vertically splicing) obtained by vertically splicing and a group of vertically-shot 2 (images 5, 6, and 7 after vertically splicing).
According to the technical scheme of the embodiment, the transverse seam line of each transversely adjacent image block pair to be spliced is determined according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system, so that the transverse seam line can be determined more quickly and accurately, and the panoramic image can be spliced more quickly and accurately subsequently.
In one embodiment, the method may further include determining a transverse seam line by: and taking the seam line with the minimum energy accumulation in each transversely adjacent image block pair to be spliced as a transverse seam line on the basis of a preset energy function through a dynamic programming algorithm.
In this embodiment, the preset energy function may be energy = energy func (i 1, i 2).
According to the technical scheme of the embodiment, the dynamic programming algorithm and the preset energy function are used for taking the seam line with the minimum energy accumulation in each transversely adjacent image block pair to be spliced as the transverse seam line, so that the optimal transverse seam line of each image block pair to be spliced can be determined more accurately, and the panoramic image can be spliced more accurately in the follow-up process.
The following application example illustrates that the method provided by the present application can be applied to panoramic image stitching of images shot by an unmanned aerial vehicle, as shown in fig. 2, the main steps include:
the method comprises the following steps: the terminal reads images to be spliced sent by the unmanned aerial vehicle and corresponding camera internal reference information and gyroscope information when shooting the images to be spliced in real time; the images to be spliced are divided into 8 groups of vertical shooting images (for example, a group of vertical shooting images comprises three images to be spliced) and two south pole images to be spliced in advance.
Step two: and the terminal extracts the image feature point set of each image to be spliced through an SIFT algorithm.
Step three: the terminal judges whether all the current group of vertical shooting images are obtained (if yes, the terminal finishes obtaining 1 group of vertical shooting images), if not, the first step is executed, and if yes, the next step is executed.
Step four: and the terminal performs image feature point matching on the adjacent images to be spliced according to the image feature point set obtained by the SIFT algorithm.
Step five: and the terminal optimizes gyroscope information corresponding to each image to be spliced according to the image feature point matching result.
Step six: and the terminal maps the images to be spliced in the same vertical direction to the panoramic image of the spherical coordinate system according to the camera internal reference information corresponding to the images to be spliced in the same vertical direction and the optimized gyroscope information.
Step seven: and the terminal determines a vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the optimized gyroscope information and the overlapped area of each image pair to be spliced adjacent in the same vertical direction in the spherical coordinate system.
Step eight: and the terminal performs fusion splicing on the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction (like a group of vertical shot images) to obtain image blocks Imagel to be spliced obtained by splicing in each vertical direction.
Step nine: and the terminal calculates the transverse joint lines among the image blocks to be spliced obtained by splicing in the vertical direction.
Step ten: and the terminal judges whether all the images to be spliced are read or not, if not, the step one is executed, and if so, the next step is executed.
Step eleven: and splicing the image blocks to be spliced obtained by splicing in the vertical directions by the terminal according to the transverse seam lines to obtain a first panoramic image without the south pole.
Step twelve: and the terminal obtains a second panoramic image through a seam finding algorithm and a fusion algorithm according to the south pole image to be spliced and the first panoramic image.
According to the application example, the shooting sequence of the unmanned aerial vehicle is preset, a mode of shooting while splicing can be adopted, the unmanned aerial vehicle only needs to splice the south pole images after shooting the images, splicing is not needed after loading all the images is completed, and the effect of quickly outputting the panoramic image after shooting the images can be achieved; furthermore, the position relation of the images can be determined in advance through gyroscope information, so that the position of the images is determined without pairwise image matching; moreover, by optimizing gyroscope information instead of homography matrixes, algorithm precision can be improved, and better panoramic image splicing effect can be obtained.
It should be understood that, although the steps in the flowcharts related to the embodiments as described above are sequentially displayed as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not limited to being performed in the exact order illustrated and, unless explicitly stated herein, may be performed in other orders. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be rotated or alternated with other steps or at least a part of the steps or stages in other steps.
Based on the same inventive concept, the embodiment of the application also provides a device for splicing the panoramic images, which is used for realizing the method for splicing the panoramic images. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the method, so specific limitations in one or more apparatus embodiments for stitching panoramic images provided below can be referred to as limitations on the method for stitching panoramic images in the foregoing, and details are not repeated here.
In one embodiment, as shown in fig. 6, there is provided an apparatus for stitching panoramic images, and the apparatus 600 may include:
the image acquisition module 601 is used for acquiring a plurality of images to be spliced and extracting an image feature point set of each image to be spliced;
a gyroscope information obtaining module 602, configured to obtain gyroscope information corresponding to each image to be stitched;
the feature point matching module 603 is configured to perform image feature point matching on the adjacent images to be stitched according to the gyroscope information;
a gyroscope information optimization module 604, configured to optimize gyroscope information corresponding to each image to be stitched according to an image feature point matching result;
a vertical seam line determining module 605, configured to determine a vertical seam line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction;
the image splicing module 606 is configured to splice the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction;
and the image block splicing module 607 is configured to splice the image blocks to be spliced obtained by splicing in the vertical directions according to the horizontal seam line.
In one embodiment, the apparatus 600 further comprises: the camera internal reference information acquisition module is used for acquiring corresponding camera internal reference information when the images to be spliced are shot; and the gyroscope information optimizing module 604 is further configured to optimize gyroscope information corresponding to each image to be stitched based on the image feature point matching result and the camera internal reference information corresponding to each image to be stitched.
In an embodiment, the gyroscope information optimizing module 604 is further configured to optimize, based on the image feature point matching result and the camera internal reference information corresponding to each image to be stitched, gyroscope information corresponding to each image to be stitched by using a least square method.
In an embodiment, the vertical seam line determining module 605 is further configured to map each image to be stitched in the same vertical direction to a spherical coordinate system according to the optimized gyroscope information corresponding to each image to be stitched in the same vertical direction; and determining a vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the overlapping area of each image pair to be spliced adjacent in the same vertical direction in the spherical coordinate system.
In one embodiment, the vertical seam line determining module 605 is further configured to use, by a dynamic planning algorithm, the seam line with the smallest energy accumulation in each vertically adjacent pair of images to be stitched based on a preset energy function as the vertical seam line.
In one embodiment, the apparatus 600 further comprises: the south pole image acquisition module is used for acquiring a south pole image to be spliced; the image block splicing module 607 is further configured to splice the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse seam line to obtain a first panoramic image without a south pole; and obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
In an embodiment, the image obtaining module 601 is further configured to extract an image feature point set of each image to be stitched through a SIFT algorithm.
In one embodiment, the apparatus 600 further comprises: the transverse seam line determining module is used for mapping the image blocks to be spliced to a spherical coordinate system according to the gyroscope information of the image blocks to be spliced; and determining a transverse joint line of each transversely adjacent image block pair to be spliced according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system.
In one embodiment, the transverse seam line determining module is further configured to use, as the transverse seam line, a seam line with the smallest energy accumulation in each pair of laterally adjacent image blocks to be stitched based on a preset energy function through a dynamic planning algorithm.
All or part of the modules in the device for splicing panoramic images can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 7. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer equipment also comprises an input/output interface, wherein the input/output interface is a connecting circuit for exchanging information between the processor and external equipment, and is connected with the processor through a bus, namely an I/O interface for short. The computer program, when executed by a processor, implements a method of stitching panoramic images. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on a shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory having a computer program stored therein and a processor that when executing the computer program performs the steps of:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced; acquiring gyroscope information corresponding to each image to be spliced; image feature point matching is carried out on adjacent images to be spliced according to gyroscope information; optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result; determining a vertical seam line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction; and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse joint lines.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
acquiring corresponding camera internal reference information when shooting each image to be spliced;
and optimizing gyroscope information corresponding to each image to be spliced based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, the processor when executing the computer program further performs the steps of:
and optimizing gyroscope information corresponding to each image to be spliced by a least square method based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, the processor when executing the computer program further performs the steps of:
mapping each image to be spliced in the same vertical direction to a spherical coordinate system according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; and determining a vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the overlapping area of each image pair to be spliced adjacent in the same vertical direction in the spherical coordinate system.
In one embodiment, the processor when executing the computer program further performs the steps of:
and taking the seam line with the minimum energy accumulation in each vertically adjacent image pair to be spliced as a vertical seam line based on a preset energy function through a dynamic programming algorithm.
In one embodiment, the processor when executing the computer program further performs the steps of:
obtaining an Antarctic image to be spliced;
splicing image blocks to be spliced obtained by splicing in the vertical directions according to a transverse joint line to obtain a first panoramic image without a south pole; and obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
In one embodiment, the processor when executing the computer program further performs the steps of:
and extracting an image feature point set of each image to be spliced through an SIFT algorithm.
In one embodiment, the processor when executing the computer program further performs the steps of:
mapping the image blocks to be spliced to a spherical coordinate system according to gyroscope information of the image blocks to be spliced; and determining a transverse joint line of each transversely adjacent image block pair to be spliced according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
and taking the seam line with the minimum energy accumulation in each transversely adjacent image block pair to be spliced as a transverse seam line on the basis of a preset energy function through a dynamic programming algorithm.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, performs the steps of:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced; acquiring gyroscope information corresponding to each image to be spliced; matching image feature points of adjacent images to be spliced according to gyroscope information; optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result; determining a vertical joint line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction; and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse joint lines.
In one embodiment, the computer program when executed by the processor further performs the steps of:
acquiring corresponding camera internal reference information when shooting each image to be spliced;
and optimizing gyroscope information corresponding to each image to be spliced based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and optimizing gyroscope information corresponding to each image to be spliced by a least square method based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, the computer program when executed by the processor further performs the steps of:
mapping each image to be spliced in the same vertical direction to a spherical coordinate system according to optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; and determining a vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the overlapping area of each image pair to be spliced adjacent in the same vertical direction in the spherical coordinate system.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and taking the seam line with the minimum energy accumulation in each vertically adjacent image pair to be spliced as a vertical seam line based on a preset energy function through a dynamic programming algorithm.
In one embodiment, the computer program when executed by the processor further performs the steps of:
obtaining a south pole image to be spliced;
splicing image blocks to be spliced obtained by splicing in the vertical directions according to a transverse joint line to obtain a first panoramic image without a south pole; and obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and extracting an image feature point set of each image to be spliced through an SIFT algorithm.
In one embodiment, the computer program when executed by the processor further performs the steps of:
mapping the image blocks to be spliced to a spherical coordinate system according to gyroscope information of the image blocks to be spliced; and determining a transverse joint line of each transversely adjacent image block pair to be spliced according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and taking the seam line with the minimum energy accumulation in each transversely adjacent image block pair to be spliced as a transverse seam line on the basis of a preset energy function through a dynamic programming algorithm.
In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, performs the steps of:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced; acquiring gyroscope information corresponding to each image to be spliced; matching image feature points of adjacent images to be spliced according to gyroscope information; optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result; determining a vertical seam line of each image pair to be spliced adjacent to each other in the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction; and splicing the image blocks to be spliced obtained by splicing in the vertical directions according to the transverse joint lines.
In one embodiment, the computer program when executed by the processor further performs the steps of:
acquiring corresponding camera internal reference information when shooting each image to be spliced;
and optimizing gyroscope information corresponding to each image to be spliced based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and optimizing gyroscope information corresponding to each image to be spliced by a least square method based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
In one embodiment, the computer program when executed by the processor further performs the steps of:
mapping each image to be spliced in the same vertical direction to a spherical coordinate system according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction; and determining the vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the overlapping area of each image pair to be spliced adjacent in the same vertical direction in the spherical coordinate system.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and taking the seam line with the minimum energy accumulation in each vertically adjacent image pair to be spliced as a vertical seam line based on a preset energy function through a dynamic programming algorithm.
In one embodiment, the computer program when executed by the processor further performs the steps of:
obtaining a south pole image to be spliced;
splicing image blocks to be spliced obtained by splicing in the vertical directions according to a transverse joint line to obtain a first panoramic image without a south pole; and obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and extracting an image feature point set of each image to be spliced through an SIFT algorithm.
In one embodiment, the computer program when executed by the processor further performs the steps of:
mapping the image blocks to be spliced to a spherical coordinate system according to gyroscope information of the image blocks to be spliced; and determining a transverse joint line of each transversely adjacent image block pair to be spliced according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and taking the seam line with the minimum energy accumulation in each transversely adjacent image block pair to be spliced as a transverse seam line based on a preset energy function through a dynamic programming algorithm.
It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, databases, or other media used in the embodiments provided herein can include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), Magnetic Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the various embodiments provided herein may be, without limitation, general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, or the like.
All possible combinations of the technical features in the above embodiments may not be described for the sake of brevity, but should be considered as being within the scope of the present disclosure as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims (11)

1. A method of stitching panoramic images, the method comprising:
acquiring a plurality of images to be spliced, and extracting an image feature point set of each image to be spliced;
acquiring gyroscope information corresponding to the images to be spliced;
matching image feature points of adjacent images to be spliced according to the gyroscope information;
optimizing gyroscope information corresponding to the images to be spliced according to the image feature point matching result;
determining a vertical joint line of each image pair to be spliced adjacent to the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction;
splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction;
splicing image blocks to be spliced obtained by splicing in the vertical directions according to a transverse seam line to obtain a first panoramic image without a south pole;
obtaining a south pole image to be spliced;
and obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
2. The method of claim 1,
the method further comprises the following steps:
acquiring corresponding camera internal reference information when the images to be spliced are shot;
and optimizing gyroscope information corresponding to the images to be spliced according to the image feature point matching result, wherein the gyroscope information comprises:
and optimizing gyroscope information corresponding to each image to be spliced based on the image feature point matching result and the camera internal reference information corresponding to each image to be spliced.
3. The method according to claim 2, wherein the optimizing gyroscope information corresponding to each image to be stitched based on the image feature point matching result and camera internal reference information corresponding to each image to be stitched comprises:
and optimizing gyroscope information corresponding to the images to be spliced by a least square method based on the image feature point matching result and the camera internal reference information corresponding to the images to be spliced.
4. The method of claim 1, wherein the determining the vertical seam line of each vertically adjacent pair of images to be stitched according to the optimized gyroscope information corresponding to the images to be stitched in the same vertical direction comprises:
mapping each image to be spliced in the same vertical direction to a spherical coordinate system according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction;
and determining a vertical seam line of each image pair to be spliced adjacent in the same vertical direction according to the overlapping area of each image pair to be spliced adjacent in the same vertical direction in the spherical coordinate system.
5. The method of claim 4, wherein the determining a vertical seam line for each vertically adjacent pair of images to be stitched comprises:
and taking the seam line with the minimum energy accumulation in each vertically adjacent image pair to be spliced as the vertical seam line based on a preset energy function through a dynamic programming algorithm.
6. The method according to claim 1, wherein the extracting the image feature point set of each image to be stitched comprises:
and extracting the image feature point set of each image to be spliced through an SIFT algorithm.
7. The method according to claim 1, wherein before the image blocks to be spliced obtained by splicing in the vertical directions are spliced according to the transverse seam lines, the method further comprises the following steps:
mapping the image blocks to be spliced to a spherical coordinate system according to the gyroscope information of the image blocks to be spliced;
and determining a transverse joint line of each transversely adjacent image block pair to be spliced according to the overlapping area of each transversely adjacent image block pair to be spliced in the spherical coordinate system.
8. The method according to claim 7, wherein the determining the transverse seam line of each transversely adjacent pair of image blocks to be stitched comprises:
and taking the seam line with the minimum energy accumulation in each transversely adjacent image block pair to be spliced as the transverse seam line on the basis of a preset energy function through a dynamic programming algorithm.
9. An apparatus for stitching panoramic images, the apparatus comprising:
the image acquisition module is used for acquiring a plurality of images to be spliced and extracting an image feature point set of each image to be spliced;
the gyroscope information acquisition module is used for acquiring gyroscope information corresponding to the images to be spliced;
the characteristic point matching module is used for matching the image characteristic points of the adjacent images to be spliced according to the gyroscope information;
the gyroscope information optimization module is used for optimizing gyroscope information corresponding to each image to be spliced according to the image feature point matching result;
the vertical joint line determining module is used for determining a vertical joint line of each image pair to be spliced adjacent to the same vertical direction according to the optimized gyroscope information corresponding to each image to be spliced in the same vertical direction;
the image splicing module is used for splicing the images to be spliced in the same vertical direction according to the vertical seam line of each image pair to be spliced adjacent in the same vertical direction;
the image block splicing module is used for splicing image blocks to be spliced which are obtained by splicing in the vertical directions according to a transverse seam line to obtain a first panoramic image without a south pole; obtaining an Antarctic image to be spliced; and obtaining a second panoramic image according to the south pole image to be spliced and the first panoramic image.
10. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 8.
11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 8.
CN202210407577.9A 2022-04-19 2022-04-19 Method, device, equipment and medium for splicing images Active CN114511448B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210407577.9A CN114511448B (en) 2022-04-19 2022-04-19 Method, device, equipment and medium for splicing images

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210407577.9A CN114511448B (en) 2022-04-19 2022-04-19 Method, device, equipment and medium for splicing images

Publications (2)

Publication Number Publication Date
CN114511448A CN114511448A (en) 2022-05-17
CN114511448B true CN114511448B (en) 2022-07-26

Family

ID=81554726

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210407577.9A Active CN114511448B (en) 2022-04-19 2022-04-19 Method, device, equipment and medium for splicing images

Country Status (1)

Country Link
CN (1) CN114511448B (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103685955A (en) * 2013-12-10 2014-03-26 福州瑞芯微电子有限公司 Panoramic shooting method of handheld device with double cameras and device
CN107111371A (en) * 2015-09-30 2017-08-29 华为技术有限公司 A kind of method, device and terminal for showing panoramic vision content
US10021295B1 (en) * 2013-06-03 2018-07-10 Amazon Technologies, Inc. Visual cues for managing image capture
CN110969145A (en) * 2019-12-19 2020-04-07 珠海大横琴科技发展有限公司 Remote sensing image matching optimization method and device, electronic equipment and storage medium
CN111260550A (en) * 2018-12-03 2020-06-09 微鲸科技有限公司 Splicing line optimization method and equipment for panoramic video
CN112866556A (en) * 2019-11-28 2021-05-28 南京理工大学 Image stabilization method and system based on gyroscope and feature point matching
CN113112518A (en) * 2021-04-19 2021-07-13 深圳思谋信息科技有限公司 Feature extractor generation method and device based on spliced image and computer equipment

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102968777B (en) * 2012-11-20 2015-01-14 河海大学 Image stitching method based on overlapping region scale-invariant feather transform (SIFT) feature points
CN104680501B (en) * 2013-12-03 2018-12-07 华为技术有限公司 The method and device of image mosaic
US20150215532A1 (en) * 2014-01-24 2015-07-30 Amazon Technologies, Inc. Panoramic image capture
WO2019080052A1 (en) * 2017-10-26 2019-05-02 深圳市大疆创新科技有限公司 Attitude calibration method and device, and unmanned aerial vehicle
CN110390640B (en) * 2019-07-29 2023-04-28 齐鲁工业大学 Template-based Poisson fusion image splicing method, system, equipment and medium

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10021295B1 (en) * 2013-06-03 2018-07-10 Amazon Technologies, Inc. Visual cues for managing image capture
CN103685955A (en) * 2013-12-10 2014-03-26 福州瑞芯微电子有限公司 Panoramic shooting method of handheld device with double cameras and device
CN107111371A (en) * 2015-09-30 2017-08-29 华为技术有限公司 A kind of method, device and terminal for showing panoramic vision content
CN111260550A (en) * 2018-12-03 2020-06-09 微鲸科技有限公司 Splicing line optimization method and equipment for panoramic video
CN112866556A (en) * 2019-11-28 2021-05-28 南京理工大学 Image stabilization method and system based on gyroscope and feature point matching
CN110969145A (en) * 2019-12-19 2020-04-07 珠海大横琴科技发展有限公司 Remote sensing image matching optimization method and device, electronic equipment and storage medium
CN113112518A (en) * 2021-04-19 2021-07-13 深圳思谋信息科技有限公司 Feature extractor generation method and device based on spliced image and computer equipment

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
实战无人机360全景摄影制作;张勇;《影视制作》;20181115(第11期);第42-46页 *

Also Published As

Publication number Publication date
CN114511448A (en) 2022-05-17

Similar Documents

Publication Publication Date Title
US10726580B2 (en) Method and device for calibration
CN109308678B (en) Method, device and equipment for repositioning by using panoramic image
CN107633526A (en) A kind of image trace point acquisition methods and equipment, storage medium
JP7220785B2 (en) Survey sampling point planning method, device, control terminal and storage medium
CN111754579A (en) Method and device for determining external parameters of multi-view camera
CN109448088B (en) Method and device for rendering three-dimensional graphic wire frame, computer equipment and storage medium
CN113034347B (en) Oblique photography image processing method, device, processing equipment and storage medium
CN106504196A (en) A kind of panoramic video joining method and equipment based on space sphere
CN113963072B (en) Binocular camera calibration method and device, computer equipment and storage medium
CN106997366B (en) Database construction method, augmented reality fusion tracking method and terminal equipment
CN114511448B (en) Method, device, equipment and medium for splicing images
CN107240149A (en) Object dimensional model building method based on image procossing
CN114022518B (en) Method, device, equipment and medium for acquiring optical flow information of image
CN116863137A (en) Optimization method and device for three-dimensional model of transmission tower and computer equipment
CN115272470A (en) Camera positioning method and device, computer equipment and storage medium
CN114255396B (en) Method, system and device for reconstructing environment of power transmission line and controller
CN114663449A (en) Road defect area calculation method and device, computer equipment and storage medium
CN114626991B (en) Image splicing method, device, equipment and medium
CN115601451B (en) External parameter data calibration method, device, computer equipment and storage medium
CN116295031B (en) Sag measurement method, sag measurement device, computer equipment and storage medium
CN116527908B (en) Motion field estimation method, motion field estimation device, computer device and storage medium
CN116442226B (en) Pose correctness judging method, pose correctness judging device, robot and storage medium
CN115953441A (en) Relative pose acquisition method and device, computer equipment and storage medium
CN117939093A (en) Projection correction method, apparatus, computer device, and storage medium
CN116664682A (en) Robot, map generation method, and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant