CN113822800B - Panoramic image splicing and fusing method and device - Google Patents
Panoramic image splicing and fusing method and device Download PDFInfo
- Publication number
- CN113822800B CN113822800B CN202110654226.3A CN202110654226A CN113822800B CN 113822800 B CN113822800 B CN 113822800B CN 202110654226 A CN202110654226 A CN 202110654226A CN 113822800 B CN113822800 B CN 113822800B
- Authority
- CN
- China
- Prior art keywords
- pyramid
- pixel
- sub
- panorama
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Abstract
The invention discloses a method and a device for splicing and fusing panoramic images, and relates to the field of computer vision and panoramic image splicing; the method specifically comprises the following steps: acquiring a bent picture corresponding to each sub-camera and a coordinate range of the bent picture on the panoramic image; calculating a definition evaluation array corresponding to each sub-camera according to the bent picture, and constructing a weight pyramid corresponding to each sub-camera according to the definition evaluation array; constructing a pixel weighting pyramid corresponding to each sub-camera according to the constructed pixel pyramid and the weight pyramid; accumulating/combining the weight pyramid and the pixel weighting pyramid corresponding to each sub-camera according to the coordinate range to obtain a panorama weighting pyramid and a panorama weighting pixel pyramid; and regularizing the panoramic image weight pyramid and the panoramic image weighted pixel pyramid to generate a panoramic image pixel pyramid, and performing image reconstruction on the panoramic image pixel pyramid to generate a spliced panoramic image.
Description
Technical Field
The disclosure relates to the field of computer vision and panorama splicing, in particular to a method and a device for splicing and fusing panoramic images.
Background
In the prior art, in order to achieve the optimal fusion effect of image splicing when a panorama is spliced, a gaussian-laplacian pyramid fusion mode is generally adopted, and this method can perform average fusion on each pixel point of a superposition region according to the pixel point value of each sub-camera at the position where any plurality of sub-camera pictures are superposed, that is, the pixel of any sub-camera appearing in the superposition region contributes to a fusion result according to the weight of 1 divided by the number of pictures. The fusion mode can actually make transition among sub-camera pictures more natural to a certain extent, but the mode cannot screen data sources of fusion results of all parts of the overlapped area according to definition, namely, the result of the overlapped area in the spliced image cannot be made to be as clear as possible.
Disclosure of Invention
Aiming at the technical problems in the prior art, the embodiment of the disclosure provides a method and a device for splicing and fusing panoramic images, which can solve the problems of insufficient definition of a splicing overlapping area of the panoramic images and the like in the prior art.
A first aspect of the embodiments of the present disclosure provides a method for stitching and fusing panoramic images, including:
acquiring a bent picture corresponding to each sub-camera and a coordinate range of the bent picture on the panoramic image;
acquiring a bent picture corresponding to each sub-camera and a coordinate range of the bent picture on the panoramic image;
constructing a corresponding pixel pyramid according to the pixels of each sub-camera;
calculating a definition evaluation array corresponding to each sub-camera according to the bent picture, and constructing a weight pyramid corresponding to each sub-camera according to the definition evaluation array;
constructing a pixel weighting pyramid corresponding to each sub-camera according to the pixel pyramid and the weight pyramid;
accumulating/combining the weight pyramid and the pixel weighting pyramid corresponding to each sub-camera according to the coordinate range to obtain a panorama weighting pyramid and a panorama weighting pixel pyramid;
and regularizing the panoramic image weight pyramid and the panoramic image weighted pixel pyramid to generate a panoramic image pixel pyramid, and performing image reconstruction on the panoramic image pixel pyramid to generate a spliced panoramic image.
In some embodiments, the method specifically comprises: and performing conversion operation on the bent picture to obtain a gray image, and performing convolution operation on the gray image by using a definition evaluation operator to obtain the definition evaluation array.
In some embodiments, the constructing the corresponding pyramid of pixels according to the pixels of the respective sub-cameras specifically includes:
performing downsampling Gaussian fuzzy operation on a pixel data array of a current layer, removing elements on a user-defined row and/or column, generating a pixel data array with the length and width of half of the length and width of the current layer, recording the pixel data array as a downsampling pixel data array, and using the downsampling pixel data array as a pixel data array of a next layer of the current layer; performing up-sampling Gaussian fuzzy operation on the down-sampling pixel data array after data filling to generate an up-sampling pixel data array; performing difference operation on the current layer pixel data array and the up-sampling pixel data array, and taking an operation result as the current layer pixel data array; repeating the operation until a pixel pyramid with a preset number of layers is generated; wherein the size of the maximum size layer of the pixel pyramid is the same as the size of the corresponding curved picture.
In some embodiments, the constructing the weight pyramid of the sub-camera corresponding to each sub-camera according to the sharpness evaluation array specifically includes:
performing down-sampling Gaussian fuzzy operation on the definition data array of the current layer, removing elements on custom lines and/or columns, generating a definition data array with the length and width of half of the length and width of the current layer, recording the definition data array as a down-sampling definition data array, and taking the down-sampling definition data array as a next-layer definition data array of the current layer; repeating the operation until a weight pyramid with a preset number of layers is generated; wherein the size of the largest size layer of the weight pyramid is the same as the size of the curved picture of the corresponding sub-camera.
In some embodiments, the method specifically comprises: multiplying the pixel data array of each layer of the pixel pyramid corresponding to each sub-camera with the definition data array of each layer of the corresponding weight pyramid element by element to generate a weighted pixel pyramid corresponding to each sub-camera; wherein the size of the weighted pixel pyramid maximum size layer is the same as the size of the curved picture of the corresponding sub-camera.
In some embodiments, the regularization process specifically includes: and carrying out quotient operation on each element in the panorama weighted pixel pyramid and each element in the panorama weighted pyramid.
In some embodiments, the method further comprises: performing an initialization operation on the panorama weight pyramid and the panorama weighted pixel pyramid.
In some embodiments, the maximum size layers of the panorama weight pyramid and the panorama weighted pixel pyramid are the same size as the panorama.
A second aspect of the embodiments of the present disclosure provides a device for stitching and fusing panoramic images, including:
the acquisition module is used for acquiring the bent picture corresponding to each sub-camera and the coordinate range of the bent picture on the panoramic image;
the sub-camera pixel pyramid construction module is used for constructing a corresponding pixel pyramid according to the pixels of each sub-camera;
the sub-camera weight pyramid construction module is used for calculating a definition evaluation array corresponding to each sub-camera according to the bent picture and constructing a weight pyramid corresponding to each sub-camera according to the definition evaluation array;
the sub-camera pixel weighting pyramid construction module is used for constructing a pixel weighting pyramid corresponding to each sub-camera according to the pixel pyramid and the weighting pyramid;
the panorama pyramid construction module is used for constructing a pixel weighting pyramid corresponding to each sub-camera according to the pixel pyramid and the weighting pyramid;
and the regularization module is used for regularizing the panoramic image weight pyramid and the panoramic image weighted pixel pyramid to generate a panoramic image pixel pyramid, and performing image reconstruction on the panoramic image pixel pyramid to generate a spliced panoramic image.
In some embodiments, the sub-camera pixel weighting pyramid construction module is specifically configured to: multiplying the pixel data array of each layer of the pixel pyramid corresponding to each sub-camera and the definition data array of each layer of the corresponding weight pyramid element by element to generate a weighted pixel pyramid corresponding to each sub-camera; wherein the size of the weighted pixel pyramid maximum size layer is the same as the size of the curved picture of the corresponding sub-camera.
A third aspect of the embodiments of the present disclosure provides an electronic device, including:
a memory and one or more processors;
wherein the memory is communicatively coupled to the one or more processors, and the memory stores instructions executable by the one or more processors, and when the instructions are executed by the one or more processors, the electronic device is configured to implement the method according to the foregoing embodiments.
A fourth aspect of the embodiments of the present disclosure provides a computer-readable storage medium having stored thereon computer-executable instructions, which, when executed by a computing device, may be used to implement the method according to the foregoing embodiments.
A fifth aspect of embodiments of the present disclosure provides a computer program product comprising a computer program stored on a computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, are operable to implement a method as in the preceding embodiments.
The beneficial effects of the embodiment of the disclosure are: the definition evaluation array is used for calculating a weight pyramid formed by mutually fusing curved pictures corresponding to all the sub-cameras so as to obtain a pixel weighting pyramid, and the panoramic image weight pyramid and the panoramic image weighting pixel pyramid are obtained by accumulating/combining the weight pyramid and the pixel weighting pyramid corresponding to all the sub-cameras; regularization processing is carried out on the panoramic image weight pyramid and the panoramic image weighted pixel pyramid, and the superposition area achieves the clearest effect according to the principle of definition priority; and the calculation amount is small, and the calculation speed is high.
Drawings
The features and advantages of the present disclosure will be more clearly understood by reference to the accompanying drawings, which are illustrative and not to be construed as limiting the disclosure in any way, and in which:
FIG. 1 is a flow diagram of a method of panoramic image stitching fusion, according to some embodiments of the present disclosure;
FIG. 2 is a block diagram of an apparatus for stitching and fusing panoramic images according to some embodiments of the present disclosure;
fig. 3 is a schematic structural diagram of an electronic device according to some embodiments of the present disclosure.
Detailed Description
In the following detailed description, numerous specific details of the disclosure are set forth by way of examples in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure may be practiced without these specific details. It should be understood that the use of the terms "system," "apparatus," "unit" and/or "module" in this disclosure is a method for distinguishing between different components, elements, portions or assemblies at different levels of sequence. However, these terms may be replaced by other expressions if they can achieve the same purpose.
It will be understood that when a device, unit or module is referred to as being "on" … … "," connected to "or" coupled to "another device, unit or module, it can be directly on, connected or coupled to or in communication with the other device, unit or module, or intervening devices, units or modules may be present, unless the context clearly dictates otherwise. For example, as used in this disclosure, the term "and/or" includes any and all combinations of one or more of the associated listed items.
The terminology used in the disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of the disclosure. As used in the specification and claims of this disclosure, the terms "a," "an," "the," and/or "the" are not intended to be inclusive in the singular, but rather are inclusive in the plural, unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" are intended to cover only the explicitly identified features, integers, steps, operations, elements, and/or components, but not to constitute an exclusive list of such features, integers, steps, operations, elements, and/or components.
These and other features and characteristics of the present disclosure, as well as the methods of operation and functions of the related elements of structure and the combination of parts and economies of manufacture, will be better understood by reference to the following description and drawings, which form a part of this specification. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the disclosure. It will be understood that the figures are not drawn to scale.
Various block diagrams are used in this disclosure to illustrate various variations of embodiments according to the disclosure. It should be understood that the foregoing and following structures are not intended to limit the present disclosure. The protection scope of the present disclosure is subject to the claims.
In the prior art, in order to achieve the optimal fusion effect of image splicing when a panorama is spliced, a gaussian-laplacian pyramid fusion mode is generally adopted, and the method can perform average fusion on each pixel point of a superposition region according to the pixel point value of each sub-camera at the position where any plurality of sub-camera pictures are superposed, that is, the pixel of any sub-camera appearing in the superposition region contributes to a fusion result according to the weight of 1 divided by the number of the pictures. The fusion mode can actually make transition among sub-camera pictures more natural to a certain extent, but the mode cannot screen data sources of fusion results of all parts of the overlapped area according to definition, namely, the result of the overlapped area in the spliced image cannot be made to be as clear as possible.
Therefore, to solve the above problem, the present embodiment discloses a method for stitching and fusing panoramic images, as shown in fig. 1, specifically including:
s101, obtaining a bent picture corresponding to each sub-camera and a coordinate range of the bent picture on a panoramic image;
s102, constructing a corresponding pixel pyramid according to the pixels of all the sub-cameras;
s103, calculating a definition evaluation array corresponding to each sub-camera according to the bent picture, and constructing a weight pyramid corresponding to each sub-camera according to the definition evaluation array;
s104, constructing a pixel weighting pyramid corresponding to each sub-camera according to the pixel pyramid and the weighting pyramid;
s105, accumulating/combining the weight pyramid and the pixel weighting pyramid corresponding to each sub-camera according to the coordinate range to obtain a panorama weighting pyramid and a panorama weighting pixel pyramid;
s106, conducting regularization processing on the panoramic image weight pyramid and the panoramic image weighted pixel pyramid to generate a panoramic image pixel pyramid, conducting image reconstruction on the panoramic image pixel pyramid, and generating a spliced panoramic image.
In some embodiments, the warping parameter of each sub-camera picture and the coordinates of the warped picture on the panorama can be obtained according to a conventional method or a method of cumulative modeling or cumulative modeling plus an adjustment module, and the warped picture corresponding to each sub-camera is obtained according to the warping parameter.
In some embodiments, there are regions of coincidence between some of the sub-camera pictures in the panorama.
In some embodiments, a sharpness evaluation array is computed for the curved picture corresponding to each sub-camera.
Specifically, a conversion operation is performed on the curved picture to obtain a corresponding gray scale image, and an operator (convolution kernel) such as Laplacian or Sobel, which is commonly used for sharpness evaluation, is used for performing convolution with a step length of 1 to obtain a sharpness evaluation array. Wherein each element in the sharpness evaluation array corresponds to a weighted gradient between each gray pixel and surrounding gray pixels. In general, when convolution is performed using Laplacian or Sobel kernels, in order to keep the vertical and horizontal dimensions (or the length and width dimensions) of the sharpness evaluation array consistent with the vertical and horizontal dimensions of the corresponding warped sub-camera pictures, it is necessary to perform boundary filling (padding) on the gray scale maps obtained by converting each warped picture; the filling mode can be common modes such as constant filling and reflection filling.
In some embodiments, a pixel-weighted gaussian-laplacian downsampling pyramid (pixel-weighted pyramid for short) is constructed for the area where each sub-camera is located according to the coordinate range of the bent picture of each sub-camera on the panorama; at the same time or then, a down-sampling pyramid of laplacian of gaussian (weight pyramid for short) of the sharpness evaluation array is constructed only for the region where each sub-camera is located.
Specifically, constructing a corresponding pixel pyramid according to the pixels of each sub-camera specifically includes: performing downsampling Gaussian fuzzy operation on a pixel data array of a current layer, removing elements on a user-defined row and/or column, generating a pixel data array with the length and width of half of the size of the current layer, recording the pixel data array as a downsampling pixel data array, and using the downsampling pixel data array as a pixel data array of a next layer of the current layer; performing upsampling Gaussian fuzzy operation on the downsampling pixel data array subjected to data filling to generate an upsampling pixel data array; performing difference operation on the current layer pixel data array and the up-sampling pixel data array, and taking an operation result as the current layer pixel data array; this operation is repeated until a predetermined number of layers of pixel pyramids are generated.
More specifically, whether the pixel data array of the current layer is the minimum size layer of the pixel pyramid is judged; if not, firstly, performing downsampling Gaussian blur operation on the pixel data array of the current layer, namely performing convolution by using a downsampling Gaussian convolution kernel, removing elements (such as elements on even rows and even columns) on custom rows and/or columns, obtaining a pixel data array with the vertical and horizontal sizes being half of the vertical and horizontal sizes of the current layer, recording the pixel data array as a downsampling pixel data array, and taking the downsampling pixel data array as the pixel data array of the next layer of the current layer; continuously judging whether the current layer is the last layer of the pixel pyramid or not; if not, the right, the lower edge and the right and the lower edge of each element in the pixel data array of the current layer are complemented by 0, then the upsampling Gaussian blur operation is executed, namely convolution is carried out by an upsampling Gaussian convolution kernel to obtain a pixel data array with the same size as the current layer, the pixel data array is marked as an upsampling pixel data array, the pixel data array of the current layer is subtracted from the upsampling pixel data array element by element, and the result is used as the final pixel data array of the current layer.
If the current layer pixel data array is already the minimum size layer of the pixel pyramid, the down/up sampling Gaussian convolution kernel operation is not performed. Further, the size of the maximum size layer of the pixel pyramid is the same as the size of the curved picture corresponding to each sub-camera.
Further, a weight pyramid corresponding to each sub-camera is constructed according to the definition evaluation array.
Specifically, firstly, judging whether a definition data array of a current layer in a definition evaluation array is a minimum size layer of a weight pyramid; if not, performing downsampling Gaussian blur operation on the definition data array of the current layer, namely performing convolution by using a downsampling Gaussian convolution kernel, and then removing elements on custom rows and/or columns (for example, removing elements on even rows and even columns) to obtain a definition data array with the length and width being half of the length and width of the current layer as the next layer; this operation is repeated until a layer of minimum size is obtained. And the size of the maximum size layer of the weight pyramid is the same as that of the corresponding curved picture.
More specifically, performing an initialization operation on the maximum size layer of the weight pyramid enables a corresponding sharpness evaluation array to be obtained.
Further, the pixel data array of each layer of the pixel pyramid obtained according to the bent picture of each sub-camera is multiplied by the definition data array of each layer of the corresponding weight pyramid element by element to obtain the weighted pixel pyramid corresponding to each sub-camera.
Specifically, the coordinates corresponding to the weighted pixel pyramid and the weighting pyramid in each layer are obtained according to the coordinates of each sub-camera picture on the panorama, and the coordinates in each smaller layer are half of the numerical values of the coordinates in the immediately adjacent larger layer.
In some embodiments, the data arrays of each layer of the weighting pyramid and the weighting pixel pyramid corresponding to the curved pictures of each sub-camera are respectively filled in each layer of the initialized panorama weighting pyramid and panorama weighting pixel pyramid according to the coordinates of each layer, so as to obtain the final panorama weighting pixel pyramid and panorama weighting pyramid.
In some embodiments, the method further comprises: and initializing a panorama weighting pixel pyramid and a panorama weighting pyramid according to the size of the panorama and the preset pyramid layer number.
The maximum layer size of the initialized weighted pixel pyramid is equal to the size of the panorama, and the number of channels is 3. The maximum layer size of the initialized weight pyramid is equal to the size of the panorama, and the number of channels is 1. Where the size of the panorama is the correct size plus the convolution fill required for the pyramid structure. Generally, the length and width of each layer of the initialized panorama weighted pixel pyramid and panorama weighted pyramid is half of the length and width of the next layer immediately below the layer, and the requirement on the size of the panorama is high because when one pyramid has n (n is a natural number greater than 0) layers, the height H and the width W of the layer data array with the largest size are both 2 n Trimming; the filling here may be performed in a common mode such as constant filling or reflective filling.
Typically, all element values in the initialized panorama weighted pixel pyramid and panorama weight pyramid are 0.
In some embodiments, the method further comprises: and accumulating/combining the weight pyramid of each sub-camera according to the coordinate range to generate a panoramic image weight pyramid.
In some embodiments, the panorama weighted pixel pyramid and the panorama weighted pyramid are regularized to generate a panorama pixel pyramid.
In some embodiments, the regularized panorama pixel pyramid is subjected to image reconstruction to generate a stitched panorama.
In the embodiment of the disclosure, the definition evaluation array is used for calculating a weight pyramid in which curved pictures corresponding to each sub-camera are fused with each other, and the panorama weighted pixel pyramid and the panorama weighted pyramid are obtained by accumulating/combining the pixel weighted pyramid and the weight pyramid corresponding to each sub-camera; regularization processing is carried out on the panoramic image weighted pixel pyramid and the panoramic image weighted pyramid, and the superposition area achieves the clearest effect according to the principle of definition priority; and the calculation amount is small, and the calculation speed is high.
The embodiment of the present disclosure still further discloses a device 200 for stitching and fusing panoramic images, which is specifically shown in fig. 2 and includes:
an obtaining module 201, configured to obtain a curved picture corresponding to each sub-camera and a coordinate range of the curved picture on the panorama;
a sub-camera pixel pyramid construction module 202, configured to construct a corresponding pixel pyramid according to pixels of each sub-camera;
a sub-camera weight pyramid construction module 203, configured to calculate a definition evaluation array corresponding to each sub-camera according to the curved picture, and construct a weight pyramid corresponding to each sub-camera according to the definition evaluation array;
a sub-camera pixel weighting pyramid construction module 204, configured to construct a pixel weighting pyramid corresponding to each sub-camera according to the pixel pyramid and the weight pyramid;
a panorama pyramid constructing module 205, configured to construct a pixel weighting pyramid corresponding to each sub-camera according to the pixel pyramid and the weighting pyramid;
and the regularization module 206 is configured to regularize the panorama weighting pyramid and the panorama weighted pixel pyramid to generate a panorama pixel pyramid, and perform image reconstruction on the panorama pixel pyramid to generate a stitched panorama.
In some embodiments, the sub-camera pixel weighting pyramid construction module is specifically configured to: multiplying the pixel data array of each layer of the pixel pyramid corresponding to each sub-camera and the definition data array of each layer of the corresponding weight pyramid element by element to generate a weighted pixel pyramid corresponding to each sub-camera; wherein the size of the weighted pixel pyramid maximum size layer is the same as the size of the curved picture of the corresponding sub-camera.
The embodiment of the present disclosure further discloses a schematic diagram of an electronic device, as shown in fig. 3. Wherein, this electronic equipment 300 includes:
a memory 330 and one or more processors 310;
wherein the memory 330 is communicatively coupled to the one or more processors 310, the memory 330 stores instructions 332 executable by the one or more processors, and the instructions 332 are executable by the one or more processors 310 to cause the one or more processors 310 to perform the methods of the foregoing embodiments of the present disclosure.
In particular, the processor 310 and the memory 330 may be connected by a bus or other means, such as by a bus 340 in FIG. 3. Processor 310 may be a Central Processing Unit (CPU) and/or a graphics processor. Graphics Processing Unit (GPU). The Processor 310 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, or any combination thereof.
The memory 330, which is a non-transitory computer-readable storage medium, may be used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 310 executes various functional applications of the processor and data processing by running non-transitory software programs, instructions, and modules 332 stored in the memory 330.
The memory 330 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, application programs required for functions; the storage data area may store data created by the processor 310, and the like. Further, memory 330 may include high speed random access memory, and may also include non-transitory memory, such as a disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 330 optionally includes memory located remotely from processor 310, which may be connected to processor 310 via a network, such as through communication interface 320. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
An embodiment of the present disclosure also provides a computer-readable storage medium, in which computer-executable instructions are stored, and the computer-executable instructions are executed to perform the method in the foregoing embodiment of the present disclosure.
The foregoing computer-readable storage media include physical volatile and nonvolatile, removable and non-removable media implemented in any manner or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media specifically include, but are not limited to, a USB flash drive, a removable hard drive, a Read-Only Memory (ROM), a Random Access Memory (RAM), an erasable programmable Read-Only Memory (EPROM), an electrically erasable programmable Read-Only Memory (EEPROM), flash Memory or other solid state Memory technology, a CD-ROM, a Digital Versatile Disk (DVD), an HD-DVD, a Blue-Ray or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer.
While the subject matter described herein is provided in the general context of execution in conjunction with the execution of an operating system and application programs on a computer system, those skilled in the art will recognize that other implementations may also be performed in combination with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Those skilled in the art will appreciate that the subject matter described herein may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like, as well as distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
Those of ordinary skill in the art will appreciate that the various illustrative elements and method steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present disclosure.
In summary, the present disclosure provides a method and an apparatus for stitching and fusing panoramic images, an electronic device and a computer-readable storage medium thereof. In the disclosure, the definition evaluation array is used for calculating the weight of mutual fusion of the curved pictures corresponding to each sub-camera, and the pixel weighting pyramid and the panorama weighting pyramid corresponding to each sub-camera are obtained by accumulating/combining the pixel weighting pyramid and the weighting pyramid corresponding to each sub-camera; regularization processing is carried out on the panoramic image weighted pixel pyramid and the panoramic image weighted pyramid, and the superposition area achieves the clearest effect according to the principle of definition priority; and the calculation amount is small, and the calculation speed is high.
It is to be understood that the above-described specific embodiments of the present disclosure are merely illustrative of or illustrative of the principles of the present disclosure and are not to be construed as limiting the present disclosure. Accordingly, any modification, equivalent replacement, improvement or the like made without departing from the spirit and scope of the present disclosure should be included in the protection scope of the present disclosure. Further, it is intended that the following claims cover all such variations and modifications that fall within the scope and bounds of the appended claims, or equivalents of such scope and bounds.
Claims (8)
1. A method for splicing and fusing panoramic images is characterized by comprising the following steps:
acquiring a bent picture corresponding to each sub-camera and a coordinate range of the bent picture on the panoramic image;
constructing a corresponding pixel pyramid according to the pixels of each sub-camera;
calculating a definition evaluation array corresponding to each sub-camera according to the bent picture, and constructing a weight pyramid corresponding to each sub-camera according to the definition evaluation array;
constructing a pixel weighting pyramid corresponding to each sub-camera according to the pixel pyramid and the weight pyramid;
accumulating/combining the weight pyramid and the pixel weighting pyramid corresponding to each sub-camera according to the coordinate range to obtain a panorama weighting pyramid and a panorama weighting pixel pyramid;
regularizing the panoramic image weight pyramid and the panoramic image weighted pixel pyramid to generate a panoramic image pixel pyramid, and performing image reconstruction on the panoramic image pixel pyramid to generate a spliced panoramic image;
further comprising: performing an initialization operation on the panorama weight pyramid and the panorama weighted pixel pyramid, wherein the size of the maximum size layer of the panorama weight pyramid and the panorama weighted pixel pyramid is the same as the size of the panorama;
further comprising: and initializing a panorama weighting pixel pyramid and a panorama weighting pyramid according to the size of the panorama and the preset pyramid layer number, wherein the maximum layer size of the initialized weighting pixel pyramid is equal to the size of the panorama, the number of channels is 3, the maximum layer size of the initialized weighting pyramid is equal to the size of the panorama, and the number of channels is 1.
2. The method according to claim 1, characterized in that it comprises in particular:
and performing conversion operation on the bent picture to obtain a gray image, and performing convolution operation on the gray image by using a definition evaluation operator to obtain the definition evaluation array.
3. The method of claim 1, wherein constructing the corresponding pyramid of pixels from the pixels of each sub-camera specifically comprises:
performing downsampling Gaussian fuzzy operation on a pixel data array of a current layer, removing elements on a user-defined row and/or column, generating a pixel data array with the length and width of half of the length and width of the current layer, recording the pixel data array as a downsampling pixel data array, and using the downsampling pixel data array as a pixel data array of a next layer of the current layer; performing up-sampling Gaussian fuzzy operation on the down-sampling pixel data array after data filling to generate an up-sampling pixel data array; performing difference operation on the current layer pixel data array and the up-sampling pixel data array, and taking an operation result as the current layer pixel data array; repeating the operation until a pixel pyramid with a preset number of layers is generated; wherein the size of the maximum size layer of the pixel pyramid is the same as the size of the corresponding curved picture.
4. The method according to claim 1, wherein the constructing a weight pyramid of the sub-camera corresponding to each sub-camera according to the sharpness evaluation array specifically comprises:
executing downsampling Gaussian fuzzy operation on the definition data array of the current layer, removing elements on custom lines and/or columns, generating a definition data array with the length and width being half of the length and width of the current layer, recording the definition data array as a downsampling definition data array, and using the downsampling definition data array as a definition data array of the next layer of the current layer; repeating the operation until a weight pyramid with a preset number of layers is generated; wherein the size of the maximum size layer of the weight pyramid is the same as the size of the curved picture of the corresponding sub-camera.
5. The method according to claim 1, characterized in that it comprises in particular: multiplying the pixel data array of each layer of the pixel pyramid corresponding to each sub-camera and the definition data array of each layer of the corresponding weight pyramid element by element to generate a weighted pixel pyramid corresponding to each sub-camera; wherein the size of the weighted pixel pyramid maximum size layer is the same as the size of the curved picture of the corresponding sub-camera.
6. The method according to claim 1, wherein the regularization process specifically comprises: and carrying out quotient operation on each element in the panorama weighted pixel pyramid and each element in the panorama weighted pyramid.
7. A panoramic image splicing and fusing device is characterized by comprising:
the acquisition module is used for acquiring the bent picture corresponding to each sub-camera and the coordinate range of the bent picture on the panoramic image;
the sub-camera pixel pyramid construction module is used for constructing a corresponding pixel pyramid according to the pixels of each sub-camera;
the sub-camera weight pyramid construction module is used for calculating a definition evaluation array corresponding to each sub-camera according to the bent picture and constructing a weight pyramid corresponding to each sub-camera according to the definition evaluation array;
the sub-camera pixel weighting pyramid construction module is used for constructing a pixel weighting pyramid corresponding to each sub-camera according to the pixel pyramid and the weighting pyramid;
the panorama pyramid construction module is used for constructing a pixel weighting pyramid corresponding to each sub-camera according to the pixel pyramid and the weighting pyramid;
the regularization module is used for regularizing the panoramic image weight pyramid and the panoramic image weighted pixel pyramid to generate a panoramic image pixel pyramid, and performing image reconstruction on the panoramic image pixel pyramid to generate a spliced panoramic image; performing an initialization operation on the panorama weight pyramid and the panorama weighted pixel pyramid, wherein the size of the maximum size layer of the panorama weight pyramid and the panorama weighted pixel pyramid is the same as the size of the panorama; and initializing a panorama weighting pixel pyramid and a panorama weighting pyramid according to the size of the panorama and the preset pyramid layer number, wherein the maximum layer size of the initialized weighting pixel pyramid is equal to the size of the panorama, the number of channels is 3, the maximum layer size of the initialized weighting pyramid is equal to the size of the panorama, and the number of channels is 1.
8. The apparatus of claim 7, wherein the sub-camera pixel weighting pyramid construction module is specifically configured to: multiplying the pixel data array of each layer of the pixel pyramid corresponding to each sub-camera and the definition data array of each layer of the corresponding weight pyramid element by element to generate a weighted pixel pyramid corresponding to each sub-camera; wherein the size of the weighted pixel pyramid maximum size layer is the same as the size of the curved picture of the corresponding sub-camera.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110654226.3A CN113822800B (en) | 2021-06-11 | 2021-06-11 | Panoramic image splicing and fusing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110654226.3A CN113822800B (en) | 2021-06-11 | 2021-06-11 | Panoramic image splicing and fusing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113822800A CN113822800A (en) | 2021-12-21 |
| CN113822800B true CN113822800B (en) | 2022-09-06 |
Family
ID=78923853
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110654226.3A Active CN113822800B (en) | 2021-06-11 | 2021-06-11 | Panoramic image splicing and fusing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113822800B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8885976B1 (en) * | 2013-06-20 | 2014-11-11 | Cyberlink Corp. | Systems and methods for performing image fusion |
| CN106934765A (en) * | 2017-03-14 | 2017-07-07 | 长沙全度影像科技有限公司 | Panoramic picture fusion method based on depth convolutional neural networks Yu depth information |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107784632A (en) * | 2016-08-26 | 2018-03-09 | 南京理工大学 | A kind of infrared panorama map generalization method based on infra-red thermal imaging system |
| CN107301620B (en) * | 2017-06-02 | 2019-08-13 | 西安电子科技大学 | Method for panoramic imaging based on camera array |
| CN107845128B (en) * | 2017-11-03 | 2021-09-03 | 安康学院 | Multi-exposure high-dynamic image reconstruction method with multi-scale detail fusion |
| CN109672872B (en) * | 2018-12-29 | 2021-05-04 | 合肥工业大学 | Method for generating naked eye 3D (three-dimensional) effect by using single image |
-
2021
- 2021-06-11 CN CN202110654226.3A patent/CN113822800B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8885976B1 (en) * | 2013-06-20 | 2014-11-11 | Cyberlink Corp. | Systems and methods for performing image fusion |
| CN106934765A (en) * | 2017-03-14 | 2017-07-07 | 长沙全度影像科技有限公司 | Panoramic picture fusion method based on depth convolutional neural networks Yu depth information |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113822800A (en) | 2021-12-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111275626B (en) | Video deblurring method, device and equipment based on ambiguity | |
| CN107967669A (en) | Method, apparatus, computer equipment and the storage medium of picture processing | |
| CN110782424B (en) | Image fusion method and device, electronic equipment and computer readable storage medium | |
| US9519996B2 (en) | Virtual view generating method and apparatus | |
| US10013741B2 (en) | Method for deblurring video using modeling blurred video with layers, recording medium and device for performing the method | |
| JP2020053062A (en) | Modification of homography | |
| CN113793284B (en) | Image restoration method for nonuniform blocking of aerodynamic optical effect space-variant blurred image | |
| US11238560B2 (en) | Image down-scaling with pixel sets selected via blue noise sampling | |
| CN111652800B (en) | Single image super-resolution method and computer readable storage medium | |
| CN113781621A (en) | Three-dimensional reconstruction processing method, device, equipment and storage medium | |
| CN110335228B (en) | Method, device and system for determining image parallax | |
| CN116912148B (en) | Image enhancement method, device, computer equipment and computer readable storage medium | |
| WO2016098323A1 (en) | Information processing device, information processing method, and recording medium | |
| CN113822800B (en) | Panoramic image splicing and fusing method and device | |
| CN113112561A (en) | Image reconstruction method and device and electronic equipment | |
| CN112164006A (en) | Image color homogenizing method and device, electronic equipment and storage medium | |
| CN114782300B (en) | Product image reconstruction method, product reconstruction model training method and device | |
| US9412188B2 (en) | Method and image processing system for removing a visual object from an image | |
| CN116071279A (en) | Image processing method, device, computer equipment and storage medium | |
| CN111524072A (en) | Super-resolution reconstruction network training method and device, electronic equipment and storage medium | |
| CN116402693B (en) | Municipal engineering image processing method and device based on remote sensing technology | |
| US20130114888A1 (en) | Image processing apparatus, computer program product, and image processing method | |
| CN117455799A (en) | Image processing method, electronic equipment and storage medium | |
| CN112950496A (en) | Image sample set generation method, image sample set generation device and non-volatile computer-readable storage medium | |
| CN114862686A (en) | Image processing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A method and device for panoramic image stitching and fusion Effective date of registration: 20230817 Granted publication date: 20220906 Pledgee: Jiangsu Jiangyin Rural Commercial Bank Co.,Ltd. Wuxi Branch Pledgor: Wuxi ankedi Intelligent Technology Co.,Ltd. Registration number: Y2023980052623 |