CN117218000A

CN117218000A - Panoramic image stitching method, device, equipment and storage medium

Info

Publication number: CN117218000A
Application number: CN202311181400.2A
Authority: CN
Inventors: 沈俊; 田虎; 关海波
Original assignee: Beijing Chengshi Wanglin Information Technology Co Ltd
Current assignee: Beijing Chengshi Wanglin Information Technology Co Ltd
Priority date: 2023-09-13
Filing date: 2023-09-13
Publication date: 2023-12-12

Abstract

The embodiment of the invention provides a panoramic image stitching method, a device, equipment and a storage medium, which comprise the following steps: acquiring a plurality of scene images of a target scene and a depth map of the target scene, marking a plurality of image overlapping areas corresponding to a plurality of groups of adjacent scene images in the depth map, selecting an image overlapping sub-area from each image overlapping area, and enabling depth gradient values corresponding to pixels in the image overlapping sub-areas to be smaller than a first preset threshold value. And determining a target depth value corresponding to each pixel point in the plurality of scene images according to the depth values corresponding to each pixel point in the plurality of image overlapping sub-areas. According to the target depth value, image expansion processing is carried out on a plurality of scene images to obtain a plurality of plane expansion images, the plurality of plane expansion images are fused to obtain a panoramic image corresponding to the target scene, the depth difference between the image overlapping area and the non-image overlapping area is reduced, and the phenomenon of distortion or cutting at the image splicing edge is effectively avoided.

Description

Panoramic image stitching method, device, equipment and storage medium

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to a panoramic image stitching method, apparatus, device, and storage medium.

Background

Currently, mainstream panoramic cameras in the market are expensive and cannot be widely used, a plurality of fisheye cameras are generally used for obtaining panoramic images, the plurality of fisheye cameras can respectively shoot the same target scene at the same moment and at different shooting angles, and panoramic stitching processing is carried out on the photographed plurality of fisheye images so as to obtain panoramic images corresponding to the target scenes.

In the existing panoramic image stitching scheme, depth values corresponding to all pixel points in the acquired multiple fisheye images are used as radiuses to determine respective corresponding projection spherical surfaces, all the pixel points are projected on the respective corresponding projection spherical surfaces to obtain target longitude and latitude coordinates corresponding to all the pixel points, the multiple fisheye images are unfolded according to the target longitude and latitude coordinates to obtain multiple unfolded images, and finally the multiple unfolded images are fused to obtain the fused panoramic image.

However, in practical application, a certain error may exist in the depth value corresponding to each obtained pixel, so that when a plurality of fisheye images are spliced, obvious distortion and splitting easily occur in the image splicing part, resulting in poor panoramic experience of a user.

Disclosure of Invention

The embodiment of the invention provides a panoramic image stitching method, a device, equipment and a storage medium, which are used for improving the stitching effect of panoramic images.

In a first aspect, an embodiment of the present invention provides a panoramic image stitching method, where the method includes:

acquiring a plurality of scene images of a target scene and a depth map of the target scene, wherein the plurality of scene images are obtained by shooting the target scene at a plurality of different angles around the same rotation shaft center by a camera;

marking a plurality of image overlapping areas corresponding to a plurality of groups of adjacent scene images in a depth map according to the camera pose and the camera view angle corresponding to each of the plurality of scene images, wherein the image overlapping area corresponding to each group of adjacent scene images refers to an overlapping area between two adjacent scene images contained in the image overlapping area;

selecting an image overlapping sub-region from each image overlapping region, wherein the depth gradient value corresponding to each pixel in the image overlapping sub-region is smaller than a first preset threshold value;

determining target depth values corresponding to all pixel points in the plurality of scene images according to the depth values corresponding to all pixel points in the plurality of image overlapping sub-areas;

Performing image expansion processing on the plurality of scene images according to the target depth value to obtain a plurality of plane expansion images;

and fusing the plurality of plane expansion images to obtain a panoramic image corresponding to the target scene.

In a second aspect, an embodiment of the present invention provides a panoramic image stitching apparatus, including:

the acquisition module is used for acquiring a plurality of scene images of a target scene and a depth map of the target scene, wherein the plurality of scene images are obtained by shooting the target scene at a plurality of different angles around the same rotation shaft center by a camera;

the calibration module is used for calibrating a plurality of image overlapping areas corresponding to a plurality of groups of adjacent scene images in the depth map according to the camera pose and the camera view angle corresponding to each of the plurality of scene images, wherein the image overlapping area corresponding to each group of adjacent scene images refers to the overlapping area between two adjacent scene images contained in the image overlapping area;

the screening module is used for selecting an image overlapping sub-region from each image overlapping region, and the depth gradient value corresponding to each pixel point in the image overlapping sub-region is smaller than a first preset threshold value;

The determining module is used for determining target depth values corresponding to all pixel points in the plurality of scene images according to the depth values corresponding to all pixel points in the plurality of image overlapping sub-areas;

the expansion module is used for carrying out image expansion processing on the plurality of scene images according to the target depth value so as to obtain a plurality of plane expansion images;

and the fusion module is used for fusing the plurality of plane expansion images to obtain a panoramic image corresponding to the target scene.

In a third aspect, an embodiment of the present invention provides an electronic device, including: a memory, a processor, a communication interface; wherein the memory has executable code stored thereon, which when executed by the processor, causes the processor to at least implement the panoramic image stitching method as described in the first aspect.

In a fourth aspect, embodiments of the present invention provide a non-transitory machine-readable storage medium having executable code stored thereon, which when executed by a processor of an electronic device, causes the processor to at least implement a panoramic image stitching method as described in the first aspect.

In the panoramic image stitching scheme provided by the embodiment of the invention, firstly, a plurality of scene images of a target scene and a depth map of the target scene are acquired. The plurality of scene images are obtained by shooting a target scene at a plurality of different angles around the same rotation shaft center by the camera. And then, marking a plurality of image overlapping areas corresponding to a plurality of groups of adjacent scene images in the depth map according to the camera pose and the camera view angle corresponding to each of the plurality of scene images, wherein the image overlapping area corresponding to each group of adjacent scene images refers to the overlapping area between two adjacent scene images contained in the image overlapping area. And selecting an image overlapping sub-region from each image overlapping region. The depth gradient value corresponding to each pixel in the image overlapping sub-region is smaller than a first preset threshold value. And then, determining the target depth value corresponding to each pixel point in the plurality of scene images according to the depth value corresponding to each pixel point in the plurality of image overlapping sub-areas. And performing image expansion processing on the plurality of scene images according to the target depth values corresponding to the pixel points in the plurality of scene images so as to obtain a plurality of plane expansion images. And finally, fusing the plurality of plane expansion images to obtain a panoramic image corresponding to the target scene.

In the above scheme, the image overlapping subareas with smaller depth gradient values corresponding to the pixel points are selected from each image overlapping area, so that the image overlapping subareas with smoother depth changes corresponding to the image overlapping areas can be obtained. And optimizing the depth value of each pixel point in the scene image according to the depth value of each pixel point in the image overlapping sub-region with smoother depth change, so as to reduce the depth difference between the image overlapping region and the non-image overlapping region, and enable the depth difference at the image splicing edge to be smaller, thereby effectively avoiding the phenomenon of distortion or cutting at the image splicing edge and further improving the splicing effect of the panoramic image.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic diagram of a panoramic image stitching system according to an embodiment of the present invention;

Fig. 2 is a flowchart of a panoramic image stitching method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of an application of identifying overlapping regions of images in a depth map according to an embodiment of the present invention;

FIG. 4 is a flow chart of selecting an image overlapping sub-region from each image overlapping region provided by an embodiment of the present invention;

FIG. 5 is a flowchart of determining a target depth value corresponding to each pixel point in a plurality of scene images according to depth values corresponding to each pixel point in a plurality of overlapping sub-regions of the images according to an embodiment of the present invention;

fig. 6 is a schematic structural diagram of a panoramic image stitching apparatus according to an embodiment of the present invention;

fig. 7 is a schematic structural diagram of an electronic device corresponding to the panoramic image stitching apparatus provided in the embodiment shown in fig. 6.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

In addition, the sequence of steps in the method embodiments described below is only an example and is not strictly limited.

Before describing the panoramic image stitching scheme provided by the embodiment of the invention, the following concepts will be described.

The rotation axis center refers to a rotation center corresponding to a camera in a three-dimensional space during panoramic shooting. It will be appreciated that in determining a panoramic image of a target scene, it is often necessary to first acquire an image of the target scene over 360 degrees. The plurality of cameras may be placed around a certain center point (rotation axis Zhou Xun) so that the plurality of cameras respectively take a target scene at different fixed angles, and the photographing ranges of the plurality of cameras cover the target scene.

In practical application, when stitching multiple scene images, firstly performing panoramic expansion on the multiple scene images to obtain multiple plane expansion images, and then stitching the plane expansion images to obtain a panoramic image. However, when the scene image is unfolded, the depth value corresponding to any pixel point is used as the radius of the target projection spherical surface corresponding to the pixel point, then each pixel point is projected to the target projection spherical surface to respectively obtain corresponding target longitude and latitude coordinates, and then the target longitude and latitude coordinates are converted into two-dimensional plane coordinates in a panoramic coordinate system, so that a plurality of target plane unfolded images with higher accuracy are obtained. However, there are errors in the depth values of the pixels in the actually acquired multiple images, and these errors may cause significant distortion and cracking at the image stitching edges, so that the quality of the obtained panoramic image is lower.

In order to solve at least one technical problem, the embodiment of the invention provides a panoramic image stitching method, in the scheme, the depth value of each pixel point in a scene image is optimized by utilizing the depth value of each pixel point in an image sub-overlapping region with smoother depth change, so as to smooth the depth difference value of each pixel point in an image stitching edge region, thereby effectively avoiding the phenomenon of distortion or splitting of the image stitching edge region.

Fig. 1 is a schematic diagram of a panoramic image stitching system according to an embodiment of the present invention, where the system shown in fig. 1 includes: camera and panorama image concatenation device.

The camera may be a fisheye camera or a non-fisheye camera, and in the embodiment of the present invention, the type of the camera is not limited. The camera is arranged at a certain shooting point, and then a plurality of scene images of the target scene are shot at a plurality of different angles around the axis of the rotating shaft. In the implementation process, optionally, the camera can be fixed on the tripod head, the camera is driven to rotate around the axis of the rotating shaft through rotation of the tripod head, and in the rotation process, a plurality of scene images of a target scene are obtained through shooting of the camera from a plurality of different angles; alternatively, a plurality of cameras are placed around the axis of the rotation shaft, and a plurality of scene images of the target scene are captured by the plurality of cameras at different fixed angles.

The panoramic image stitching device can be electronic equipment such as a smart phone and a notebook computer. The panoramic image stitching device is mainly used for acquiring a plurality of scene images of a target scene and a depth map of the target scene, wherein the plurality of scene images are obtained by shooting the target scene at a plurality of different angles around the same rotation shaft center by a camera. And then, marking a plurality of image overlapping areas corresponding to a plurality of groups of adjacent scene images in the depth map according to the camera pose and the camera view angle corresponding to each of the plurality of scene images, wherein the image overlapping area corresponding to each group of adjacent scene images refers to the overlapping area between two adjacent scene images contained in the image overlapping area. And selecting an image overlapping sub-region from each image overlapping region, wherein the depth gradient value corresponding to each pixel in the image overlapping sub-region is smaller than a first preset threshold value. And then, determining target depth values corresponding to all pixel points in the plurality of scene images according to the depth values corresponding to all pixel points in the plurality of image overlapping sub-areas, performing image unfolding processing on the plurality of scene images according to the target depth values to obtain a plurality of plane unfolding images, and fusing the plurality of plane unfolding images to obtain a panoramic image corresponding to the target scene.

Optionally, the panoramic image stitching device may further include a laser scanner, and the laser scanner may collect point cloud data corresponding to the target scene, where the point cloud data includes depth information between the laser scanner and an object point in the three-dimensional space. And converting the point cloud data according to a first position relation between the laser scanner and the camera calibrated in advance and a second position relation between the camera and the axis of the rotating shaft so as to determine a depth map corresponding to the target scene and depth values corresponding to each pixel point in the depth map. The external parameters of the camera comprise a second position relation between the camera and the axis of the rotating shaft, and the second position relation can be obtained through pre-calibration.

The camera can be in communication connection with the panoramic image stitching device, and after shooting is completed, a plurality of scene images can be sent to the panoramic image stitching device, so that the panoramic image stitching device can perform stitching processing on the received plurality of scene images, and a panoramic image corresponding to a target scene can be obtained. In an optional embodiment, the panoramic image stitching system may further include a mobile device corresponding to the user side, after the mobile device acquires the plurality of scene images shot by the camera, a panoramic image stitching request is generated based on the plurality of scene images, and the panoramic image stitching request is sent to the panoramic image stitching device, and the panoramic image stitching device performs stitching processing on the plurality of scene images based on the request, so as to obtain a stitched panoramic image, and sends the stitched panoramic image to the mobile device corresponding to the user.

Fig. 2 is a flowchart of a panoramic image stitching method according to an embodiment of the present invention, as shown in fig. 2, may include the following steps:

201. and acquiring a plurality of scene images of the target scene and a depth map of the target scene, wherein the plurality of scene images are obtained by shooting the target scene at a plurality of different angles around the same rotation shaft center by a camera.

202. And according to the camera pose and the camera view angle corresponding to each of the plurality of scene images, a plurality of image overlapping areas corresponding to a plurality of groups of adjacent scene images are marked in the depth map, wherein the image overlapping area corresponding to each group of adjacent scene images refers to the overlapping area between two adjacent scene images contained in the image overlapping area.

203. And selecting an image overlapping sub-region from each image overlapping region, wherein the depth gradient value corresponding to each pixel in the image overlapping sub-region is smaller than a first preset threshold value.

204. And determining a target depth value corresponding to each pixel point in the plurality of scene images according to the depth values corresponding to each pixel point in the plurality of image overlapping sub-areas.

205. And performing image unfolding processing on the plurality of scene images according to the target depth value to obtain a plurality of plane unfolding images.

206. And fusing the plurality of plane expansion images to obtain a panoramic image corresponding to the target scene.

In the panoramic image stitching scheme provided by the embodiment of the invention, a plurality of scene images corresponding to the acquired target scene can be stitched to obtain the stitched panoramic image. The scene image may be a fisheye image, a non-fisheye image, etc., and the type of the image is not limited. In addition, after a plurality of scene images of a target scene are acquired, the depth values corresponding to all the pixel points in the acquired plurality of scene images are processed to reduce the depth difference value corresponding to all the pixel points in the image edge area, and then the scene images are spliced based on the processed depth values corresponding to all the pixel points to obtain the panoramic image.

Specifically, a plurality of scene images of a target scene and a depth map of the target scene are first acquired. The target scene can be an outdoor scene, an indoor scene and the like, and the plurality of scene images are obtained by shooting the same target scene at a plurality of different angles around the same rotation shaft center by the camera.

In addition, a depth map corresponding to the target scene may be acquired by using a laser scanner. For example, a laser scanner is used to scan the whole target scene to obtain point cloud data corresponding to the target scene, wherein the point cloud data comprises depth values corresponding to each point cloud in the target scene. However, before the point cloud data corresponding to the target scene is acquired by using the laser scanner, the positions of the laser scanner and the camera for acquiring the images of the multiple scenes can be calibrated, and then the point cloud data acquired under the coordinate system of the laser scanner is converted into the coordinate system corresponding to the camera according to the position relationship between the laser scanner and the camera, so that the depth value acquired by the laser scanner can be directly utilized in the follow-up. And after the point cloud data obtained under the coordinate system of the laser scanner is converted into the coordinate system corresponding to the camera, generating a depth map corresponding to the target scene according to the converted point cloud data.

In an alternative embodiment, a laser scanner may be used to scan the scene corresponding to each scene image. And setting a laser scanner at a position corresponding to a camera shooting each scene image, and then scanning the scene by using the laser scanner to obtain point cloud data in the scene, and generating a depth map corresponding to the scene image according to the point cloud data. And splicing the depth maps corresponding to the plurality of scene images by utilizing the pose relation of the cameras when shooting each scene image so as to obtain the depth map corresponding to the target scene.

In addition, a depth map corresponding to the target scene can be obtained by utilizing a pre-trained depth prediction model. For example, after a plurality of scene images of a target scene are acquired, performing panoramic expansion on the plurality of scene images to obtain plane expansion images corresponding to the plurality of scene images, stitching the plurality of plane expansion images to obtain an initial panoramic image of the target scene, and inputting the initial panoramic image into a pre-trained depth prediction model to obtain depth values corresponding to all pixel points in the initial panoramic image and a depth map corresponding to the initial panoramic image. Or after a plurality of scene images of the target scene are acquired, respectively inputting the plurality of scene images into a pre-trained depth prediction model, processing the depth values of the pixel points in the image overlapping area corresponding to each scene image to enable the depth values to be more approximate, generating a depth map corresponding to the target scene based on the processed depth values of the pixel points, and inputting the depth map.

After a depth map corresponding to a target scene is acquired, a plurality of image overlapping areas corresponding to a plurality of groups of adjacent scene images are marked in the depth map according to camera pose and camera view angles corresponding to the plurality of scene images. The image overlapping area corresponding to each group of adjacent scene images refers to the overlapping area between two adjacent scene images contained in the image overlapping area. The camera pose information corresponding to each of the plurality of scene images can be determined by acquiring the shooting angle of the camera. For example, the camera shoots the target scene at two angles of 0 degrees and 90 degrees around the same rotation axis, and the view angle of the camera is 180 degrees, so that the shooting angle of the target scene obtained by the camera at the position 1 is (-90 degrees, 90 degrees), the shooting angle of the target scene obtained by the camera at the position 1 is (0 degrees, 180 degrees), and the shooting angle (0 degrees, 90 degrees) is the image overlapping area. And directly marking the corresponding image overlapping region in the depth map according to the shooting angle.

In a specific application, as shown in fig. 3, for a target scene, 3 scene images, namely a scene image 1, a scene image 2 and a scene image 3 are acquired, wherein the scene image 1 and the scene image 2 are a group of adjacent scene images, an overlapping area between the two images is a, the overlapping area a is determined as an image overlapping area corresponding to the scene image 1 and the scene image 2, and an image overlapping area corresponding to the scene image 1 and the scene image 2 is marked in a depth map. Scene image 2 and scene image 3 are a group of adjacent scene images, the overlapping area between the two images is B, the overlapping area B is determined as the image overlapping area corresponding to the scene image 2 and the scene image 3, and the image overlapping area corresponding to the scene image 2 and the scene image 3 is marked in the depth map.

After a plurality of image overlapping areas corresponding to a plurality of groups of adjacent scene images are determined, an image overlapping sub-area is selected from each image overlapping area, and a target depth value corresponding to each pixel point in a plurality of scene images is determined according to the depth values corresponding to each pixel point in the plurality of image overlapping sub-areas. The depth gradient value corresponding to each pixel in the image overlapping sub-region is smaller than a first preset threshold value, and the corresponding first preset threshold value can be set according to actual conditions. The method comprises the steps of selecting an image overlapping subarea with a small depth gradient value from each image overlapping area, and processing the depth value corresponding to each pixel point in a plurality of scene images by taking the depth value of each pixel point in the image overlapping subarea as a standard to obtain a target depth value corresponding to each pixel point in the plurality of scene images, so that the actually measured depth error is reduced.

In practical application, a certain error exists in the depth values corresponding to all pixel points in a plurality of scene images, and the error is very easy to generate distortion or fracture in the image splicing edge area with inconsistent actual depth. In order to solve the problem, in the embodiment of the invention, a small image overlapping sub-region with smooth depth value change is selected from each image overlapping region by utilizing a depth map of a target scene, then an overall depth baseline corresponding to the target scene is known according to the depth values corresponding to each pixel point in each image overlapping sub-region, and the depth values corresponding to each pixel point in a plurality of scene images are adjusted according to the baseline and the depth values corresponding to each pixel point in a plurality of image overlapping sub-regions, so that the depth difference between the image overlapping region and a non-image overlapping region is reduced, and the phenomenon of distortion or cracking in an image splicing edge region is effectively avoided.

And processing the depth values corresponding to the pixel points in the plurality of scene images, and performing image unfolding processing on the plurality of scene images according to the target depth values corresponding to the pixel points in the plurality of scene images after obtaining the corresponding target depth values so as to obtain a plurality of plane unfolding images.

Specifically, the axis center of the rotation shaft is taken as a projection center, the target depth value corresponding to each pixel point is taken as a radius, the target projection spherical surface corresponding to each pixel point is determined, and the target longitude and latitude coordinates of each pixel point projected onto the corresponding target projection spherical surface are determined according to the internal parameters and the external parameters of the camera calibrated in advance. And further, converting the longitude and latitude coordinates of the targets corresponding to the pixel points into two-dimensional plane coordinates in a panoramic coordinate system, so as to obtain a plurality of plane expansion images with higher accuracy. Finally, the method of acquiring other characteristic points by utilizing the optical flow further processes the image overlapping area between the plurality of plane expansion images, then performs image fusion processing on the plurality of plane expansion images, fuses the plurality of target plane expansion images to obtain a panoramic image corresponding to the target scene, and thus the obtained panoramic image has better effect and avoids the phenomenon of distortion or splitting in the image splicing area.

In addition, in practical application, the multiple scene images are obtained by shooting the target scene at multiple different angles around the same rotation axis by the camera, so that the illumination conditions are different when the multiple scene images are shot, and therefore obvious illumination differences exist in each region in the spliced panoramic image, and the display effect of the panoramic image is affected. Therefore, in the embodiment of the invention, after the plurality of plane expansion images corresponding to the plurality of scene images are acquired, the plurality of plane expansion images can be subjected to illumination unified processing, and then the plurality of plane expansion images are fused.

Wherein, the illumination unification processing can be performed on the plurality of plane expansion images by utilizing the plurality of image overlapping areas. Specifically, first, a plurality of plane-expanded images are subjected to color space conversion into a format having luminance channel values, such as HSV format or YUV format. Then, the gain g and the deviation b are adjusted so that the luminance channel values corresponding to the image overlapping regions in the two plane expansion images are the same by using the information of the plurality of image overlapping regions.

For example, first according to formula g _i *δ _ij ＝g _j *δ _ji Separately calculateGain g corresponding to two plane expansion images _i 、g _j . Wherein g _i Representing the gain corresponding to the i-th plane expansion image, representing the gain corresponding to the j-th plane expansion image, delta _ij Representing the standard deviation, delta, corresponding to the image overlapping region in the ith plane expansion image _ji And the standard deviation corresponding to the image overlapping area in the jth plane expansion image is represented. Then, the corresponding gains of the two plane expansion images and the corresponding average value of the corresponding image overlapping areas are utilized to determine the corresponding deviation of the two plane expansion images. For example, according to formula g _i *m _ij +b _i ＝g _j *m _ji +b _j And calculating the corresponding deviation of the two plane expansion images. Wherein m is _ij Representing the average value, m, corresponding to the image overlapping region in the ith plane expansion image _ji Representing the mean value corresponding to the image overlapping region in the jth plane expansion image, b _i Representing the corresponding deviation in the i-th plane expansion image, b _j Representing the deviation corresponding to the jth plane expansion image. And finally, calculating the brightness channel value of the adjusted image overlapping area according to the corresponding gain and deviation of the two plane expansion images. For example, according to formula I _i ’＝g _i *I _i +b _i Formula I _j ’＝g _j *I _j +b _j And obtaining the brightness channel value corresponding to the image overlapping region in the i-th plane expansion image after adjustment and the brightness channel value corresponding to the image overlapping region in the j-th plane expansion image, and restoring the plane expansion image according to the brightness channel value corresponding to the image overlapping region after adjustment to obtain the image with uniform illumination. Wherein I is _i Is the brightness channel value corresponding to the I-th plane expansion image, I _i ' luminance channel value representing adjusted image overlap region corresponding to the I-th plane expansion image, I _j Is the brightness channel value corresponding to the jth plane expansion image, I _j ' represents the luminance channel value of the adjusted image overlap region corresponding to the jth plane expanded image. By adjusting g and b, the luminance channel values of the image overlapping areas are made uniform.

And then, carrying out fusion processing on the plurality of plane expansion images subjected to illumination unification processing so as to obtain a panoramic image corresponding to the target scene, and further improving the quality of the panoramic image.

In the embodiment of the invention, the image overlapping subareas with smaller depth gradient values corresponding to the pixel points are selected from each image overlapping area, so that the image overlapping subareas with smoother depth changes corresponding to the image overlapping areas can be obtained. And optimizing the depth value of each pixel point in the scene image according to the depth value of each pixel point in the image overlapping sub-region with smoother depth change, so as to reduce the depth difference between the image overlapping region and the non-image overlapping region, and enable the depth difference at the image splicing edge to be smaller, thereby effectively avoiding the phenomenon of distortion or cutting at the image splicing edge and further improving the splicing effect of the panoramic image.

In the above embodiment, it is described that, by selecting an image overlapping sub-region from each image overlapping region, determining a target depth value corresponding to each pixel point in the multiple scene images according to the depth values corresponding to each pixel point in the multiple image overlapping sub-regions, and performing image expansion processing on the multiple scene images according to the adjusted target depth values, a target plane expansion image with higher accuracy can be obtained, and then the panoramic image stitching effect is improved. In order to facilitate understanding of the specific implementation procedure of selecting the image overlapping sub-areas in the above embodiment, in the following embodiment, an implementation procedure of selecting the image overlapping sub-areas from each image overlapping area is exemplarily described with reference to fig. 4.

Fig. 4 is a flowchart of selecting an image overlapping sub-area from each image overlapping area according to an embodiment of the present invention, and as shown in fig. 4, the method may include the following steps:

401. and dividing the target image overlapping region based on a preset sliding window size to obtain a plurality of first sliding window regions corresponding to the target image overlapping region, wherein the target image overlapping region is any one of the plurality of image overlapping regions.

402. And determining an image overlapping sub-region corresponding to the target image overlapping region from the plurality of first sliding window regions according to the depth values of the pixel points in the plurality of first sliding window regions.

Because a certain error exists in the depth values corresponding to the pixel points in the scene images obtained through actual measurement, and when the obtained depth values are sparse, the problem that the depth values are not completely matched in the image splicing edge area occurs, so that the image splicing edge area is distorted or split. In the embodiment of the invention, the image overlapping subareas with smaller depth gradient values are selected from the image overlapping areas, so that the problems of depth value errors and sparse depth values of all the pixel points in the scene image are solved by the depth values corresponding to all the pixel points in the image overlapping subareas.

Specifically, firstly, based on a preset sliding window size, a target image overlapping region is subjected to segmentation processing, so that a plurality of first sliding window regions corresponding to the target image overlapping region are obtained. The corresponding sliding window size may be set according to actual requirements, for example, the image overlapping area is 20mm, and the sliding window size may be 1mm. The sliding window size cannot be changed during the sliding window movement. The target image overlapping region is any one of a plurality of image overlapping regions, and the plurality of image overlapping regions may be windowed in the same way to obtain a plurality of window regions corresponding to the plurality of image overlapping regions, respectively.

After the plurality of first sliding window areas are obtained, determining an image overlapping sub-area corresponding to the target image overlapping area from the plurality of first sliding window areas according to the depth values of all pixel points in the plurality of first sliding window areas. The depth gradient change corresponding to the selected image overlapping sub-region is smaller and smoother, namely the obtained depth value corresponding to each pixel point in the image overlapping sub-region is closer to the depth value corresponding to each pixel point in the non-image overlapping region, and the depth difference between the image overlapping region and the non-image overlapping region can be smoothed, so that distortion or cracks in the image splicing edge region are avoided.

From the above description, it is clear that: in the embodiment of the invention, in order to reduce the depth value error corresponding to each collected pixel point, the depth difference exists between the image overlapping area and the non-image overlapping area, when the depth value corresponding to each pixel point is adjusted, the image overlapping subarea with smaller depth gradient change and smoother depth gradient can be selected from the plurality of first sliding window areas corresponding to each image overlapping area, namely, the image overlapping subarea with the depth value corresponding to each pixel point in the non-image overlapping area being closer is selected, the image overlapping subarea is taken as the depth value corresponding to the image overlapping area, and then the target depth value corresponding to each pixel point in each scene image is determined according to the depth value corresponding to each pixel point in each image overlapping subarea.

In an optional embodiment, according to depth values of each pixel point in the plurality of first sliding window areas, a specific implementation manner of determining an image overlapping sub-area corresponding to the target image overlapping area from the plurality of first sliding window areas may be: calculating depth average values and total depth gradient values corresponding to the first sliding window areas; determining energy values corresponding to the first sliding window areas respectively according to the depth average value and the total depth gradient value; selecting a first sliding window area with the energy value smaller than a second preset threshold value from the plurality of first sliding window areas according to the energy value; and determining an image overlapping sub-region corresponding to the target image overlapping region from the selected first sliding window region according to the respective depth average value of the selected first sliding window region. The first sliding window area with smaller energy value can be selected from the plurality of first sliding window areas, and then the depth average value close to the depth average value of the adjacent area is selected from the first sliding window area with smaller energy, so that the finally selected image overlapping subarea can better reflect the depth value of the whole target scene.

The energy value corresponding to each first sliding window area can be calculated through an energy function. Specifically, in an alternative embodiment, an energy function E is constructed according to the depth average value and the total depth gradient value corresponding to the first sliding window area, where the energy function is mainly used to find the first sliding window area with similar depth values of the pixel points. Concrete embodiments Wherein E represents an energy value, alpha ₁ 、α ₂ 、α ₃ 、α ₄ The weights corresponding to the energy terms are empirical values, and are mainly used for enabling the depth values of the pixel points in the image overlapping region to be closer to each other, d _max Represents the maximum depth, d, in the region of the first sliding window _min Represents a depth minimum in the first sliding window region, d _median Represents the median depth, h, in the first sliding window region _i Representing the ordinate of the pixel point, wherein H is the image height, d is the depth average value corresponding to the first sliding window area, and d _i Representing the depth value of the pixel point, and grad represents the total depth gradient value corresponding to the first sliding window area. Alpha ₃ The corresponding factors are understood to be depths that deviate from the average depth, the depths of deviation being related to the depth values and the ordinate.

From the energy function, it can be seen that: when the difference value between the maximum value of the depth in the first sliding window area and the minimum value of the depth in the first sliding window area is larger, the depth value of each pixel point in the first sliding window area is closer; when the depth median in the first sliding window area is smaller, the depth value of each pixel point in the first sliding window area is closer; when the occupation weight of the image is larger at the position closer to the middle area of the image, the depth value of each pixel point in the first sliding window area is closer to the position; the smaller the total depth gradient value corresponding to the first sliding window area, the closer the depth value of each pixel point in the first sliding window area is.

In order to accurately acquire the actual depth value corresponding to each pixel point in the scene image, when the depth value corresponding to each pixel point in the scene image is adjusted, the base value of the overall depth of the target scene can be known, the depth value corresponding to each pixel point in the scene image is adjusted based on the base value of the overall depth and the depth value corresponding to each pixel point in the image overlapping sub-region, so that the depth value which is closer to the actual depth value of each pixel point can be acquired, and the difference between the image overlapping region and the non-image overlapping region can be smoothed better, so that a better splicing effect can be obtained.

In an alternative embodiment, a depth average value corresponding to the overlapping region of the adjacent images may be obtained, and the image overlapping sub-region corresponding to the target image overlapping region may be determined according to a plurality of sliding window regions with smaller energy values corresponding to the overlapping region of the adjacent images. Specifically, first, determining an adjacent image overlapping region of the target image overlapping region from the plurality of image overlapping regions, and then acquiring a plurality of second sliding window regions separated from the adjacent image overlapping region, wherein energy values corresponding to the plurality of second sliding window regions are smaller than a second preset threshold value. And determining a target first sliding window area from the selected first sliding window area according to the depth average value corresponding to each of the plurality of second sliding window areas and the depth average value corresponding to each of the selected first sliding window area, wherein the depth average value corresponding to the target first sliding window area is closest to the depth average value corresponding to each of the plurality of second sliding window areas in the selected first sliding window area. And finally, determining the target first sliding window area as an image overlapping sub-area corresponding to the target image overlapping area.

For example, assume that 4 scene images, namely scene image 1, scene image 2, scene image 3, and scene image 4, are captured for target scene a, and scene image 1 is adjacent to scene image 2, the image overlap region is a, scene image 2 is adjacent to scene image 3, the image overlap region is B, scene image 3 is adjacent to scene image 4, and the image overlap region is C. Assume that 3 first sliding window areas with energy values smaller than a second preset threshold value are selected from the image overlapping area a, namely a first sliding window area 1, a first sliding window area 2 and a first sliding window area 3, wherein the depth average value corresponding to the first sliding window area 1 is 1.1, the depth average value corresponding to the first sliding window area 2 is 1.5, and the depth average value corresponding to the first sliding window area 3 is 1.3. Assume that 3 second sliding window areas with energy values smaller than a second preset threshold value are selected from the image overlapping area B, namely a second sliding window area 1, a second sliding window area 2 and a second sliding window area 3, wherein the depth average value corresponding to the second sliding window area 1 is 2, the depth average value corresponding to the second sliding window area 2 is 2.5, and the depth average value corresponding to the second sliding window area 3 is 1.9.

When determining the image overlapping sub-region corresponding to the image overlapping region a from the three selected first sliding window regions, the image overlapping region adjacent to the image overlapping region a may be determined as the image overlapping region B, and then three second sliding window regions corresponding to the image overlapping region B may be acquired. Namely, the depth average values of the first sliding window area corresponding to the current image overlapping area A are respectively 1.1, 1.5 and 1.3, the depth average values of the second sliding window area corresponding to the current image overlapping area B are respectively 2, 2.5 and 1.9, the depth average values are sequenced, and the first sliding window area closest to the second sliding window area is selected to be determined as the image overlapping subarea corresponding to the image overlapping area A. I.e. the result after sorting is 1.1, 1.3, 1.5, 1.9, 2, 2.5, and the first sliding window area with the depth average value of 1.5 is determined as the image overlapping sub-area, i.e. the first sliding window area 2 is finally determined as the image overlapping sub-area corresponding to the image overlapping area a.

In the embodiment of the invention, based on the preset sliding window size, the target image overlapping area is segmented to obtain a plurality of first sliding window areas corresponding to the target image overlapping area, and the image overlapping subarea corresponding to the target image overlapping area is determined from the plurality of first sliding window areas according to the depth values of all the pixel points in the plurality of first sliding window areas, so that the obtained depth values of all the pixel points in the image overlapping subarea change smoothly and are closer to the depth values of all the pixel points in the adjacent areas, thereby facilitating the subsequent adjustment of all the pixel points in the scene image by utilizing the depth values of all the pixel points in the image subarea so as to achieve the difference of the depth values of all the pixel points in the smooth image overlapping area and the non-image overlapping area.

The above embodiments describe a specific implementation of selecting an image overlapping sub-region from each image overlapping region. After the image overlapping subareas corresponding to the image overlapping areas are selected, the depth values corresponding to the pixel points in the plurality of scene images are adjusted according to the depth values corresponding to the pixel points in the image overlapping subareas so as to determine the target depth values corresponding to the pixel points in the plurality of scene images. In order to facilitate understanding of the specific implementation process of selecting the image overlapping sub-regions in the above embodiment, in the following embodiment, an implementation process of determining, according to depth values corresponding to respective pixels in a plurality of image overlapping sub-regions, target depth values corresponding to respective pixels in a plurality of scene images is exemplarily described with reference to fig. 5.

Fig. 5 is a flowchart of determining a target depth value corresponding to each pixel point in a plurality of scene images according to a depth value corresponding to each pixel point in a plurality of overlapping sub-areas of the images, where, as shown in fig. 5, the method may include the following steps:

501. and respectively determining the depth average value of the plurality of image overlapping subareas according to the depth values corresponding to the pixel points in the plurality of image overlapping subareas.

502. And determining the depth average value corresponding to the depth map according to the depth average values of the overlapping subareas of the plurality of images.

503. And determining the depth average value corresponding to the depth map as the depth average value corresponding to the non-image overlapping region in the depth map.

504. And performing polynomial fitting on the depth average values of the plurality of image overlapping sub-areas and the depth average values corresponding to the non-image overlapping areas to obtain fitted depth values corresponding to each pixel of a center line of the depth map, wherein the center line refers to a line positioned at the center position of the depth map.

505. And determining target depth values corresponding to all the pixel points in the plurality of scene images according to the fitted depth values corresponding to all the pixel points in the center line of the depth map.

When processing the depth values corresponding to the pixels in the scene image, firstly, respectively determining the depth average value of the image overlapping subareas according to the depth values corresponding to the pixels in the image overlapping subareas. The depth average value of the image overlapping sub-region corresponding to each image overlapping region can be directly determined as the depth average value corresponding to the image overlapping region.

And then, determining the depth average value corresponding to the depth map according to the depth average value of the overlapped subareas of the plurality of images. Alternatively, the average value of the depth average values of the overlapping sub-regions of the plurality of images may be calculated and determined as the depth average value corresponding to the depth map. The depth average value corresponding to the depth map is equal to the depth average value corresponding to the target scene, namely, the depth average value corresponding to the target scene is determined according to the depth average values of the overlapping subareas of the plurality of images.

And then determining the depth average value corresponding to the depth map as the depth average value corresponding to the non-image overlapping region in the depth map, and performing polynomial fitting on the depth average values of the plurality of image overlapping sub-regions and the depth average value corresponding to the non-image overlapping region to obtain fitted depth values corresponding to each pixel in the center line of the depth map. Wherein the center row refers to a row located at the center of the depth map. And finally, determining target depth values corresponding to all the pixel points in the plurality of scene images according to the fitted depth values corresponding to all the pixel points in the center line of the depth map.

In an optional embodiment, according to the fitted depth values corresponding to the pixels in the center line of the depth map, a specific implementation manner of determining the target depth values corresponding to the pixels in the multiple scene images may include: the fitted depth values corresponding to the pixel points in the center line of the depth map are determined to be the depth values corresponding to the pixel points in the same column with the pixel points in the center line; and determining target depth values corresponding to the pixel points in the plurality of scene images according to the depth values corresponding to the pixel points in each column in the depth map. From the above description, it is clear that: and sequentially and respectively determining the depth values corresponding to the pixels in each column of the depth map according to the fitted depth values corresponding to the pixels in the center row of the depth map, and determining the target depth values corresponding to the pixels in the plurality of scene images based on the conversion relation between the plurality of scene images and the depth values corresponding to the pixels in each column of the depth map. Namely, the depth value corresponding to each pixel point in each column in the depth map takes the fitted depth value corresponding to the pixel point in the column in the central row.

In the embodiment of the invention, the depth average value of the plurality of image overlapping subareas is respectively determined according to the depth values corresponding to the pixel points in the plurality of image overlapping subareas, and the depth average value corresponding to the depth map is determined according to the depth average value of the plurality of image overlapping subareas. And determining the depth average value corresponding to the depth map as the depth average value corresponding to the non-image overlapping region in the depth map. Polynomial fitting is carried out on the depth average value of the plurality of image overlapping sub-areas and the depth average value corresponding to the non-image overlapping areas, fitted depth values corresponding to all pixels in the center line of the depth image are obtained, target depth values corresponding to all pixels in the plurality of scene images are determined according to the fitted depth values corresponding to all pixels in the center line of the depth image, the determined target depth values corresponding to all pixels are very close, the problem of inconsistent depth between all pixels in the image splicing edge area can be reduced, and therefore the problem of distortion or cracking during splicing and fusion of a plurality of scene images is avoided.

A panoramic image generation apparatus of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will appreciate that these means may be configured by the steps taught by the present solution using commercially available hardware components.

Fig. 6 is a schematic structural diagram of a panoramic image stitching device according to an embodiment of the present invention, as shown in fig. 6, where the device includes: the device comprises an acquisition module 11, a calibration module 12, a screening module 13, a determination module 14, a deployment module 15 and a fusion module 16.

The obtaining module 11 is configured to obtain a plurality of scene images of a target scene and a depth map of the target scene, where the plurality of scene images are obtained by photographing the target scene by a camera around the same rotation axis and at a plurality of different angles.

And the calibration module 12 is configured to identify, in the depth map, a plurality of image overlapping areas corresponding to a plurality of sets of adjacent scene images according to the camera pose and the camera view angle corresponding to each of the plurality of scene images, where the image overlapping area corresponding to each set of adjacent scene images refers to an overlapping area between two adjacent scene images included in the image overlapping area.

And the screening module 13 is configured to select an image overlapping sub-region from each image overlapping region, where a depth gradient value corresponding to each pixel point in the image overlapping sub-region is smaller than a first preset threshold.

The determining module 14 is configured to determine a target depth value corresponding to each pixel point in the multiple scene images according to the depth values corresponding to each pixel point in the overlapping sub-regions of the multiple images.

And the expansion module 15 is used for carrying out image expansion processing on the plurality of scene images according to the target depth value so as to obtain a plurality of plane expansion images.

And the fusion module 16 is used for fusing the plurality of plane expansion images to obtain a panoramic image corresponding to the target scene.

In an alternative embodiment, the screening module 13 may specifically be configured to: dividing the target image overlapping area based on a preset sliding window size to obtain a plurality of first sliding window areas corresponding to the target image overlapping area, wherein the target image overlapping area is any one of the plurality of image overlapping areas; and determining an image overlapping sub-region corresponding to the target image overlapping region from the plurality of first sliding window regions according to the depth values of the pixel points in the plurality of first sliding window regions.

In an alternative embodiment, the screening module 13 may specifically be configured to: calculating depth average values and total depth gradient values corresponding to the plurality of first sliding window areas; determining energy values corresponding to the first sliding window areas according to the depth average value and the total depth gradient value; selecting a first sliding window area with the energy value smaller than a second preset threshold value from the plurality of first sliding window areas according to the energy value; and determining an image overlapping sub-region corresponding to the target image overlapping region from the selected first sliding window region according to the respective depth average value of the selected first sliding window region.

In an alternative embodiment, the screening module 13 may specifically be configured to: determining adjacent image overlapping areas of the target image overlapping areas from the plurality of image overlapping areas; acquiring a plurality of second sliding window areas separated from the adjacent image overlapping areas, wherein the energy values corresponding to the second sliding window areas are smaller than the second preset threshold value; determining a target first sliding window area from the selected first sliding window area according to the depth average value corresponding to each of the plurality of second sliding window areas and the depth average value corresponding to each of the selected first sliding window areas, wherein the depth average value corresponding to the target first sliding window area is closest to the depth average value corresponding to each of the plurality of second sliding window areas in the selected first sliding window area; and determining the target first sliding window area as an image overlapping sub-area corresponding to the target image overlapping area.

In an alternative embodiment, the determining module 14 may specifically be configured to: respectively determining the depth average value of the plurality of image overlapping subareas according to the depth values corresponding to the pixels in the plurality of image overlapping subareas; determining a depth average value corresponding to the depth map according to the depth average values of the overlapping subareas of the plurality of images; determining a depth average value corresponding to the depth map as a depth average value corresponding to a non-image overlapping region in the depth map; performing polynomial fitting on the depth average values of the plurality of image overlapping sub-areas and the depth average values corresponding to the non-image overlapping areas to obtain fitted depth values corresponding to pixels in a center line of the depth map, wherein the center line is a line positioned at the center position of the depth map; and determining target depth values corresponding to all the pixel points in the plurality of scene images according to the fitted depth values corresponding to all the pixel points in the center line of the depth map.

In an alternative embodiment, the determining module 14 may specifically be configured to: the fitted depth values corresponding to the pixel points in the center line of the depth map are determined to be the depth values corresponding to the pixel points in the same column with the pixel points in the center line; and determining target depth values corresponding to all the pixel points in the plurality of scene images according to the depth values corresponding to all the pixel points in each column in the depth map.

In an alternative embodiment, the deployment module 15 may be specifically configured to: taking the axis of the rotating shaft as a projection center and taking a target depth value corresponding to each pixel point as a radius to determine a target projection spherical surface corresponding to each pixel point of the plurality of scene images; projecting each pixel point to a target projection spherical surface so as to determine the longitude and latitude coordinates of the target of each pixel point projected to the corresponding target projection spherical surface; and converting the longitude and latitude coordinates of the target corresponding to each pixel point into two-dimensional plane coordinates under a panoramic coordinate system to obtain plane expansion images corresponding to the scene images.

The apparatus shown in fig. 6 may perform the steps described in the foregoing embodiments, and detailed execution and technical effects are referred to in the foregoing embodiments and are not described herein.

In one possible design, the structure of the panoramic image stitching apparatus shown in fig. 6 may be implemented as an electronic device, as shown in fig. 7, where the electronic device may include: memory 21, processor 22, communication interface 23. Wherein the memory 21 has stored thereon executable code which, when executed by the processor 22, causes the processor 22 to at least implement the panoramic image stitching method as provided in the foregoing embodiments.

In addition, embodiments of the present invention provide a non-transitory machine-readable storage medium having executable code stored thereon, which when executed by a processor of an electronic device, causes the processor to at least implement the panoramic image stitching method as provided in the previous embodiments.

The apparatus embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by adding necessary general purpose hardware platforms, or may be implemented by a combination of hardware and software. Based on such understanding, the foregoing aspects, in essence and portions contributing to the art, may be embodied in the form of a computer program product, which may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A panoramic image stitching method, comprising:

2. The method of claim 1, wherein said selecting an image overlap sub-region from each of said image overlap regions comprises:

dividing the target image overlapping area based on a preset sliding window size to obtain a plurality of first sliding window areas corresponding to the target image overlapping area, wherein the target image overlapping area is any one of the plurality of image overlapping areas;

and determining an image overlapping sub-region corresponding to the target image overlapping region from the plurality of first sliding window regions according to the depth values of the pixel points in the plurality of first sliding window regions.

3. The method according to claim 2, wherein the determining, from the plurality of first sliding window areas, the image overlapping sub-area corresponding to the target image overlapping area according to the depth values of the respective pixel points in the plurality of first sliding window areas includes:

calculating depth average values and total depth gradient values corresponding to the plurality of first sliding window areas;

determining energy values corresponding to the first sliding window areas according to the depth average value and the total depth gradient value;

selecting a first sliding window area with the energy value smaller than a second preset threshold value from the plurality of first sliding window areas according to the energy value;

And determining an image overlapping sub-region corresponding to the target image overlapping region from the selected first sliding window region according to the respective depth average value of the selected first sliding window region.

4. The method according to claim 3, wherein the determining, from the selected first sliding window areas, the image overlapping sub-area corresponding to the target image overlapping area according to the respective depth average value of the selected first sliding window areas includes:

determining adjacent image overlapping areas of the target image overlapping areas from the plurality of image overlapping areas;

acquiring a plurality of second sliding window areas separated from the adjacent image overlapping areas, wherein the energy values corresponding to the second sliding window areas are smaller than the second preset threshold value;

determining a target first sliding window area from the selected first sliding window area according to the depth average value corresponding to each of the plurality of second sliding window areas and the depth average value corresponding to each of the selected first sliding window areas, wherein the depth average value corresponding to the target first sliding window area is closest to the depth average value corresponding to each of the plurality of second sliding window areas in the selected first sliding window area;

And determining the target first sliding window area as an image overlapping sub-area corresponding to the target image overlapping area.

5. The method according to any one of claims 1-4, wherein determining the target depth value for each pixel in the plurality of scene images based on the depth values for each pixel in the plurality of image overlapping sub-regions comprises:

respectively determining the depth average value of the plurality of image overlapping subareas according to the depth values corresponding to the pixel points in the plurality of image overlapping subareas;

determining a depth average value corresponding to the depth map according to the depth average values of the overlapping subareas of the plurality of images;

determining a depth average value corresponding to the depth map as a depth average value corresponding to a non-image overlapping region in the depth map;

performing polynomial fitting on the depth average values of the plurality of image overlapping sub-areas and the depth average values corresponding to the non-image overlapping areas to obtain fitted depth values corresponding to pixels in a center line of the depth map, wherein the center line is a line positioned at the center position of the depth map;

and determining target depth values corresponding to all the pixel points in the plurality of scene images according to the fitted depth values corresponding to all the pixel points in the center line of the depth map.

6. The method of claim 5, wherein determining the target depth value for each pixel in the plurality of scene images based on the fitted depth values for each pixel in the center line of the depth map comprises:

the fitted depth values corresponding to the pixel points in the center line of the depth map are determined to be the depth values corresponding to the pixel points in the same column with the pixel points in the center line;

and determining target depth values corresponding to all the pixel points in the plurality of scene images according to the depth values corresponding to all the pixel points in each column in the depth map.

7. The method of claim 1, wherein performing image expansion processing on the plurality of scene images according to the target depth value to obtain a plurality of plane expansion images comprises:

taking the axis of the rotating shaft as a projection center and taking a target depth value corresponding to each pixel point as a radius to determine a target projection spherical surface corresponding to each pixel point of the plurality of scene images;

projecting each pixel point to a target projection spherical surface so as to determine the longitude and latitude coordinates of the target of each pixel point projected to the corresponding target projection spherical surface;

And converting the longitude and latitude coordinates of the target corresponding to each pixel point into two-dimensional plane coordinates under a panoramic coordinate system to obtain plane expansion images corresponding to the scene images.

8. A panoramic image stitching device, comprising:

9. An electronic device, comprising: a memory, a processor, a communication interface; wherein the memory has stored thereon executable code which, when executed by the processor, causes the processor to perform the panoramic image stitching method of any of claims 1 to 7.

10. A non-transitory machine-readable storage medium having stored thereon executable code, which when executed by a processor of an electronic device, causes the processor to perform the panoramic image stitching method of any of claims 1-7.