WO2022140970A1

WO2022140970A1 - Panoramic image generation method and apparatus, movable platform and storage medium

Info

Publication number: WO2022140970A1
Application number: PCT/CN2020/140351
Authority: WO
Inventors: 谭卓; 郭浩铭; 李琛
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2020-12-28
Filing date: 2020-12-28
Publication date: 2022-07-07

Abstract

A panoramic image generation method and apparatus, a movable platform and a storage medium. The method comprises: acquiring at least one first image, which is photographed, by means of a main photographic apparatus mounted on a movable platform, within the field of view of the main photographic apparatus (S101); acquiring at least one second image, which is photographed, by means of at least one auxiliary photographic apparatus mounted on the movable platform, within the field of view range of the at least one auxiliary photographic apparatus, wherein the field of view range of the at least one auxiliary photographic apparatus comprises an upper area of the movable platform, and there is an overlapping range between the field of view range of the main photographic apparatus and the field of view range of the at least one auxiliary photographic apparatus (S102); and performing image fusion processing on the at least one first image and the at least one second image, so as to generate a corresponding panoramic image (S103).

Description

Panoramic image generation method, device, movable platform and storage medium

technical field

The present application relates to the technical field of movable platforms, and in particular, to a method, device, movable platform and storage medium for generating panoramic images.

Background technique

Cameras mounted on mobile platforms (such as drones) have important applications and can be used in aerial photography, inspection and other fields. At present, in the design of the movable platform, the maximum limit of the pitch axis of the gimbal is generally not more than 30 degrees. In addition, there is a fuselage occlusion above the camera, so that the camera cannot capture the image directly above the movable platform. Therefore, when performing panoramic shooting, the generated panoramic image will lack the upper part of the movable platform such as the "sky", which needs to be filled by certain means. The filling operation is cumbersome, and the filled "sky" is false and the effect is poor. unnatural.

Therefore, how to improve the quality of the panoramic image captured by the movable platform has become an urgent problem to be solved.

SUMMARY OF THE INVENTION

Based on this, the present application provides a panorama image generating method, device, movable platform and storage medium, so as to improve the quality of panorama images captured and generated based on the movable platform.

In a first aspect, the present application provides a method for generating a panoramic image, the method comprising:

acquiring at least one first image captured by the main camera mounted on the movable platform within the field of view of the main camera;

Obtain at least one second image captured by at least one auxiliary camera device mounted on the movable platform within the field of view of the auxiliary camera device, where the field of view of the auxiliary camera device includes the movable platform. In the upper area, there is an overlapping range between the field of view of the main camera device and the field of view of the auxiliary camera device;

Perform image fusion processing on the at least one first image and the at least one second image to generate a corresponding panoramic image.

In a second aspect, the present application further provides a panoramic image generating device, the panoramic image generating device comprising a memory and a processor;

the memory is used to store computer programs;

The processor is configured to execute the computer program and implement the following steps when executing the computer program:

In a third aspect, the present application further provides a movable platform, where the movable platform includes the above-mentioned panoramic image generating apparatus.

In a fourth aspect, the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor can realize the above-mentioned panoramic image generation method.

The panorama image generating method, device, movable platform and storage medium disclosed in the present application, by acquiring at least one first image captured by the main camera device mounted on the movable platform within the field of view of the main camera device, and acquiring a possible first image. At least one second image captured by at least one auxiliary camera device mounted on the mobile platform within the field of view of the auxiliary camera device, wherein the field of view of the auxiliary camera device includes the upper area of the movable platform, and the field of view of the main camera device. There is an overlapping range between the field range and the field of view range of the auxiliary camera device, and image fusion processing is performed on the acquired at least one first image and at least one second image to generate a corresponding panoramic image. The panoramic image obtained by the operation is more natural and real, and the quality of the panoramic image is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the present application.

Description of drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. For those of ordinary skill, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic block diagram of a movable platform provided by an embodiment of the present application;

2 is a schematic structural diagram of an unmanned aerial vehicle provided by an embodiment of the present application;

3 is a schematic flowchart of steps of a method for generating a panoramic image provided by an embodiment of the present application;

4 is a schematic diagram of an auxiliary camera device selection interface provided by an embodiment of the present application;

5 is a schematic flowchart of steps of another panoramic image generation method provided by an embodiment of the present application;

6 is a schematic flowchart of steps of another panoramic image generation method provided by an embodiment of the present application;

FIG. 7 is a schematic block diagram of a panoramic image generating apparatus provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The flowcharts shown in the figures are for illustration only, and do not necessarily include all contents and operations/steps, nor do they have to be performed in the order described. For example, some operations/steps can also be decomposed, combined or partially combined, so the actual execution order may be changed according to the actual situation.

It should be understood that the terms used in the specification of the present application herein are for the purpose of describing particular embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly dictates otherwise.

It will also be understood that, as used in this specification and the appended claims, the term "and/or" refers to and including any and all possible combinations of one or more of the associated listed items .

Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments and features of the embodiments may be combined with each other without conflict.

The embodiments of the present application provide a panorama image generating method, device, movable platform and storage medium, which are used to improve the quality of panorama images captured and generated based on the movable platform.

Wherein, the movable platform includes but is not limited to unmanned aerial vehicles, such as rotary-wing aircraft, including single-rotor aircraft, dual-rotor aircraft, tri-rotor aircraft, quad-rotor aircraft, hexa-rotor aircraft, octa-rotor aircraft, ten-rotor aircraft, twelve-rotor aircraft Rotorcraft, etc. Of course, the movable platform may also be other types of unmanned aerial vehicles or movable devices, such as fixed-wing unmanned aerial vehicles, and the embodiment of the present application is not limited thereto.

Please refer to FIG. 1 , which is a schematic block diagram of a movable platform according to an embodiment of the present application. As shown in FIG. 1 , the movable platform 1000 may include a body 100 , a power system 200 provided in the body 100 , a panoramic image generating device 300 , a main camera device 400 and at least one auxiliary camera device 500 . Wherein, the power system 200 is used to provide power for the movable platform 1000; the main camera device 400 and at least one auxiliary camera device 500 include but are not limited to cameras, which are used to capture images in different fields of view; the panoramic image generation device 300 is used for Panoramic images are generated from images captured by the main camera 400 and the at least one auxiliary camera 500 in different fields of view.

Exemplarily, at least one auxiliary camera device 500 includes two or more up-view cameras, the up-view cameras are usually used for obstacle avoidance, and the images captured by the up-view cameras are generally of low resolution and poor signal-to-noise ratio. , the dynamic range is small.

Exemplarily, the power system 200 may include one or more electronic governors (referred to as ESCs for short), one or more propellers, and one or more motors corresponding to the one or more propellers, wherein the motors are connected to the electronic between the governor and the propeller. The electronic governor is used to provide driving current to the motor to control the speed of the motor. The motor is used to drive the propeller to rotate, thereby providing power for the flight of the movable platform 1000, which enables the movable platform 1000 to achieve one or more degrees of freedom movement. In certain embodiments, the movable platform 1000 can rotate about one or more axes of rotation. It should be understood that the motor may be a DC motor or an AC motor. In addition, the motor may be a brushless motor or a brushed motor.

Exemplarily, taking the movable platform 1000 as an unmanned aerial vehicle as an example, as shown in FIG. 2 , the unmanned aerial vehicle includes a plurality of propellers, an anti-shake gimbal, a camera and at least one upward-looking camera (not shown). Among them, operations such as UAV attitude adjustment and UAV hovering can be realized by controlling the rotation of the propeller. The first image can be captured by a camera, and the second image can be captured by at least one upward-looking camera.

It can be understood that the above naming of the components of the movable platform 1000 is only for the purpose of identification, and therefore does not limit the embodiments of the present application.

The panoramic image generation method provided by the embodiments of the present application will be described in detail below based on the movable platform 1000 . It should be noted that the movable platform 1000 in FIG. 1 is only used to explain the panoramic image generation method provided by the embodiment of the present application, but does not constitute a limitation on the application scenario of the panoramic image generation method provided by the embodiment of the present application.

Please refer to FIG. 3 , which is a schematic flowchart of a method for generating a panoramic image provided by an embodiment of the present application. The method can be used in any movable platform provided by the above embodiments, so as to improve the quality of the panoramic image captured and generated based on the movable platform.

As shown in FIG. 3 , the panoramic image generation method specifically includes steps S101 to S103 .

S101. Acquire at least one first image captured by a main camera device mounted on a movable platform within a field of view of the main camera device.

S102: Acquire at least one second image captured by at least one auxiliary camera device mounted on the movable platform within the field of view of the auxiliary camera device, where the field of view of the auxiliary camera device includes the movable platform In the upper area of the platform, there is an overlapping range between the field of view of the main camera device and the field of view of the auxiliary camera device.

The movable platform is loaded with a main camera device and one or more auxiliary camera devices. Exemplarily, the field of view of the main camera device includes the front, rear, left, and right regions of the movable platform and the movable platform. The field of view of the auxiliary camera device includes the upper region of the movable platform, wherein the field of view of the main camera device and the field of view of the auxiliary camera device have overlapping and non-overlapping ranges.

One or more images are taken within the field of view of the main camera by using the main camera, and one or more images are taken within the field of view of the corresponding auxiliary camera by using one or more auxiliary cameras. For convenience of description, the image captured by the main camera device is hereinafter referred to as the first image, and the image captured by the auxiliary camera device is referred to as the second image.

In some embodiments, using a main camera device to capture multiple rows and multiple columns of first images within the field of view of the main camera device, the panoramic image generation method may further include: when the main camera device is in the main camera device When capturing the first image of each column within the field of view of the camera device, the at least one auxiliary camera device captures the second image within the field of view of the auxiliary camera device.

For example, it is assumed that 3 rows and 8 columns of first images are captured by the main camera within the field of view of the main camera. When the main camera captures the first image of each column within the field of view of the main camera, the One or more auxiliary camera devices capture second images within the field of view of the auxiliary camera devices, so as to obtain a plurality of first images and a plurality of second images. The panoramic image generated by using the multiple first images and the multiple second images improves the image quality of the panoramic image compared to the panoramic image generated only by using the single first image and the single second image.

In some embodiments, when a plurality of auxiliary cameras are mounted on the movable platform, some of the plurality of auxiliary cameras may be selected to capture the second image, and the panoramic image generation method may further include: According to the FOV (Field of view, field of view) information corresponding to the plurality of auxiliary camera devices loaded on the movable platform and the FOV information corresponding to the main camera device, the selected auxiliary camera device is selected from the plurality of auxiliary camera devices. at least one auxiliary camera device.

Since the FOV corresponding to different auxiliary camera devices is different, one or more auxiliary camera devices are selected from the plurality of auxiliary camera devices to capture the second image according to the FOV information corresponding to the main camera device and the FOV information corresponding to each auxiliary camera device. , there are overlapping ranges and non-overlapping ranges between the field of view range of the selected one or more auxiliary camera devices and the field of view range of the main camera device, so that there may be an overlapping portion between the second image captured and the first image, according to The second image and the first image may generate a panoramic image. In addition, since not all the auxiliary camera devices are used to capture the second image, the number of the second image is not so many, thereby reducing the workload of subsequent processing of the second image.

Exemplarily, the at least one auxiliary camera is selected from the plurality of auxiliary camera devices according to the FOV information corresponding to the plurality of auxiliary camera devices loaded on the movable platform and the FOV information corresponding to the main camera device The apparatus may include selecting a single auxiliary camera device from the plurality of auxiliary camera devices if the FOV of the auxiliary camera device is greater than or equal to a preset FOV threshold.

When the FOV of a certain auxiliary camera device is greater than or equal to the preset FOV threshold, that is, when the FOV of the auxiliary camera device is larger, the captured second image also covers a larger area. A single of the secondary cameras is selected for the second image capture.

For example, the movable platform can be rotated to cover -180 to 180 degrees in the longitude direction. As long as a certain auxiliary camera device and the main camera device can completely cover the latitude direction of -90 to 90 degrees, then the auxiliary camera device and the main camera device can completely cover the latitude direction of -90 to 90 degrees. It can cover the area corresponding to the entire 360*180 spherical panorama.

In some embodiments, acquiring at least one second image captured by at least one auxiliary camera device mounted on the movable platform within the field of view of the auxiliary camera device may include: if the FOV of the auxiliary camera device greater than or equal to the preset FOV threshold, obtain a single second image captured by a single auxiliary camera; if the FOV of the auxiliary camera is less than the preset FOV threshold, obtain a single auxiliary camera The plurality of second images captured at different orientations are rotated, or the plurality of second images captured by one or more of the auxiliary camera devices are acquired.

When the FOV of a certain auxiliary camera device is greater than or equal to the preset FOV threshold, that is, when the FOV of the auxiliary camera device is relatively large, for example, an area within a preset angle above the movable platform can be photographed once through the auxiliary camera device , a single second image is captured by the auxiliary camera device. For example, when the main camera device captures the first image of the first column within the field of view of the main camera device, a second image is captured by the auxiliary camera device.

Conversely, if the FOV of a single auxiliary camera device is less than the preset FOV threshold, that is, when the FOV of a single auxiliary camera device is small, for example, only a local area within a preset angle above the movable platform can be captured by a single auxiliary camera device at a time, Then, a plurality of second images are captured at different orientations by rotating the single auxiliary camera device, or a plurality of second images are captured by using a plurality of auxiliary camera devices. Regardless of whether a plurality of second images are captured in different directions by rotating a single auxiliary camera device, or a plurality of second images are captured by a plurality of sub-camera devices, an area within a preset angle above the movable platform can be covered.

For example, when the main camera device captures the first images of the first column and the fifth column within the field of view of the main camera device, each second image is captured by a single auxiliary camera device, thereby obtaining two second images. Of course, when the main camera device captures the first images of the first column and the fifth column within the field of view of the main camera device, each of the plurality of auxiliary camera devices can also capture a second image, so as to obtain a plurality of first images. For two images, for example, if one second image is captured in each of the first column and the fifth column by two auxiliary camera devices, four second images are obtained.

In some embodiments, the panoramic image generating method may further include: displaying an auxiliary camera device selection interface for a user to select from a plurality of auxiliary camera devices mounted on the movable platform displayed on the auxiliary camera device selection interface , select a target auxiliary camera device; and determine the target auxiliary camera device selected by the user as the at least one auxiliary camera device.

For a plurality of auxiliary camera devices loaded on the movable platform, before the second image is captured, a preset auxiliary camera device selection interface is displayed, wherein the auxiliary camera device selection interface displays a plurality of auxiliary camera devices loaded on the movable platform. For the auxiliary camera device, the user can select one or more target auxiliary camera devices from the plurality of auxiliary camera devices mounted on the movable platform displayed on the auxiliary camera device selection interface. Based on the user's selection operation, one or more target auxiliary camera apparatuses selected by the user are determined as at least one auxiliary camera apparatus for capturing the second image. For example, as shown in FIG. 4 , the auxiliary camera device 1, the auxiliary camera device 2, the auxiliary camera device 3, and the auxiliary camera device i loaded on the movable platform are displayed on the auxiliary camera device selection interface. In the device, if the auxiliary camera 1 and the auxiliary camera 2 are selected, the auxiliary camera 1 and the auxiliary camera 2 are determined as the auxiliary camera for capturing the second image. Determining the at least one auxiliary camera device through a user operation further improves the user's interactive experience.

In some embodiments, the panoramic image generating method may further include: selecting the at least one auxiliary camera device from a plurality of auxiliary camera devices loaded on the movable platform according to the current shooting mode.

Shooting based on a movable platform includes a variety of shooting modes, including but not limited to vertical panorama mode, spherical panorama mode, and the like. For different shooting modes, correspondingly, different auxiliary camera devices are used for shooting the second image. During actual shooting, according to the current shooting mode, one or more auxiliary camera devices are selected from the plurality of auxiliary camera devices mounted on the movable platform for shooting the second image.

Exemplarily, the selecting the at least one auxiliary camera device from the plurality of auxiliary camera devices loaded on the movable platform according to the current shooting mode may include: if the current shooting mode is the vertical panorama mode, from the A single auxiliary camera device is selected from the plurality of auxiliary camera devices; if the current shooting mode is a spherical panorama mode, more than one auxiliary camera device is selected from the plurality of auxiliary camera devices.

For the vertical panorama mode, the area above the movable platform is not required to be photographed, and a single auxiliary camera device can meet the requirements. Therefore, if the current shooting mode is the vertical panorama mode, multiple auxiliary camera devices loaded on the movable platform are required. to select a single secondary camera for capturing the second image. For spherical panorama mode, it is required to capture the entire spherical area of the movable platform, and a single auxiliary camera device may not meet the requirements. Therefore, if the current shooting mode is spherical panorama mode, multiple auxiliary camera devices mounted on the movable platform One or more auxiliary camera devices are selected for capturing the second image. Selecting the auxiliary camera device according to the current shooting mode can not only meet the shooting requirements, but also avoid the use of too many auxiliary camera devices, resulting in waste of resources.

S103. Perform image fusion processing on the at least one first image and the at least one second image to generate a corresponding panoramic image.

After obtaining at least one first image and at least one second image through the main camera device and the auxiliary camera device, image fusion processing is performed on the at least one first image and the at least one second image. For example, if a first image is captured by the main camera device, and a second image is captured by each of the two auxiliary camera devices, image fusion processing is performed on the first image and the two second images obtained by shooting to generate the corresponding panoramic image.

In some embodiments, as shown in FIG. 5 , step S104 may be included before step S103 , and step S103 may include sub-step S1031 .

S104: Perform image preprocessing on the at least one first image or the at least one second image, so that the resolution of the at least one first image is the same as the resolution of the at least one second image match.

S1031. Perform image fusion processing on the at least one first image and the at least one second image whose resolutions are matched.

Since at least one first image and at least one second image are obtained by using the main camera device and the auxiliary camera device to capture, respectively, the resolutions of the first image and the second image will be different. Therefore, in order to achieve a better fusion effect, after obtaining at least one first image and at least one second image, image preprocessing is performed on the at least one first image or the at least one second image. The preprocessing matches the resolution of the at least one first image with the resolution of the at least one second image. The image preprocessing includes but is not limited to image upsampling processing, image downsampling processing, and the like. Then, image fusion processing is performed on the at least one first image and the at least one second image whose resolutions are matched, compared to directly performing image fusion on the obtained at least one first image and at least one second image The fusion process further improves the image quality of the generated panoramic image.

In some embodiments, performing image preprocessing on the at least one first image or the at least one second image may include: if the resolution of the at least one second image is smaller than that of the at least one first image If the resolution of the at least one second image is larger than the resolution of the at least one first image, perform image upsampling processing on the at least one second image; Perform image downsampling processing on the at least one second image, or perform image upsampling processing on the at least one first image.

In one case, the resolution of the second image is smaller than the resolution of the first image, that is, the resolution of the first image is high, and the resolution of the second image is low. The sampling process increases the resolution of the second image, so that the resolution of the at least one first image matches the resolution of the at least one second image.

In another case, the resolution of the second image is greater than the resolution of the first image, that is, the resolution of the first image is low, and the resolution of the second image is high. The down-sampling process increases the resolution of the second image, or performs image up-sampling processing on each first image, so that the resolution of at least one first image matches the resolution of at least one second image.

In some embodiments, performing image upsampling processing on the at least one second image may include: inputting the at least one second image into a neural network model to perform superdivision upsampling processing; or, using a bicubic interpolation algorithm Perform image upsampling processing on the at least one second image.

In order to improve the resolution of the second image, exemplarily, a neural network super-resolution technique can be used to input at least one second image into a corresponding neural network model for super-resolution and up-sampling processing, and output a high-resolution second image . Alternatively, an interpolation algorithm, such as a bicubic interpolation algorithm, is used to perform image upsampling processing on at least one second image to obtain at least one second image with high resolution.

In some embodiments, as shown in FIG. 6 , step S105 may be included before step S103 , and step S103 may include sub-step S1032 .

S105. Perform HDR fusion processing on a plurality of the second images captured by each of the auxiliary camera devices, respectively, to generate a high dynamic range image corresponding to each of the auxiliary camera devices.

S1032. Perform image fusion processing on the at least one first image and the high dynamic range image.

If multiple second images are captured by one or more auxiliary camera devices, in order to further improve the quality of the panoramic image, before performing the image fusion processing of the first image and the second image, the HDR (High-Dynamic Range, high dynamic lighting rendering) fusion processing is performed on the plurality of second images to generate a high dynamic range image corresponding to each auxiliary camera device. For example, if the auxiliary camera 1 and the auxiliary camera 2 are used to capture multiple second images, respectively, the multiple second images captured by the auxiliary camera 1 are subjected to HDR fusion processing to generate the corresponding first high dynamic range image. , performing HDR fusion processing on the plurality of second images captured by the auxiliary camera device 2 to generate a corresponding second high dynamic range image. Then, image fusion processing is performed on at least one first image captured by the main camera device and a high dynamic range image generated by performing HDR fusion processing. For example, image fusion processing is performed on at least one first image, the first high dynamic range image, and the second high dynamic range image. Compared with directly performing image fusion processing on the obtained at least one first image and at least one second image, the image quality of the generated panoramic image is further improved.

In some embodiments, before performing the image fusion processing of the first image and the second image, HDR fusion processing is performed on a plurality of second images captured by each auxiliary camera device to generate a high dynamic image corresponding to each auxiliary camera device. range image. If the resolution of the at least one high dynamic range image is smaller than the resolution of the at least one first image, perform image upsampling processing on the at least one high dynamic range image. If the resolution of at least one high dynamic range image is greater than the resolution of at least one first image, perform image downsampling processing on at least one high dynamic range image, or perform image upsampling processing on at least one first image . Match the resolution of the at least one first image with the resolution of the at least one high dynamic range image, and then perform image fusion processing on the at least one first image and the at least one high dynamic range image whose resolutions match, to generate the corresponding panoramic image. Through HDR fusion processing and image up/down sampling processing, the quality of panoramic images is further improved.

In some embodiments, before performing image fusion processing on the at least one first image and the at least one second image, the process may include: when the first image includes multiple images, performing image fusion processing on the multiple first images. Perform image fusion processing on the images to obtain a corresponding first fusion image; when the second images include multiple images, perform image fusion processing on the plurality of second images to obtain a corresponding second fusion image; Performing image fusion processing on the at least one first image and the at least one second image may include: performing image fusion processing on the first fusion image and the second fusion image.

For example, it is assumed that 3 rows and 8 columns of first images are captured by the main camera device within the field of view of the main camera device to obtain multiple first images, and image fusion processing is performed on the multiple first images to obtain the corresponding first images. a fused image. Correspondingly, if a plurality of second images are captured by at least one auxiliary camera device within the field of view of the auxiliary camera device, image fusion processing is also performed on the plurality of second images to obtain corresponding second fusion images. Then, image fusion processing is performed on the first fused image and the second fused image to generate a corresponding panoramic image.

In some embodiments, before performing image fusion processing on the plurality of first images, illumination compensation is performed on the plurality of first images, so that the brightness of the plurality of first images is consistent, thereby avoiding the fusion of the plurality of first images. Brightness differences appear in the first fused image.

Similarly, before performing image fusion processing on the plurality of second images, illumination compensation is performed on the plurality of second images, so that the brightness of the plurality of second images is consistent. And, before performing image fusion processing on the first fusion image and the second fusion image, first perform illumination compensation on the first fusion image and the second fusion image, so that the brightness of the first fusion image and the second fusion image are consistent, so as to ensure The quality of the resulting panorama image.

In some embodiments, performing image fusion processing on the plurality of second images to obtain the corresponding second fused images may include: performing latitude and longitude mapping on the plurality of second images to obtain the corresponding plurality of second local regions image; perform a first seam finding operation on a plurality of the second local area images; according to the results of the first seam finding operation, perform multi-band fusion () processing on a plurality of the second local area images to obtain the second Fused images.

For example, assuming that the second image includes the img_up_1 image and the img_up_2 image, respectively perform latitude and longitude mapping on the img_up_1 image and the img_up_2 image to obtain the second local area image sph_up_1 corresponding to the img_up_1 image and the second local area image sph_up_2 corresponding to the img_up_2 image. After that, a seam finding operation is performed on the two second local area images sph_up_1 and sph_up_2. In order to facilitate the distinguishing description, the seam finding operation of the two second local area images sph_up_1 and sph_up_2 is hereinafter referred to as the first seam finding operation.

In some embodiments, performing the first seam finding operation on the plurality of second partial area images may include: acquiring a first overlapping area of the plurality of second partial area images; performing the first overlapping area according to the first overlapping area. The first seam finding operation is described.

Still taking the above-mentioned two second local area images sph_up_1 and sph_up_2 as examples, extract the overlapping area of the two second local area images sph_up_1 and sph_up_2, perform a seam finding operation according to the overlapping area, and find a curve with the smallest difference in the overlapping area. In order to facilitate the distinguishing description, the overlapping area of the two second partial area images sph_up_1 and sph_up_2 is hereinafter referred to as the first overlapping area.

In some embodiments, before performing latitude and longitude mapping on the plurality of second images may include: optimizing the internal parameter matrix and the external parameter matrix corresponding to the plurality of second images; Performing latitude and longitude mapping may include: performing latitude and longitude mapping on a plurality of the second images according to the optimized internal parameter matrix and the external parameter matrix corresponding to a plurality of the second images to obtain a plurality of the second partial images. area images, and multiple mask images corresponding to the second partial area images; the acquiring the first overlapping area of the multiple second partial area images may include: corresponding to the multiple second partial area images The mask image is obtained, and the first overlapping area of a plurality of the second partial area images is obtained.

Among them, the internal and external parameters of the main camera device and the auxiliary camera device are first calibrated, and the main camera device is centered, that is, under the condition that the yaw, pitch, and roll angles are all set to 0, the internal and external parameters of the auxiliary camera device are calibrated to obtain the auxiliary camera. The internal parameter matrix K and the external parameter matrix (including the rotation matrix R and the translation matrix T) of the device are generally ignored when the translation matrix T is used. For example, if the auxiliary camera device includes the auxiliary camera device 1 and the auxiliary camera device 2, the internal parameter matrix K_up_1 and the external parameter matrix (R_up_1, T_up_1) of the auxiliary camera device 1 and the internal parameter matrix K_up_2 and the external parameter matrix ( R_up_2, T_up_2).

Based on the calibrated internal parameter matrix and external parameter matrix, the internal parameter matrix corresponding to the first image captured by the main camera device and the second image captured by the auxiliary camera device are known, and the pose relationship between the two first images can be directly obtained by The IMU (Inertial measurement unit, inertial measurement unit) data of the two first images is derived, and the pose relationship between the first image and the second image can be derived through the IMU data or the calibrated rotation matrix R, so as to obtain the external image of the second image. parameter matrix.

In use, by optimizing the internal parameter matrix and the external parameter matrix corresponding to the plurality of second images, more accurate internal parameter matrix and external parameter matrix corresponding to the second image are obtained. In some embodiments, the optimizing the internal parameter matrices and the external parameter matrices corresponding to the plurality of second images may include: based on the at least one first image, determining, based on the at least one first image, to perform the process with the plurality of second images. The target first image matched by the image feature points; the determined first target image and a plurality of the second images are subjected to image feature point matching to obtain corresponding image feature point matching information; the image feature point matching information , and the internal parameter matrix and external parameter matrix corresponding to the first image of the target, the internal parameter matrix and the external parameter matrix corresponding to a plurality of the second images, input beam method adjustment model processing, and obtain a plurality of the second images corresponding to The optimized internal parameter matrix and the external parameter matrix.

Exemplarily, if the first image includes a single image, the first image of the single image is determined as the target first image; if the first image includes multiple images, all the first images are determined as the target first image. , or select several first images from the plurality of first images according to preset rules, and determine them as the target first images.

Exemplarily, the main camera device captures first images of multiple rows and columns within the field of view of the main camera device to obtain first images of multiple rows and columns. By selecting the first image of the interval column from the first images of the plurality of columns, it is determined as the target first image. For example, the first images in columns 1, 3, 5, and 7 are selected to be determined as the target first images.

The external parameter matrix (rotation matrix R) corresponding to each second image includes the pose data between each second image and other second images, and the pose data between each second image and the target first image . The pose data between each second image and the target first image is derived from the calibrated rotation matrix R, or derived from the IMU data of each second image and the IMU data of the target first image.

After that, perform image feature point matching on the determined target first image and multiple second images to obtain corresponding image feature point matching information, and match the obtained image feature point matching information, as well as the internal parameter matrix and external parameter matrix corresponding to the target first image. The parameter matrix, the internal parameter matrix and the external parameter matrix corresponding to multiple second images, input the Bundle Adjustment beam method adjustment model for processing, and obtain the optimized internal parameter matrix and external parameter matrix corresponding to the target first image, as well as multiple second images. The corresponding optimized internal parameter matrix and external parameter matrix. For example, it is still assumed that the second image includes the img_up_1 image and the img_up_2 image, and the optimized internal parameter matrix K_up_refine1 and the external parameter matrix R_up_refine_1 corresponding to the img_up_1 image, and the optimized internal parameter matrix K_up_refine2 and the external parameter matrix R_up_refine_2 corresponding to the img_up_2 image are obtained.

According to the optimized internal parameter matrix and external parameter matrix corresponding to the img_up_1 image and the img_up_2 image, map the img_up_1 image and the img_up_2 image to the latitude and longitude to obtain the second local area image corresponding to the img_up_1 image sph_up_1, sph_up_1 corresponding mask image mask_up_1, and img_up_2 The mask image mask_up_2 corresponding to the second local area images sph_up_2 and sph_up_2 corresponding to the images. According to the mask image mask_up_1 and the mask image mask_up_2, extract the first overlapping area of the two second local area images sph_up_1 and sph_up_2.

The first seam finding operation is performed according to the first overlapping area, and a curve with the smallest difference is found in the first overlapping area. After that, based on the curve with the smallest difference obtained by the first seam finding operation, perform multi-band blend processing on the two second local area images sph_up_1 and sph_up_2 to obtain the second fusion image sph_up and the second The mask image mask_up corresponding to the fusion image sph_up.

At the same time, ERP (equirectangular projection, spherical projection) processing is performed on the first image to obtain a first fusion image sph_side corresponding to the first image and a mask image mask_side corresponding to the first fusion image sph_side.

Then, according to the mask image mask_side and the mask image mask_up, the second overlapping area of the first fused image sph_side and the second fused image sph_up is extracted. A second seam finding operation is performed according to the second overlapping area, and a curve with the smallest difference is found in the second overlapping area. After that, based on the curve with the smallest difference obtained by the second seam finding operation, multi-band fusion processing is performed on the first fusion image sph_side and the second fusion image sph_up to obtain a corresponding panoramic image.

In the above-mentioned embodiments, at least one first image captured by the main camera mounted on the movable platform within the field of view of the main camera is acquired, and at least one auxiliary camera mounted on the movable platform is acquired in the field of view of the auxiliary camera. At least one second image captured within the field of view, wherein the field of view of the auxiliary camera device includes the upper area of the movable platform, and there is an overlapping range between the field of view of the main camera device and the field of view of the auxiliary camera device , and perform image fusion processing on the obtained at least one first image and at least one second image to generate a corresponding panoramic image, which is more natural and realistic than the panoramic image obtained by the filling operation, and improves the panoramic image. the quality of.

Please refer to FIG. 7 , which is a schematic block diagram of a panoramic image generating apparatus provided by an embodiment of the present application.

As shown in FIG. 7 , the panoramic image generating apparatus 300 may include a processor 311 and a memory 312, and the processor 311 and the memory 312 are connected through a bus, such as an I2C (Inter-integrated Circuit) bus.

Specifically, the processor 311 may be a micro-controller unit (Micro-controller Unit, MCU), a central processing unit (Central Processing Unit, CPU), or a digital signal processor (Digital Signal Processor, DSP) or the like.

Specifically, the memory 312 may be a Flash chip, a read-only memory (ROM, Read-Only Memory) magnetic disk, an optical disk, a U disk, or a mobile hard disk, and the like. Various computer programs to be executed by the processor 311 are stored in the memory 312 .

Wherein, the processor is used for running the computer program stored in the memory, and implements the following steps when executing the computer program:

In some embodiments, before performing the image fusion processing on the at least one first image and the at least one second image, the processor is configured to:

performing image preprocessing on the at least one first image or the at least one second image so that the resolution of the at least one first image matches the resolution of the at least one second image;

When implementing the image fusion processing on the at least one first image and the at least one second image, the processor is configured to implement:

Perform image fusion processing on the at least one first image and the at least one second image whose resolutions are matched.

In some embodiments, when implementing the image preprocessing on the at least one first image or the at least one second image, the processor is configured to implement:

If the resolution of the at least one second image is smaller than the resolution of the at least one first image, perform image upsampling processing on the at least one second image;

If the resolution of the at least one second image is greater than the resolution of the at least one first image, perform image downsampling processing on the at least one second image, or perform image downsampling on the at least one second image, or perform an image downsampling process on the at least one second image. An image is subjected to image upsampling processing.

In some embodiments, when implementing the image upsampling process on the at least one second image, the processor is configured to implement:

inputting the at least one second image into a neural network model for super-sampling and upsampling; or

An image upsampling process is performed on the at least one second image by using a bicubic interpolation algorithm.

respectively performing HDR fusion processing on a plurality of the second images captured by each of the auxiliary cameras to generate a high dynamic range image corresponding to each of the auxiliary cameras;

Perform image fusion processing on the at least one first image and the high dynamic range image.

When the first images include multiple sheets, image fusion processing is performed on the multiple sheets of the first images to obtain corresponding first fusion images;

When the second images include multiple sheets, image fusion processing is performed on the multiple sheets of the second images to obtain corresponding second fusion images;

When implementing the image fusion processing on the at least one first image and the at least one second image, the processor is used to implement:

Perform image fusion processing on the first fusion image and the second fusion image.

In some embodiments, when implementing the image fusion processing on the plurality of second images to obtain the corresponding second fused images, the processor is configured to implement:

Performing latitude and longitude mapping on a plurality of the second images to obtain a plurality of corresponding second local area images;

performing a first seam finding operation on a plurality of the second local area images;

According to the result of the first seam finding operation, multi-band fusion processing is performed on a plurality of the second local area images to obtain the second fusion image.

In some embodiments, when implementing the first seam finding operation on the plurality of second partial region images, the processor is configured to implement:

acquiring a first overlapping area of a plurality of the second partial area images;

The first seam finding operation is performed according to the first overlapping area.

In some embodiments, before the processor implements the latitude and longitude mapping on the plurality of second images, the processor is configured to implement:

Optimizing the internal parameter matrix and the external parameter matrix corresponding to the plurality of second images;

When implementing the latitude and longitude mapping on the plurality of second images, the processor is configured to implement:

According to the optimized internal parameter matrix and the external parameter matrix corresponding to the plurality of second images, perform latitude and longitude mapping on the plurality of second images to obtain a plurality of the second local area images and a plurality of a mask image corresponding to the second local area image;

When implementing the acquiring of the first overlapping area of the plurality of second partial area images, the processor is configured to implement:

The first overlapping area of the plurality of second partial area images is acquired according to the mask images corresponding to the plurality of second partial area images.

In some embodiments, when implementing the optimization of the internal parameter matrices and the external parameter matrices corresponding to the plurality of second images, the processor is configured to implement:

determining, based on the at least one first image, a target first image to be matched with a plurality of the second images for image feature point matching;

Perform image feature point matching on the determined first image of the target and a plurality of the second images to obtain corresponding image feature point matching information;

The matching information of the image feature points, the internal parameter matrix and the external parameter matrix corresponding to the first image of the target, the internal parameter matrix and the external parameter matrix corresponding to the plurality of second images, and the input beam method adjustment model are processed to obtain The optimized internal parameter matrix and the external parameter matrix corresponding to the plurality of second images.

In some embodiments, the extrinsic parameter matrix corresponding to each of the second images includes pose data between each of the second images and other second images, and each of the second images and the The pose data between the first images of the target, wherein the pose data between each of the second images and the first image of the target is determined by a calibrated rotation matrix, or the IMU of each second image. The data is generated with the IMU data of the first image of the target.

In some embodiments, when implementing the determining, based on the at least one first image, the target first image to be matched with the image feature points of the plurality of second images, the processor is configured to implement:

If the at least one first image includes a single image, determining the single first image as the target first image;

If the at least one first image includes multiple images, determine all the first images as the target first image, or select several first images from the multiple first images according to a preset rule an image, determined as the first image of the target.

In some embodiments, the plurality of first images include multiple columns, and the processor selects a plurality of first images from the plurality of first images according to a preset rule, and determines them as the target The first image is used to achieve:

From the first images in the plurality of columns, the first images in the interval column are selected and determined as the target first images.

In some embodiments, the processor is further configured to:

When the main camera device captures the first image of each column within the field of view of the main camera device, the at least one auxiliary camera device performs the first image capture within the field of view of the auxiliary camera device The second image is taken.

In some embodiments, the processor is further configured to:

The at least one auxiliary camera device is selected from the plurality of auxiliary camera devices according to the FOV information corresponding to the plurality of auxiliary camera devices mounted on the movable platform and the FOV information corresponding to the main camera device.

In some embodiments, the processor implements the method according to the FOV information corresponding to the plurality of auxiliary camera devices loaded on the movable platform and the FOV information corresponding to the main camera device, from the plurality of auxiliary camera devices. When the at least one auxiliary camera device is selected in the camera device, it is used to realize:

If the FOV of the auxiliary camera device is greater than or equal to a preset FOV threshold, a single auxiliary camera device is selected from the plurality of auxiliary camera devices.

In some embodiments, when the processor implements the acquiring at least one second image captured by at least one auxiliary camera device mounted on the movable platform within the field of view of the auxiliary camera device, use To achieve:

If the FOV of the auxiliary camera device is greater than or equal to a preset FOV threshold, acquiring a single second image captured by a single auxiliary camera device;

If the FOV of the auxiliary camera device is less than the preset FOV threshold, acquire multiple second images shot by a single auxiliary camera device rotated in different directions, or acquire images shot by more than one auxiliary camera device a plurality of the second images.

In some embodiments, the processor is further configured to:

displaying an auxiliary camera device selection interface for the user to select a target auxiliary camera device from a plurality of auxiliary camera devices mounted on the movable platform displayed on the auxiliary camera device selection interface;

The target auxiliary camera device selected by the user is determined as the at least one auxiliary camera device.

In some embodiments, the processor is further configured to:

The at least one auxiliary camera device is selected from a plurality of auxiliary camera devices loaded on the movable platform according to the current shooting mode.

In some embodiments, when the processor selects the at least one auxiliary camera device from a plurality of auxiliary camera devices loaded on the movable platform according to the current shooting mode, the processor specifically implements:

If the current shooting mode is a vertical panorama mode, selecting a single auxiliary camera device from the plurality of auxiliary camera devices;

If the current shooting mode is a spherical panorama mode, selecting one or more of the auxiliary camera devices from the plurality of auxiliary camera devices.

The embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, the computer program includes program instructions, and the processor executes the program instructions to implement the present application The steps of the panorama image generating method provided by the embodiment.

Wherein, the computer-readable storage medium may be the internal storage unit of the movable platform or the panoramic image generating device described in the foregoing embodiments, such as a hard disk or memory of the movable platform or the panoramic image generating device. The computer-readable storage medium can also be an external storage device of the movable platform or the panoramic image generation device, such as a plug-in hard disk equipped on the movable platform or the panoramic image generation device, a smart memory card (Smart Media) Card, SMC), secure digital (Secure Digital, SD) card, flash memory card (Flash Card), etc.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalents within the technical scope disclosed in the present application. Modifications or substitutions shall be covered by the protection scope of this application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

A method for generating panoramic images, comprising:

acquiring at least one first image captured by the main camera mounted on the movable platform within the field of view of the main camera;

Obtain at least one second image captured by at least one auxiliary camera device mounted on the movable platform within the field of view of the auxiliary camera device, where the field of view of the auxiliary camera device includes the movable platform. In the upper area, there is an overlapping range between the field of view of the main camera device and the field of view of the auxiliary camera device;

Perform image fusion processing on the at least one first image and the at least one second image to generate a corresponding panoramic image.
The method according to claim 1, wherein before performing image fusion processing on the at least one first image and the at least one second image, the method comprises:

performing image preprocessing on the at least one first image or the at least one second image so that the resolution of the at least one first image matches the resolution of the at least one second image;

The performing image fusion processing on the at least one first image and the at least one second image includes:

Perform image fusion processing on the at least one first image and the at least one second image whose resolutions are matched.
The method according to claim 2, wherein the performing image preprocessing on the at least one first image or the at least one second image comprises:

If the resolution of the at least one second image is smaller than the resolution of the at least one first image, perform image upsampling processing on the at least one second image;

If the resolution of the at least one second image is greater than the resolution of the at least one first image, perform image down-sampling processing on the at least one second image, or perform image downsampling on the at least one second image, or perform an image downsampling process on the at least one first image. The image is subjected to image upsampling processing.
The method according to claim 3, wherein the performing image upsampling processing on the at least one second image comprises:

inputting the at least one second image into a neural network model for super-sampling and upsampling; or

An image upsampling process is performed on the at least one second image by using a bicubic interpolation algorithm.
The method according to claim 1, wherein before performing image fusion processing on the at least one first image and the at least one second image, the method comprises:

respectively performing HDR fusion processing on a plurality of the second images captured by each of the auxiliary cameras to generate a high dynamic range image corresponding to each of the auxiliary cameras;

The performing image fusion processing on the at least one first image and the at least one second image includes:

Perform image fusion processing on the at least one first image and the high dynamic range image.
The method according to claim 1, wherein before performing image fusion processing on the at least one first image and the at least one second image, the method comprises:

When the first images include multiple sheets, image fusion processing is performed on the multiple sheets of the first images to obtain corresponding first fusion images;

When the second images include multiple sheets, image fusion processing is performed on the multiple sheets of the second images to obtain corresponding second fusion images;

The performing image fusion processing on the at least one first image and the at least one second image includes:

Perform image fusion processing on the first fusion image and the second fusion image.
The method according to claim 6, wherein the performing image fusion processing on a plurality of the second images to obtain the corresponding second fusion images, comprising:

Performing latitude and longitude mapping on a plurality of the second images to obtain a plurality of corresponding second local area images;

performing a first seam finding operation on a plurality of the second local area images;

According to the result of the first seam finding operation, multi-band fusion processing is performed on a plurality of the second local area images to obtain the second fusion image.
The method according to claim 7, wherein the performing the first seam finding operation on the plurality of second partial region images comprises:

acquiring a first overlapping area of a plurality of the second partial area images;

The first seam finding operation is performed according to the first overlapping area.
The method according to claim 7 or 8, wherein before the performing latitude and longitude mapping on a plurality of the second images, the method comprises:

Optimizing the internal parameter matrix and the external parameter matrix corresponding to the plurality of second images;

The performing latitude and longitude mapping on the plurality of second images includes:

According to the optimized internal parameter matrix and the external parameter matrix corresponding to the plurality of second images, perform latitude and longitude mapping on the plurality of second images to obtain a plurality of the second local area images and a plurality of a mask image corresponding to the second local area image;

The acquiring the first overlapping area of the second partial area images includes:

The first overlapping area of the plurality of second partial area images is acquired according to the mask images corresponding to the plurality of second partial area images.
The method according to claim 9, wherein the optimizing the internal parameter matrix and the external parameter matrix corresponding to the plurality of second images comprises:

determining, based on the at least one first image, a target first image to be matched with a plurality of the second images for image feature point matching;

Perform image feature point matching on the determined first image of the target and a plurality of the second images to obtain corresponding image feature point matching information;

The matching information of the image feature points, the internal parameter matrix and the external parameter matrix corresponding to the first image of the target, the internal parameter matrix and the external parameter matrix corresponding to the plurality of second images, and the input beam method adjustment model are processed to obtain The optimized internal parameter matrix and the external parameter matrix corresponding to the plurality of second images.
The method according to claim 10, wherein the extrinsic parameter matrix corresponding to each second image comprises pose data between each second image and other second images, and each The pose data between the second image and the target first image, wherein the pose data between each second image and the target first image is determined by a calibrated rotation matrix, or each The IMU data of the second image is generated with the IMU data of the target first image.
The method according to claim 10 or 11, wherein the determining, based on the at least one first image, the target first image to be matched with the image feature points of the plurality of second images comprises:

If the at least one first image includes a single image, determining the single first image as the target first image;

If the at least one first image includes multiple images, determine all the first images as the target first image, or select several first images from the multiple first images according to a preset rule an image, determined as the first image of the target.
The method according to claim 12, wherein the plurality of first images include multiple columns, and the plurality of first images are selected from the plurality of first images according to a preset rule, and are determined as the Target first image, including:

From the first images in the plurality of columns, the first images in the interval column are selected and determined as the target first images.
The method according to any one of claims 1 to 13, wherein the method further comprises:

When the main camera device captures the first image of each column within the field of view of the main camera device, the at least one auxiliary camera device performs the first image capture within the field of view of the auxiliary camera device The second image is taken.
The method according to any one of claims 1 to 14, wherein the method further comprises:

The at least one auxiliary camera device is selected from the plurality of auxiliary camera devices according to the FOV information corresponding to the plurality of auxiliary camera devices mounted on the movable platform and the FOV information corresponding to the main camera device.
The method according to claim 15, wherein, according to the FOV information corresponding to the plurality of auxiliary camera devices and the FOV information corresponding to the main camera device loaded on the movable platform, the data are obtained from the plurality of auxiliary cameras. Selecting the at least one auxiliary camera device from the auxiliary camera devices includes:

If the FOV of the auxiliary camera device is greater than or equal to a preset FOV threshold, a single auxiliary camera device is selected from the plurality of auxiliary camera devices.
The method according to any one of claims 1 to 16, wherein the acquiring at least one image captured by at least one auxiliary camera device mounted on the movable platform within the field of view of the auxiliary camera device Second image, including:

If the FOV of the auxiliary camera device is greater than or equal to a preset FOV threshold, acquiring a single second image captured by a single auxiliary camera device;

If the FOV of the auxiliary camera device is less than the preset FOV threshold, acquire multiple second images shot by a single auxiliary camera device rotated in different directions, or acquire images shot by more than one auxiliary camera device a plurality of the second images.
The method according to any one of claims 1 to 14, wherein the method further comprises:

displaying an auxiliary camera device selection interface for the user to select a target auxiliary camera device from a plurality of auxiliary camera devices mounted on the movable platform displayed on the auxiliary camera device selection interface;

The target auxiliary camera device selected by the user is determined as the at least one auxiliary camera device.
The method according to any one of claims 1 to 14, wherein the method further comprises:

The at least one auxiliary camera device is selected from a plurality of auxiliary camera devices loaded on the movable platform according to the current shooting mode.
The method according to claim 19, wherein the selecting the at least one auxiliary camera device from a plurality of auxiliary camera devices loaded on the movable platform according to the current shooting mode comprises:

If the current shooting mode is a vertical panorama mode, selecting a single auxiliary camera device from the plurality of auxiliary camera devices;

If the current shooting mode is a spherical panorama mode, selecting one or more of the auxiliary camera devices from the plurality of auxiliary camera devices.
A panoramic image generation device, characterized in that the panoramic image generation device includes a memory and a processor;

the memory is used to store computer programs;

The processor is configured to execute the computer program and implement the following steps when executing the computer program:

acquiring at least one first image captured by the main camera mounted on the movable platform within the field of view of the main camera;

Obtain at least one second image captured by at least one auxiliary camera device mounted on the movable platform within the field of view of the auxiliary camera device, where the field of view of the auxiliary camera device includes the movable platform. In the upper area, there is an overlapping range between the field of view of the main camera device and the field of view of the auxiliary camera device;

Perform image fusion processing on the at least one first image and the at least one second image to generate a corresponding panoramic image.
The apparatus according to claim 21, wherein, before performing the image fusion processing on the at least one first image and the at least one second image, the processor is configured to:

performing image preprocessing on the at least one first image or the at least one second image so that the resolution of the at least one first image matches the resolution of the at least one second image;

When implementing the image fusion processing on the at least one first image and the at least one second image, the processor is configured to implement:

Perform image fusion processing on the at least one first image and the at least one second image whose resolutions are matched.
The apparatus according to claim 22, wherein, when the processor performs image preprocessing on the at least one first image or the at least one second image, the processor is configured to:

If the resolution of the at least one second image is smaller than the resolution of the at least one first image, perform image upsampling processing on the at least one second image;

If the resolution of the at least one second image is greater than the resolution of the at least one first image, perform image downsampling processing on the at least one second image, or perform image downsampling on the at least one second image, or perform an image downsampling process on the at least one second image. An image is subjected to image upsampling processing.
The apparatus according to claim 23, wherein, when the processor implements the image upsampling processing on the at least one second image, the processor is configured to implement:

inputting the at least one second image into a neural network model for super-sampling and upsampling; or

An image upsampling process is performed on the at least one second image by using a bicubic interpolation algorithm.
The apparatus according to claim 21, wherein before performing the image fusion processing on the at least one first image and the at least one second image, the processor is configured to:

respectively performing HDR fusion processing on a plurality of the second images captured by each of the auxiliary cameras to generate a high dynamic range image corresponding to each of the auxiliary cameras;

When implementing the image fusion processing on the at least one first image and the at least one second image, the processor is configured to implement:

Perform image fusion processing on the at least one first image and the high dynamic range image.
The apparatus according to claim 21, wherein before performing the image fusion processing on the at least one first image and the at least one second image, the processor is configured to:

When the first images include multiple sheets, image fusion processing is performed on the multiple sheets of the first images to obtain corresponding first fusion images;

When the second images include multiple sheets, image fusion processing is performed on the multiple sheets of the second images to obtain corresponding second fusion images;

When implementing the image fusion processing on the at least one first image and the at least one second image, the processor is configured to implement:

Perform image fusion processing on the first fusion image and the second fusion image.
The device according to claim 26, wherein, when the processor implements the image fusion processing on a plurality of the second images to obtain the corresponding second fusion images, the processor is configured to implement:

Performing latitude and longitude mapping on a plurality of the second images to obtain a plurality of corresponding second local area images;

performing a first seam finding operation on a plurality of the second local area images;

According to the result of the first seam finding operation, multi-band fusion processing is performed on a plurality of the second local area images to obtain the second fusion image.
The device according to claim 27, wherein, when the processor performs the first seam finding operation on the plurality of second partial region images, the processor is configured to:

acquiring a first overlapping area of a plurality of the second partial area images;

The first seam finding operation is performed according to the first overlapping area.
The apparatus according to claim 27 or 28, wherein, before the processor performs the latitude and longitude mapping on the plurality of second images, the processor is configured to:

Optimizing the internal parameter matrix and the external parameter matrix corresponding to the plurality of second images;

When implementing the latitude and longitude mapping on the plurality of second images, the processor is configured to implement:

According to the optimized internal parameter matrix and the external parameter matrix corresponding to the plurality of second images, perform latitude and longitude mapping on the plurality of second images to obtain a plurality of the second local area images and a plurality of a mask image corresponding to the second local area image;

When implementing the acquiring of the first overlapping area of the plurality of second partial area images, the processor is configured to implement:

The first overlapping area of the plurality of second partial area images is acquired according to the mask images corresponding to the plurality of second partial area images.
The apparatus according to claim 29, wherein, when the processor implements the optimization of the internal parameter matrices and the external parameter matrices corresponding to the plurality of second images, the processor is configured to implement:

determining, based on the at least one first image, a target first image to be matched with a plurality of the second images for image feature point matching;

Perform image feature point matching on the determined first image of the target and a plurality of the second images to obtain corresponding image feature point matching information;

The matching information of the image feature points, the internal parameter matrix and the external parameter matrix corresponding to the first image of the target, the internal parameter matrix and the external parameter matrix corresponding to the plurality of second images, and the input beam method adjustment model are processed to obtain The optimized internal parameter matrix and the external parameter matrix corresponding to the plurality of second images.
The device according to claim 30, wherein the extrinsic parameter matrix corresponding to each second image comprises pose data between each second image and other second images, and each The pose data between the second image and the target first image, wherein the pose data between each second image and the target first image is determined by a calibrated rotation matrix, or each The IMU data of the second image is generated with the IMU data of the target first image.
The apparatus according to claim 30 or 31, wherein the processor is performing the determining, based on the at least one first image, the target to be matched with the image feature points of the plurality of second images The first image is used to achieve:

If the at least one first image includes a single image, determining the single first image as the target first image;

If the at least one first image includes multiple images, determine all the first images as the target first image, or select several first images from the multiple first images according to a preset rule an image, determined as the first image of the target.
The apparatus according to claim 32, wherein the plurality of first images includes multiple columns, and the processor selects a plurality of first images according to a preset rule in implementing the selection from the plurality of first images The image, when determined as the first image of the target, is used to achieve:

From the first images in the plurality of columns, the first images in the interval column are selected and determined as the target first images.
The apparatus according to any one of claims 21 to 33, wherein the processor is further configured to implement:

When the main camera device captures the first image of each column within the field of view of the main camera device, the at least one auxiliary camera device performs the first image capture within the field of view of the auxiliary camera device The second image is taken.
The apparatus according to any one of claims 21 to 34, wherein the processor is further configured to implement:

The at least one auxiliary camera device is selected from the plurality of auxiliary camera devices according to the FOV information corresponding to the plurality of auxiliary camera devices mounted on the movable platform and the FOV information corresponding to the main camera device.
The device according to claim 35, wherein the processor implements the FOV information corresponding to the plurality of auxiliary camera devices loaded on the movable platform and the FOV information corresponding to the main camera device , when the at least one auxiliary camera device is selected from the plurality of auxiliary camera devices, it is used to implement:

If the FOV of the auxiliary camera device is greater than or equal to a preset FOV threshold, a single auxiliary camera device is selected from the plurality of auxiliary camera devices.
The device according to any one of claims 21 to 36, characterized in that, when the processor realizes the acquisition of the field of view of at least one auxiliary camera device loaded on the movable platform in the auxiliary camera device When at least one second image is captured within, it is used to achieve:

If the FOV of the auxiliary camera device is greater than or equal to a preset FOV threshold, acquiring a single second image captured by a single auxiliary camera device;

If the FOV of the auxiliary camera device is less than the preset FOV threshold, acquire multiple second images shot by a single auxiliary camera device rotated in different directions, or acquire images shot by more than one auxiliary camera device a plurality of the second images.
The apparatus according to any one of claims 21 to 34, wherein the processor is further configured to implement:

displaying an auxiliary camera device selection interface for the user to select a target auxiliary camera device from a plurality of auxiliary camera devices mounted on the movable platform displayed on the auxiliary camera device selection interface;

The target auxiliary camera device selected by the user is determined as the at least one auxiliary camera device.
The apparatus according to any one of claims 21 to 34, wherein the processor is further configured to implement:

The at least one auxiliary camera device is selected from a plurality of auxiliary camera devices loaded on the movable platform according to the current shooting mode.
The device according to claim 39, wherein when the processor selects the at least one auxiliary camera device from a plurality of auxiliary camera devices loaded on the movable platform according to the current shooting mode ,Implementation:

If the current shooting mode is a vertical panorama mode, selecting a single auxiliary camera device from the plurality of auxiliary camera devices;

If the current shooting mode is a spherical panorama mode, selecting one or more of the auxiliary camera devices from the plurality of auxiliary camera devices.
A movable platform, characterized in that, the movable platform comprises the panoramic image generating device according to any one of claims 21 to 40.
A computer-readable storage medium, characterized in that, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method described in any one of claims 1 to 20. The panoramic image generation method described above.