WO2022027313A1

WO2022027313A1 - Panoramic image generation method, photography apparatus, flight system, and storage medium

Info

Publication number: WO2022027313A1
Application number: PCT/CN2020/107101
Authority: WO
Inventors: 翁松伟; 付洁; 郝贵伟
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2020-08-05
Filing date: 2020-08-05
Publication date: 2022-02-10

Abstract

A panoramic image generation method, a photography apparatus, a flight system, and a storage medium, the method comprising: obtaining a plurality of images to be stitched, the plurality of images to be stitched being obtained from at least one photography apparatus continuously photographing in different directions following the movement of a mount platform and/or a movable platform, said directions comprising a direction toward the sky; and performing stitching on the plurality of images to be stitched according to the directions, and generating a panoramic image. By means of the photography apparatus photographing in a direction towards the sky and obtaining an image of the sky, completion does not need to be performed on a sky portion, removing an associated sky completion step, facilitating increasing panoramic image generation efficiency, and because the sky image has been really photographed, there is no distortion problem, facilitating improving panoramic image quality.

Description

Panoramic image generation method, shooting device, flight system and storage medium

technical field

The present application relates to the technical field of image processing, and in particular, to a panoramic image generation method, a photographing device, a flight system and a storage medium.

Background technique

With the continuous development of image processing technology, panoramic images have been widely welcomed by people due to their wide viewing range and many image contents.

Please refer to FIG. 1A . Currently, the panoramic image obtained by using a movable platform such as an unmanned aerial vehicle equipped with a photographing device has the problem that the sky part at the top is missing. The texture information or color information is extracted from the top edge of the image to complete the missing sky part. Usually, the edge area where the texture change or the color change is relatively gentle in the top edge of other images with the sky part is regarded as the sky part, please refer to Figure 1B or The dashed part in Figure 2 is the completed sky part, which can be seen. This will make the extracted texture information or color information smoother, causing the completed sky part to have a distortion problem, resulting in a decrease in the quality of the panoramic image and a poor user experience.

SUMMARY OF THE INVENTION

In view of this, one of the objectives of the present application is to provide a panoramic image generation method, a photographing device, a flight system and a storage medium.

In a first aspect, an embodiment of the present application provides a method for generating a panoramic image, including:

Acquiring multiple images to be spliced, the multiple images to be spliced are continuously photographed by at least one photographing device along with the movement of the pan/tilt and/or movable platform toward different orientations, and the orientations include orientations toward the sky;

The plurality of images to be spliced are spliced according to the orientation to generate a panoramic image.

In a second aspect, an embodiment of the present application provides a photographing device, the photographing device is installed on a pan/tilt and/or a movable platform, and the photographing device includes a photographing unit, a memory and a processor;

the memory for storing executable instructions;

The photographing unit is used for continuously photographing multiple images to be spliced along with the movement of the PTZ and/or the movable platform toward different azimuths, and the azimuths include azimuths toward the sky;

When the processor executes the executable instruction, it is configured to: acquire the plurality of images to be stitched; and perform stitching processing on the plurality of images to be stitched according to the orientation to generate a panoramic image.

In a third aspect, an embodiment of the present application provides a flight system, including: a movable platform, at least one pan-tilt mounted on the movable platform, and at least one photographing device mounted on the pan-tilt;

The shooting device continuously shoots a plurality of images to be spliced in different directions along with the movement of the pan/tilt and/or the movable platform, and the directions include a direction toward the sky; The plurality of images to be spliced are spliced to generate a panoramic image.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the first aspect any one of the methods

In a panoramic image generation method, a photographing device, a flight system and a storage medium provided by the embodiments of the present application, at least one photographing device continuously photographs to different directions along with the movement of the pan/tilt and/or movable platform to obtain a plurality of to-be-received images. Stitching images, in this embodiment, the photographing device can photograph in an orientation toward the sky with the movement of the gimbal and/or the movable platform, so that the sky image at the top can be photographed, and then the sky image can be captured according to the orientation. Stitching with other images to be stitched to generate a panoramic image does not need to complete the sky part, reducing the related sky completion steps, which is beneficial to improve the efficiency of panoramic image generation, and because it is an actual sky image, there is no distortion. problem, which is beneficial to improve the quality of panoramic images.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative labor.

FIG. 1A is a schematic diagram of the top part of the sky shown in the background art missing;

FIG. 1B and FIG. 2 are schematic diagrams of the sky part at the top shown in the background art;

3 is a schematic flowchart of a method for generating a panoramic image according to an embodiment of the present application;

4 is a schematic diagram of the distribution of a pitch axis, a yaw axis, and a roll axis provided by an embodiment of the present application;

5 is a schematic structural diagram of an unmanned aerial vehicle provided by an embodiment of the present application;

6 is a schematic structural diagram of another unmanned aerial vehicle provided by an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a photographing device provided by an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a flight system provided by an embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The generation of panoramic images usually requires two steps of shooting and splicing. In the related art, when the shooting device is installed on the movable platform for shooting, it is limited by the moving range of the movable platform, and it is usually impossible to shoot the top part during the shooting process. The sky, or the captured sky at the top is incomplete, after stitching into a panoramic image, please refer to Figure 1A, there is a problem that the sky part at the top is missing.

In the related art, in order to complete the sky part at the top, texture information or color information is extracted from the top edges of other images with sky parts to complete the missing sky parts, usually the texture information or color information is extracted from the top edges of other images with sky parts. The edge area with gentle change or color change is considered to be the sky part. Please refer to the completed sky part in Figure 1B or the dotted line part in Figure 2. This will make the extracted texture information or color information smoother, which is different from the sky with clouds. Parts are separated, resulting in a distortion problem in the completed sky part, the quality of the panoramic image is degraded, and the user experience is not good.

Based on this, an embodiment of the present application provides a method for generating a panoramic image, in which at least one photographing device moves to different directions along with the movement of the pan/tilt and/or (and/or representing either or both) of the movable platform Continuous shooting to obtain a plurality of images to be spliced, in this embodiment, the shooting device can shoot in the direction of the sky along with the movement of the head and/or the movable platform, so that the sky image at the top can be captured, and then according to the above The sky image is spliced with other images to be spliced to generate a panoramic image, and there is no need to complete the sky part, which reduces the related sky completion steps, which is beneficial to improve the efficiency of panoramic image generation, and because the sky is actually shot Image, there is no problem of distortion, which is conducive to improving the quality of panoramic images.

The movable platform includes, but is not limited to, an unmanned aerial vehicle, an unmanned vehicle, an unmanned vessel, or a mobile robot.

The panoramic image includes, but is not limited to, a spherical panoramic image or a cylindrical panoramic image. The spherical panorama image can achieve the effect of 360 degrees horizontally and 180 degrees up and down. In the spherical panorama image, the user seems to be in the center of the ball, and can observe any angle through the operation of the mouse, keyboard or touch, completely integrated into the virtual In the environment; the cylindrical panoramic image can achieve a horizontal 360-degree effect, and the user can see the surrounding content in a horizontal 360-degree view of the ring.

Exemplarily, based on the feature that a large area can be observed based on the panoramic image, the panoramic image can be applied to a rescue scene, a survey scene, a monitoring scene, or a vehicle driving scene, and the like. In one example, in a monitoring scenario, such as in a large-scale public place, a traffic hub or a traffic intersection, a photographing device for generating a panoramic image is arranged, so that the staff can conduct global overall planning and scheduling through the generated panoramic image. In another example, in a fire rescue scenario, an unmanned aerial vehicle equipped with a photographing device flies to the location of the fire, and then the photographing device performs photography to obtain a plurality of images to be spliced, and then the photographing device or the unmanned aerial vehicle The human aircraft or other terminal equipment associated with the photographing device performs stitching processing on the plurality of images to be stitched to obtain a panoramic image, and then the staff can make overall planning and scheduling through the panoramic image.

Please refer to FIG. 3 , which is a schematic flowchart of a method for generating a panoramic image according to an embodiment of the present application. The method may be generated by a photographing apparatus, a pan/tilt, the movable platform, or other terminal equipment associated with the photographing apparatus. Execution, the following description takes the photographing device as an execution subject as an example, and the method includes:

In step S101, a plurality of images to be spliced are acquired, and the plurality of images to be spliced are continuously photographed by at least one photographing device in different directions along with the movement of the gimbal and/or the movable platform, and the directions include facing the sky orientation.

In step S102, stitching processing is performed on the plurality of images to be stitched according to the orientation to generate a panoramic image.

In this embodiment, a plurality of images to be spliced are continuously captured by at least one photographing device in different directions along with the movement of the PTZ and/or the movable platform, wherein, with the movement of the PTZ and/or the movable platform motion, the photographing device can capture a sky image in a direction facing the sky, and then splices the sky image and other images to be spliced according to the orientation to generate a panoramic image, without the need to complete the sky part, reducing the number of related The sky completion step is beneficial to improve the efficiency of panoramic image generation, and because it is an actual sky image, there is no distortion problem, which is beneficial to improve the quality of the panoramic image.

It can be understood that the embodiments of the present application do not impose any limitation on the number of the photographing devices, and specific settings may be made according to actual application scenarios.

In one embodiment, the number of the photographing device may be one, the photographing device is installed on the pan/tilt, and the pan/tilt can be rotated upward so that the photographing device can follow the movement of the pan/tilt. Turn towards the sky to shoot. Wherein, the rotation range of the PTZ is not less than 180°, so that with the rotation of the PTZ, the shooting device can shoot towards the sky, and can also shoot towards other directions such as the ground. As one of the implementations, please refer to FIG. 4 , the rotation range of the PTZ includes the rotation range of the PTZ around the pitch axis, that is to say, the rotation range of the PTZ around the pitch axis is not less than 180° °, so that when the gimbal rotates around the pitch axis, it can drive the photographing device to face the sky, and can also drive the shooting device to face other directions such as the ground, for example, the gimbal rotates from the horizontal direction to the direction of the sky. The pitch axis is rotated 90°, thereby driving the camera to shoot toward the sky or the ground; or the gimbal can also be rotated 45° obliquely upward from the horizontal direction around the pitch axis to make the camera face Shoot in this direction.

Further, the pan/tilt can be installed on the side of the fuselage of the movable platform. When the pan/tilt is mounted on the side of the movable platform, the movement range of the pan/tilt is limited. For example, the pan/tilt cannot rotate 360° around the yaw axis. In this embodiment, the camera and the pan/tilt can be driven to move with the movement of the movable platform, so that the camera can move toward more Shoot in different directions. Exemplarily, the movable platform can be rotated at the current position, and the rotation of the movable platform drives the pan-tilt and the photographing device to rotate, so that the photographing device can be directed to more different directions. shoot. In this embodiment, at the same time, only the pan/tilt may move, or only the movable platform may move, or the pan/tilt and the movable platform may move at the same time, which is not limited in this embodiment. Specific settings can be made according to actual application scenarios.

Here, the movable platform is an unmanned aerial vehicle as an example for illustration: the pan/tilt can be installed on the side of the fuselage of the unmanned aerial vehicle. In an optional implementation, please refer to FIG. 5 . The occlusion effect of the wings of the human aircraft 200 on the shooting range of the shooting device 202, the gimbal 201 can be installed in a place that will not be blocked by the wings, such as installed on the fuselage of the unmanned aircraft 200 At the front end or the rear end of the fuselage, the photographing device 202 is installed on the gimbal 201. With the rotation of the gimbal 201 or the unmanned aerial vehicle 200, the photographing device 202 can shoot in different directions. .

The rotation range of the gimbal around the pitch axis is not less than 180°, so that during the rotation of the gimbal around the pitch axis, the photographing device can be directed to different orientations (including the orientation towards the sky). For shooting, further, considering the problem of the limited rotation range of the pan/tilt, the unmanned aerial vehicle may be rotated to drive the shooting device to shoot in more different directions. For example, the drone can rotate around the yaw axis, thereby driving the gimbal to rotate to different positions, so that the photographing device can photograph in more different directions.

In one embodiment, the number of the photographing devices may be two, and one of the photographing devices may be installed above the body of the movable platform so that the photographing device can shoot toward the sky, and the other photographing device may be installed at the top of the movable platform. The movable platform is under the fuselage or on the side of the fuselage. In this way, as the movable platform is rotated, for example, the movable platform can be rotated about the pitch, yaw and/or roll axes (and/or any combination of the three) such that both all The photographing device can take pictures in different directions to obtain images to be spliced. In this embodiment, the simultaneous shooting by two shooting devices can reduce the shooting time and improve the shooting efficiency and the efficiency of generating the panoramic image.

Further, two of the photographing devices can be installed on two cradles respectively; one of the cradles is installed above the fuselage of the movable platform so that the photographing devices can shoot towards the sky, and the other cradle is installed on the movable platform. The lower part of the movable platform or the side of the fuselage. In this way, with the rotation of the two pan-tilts and/or the movable platform, the two shooting devices can take pictures in different directions to obtain images to be spliced, and since one of the pan-tilts is installed on the movable platform Above the fuselage, with the rotation of the gimbal, the shooting device installed on the gimbal can be made to face the sky to shoot. efficiency of the panoramic image.

In this embodiment, at the same time, only one of the PTZs may move, or only the movable platform may move, or both PTZs may move at the same time, or the PTZ and the movable platform may move at the same time. For example, there is no restriction on this, and specific settings can be made according to the actual application scenario.

Here, the movable platform is an unmanned aerial vehicle as an example for description: in the related art, a gimbal 301 is usually installed under the fuselage of the unmanned aerial vehicle 300, and the gimbal 301 is equipped with a photographing device 302. In order to save costs, please Referring to FIG. 6 , the embodiment of the present application may, on this basis, install another pan/tilt 301 capable of carrying the photographing device 302 above the fuselage of the unmanned aerial vehicle 300 . A pan/tilt 301 capable of carrying a photographing device 302 is installed below each, and the pan/tilt 301 installed above the fuselage of the drone 300 can be rotated to drive the photographing device 302 to shoot towards the sky and also to other different directions. For shooting, the rotation of the pan/tilt 301 installed under the fuselage of the drone 300 can drive the shooting device 302 to shoot towards the ground and also to shoot at other different directions, so as to obtain the image obtained by the two shooting devices. Images to be stitched to generate a panoramic image. In this embodiment, the simultaneous shooting by two shooting devices can reduce the shooting time and improve the shooting efficiency and the efficiency of generating the panoramic image.

In an embodiment, the number of the photographing devices may be three, the first photographing device may be installed above the body of the movable platform so that the photographing device can be photographed towards the sky, and the second photographing device may be Installed under the fuselage of the movable platform, the third photographing device can be installed on the side of the fuselage of the movable platform. In this way, with the rotation of the movable platform, for example, the movable platform can rotate around the pitch axis, the yaw axis and/or the roll axis (and/or any combination of the three), three of the shooting The device can take pictures in different directions to obtain images to be spliced. Further, the simultaneous shooting by the three shooting devices can reduce the shooting time and improve the shooting efficiency and the efficiency of generating the panoramic image. Wherein, the three photographing devices can be installed on the movable platform through three PTZs, or can be directly installed on the movable platform, which is not limited in this embodiment.

Wherein, when the number of the shooting devices is greater than one, in order to improve the processing efficiency, the panoramic image generation method can be applied to one of the shooting devices, and the other shooting devices can transmit the images to be stitched captured by themselves to the one of the shooting devices. a camera for centralized processing by said one of the cameras.

In one embodiment, the size of the field of view of the photographing device determines the photographing range of the photographing device, and the number of the images to be stitched and/or the orientation of the photographing device can be determined according to the shooting range of the photographing device. determined by the field of view. The larger the field of view of the photographing device, the smaller the number of the images to be stitched that need to be photographed, and accordingly, the photographing device can be photographed in fewer directions; the smaller the field of view of the photographing device is , the greater the number of the to-be-spliced images that need to be photographed, the correspondingly, the photographing device needs to be photographed in more directions.

In one embodiment, the plurality of images to be spliced include images to be spliced obtained by the photographing device facing the sky; and with each rotation of the pan/tilt and/or the movable platform by a specified angle, the photographing device moves toward the current orientation. The captured image to be stitched. Wherein, the pan/tilt and/or the movable platform can be rotated around the pitch axis, the yaw axis and/or the roll axis (and/or any combination of the three), so that the photographing device can be oriented in different directions The image to be spliced is obtained by azimuth shooting.

Wherein, considering that the size of the field of view of the shooting device determines the shooting range of the shooting device, the specified angle rotated by the pan/tilt and/or the movable platform can be determined according to the field of view of the shooting device; Further, in order to ensure the accurate splicing of the images to be spliced in the subsequent splicing process, the images to be spliced need to have a certain overlap between them, and the specified angle can be set to be smaller than the field of view angle, so that the shooting There is an overlapping portion between two adjacent images to be spliced, which ensures accurate splicing between the subsequent images to be spliced.

Wherein, when the number of the photographing devices is one, along with the rotation of the pan/tilt or the movable platform, the photographing devices are directed toward different directions to obtain images to be stitched. In one example, the photographing device is installed on a pan-tilt, the pan-tilt is mounted on the side of the body of the movable platform, and the rotation range of the pan-tilt around the pitch axis is not less than 180°, then when the camera is acquired When the multiple images to be stitched are described, the generated panoramic image is a spherical panoramic image as an example for illustration: the pan/tilt rotates ±90° from the horizontal direction around the pitch axis to drive the shooting device to shoot towards the sky or towards the ground The images to be spliced are obtained, and 2 images to be spliced are obtained here; 45°, the photographing device captures the images to be spliced at the current orientation, and 8 images to be spliced are obtained here; Or every time the movable platform rotates 45° around the yaw axis, the photographing device captures the images to be spliced at the current orientation, and here 8 images to be spliced are obtained; Every time the mobile platform rotates around the yaw axis by 45°, the photographing device captures the images to be stitched at the current orientation, and here 8 images to be stitched are obtained; then a total of 26 images to be stitched are obtained. It can be understood that the above is only an example, and the present application does not limit the rotation angle of the pan/tilt or the movable platform around the pitch axis or the yaw axis.

When the number of the photographing devices is multiple, with the rotation of the pan/tilt or the movable platform, the multiple photographing devices are directed towards different directions to photograph the images to be stitched, and the simultaneous photographing by the multiple photographing devices can reduce the shooting time. time, improve the shooting efficiency and the efficiency of generating the panoramic image.

In one example, the number of the photographing devices may be two, and the two photographing devices may be respectively installed on two pans; one of the pans is installed above the body of the movable platform so that the shooting The device can shoot towards the sky, and another pan/tilt is installed below the movable platform, then when acquiring the plurality of images to be stitched, the generated panoramic image is a spherical panoramic image as an example to illustrate: For the convenience of description, here the gimbal installed above the fuselage of the movable platform is referred to as the upper gimbal, and the gimbal installed under the fuselage of the movable platform is referred to as the lower gimbal; The gimbal rotates 90° from the horizontal direction around the pitch axis to drive the photographing device towards the sky to capture the image to be spliced, here is one image to be spliced; After rotating 45° obliquely upward, the upper-mounted pan/tilt head or the movable platform rotates by 45° around the yaw axis, and the photographing device captures the images to be spliced in the current orientation, and 8 images to be spliced are obtained here; and, After the bottom-mounted pan/tilt or the movable platform is rotated 45° obliquely downward around the pitch axis, the bottom-mounted pan/tilt or the movable platform is rotated by 45° around the yaw axis, and the photographing device is in the current orientation. The images to be spliced are obtained by shooting, and 8 images to be spliced are obtained here; The photographing device captures the images to be spliced at the current orientation, and 8 images to be spliced are obtained here; and, the lower pan/tilt rotates 90° from the horizontal direction around the pitch axis to drive the photographing device toward the ground to photograph the images to be spliced, Here, 1 image to be stitched is obtained; then a total of 26 images to be stitched are obtained. It can be understood that the above is only an example, and the present application does not limit the rotation angle of the upper-mounted pan/tilt head, the bottom-mounted pan/tilt head or the movable platform about the pitch axis or the yaw axis.

After acquiring the plurality of images to be stitched, the photographing device performs stitching processing on the plurality of images to be stitched according to the orientation of the images to be stitched to generate a panoramic image. Specifically, acquire the attitude parameters of the gimbal and/or the movable platform when each of the shooting devices shoots the images to be spliced in different directions, and then according to the attitude parameters of the gimbal and/or the attitude parameters of the movable platform Or the pose parameters of the movable platform perform splicing processing on the plurality of images to be spliced.

Wherein, when the number of the shooting devices is greater than one, the panoramic image generation method is applied to one of the shooting devices; the other shooting devices are also used to capture the images to be spliced obtained by themselves and to capture the images to be spliced. At the same time, the attitude parameters of the pan-tilt head and/or the attitude parameters of the movable platform are transmitted to the one of the photographing devices, and the one of the photographing devices performs stitching processing on the images to be stitched.

In one example, when the number of the photographing devices is two, the two photographing devices may be respectively installed on two pans; one of the pans is installed above the body of the movable platform so that the The shooting device can shoot towards the sky, and another pan/tilt is installed under the movable platform or on the side of the fuselage; for the convenience of description, the pan/tilt mounted above the fuselage of the movable platform is referred to as the upper cloud. The PTZ installed under the fuselage of the movable platform is called the lower PTZ. If the upper PTZ, the lower PTZ and all the The movable platforms have all been rotated, and for the images to be spliced shot by the photographing device on the top of the pan/tilt head, it is necessary to obtain the posture of the top-mounted pan/tilt head when the photographing device shoots the images to be stitched in different directions. parameters and the attitude parameters of the movable platform; for the images to be spliced captured by the camera with the lower pan/tilt, it is necessary to obtain the lower pan/tilt when the camera shoots the images to be stitched in different directions. and the pose parameters of the movable platform, so as to ensure the accuracy of the generated panoramic image.

In one embodiment, when generating the panoramic image, first obtain a sky image obtained by the photographing device facing the sky from a plurality of images to be spliced, and extract the edge feature points of the sky image, and then according to the Edge feature points and the attitude parameters of the gimbal and/or the movable platform when the shooting device shoots toward the sky, splicing the sky image with other images to be spliced, and generating the panoramic image .

When acquiring the sky image, consider that the image to be stitched obtained by the camera facing the sky is not necessarily a sky image. For example, when the unmanned aerial vehicle is in a bridge hole, the camera mounted on the unmanned aerial vehicle faces the sky. Therefore, in order to determine the accuracy of the obtained sky image, in this embodiment, when obtaining the sky image, the image to be spliced obtained by the photographing device facing the sky is first identified, and the identification is obtained. As a result, the sky image is then determined based on the recognition result.

In an implementation manner, feature extraction may be performed on the to-be-spliced image captured by the photographing device facing the sky, image feature information may be obtained, and the image feature information may be matched with preset sky feature information, such as determining the The matching degree between the image feature information and the preset sky feature information, the sky feature information includes but is not limited to cloud layer feature information, starry sky feature information, daytime sky feature information or night sky feature information, etc. Within the preset range, it can be determined that the image to be spliced is a sky image, and if the matching is inconsistent or the matching degree is not within a preset range, it can be determined that the image to be spliced obtained by the photographing device facing the sky is not a sky image. This embodiment ensures that an accurate sky image is obtained by effectively identifying the images to be stitched obtained by the photographing device facing the sky.

Correspondingly, referring to FIG. 7 , an embodiment of the present application further provides a photographing device 20 . The photographing device is installed on a pan/tilt and/or a movable platform, and the photographing device includes a photographing unit 21 , a memory 22 and a processing unit 20 . device 23.

The memory 22 is used to store executable instructions;

The photographing unit 21 is used for continuously photographing a plurality of images to be stitched in different directions along with the movement of the pan/tilt and/or the movable platform, and the directions include directions facing the sky.

When the processor 23 executes the executable instructions, it is configured to: acquire the plurality of images to be stitched; perform stitching processing on the plurality of images to be stitched according to the orientation to generate a panoramic image.

The processor 23 executes the executable instructions included in the memory 21, and the processor 23 may be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processors) Processor, DSP), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 21 stores executable instructions of the panoramic image generation method, and the memory 21 may include at least one type of storage medium, including flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory). etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), Magnetic Memory , disks, CDs, etc. Also, the photographing device may cooperate with a network storage device that performs the storage function of the memory through a network connection. The memory 21 may be an internal storage unit of the photographing apparatus 20 , such as a hard disk or a memory of the photographing apparatus 20 . The memory 21 can also be an external storage device of the photographing device 20, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card (Flash card) equipped on the photographing device 20. Card), etc. Further, the memory 21 may also include both an internal storage unit of the photographing apparatus 20 and an external storage device. Memory 21 is used to store executable instructions and other programs and data required by the device. The memory 21 can also be used to temporarily store data that has been output or is to be output.

The various embodiments described herein can be implemented using computer readable media such as computer software, hardware, or any combination thereof. For hardware implementation, the embodiments described herein can be implemented using application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays ( FPGA), processors, controllers, microcontrollers, microprocessors, electronic units designed to perform the functions described herein are implemented. For software implementation, embodiments such as procedures or functions may be implemented with separate software modules that allow the performance of at least one function or operation. The software codes may be implemented by a software application (or program) written in any suitable programming language, which may be stored in memory and executed by a controller.

The photographing device may include, but is not limited to, a photographing unit 21 , a processing unit 23 , and a memory 22 . Those skilled in the art can understand that FIG. 7 is only an example of the photographing device 20 , and does not constitute a limitation on the photographing device 20 , and may include more or less components than shown, or combine some components, or different components For example, the device may also include an input and output device, a network access device, a bus, and the like.

In one embodiment, the number of the photographing devices is two, and one of the photographing devices is installed above the body of the movable platform so that the photographing device can shoot toward the sky.

In one embodiment, another photographing device is installed under the fuselage of the movable platform or on the side of the fuselage.

In one embodiment, the two photographing devices are respectively mounted on the movable platform through two pans.

In one embodiment, the number of the photographing devices is one, the photographing device is installed on a pan-tilt, the pan-tilt is mounted on the side of the fuselage of the movable platform, and the pan-tilt can be rotated upward to So that the photographing device can photograph toward the sky along with the rotation of the gimbal.

In one embodiment, the rotation range of the pan/tilt head is not less than 180°.

In one embodiment, the rotation range of the PTZ includes the rotation range of the PTZ around the pitch axis.

In one embodiment, the number of the images to be stitched and/or the orientation of the photographing device to be photographed is determined according to the angle of view of the photographing device.

In one embodiment, the processor 23 is further configured to: acquire the image to be stitched obtained by the photographing device facing the sky; The image to be stitched obtained by photographing the photographing device toward the current orientation.

In one embodiment, the designated angle is smaller than the field angle.

In one embodiment, the pan/tilt and/or movable platform is rotated about a pitch axis, a yaw axis and/or a roll axis.

In one embodiment, the processor 23 is further configured to: acquire the attitude parameters of the gimbal and/or the movable platform when each of the shooting devices shoots in different directions; The pose parameters of the gimbal and/or the pose parameters of the movable platform perform splicing processing on the plurality of images to be spliced.

In one embodiment, the processor 23 is further configured to: acquire a sky image obtained by the photographing device facing the sky from a plurality of images to be spliced, and extract edge feature points of the sky image; Edge feature points and the attitude parameters of the gimbal and/or the movable platform when the photographing device shoots toward the sky, the sky image and other images to be spliced are stitched together.

In an embodiment, the processor 23 is further configured to: identify the images to be stitched obtained by the photographing device facing the sky to obtain a recognition result; and determine the sky image according to the recognition result.

In one embodiment, the processor 23 is further configured to: perform feature extraction on the to-be-spliced image obtained by the photographing device facing the sky to obtain image feature information; compare the image feature information with the preset sky feature information to match.

In an embodiment, the sky feature information includes at least one of the following: cloud layer feature information, starry sky feature information, daytime sky feature information, or night sky feature information.

In one embodiment, when the number of the photographing devices used for photographing the images to be stitched is more than one, the processor 23 is further configured to receive images to be stitched obtained by other photographing devices.

For details of the implementation process of the functions and functions of each unit in the above device, please refer to the implementation process of the corresponding steps in the above method, which will not be repeated here.

Correspondingly, referring to FIG. 8 , an embodiment of the present application further provides a flight system, including: a movable platform 30 , at least one pan/tilt 40 mounted on the movable platform 30 , and a pan/tilt mounted on the pan/tilt 40 . At least one camera 20 .

The photographing device 20 continuously shoots a plurality of images to be spliced in different directions along with the movement of the pan/tilt 40 and/or the movable platform 30, and the directions include directions toward the sky; and, according to the The orientation performs stitching processing on the plurality of images to be stitched to generate a panoramic image.

For details of the implementation process of the functions and functions of each device in the above system, please refer to the implementation process of the corresponding steps in the above method, which will not be repeated here.

In an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium, such as a memory including instructions, executable by a processor of an apparatus to perform the above-described method. For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the photographing apparatus, enable the photographing apparatus to perform the above method.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. The terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion such that a process, method, article or device comprising a list of elements includes not only those elements, but also other not expressly listed elements, or also include elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The methods and devices provided by the embodiments of the present application have been introduced in detail above, and specific examples are used to illustrate the principles and implementations of the present application. At the same time, for those of ordinary skill in the art, according to the idea of the application, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be construed as a limitation to the application. .

Claims

A method for generating panoramic images, comprising:

Acquiring multiple images to be spliced, the multiple images to be spliced are continuously photographed by at least one photographing device along with the movement of the pan/tilt and/or movable platform toward different orientations, and the orientations include orientations toward the sky;

The plurality of images to be spliced are spliced according to the orientation to generate a panoramic image.
The method according to claim 1, wherein the number of the photographing devices is two, and one of the photographing devices is installed above the body of the movable platform so that the photographing device can shoot toward the sky.
The method according to claim 2, wherein another photographing device is installed under the fuselage of the movable platform or on the side of the fuselage.
The method according to claim 2, wherein the two photographing devices are respectively mounted on the movable platform through two pans.
The method according to claim 1, wherein the number of the photographing device is one, and the photographing device is mounted on a pan/tilt, and the pan/tilt is mounted on the side of the fuselage of the movable platform. The gimbal can be rotated upwards so that the photographing device can shoot towards the sky along with the rotation of the gimbal.
The method according to claim 5, wherein the rotation range of the head is not less than 180°.
The method according to claim 6, wherein the rotation range of the PTZ includes the rotation range of the PTZ around the pitch axis.
The method according to claim 1, wherein the number of the images to be stitched and/or the orientation of the photographing device to be photographed is determined according to the field of view of the photographing device.
The method according to claim 1, wherein the acquiring a plurality of images to be spliced comprises:

Acquiring the images to be stitched obtained by the photographing device facing the sky; and acquiring the images to be stitched obtained by photographing the photographing device facing the current orientation with each rotation of the pan/tilt and/or the movable platform by a specified angle.
The method of claim 9, wherein the specified angle is smaller than the field of view.
The method according to claim 9, wherein the pan/tilt and/or the movable platform rotates around a pitch axis, a yaw axis and/or a roll axis.
The method according to claim 1, wherein the performing stitching processing on the plurality of images to be stitched according to the orientation comprises:

Obtaining the attitude parameters of the gimbal and/or the movable platform when each of the shooting devices shoots in different directions;

The plurality of images to be spliced are spliced according to the attitude parameters of the PTZ and/or the movable platform.
The method according to claim 12, wherein the splicing processing of the plurality of images to be spliced according to the attitude parameters of the gimbal and/or the attitude parameters of the movable platform, comprising:

Obtaining a sky image obtained by the photographing device facing the sky from a plurality of images to be spliced, and extracting edge feature points of the sky image;

The sky image and other images to be spliced are spliced according to the edge feature points and the attitude parameters of the gimbal and/or the movable platform when the photographing device shoots toward the sky.
The method according to claim 13, wherein the acquiring, from a plurality of images to be stitched, the sky image obtained by the photographing device facing the sky, further comprising:

Identifying the images to be stitched obtained by the shooting device facing the sky to obtain the identification result;

The sky image is determined according to the recognition result.
The method according to claim 14, wherein the identifying the images to be stitched obtained from the orientation of the photographing device facing the sky comprises:

Perform feature extraction on the images to be spliced obtained by the shooting device facing the sky to obtain image feature information;

Matching the image feature information with preset sky feature information.
The method according to claim 15, wherein the sky feature information includes at least one of the following: cloud layer feature information, starry sky feature information, daytime sky feature information, or night sky feature information.
The method according to claim 1, wherein when the number of the photographing devices is greater than one, the panoramic image generation method is applied to one of the photographing devices;

The other photographing devices are also used for transmitting the images to be spliced obtained by themselves to the one of the photographing devices.
A photographing device, characterized in that the photographing device is installed on a pan-tilt and/or a movable platform, and the photographing device comprises a photographing unit, a memory and a processor;

the memory for storing executable instructions;

The photographing unit is used for continuously photographing a plurality of images to be spliced in different directions along with the movement of the head and/or the movable platform, and the directions include directions toward the sky;

When the processor executes the executable instruction, it is configured to: acquire the plurality of images to be stitched; and perform stitching processing on the plurality of images to be stitched according to the orientation to generate a panoramic image.
The device according to claim 18, wherein the number of the photographing devices is two, and one of the photographing devices is installed above the body of the movable platform so that the photographing device can shoot toward the sky.
The device according to claim 19, wherein another photographing device is installed under the fuselage of the movable platform or on the side of the fuselage.
The device according to claim 19, wherein the two photographing devices are respectively mounted on the movable platform through two pans.
The device according to claim 18, wherein the number of the photographing device is one, the photographing device is mounted on a pan-tilt, and the pan-tilt is mounted on the side of the fuselage of the movable platform, the The gimbal can be rotated upwards so that the photographing device can shoot towards the sky along with the rotation of the gimbal.
The device according to claim 22, wherein the rotation range of the head is not less than 180°.
The device according to claim 23, wherein the rotation range of the PTZ includes the rotation range of the PTZ around the pitch axis.
The device according to claim 18, wherein the number of the images to be stitched and/or the orientation of the photographing device to be photographed is determined according to the field of view of the photographing device.
The device according to claim 18, wherein the processor is further configured to: acquire an image to be stitched obtained by the photographing device facing the sky; The specified angle is obtained, and the image to be stitched obtained by the photographing device at the current orientation is obtained.
27. The apparatus of claim 26, wherein the designated angle is smaller than the field of view.
The apparatus according to claim 26, wherein the pan/tilt and/or the movable platform rotates around a pitch axis, a yaw axis and/or a roll axis.
The device according to claim 18, wherein the processor is further configured to: acquire attitude parameters of the pan-tilt and/or the movable platform when each of the shooting devices shoots in different directions Attitude parameters; according to the attitude parameters of the pan/tilt head and/or the attitude parameters of the movable platform, the plurality of images to be spliced are spliced.
The device according to claim 29, wherein the processor is further configured to: acquire a sky image obtained by the photographing device facing the sky from a plurality of images to be spliced, and extract the edge of the sky image feature points; according to the edge feature points and the attitude parameters of the head and/or the movable platform when the shooting device shoots toward the sky, the sky image and other images to be spliced are stitched together .
The device according to claim 30, wherein the processor is further configured to: identify the images to be stitched obtained by the photographing device facing the sky to obtain a recognition result; determine the sky image according to the recognition result .
The device according to claim 31, wherein the processor is further configured to: perform feature extraction on the images to be stitched obtained by the photographing device facing the sky to obtain image feature information; Match with preset sky feature information.
The device according to claim 32, wherein the sky feature information includes at least one of the following: cloud layer feature information, starry sky feature information, daytime sky feature information, or night sky feature information.
The device according to claim 18, wherein when the number of the shooting devices for shooting the images to be spliced is greater than one, the processor is further configured to receive the images to be spliced obtained from other shooting devices. image.
A flight system, characterized in that it comprises: a movable platform, at least one pan-tilt mounted on the movable platform, and at least one photographing device mounted on the pan-tilt;

The shooting device continuously shoots a plurality of images to be spliced in different directions along with the movement of the pan/tilt and/or the movable platform, and the directions include a direction toward the sky; The plurality of images to be spliced are spliced to generate a panoramic image.
A computer-readable storage medium, characterized in that, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method described in any one of claims 1-17. method described.