WO2022040868A1

WO2022040868A1 - Panoramic photography method, electronic device, and storage medium

Info

Publication number: WO2022040868A1
Application number: PCT/CN2020/110835
Authority: WO
Inventors: 翁松伟; 刘嘉玮
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2020-08-24
Filing date: 2020-08-24
Publication date: 2022-03-03
Also published as: CN113454980A

Abstract

The present disclosure relates to a panoramic photography method, an electronic device, and a computer-readable storage medium. The panoramic photography method comprises: receiving a photographic instruction; and on the basis of the photographic instruction, adjusting device parameters of a movable platform, and photographing multiple images, wherein the device parameters comprise one or more of a device attitude, a height, a camera orientation or a photographic parameter, the multiple images are used for generating an eye-in-the-sky image, and the eye-in-the-sky image is an image obtained by performing coordinate transformation with a sky area in the image as a rotation center.

Description

Panoramic shooting method, electronic device and storage medium

technical field

The present disclosure relates to the field of unmanned aerial vehicles, and more particularly, to a panoramic shooting method, electronic device and storage medium.

Background technique

With the development of society, in order to meet people's needs for the diversity of image expressions, panoramic images emerge as the times require. At present, panoramic images are widely used in people's lives.

In the related art, multiple image materials are first collected, and then the collected image materials are processed by image processing software such as PS (Photoshop, image processing software) in the computer. The post-processing of the images is difficult, the processing speed is slow, and the panoramic Image generation is inefficient.

It should be noted that the information disclosed in the above Background section is only for enhancement of understanding of the background of the present disclosure, and therefore may contain information that does not form the prior art that is already known to a person of ordinary skill in the art.

SUMMARY OF THE INVENTION

The purpose of the present disclosure is to provide a panorama shooting method, which can automatically and quickly generate an eye-of-the-sky image, saves the process of other software processing in the later stage, is simple in operation, can reduce the difficulty of operation, and improve the user's interest in use.

Other features and advantages of the present disclosure will become apparent from the following detailed description, or be learned in part by practice of the present disclosure.

According to a first aspect of the present disclosure, there is provided a panorama shooting method, comprising: acquiring a shooting instruction; adjusting the posture of a movable platform based on the shooting instruction, and shooting multiple images; performing coordinate transformation according to the multiple images to generate a sky image The eye image, the eye image of the sky is an image obtained by coordinate transformation with the sky area in the image as the rotation center.

According to a second aspect of the present disclosure, there is provided an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to Execute a panoramic shooting method, the panoramic shooting method includes: acquiring a shooting instruction; adjusting the posture of the movable platform based on the shooting instruction, and shooting multiple images; performing coordinate transformation according to the multiple images to generate an eye of the sky image, the The eye of the sky image is an image obtained by coordinate transformation with the sky area in the image as the center of rotation.

According to a third aspect of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the panorama described in the first aspect of the embodiment of the present disclosure is realized Shooting method.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present disclosure more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some of the present disclosure. In the embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

1 is a schematic diagram of a system architecture provided according to an exemplary embodiment of the present disclosure;

FIG. 2 is a flowchart of a panorama shooting method according to an exemplary embodiment of the present disclosure;

FIG. 3 is a panoramic image of a banner according to an exemplary embodiment of the present disclosure;

FIG. 4 is a spherical panoramic image according to an exemplary embodiment of the present disclosure;

Fig. 5 is a kind of eye-of-the-sky image shown according to an exemplary embodiment of the present disclosure;

6 is a schematic diagram of a spherical coordinate system according to an exemplary embodiment of the present disclosure;

7A is a schematic diagram of a coordinate transformation according to an exemplary embodiment of the present disclosure;

7B is a schematic diagram of another coordinate transformation shown according to an exemplary embodiment of the present disclosure;

7C is a schematic diagram of another coordinate transformation shown according to an exemplary embodiment of the present disclosure;

FIG. 8 is a flowchart of another panorama shooting method shown in an exemplary embodiment of the present disclosure;

FIG. 9 is another eye image of the sky shown according to an exemplary embodiment of the present disclosure;

FIG. 10 is a flowchart of another panorama shooting method shown in an exemplary embodiment of the present disclosure;

FIG. 11 is a flowchart of another panorama shooting method shown in an exemplary embodiment of the present disclosure;

FIG. 12 is a flowchart of another panorama shooting method shown in an exemplary embodiment of the present disclosure;

FIG. 13 is a flowchart of another panorama shooting method shown in an exemplary embodiment of the present disclosure;

14 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure;

FIG. 15 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure.

detailed description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments, however, can be embodied in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be employed. In other instances, well-known solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.

In addition, the drawings are merely schematic illustrations of the present disclosure, and the same reference numerals in the drawings denote the same or similar parts, and thus their repeated descriptions will be omitted. Some of the block diagrams shown in the figures are functional entities that do not necessarily necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

The exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

FIG. 1 is a schematic diagram of a system architecture provided according to an exemplary embodiment of the present disclosure.

As shown in FIG. 1 , the system architecture 100 may include a user terminal 101 , a network 102 and a mobile platform 103 . The network 102 may provide the medium of the communication link between the user terminal 101 and the removable platform 103. The network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

A user may use the user terminal 101 to interact with the mobile platform 103 through the network 102 to receive or send messages and the like. Among them, the user terminal 101 may be various electronic devices having a display screen and supporting the ability to connect to the network 102, including but not limited to remote controls, smart phones, tablet computers, laptop computers, desktop computers, wearable devices, virtual Reality devices, augmented reality devices, gamepads, smart homes, and more.

The movable platform 101 can be, for example, a drone, the user can send a shooting instruction to the drone through the user terminal 101, the drone can receive the shooting instruction sent by the user terminal 101, and the drone can adjust the drone based on the shooting instruction. posture, and shoot multiple images, the drone can store the multiple captured images in the memory of the drone, or send the multiple captured images to the user terminal 101, and the drone can record the multiple captured images. The image is processed to generate the eye of the sky image, the drone can send the generated eye of the sky image to the user terminal 101, and the user terminal 101 can also process multiple images captured by the drone to generate the eye of the sky image.

It should be understood that the numbers of user terminals, networks and movable platforms in FIG. 1 are only illustrative, and there may be any number of user terminals, networks and movable platforms according to actual needs.

FIG. 2 is a flowchart of a panorama shooting method according to an exemplary embodiment of the present disclosure. The panorama shooting method provided by the embodiment of the present disclosure may be executed by any electronic device with computing processing capability, for example, may be executed by the movable platform shown in FIG. 1 . As shown in FIG. 2 , the panorama shooting method provided by the embodiment of the present disclosure may include steps S21-S24.

In step S21, a shooting instruction is acquired.

In some embodiments, the movable platform can receive the shooting instruction sent by the user through the terminal device.

The photographing instruction may be, for example, an instruction to photograph for generating an image of the eye of the sky.

For example, the user can select the eye-to-sky mode on the terminal device, and the terminal device can send a shooting instruction to the movable platform according to the eye-to-sky mode selected by the user.

The shooting instruction may include one or more of shooting position information, shooting object information, shooting parameter information and the number of shooting images.

The shooting location information can be, for example, coordinates, altitude, longitude and latitude, etc. The shooting object information can include, for example, objects the user wants to shoot, such as buildings, rivers, mountains, bridges, etc. The shooting parameter information can include, for example, sensitivity, aperture, focal length, etc.

In the embodiment of the present disclosure, the movable platform may be, for example, an unmanned aerial vehicle, a crossing aircraft, an unmanned vehicle, an unmanned boat, etc. The following description will be given by taking an unmanned aerial vehicle as an example, but the movable platform of the present disclosure is not limited thereto.

In step S22, the posture of the movable platform is adjusted based on the photographing instruction, and a plurality of images are photographed.

Wherein, the device parameters of the movable platform can also be adjusted based on the shooting instruction, and the device parameters include one of the height of the movable platform of the device, the orientation of the camera, or the shooting parameters (such as aperture, shutter, ISO, etc.) of the camera on the movable platform or more.

For example, the drone can adjust the device attitude, height, camera orientation, yaw angle of the drone, roll angle of the drone, The shooting parameters of the camera can be adjusted according to the shooting parameter information in the shooting instruction, and multiple images can be shot according to the number of shooting images in the shooting instruction. Or, the drone adjusts any one or more of the attitude, orientation, roll angle, yaw angle, and pitch angle of the gimbal according to the shooting instructions to shoot perspective images of different azimuths or heights. Among them, a camera can be mounted on the gimbal. When the camera on the drone can shoot a panorama of 360 in the horizontal direction and 180/360 in the vertical direction, the method of the present application can use the panorama in the horizontal direction and 180/360 in the vertical direction to generate a wide-angle image of the eye of the sky . If due to the limitation of the structure of the drone itself or the limitation of the maximum rotation angle of the gimbal, when the camera cannot achieve panorama shooting of 360 in the horizontal direction and 360 in the vertical direction, you can use the maximum rotation angle of the drone/gimbal to achieve wide-angle For image shooting, or based on the angle input by the user (180-degree panorama, 360-degree panorama, or 270-degree panorama), the attitude of the drone or the change of the gimbal attitude can be adaptively planned to realize the shooting of panoramic images. For example, with the help of a drone or a gimbal, the camera lens can achieve pitch: -90° to +30° rotation, yaw: -75° to +75° rotation, then the camera can shoot a horizontal range of 120° and a vertical range of 120°. 150° panoramic image.

In the above process, the UAV and the gimbal use a unified yaw-pitch-roll coordinate system. When the pan/tilt and the drone are rotated to realize the camera can take panoramic images, you can also use the camera to not capture the aircraft body as a criterion (it needs to be differentiated according to different aerial photography devices, because the purpose of rotating the plane and the pan/tilt is actually to rotate Camera): If you can reach the camera to shoot the panorama only by rotating the gimbal and will not shoot the aircraft body, you do not need to rotate the aircraft, otherwise you need to rotate the aircraft so that the shooting picture will not be blocked by the aircraft body.

Exemplarily, the drone can also automatically adjust the shooting parameters according to the weather, light intensity, etc. when shooting.

For example, the drone can adjust the shooting mode according to the light intensity. For example, when the light intensity is high during the day, the drone can use the day mode to shoot, and when the light intensity is low at night, the drone can use the night scene mode to shoot.

For example, the drone can rotate and shoot the subject in a clockwise or counterclockwise direction according to the adjusted device attitude, height, and camera orientation, and obtain multiple images. Wherein, the angle of rotation shooting may be, for example, 360°, or may be greater than 360°.

Among the multiple images captured by the drone, some or all of the images may have overlapping areas or adjacent image content.

The multiple captured images can be used to generate a sky eye image, and the sky eye image is an image obtained by performing coordinate transformation on the sky area in the image as the rotation center.

Fig. 3 is a panorama image of a banner according to an exemplary embodiment of the present disclosure.

FIG. 4 is a spherical panoramic image according to an exemplary embodiment of the present disclosure.

FIG. 5 is an image of an eye in the sky according to an exemplary embodiment of the present disclosure.

As shown in FIG. 3 , the upper part of the banner panorama image may be the sky area 301 , the middle may be the photographed scene 302 , for example, the photographed scene 302 may include buildings, plants, etc., and the lower part may be the ground area 303 .

As shown in FIG. 4 , in the spherical panorama image, the ground area 303 is located in the center of the screen, the photographed scene 302 is located near the center of the screen, and the sky area 301 is located around the screen.

In the spherical panoramic image shown in FIG. 4 , the shooting scene occupies a small area, and the sky area 301 needs to undergo complex sky-filling processing.

In some embodiments, as shown in FIG. 5 , in the eye of the sky image, the sky area 301 is located in the center of the picture, and the shooting scene 302 outside the sky is located around the picture.

Wherein, the center of the picture includes but is not limited to the exact center of the picture. Relative to the shooting scene, the sky area may be in the center of the picture.

In the eye of the sky image, the shooting scene occupies a larger area, and the display is clearer, which is beneficial to the user's viewing and can improve the user experience; the middle area of the eye of the sky image is the sky area 301, which does not need to undergo complex sky patching processing. easy to use.

In this embodiment of the present disclosure, the image of the eye of the sky may be directly generated by a drone, or may be generated by a terminal device using multiple images captured by the drone.

In some embodiments, the panorama shooting method further includes: sending multiple images to the terminal device, where the multiple images are used for the terminal device to generate an image of the eye of the sky.

The drone can send multiple images taken to the terminal device, and the terminal device can automatically process the multiple images. The processing process can be the same as that of the drone, and the terminal device can also process multiple images according to the user's operation. Processed to generate the eye of the sky image.

In step S23, coordinate transformation is performed according to a plurality of images to generate an image of the eye of the sky.

For example, a drone can process multiple images to generate an eye-to-sky image.

In some embodiments, the plurality of images may be mapped to the spherical coordinate system based on the spherical coordinate system corresponding to the movable platform to generate a spherical panorama; according to the conversion relationship between the spherical coordinate system and the polar coordinate system, Map the spherical panorama to the polar coordinate system to generate the eye of the sky image.

For example, an image captured by a drone can be mapped into a spherical coordinate system to generate a spherical panoramic image, and according to the coordinate transformation, the spherical panoramic image can be mapped into a polar coordinate system to generate an image of the eye of the sky.

In some embodiments, based on the GPS (Global Positioning System, global positioning system) coordinates of the multiple images captured by the movable platform, the multiple captured images may be processed through coordinate transformation to generate the eye image of the sky.

For example, based on the GPS coordinates and spherical coordinate system of the drone when the multiple images captured by the drone, coordinate transformation can be performed on the multiple images, and the images are mapped to the spherical coordinate system. One vertex of the spherical coordinate system is the sky area. The center of , the area within a certain radius is the sky area, and the rotation change is performed with the vertex as the center, and the image of the eye of the sky can be obtained.

The radius of the sky area can be set as required. For example, if the user wants the sky area in the middle of the eye of the sky image to be larger, the radius of the sky area can be set larger.

FIG. 6 is a schematic diagram of a spherical coordinate system according to an exemplary embodiment of the present disclosure.

Fig. 7A is a schematic diagram showing a coordinate transformation according to an exemplary embodiment of the present disclosure.

For example, multiple images captured by the drone can be pasted into the spherical coordinate system as shown in Figure 6. For example, the coordinates of A in the spherical coordinate system can be determined according to the longitude and latitude of the image A when the drone captured the image, and the coordinates of A in the spherical coordinate system can be determined according to the Coordinate range or area in spherical coordinate system, put A in the corresponding position.

As shown in FIG. 7A , the coordinates of the projection center point can be set in a spherical coordinate system, and a straight line can be obtained by connecting the projection center point and a point A on the spherical surface. Establish a projection plane, which can be tangent to the spherical coordinate system at the south pole or at other points. By calculating the intersection of the straight line and the projection plane, the projected coordinates can be obtained. Through coordinate transformation, the eye-of-the-sky image of the multiple images captured can be obtained.

The coordinate transformation may be polar coordinate transformation.

In some embodiments, the projection center point may be set at the south pole, for example, or may be set at other points.

The spherical coordinate system may be a spherical coordinate system centered on the drone itself. When the camera on the drone takes each picture, it has the corresponding attitude of the drone (such as GPS position and orientation, etc.) and the attitude of the gimbal. , and map each image to a spherical coordinate system for fusion. For example, the drone hovers at the initial position 1 to take a picture 1 straight ahead, rotates to the left by angle 1 to take a photo 2, rotates to the right based on position 1 and takes a photo 3, and moves vertically upward based on position 1 Rotate angle 1 to take a picture 4, rotate angle 1 vertically downward based on position 1 to take a picture 5, and rotate 180 degrees horizontally to take a picture 6 based on position 1, where angle 1 includes but is not limited to 90 degrees, can be Shoot 360-degree panoramic images in a similar way to horizontal rotation and vertical rotation. In the process of mapping the image to the spherical coordinate system, you can choose a horizontal direction with the center of the spherical coordinate system as the starting point to map the positive direction of the photo 1, and then use the position and orientation of the drone at the time of shooting to sequentially map each photo. Map to the spherical coordinate system, for example, map photo 1 to the front direction of the spherical surface, map photo 2 to the left side of the spherical surface, map photo 3 to the right side of the spherical surface, and map photo 4 to the spherical surface. For the top side, map photo 5 to the bottom side of the spherical surface, and map photo 6 to the back of the spherical surface. As shown in Figure 7B, the picture on the left is a photo taken by the drone at an over-looking angle, and the picture on the right is a photo that is about to be taken at a downward viewing angle to the bottom side of the spherical coordinate system.

In some embodiments, the spherical coordinate system corresponding to the movable platform can be used as a reference to map multiple images to the spherical coordinate system to generate a spherical panorama; the spherical panorama can be expanded with a preset position on the spherical coordinate system as the center , to get the banner image; according to the conversion relationship between the plane coordinate system where the banner image is located and the polar coordinate system, map the spherical panorama to the polar coordinate system to generate the eye image of the sky.

For example, after the spherical panoramic image is generated, the spherical panoramic image can be expanded to generate a banner panoramic image, and then the banner panoramic image can be mapped into polar coordinates according to the coordinate transformation to generate the eye of the sky image.

FIG. 7C is a schematic diagram showing another coordinate transformation according to an exemplary embodiment of the present disclosure.

For example, a spherical panorama can be expanded into a banner image centered on the direction the drone initially shot straight ahead. For example, the picture on the left in FIG. 7C is a schematic diagram of the expanded spherical panorama image. In order to facilitate the explanation of the principle of coordinate transformation, the banner panorama image shown in FIG. 3 can be simplified as shown on the left side of FIG. 7C . The banner panorama image in the Cartesian coordinate system. The banner panorama image has a length of b and a width of 2a. The image content is not shown in Figure 7C, only the image shape is shown. The upper part of the image is the sky area 301 shown in FIG. 3 , the lower part of the picture is the ground area 303 shown in FIG. 3 , and the center area is the shooting scene 302 shown in FIG. 3 .

After each photo is mapped to the spherical coordinate system, image processing can also be performed on the edges of adjacent photos to eliminate the problem of screen jumping.

The points of the sky area of the banner panoramic image in the left rectangular coordinate system in FIG. 7C can be mapped to the center area of the new right coordinate system, and the points of the ground area of the banner panoramic image in the left rectangular coordinate system can be mapped to the right. The outer region of the new coordinate system.

For example, map the point whose ordinate is 2a and whose abscissa is between 0 and b in the left rectangular coordinate system in FIG. 7C (that is, y=2a, and 0≤x≤b) to the origin of the new right coordinate system, Map the point where the ordinate is 0 and the abscissa is between 0 and b in the left rectangular coordinate system in Figure 7C (that is, y=0, and 0≤x≤b) to the new right coordinate system with R as the radius On the circle of , map the point where the ordinate is a and the abscissa is between 0 and b in the left rectangular coordinate system in Figure 7C (that is, y=a, and 0≤x≤b) to the new right coordinate system On a circle with a radius of R/2, for example, point M in the left Cartesian coordinate system can be mapped to M' in the new coordinate system on the right. After coordinate transformation, the image of the eye of the sky as shown in Figure 5 can be obtained.

Optionally, it is also possible to directly perform the mapping from the spherical coordinate system to the polar coordinate system, that is, directly map the generated spherical panorama to the polar coordinate system to generate the eye image of the sky.

In some embodiments, the panorama shooting method further includes: storing or sending the image of the eye of the sky to the terminal device.

For example, the drone can store the generated image of the eye of the sky in the memory of the drone, or send the image of the eye of the sky to the user terminal, and the user terminal can show the generated image of the eye of the sky to the user, and the user can The Eye of the Sky image is edited and processed.

In some embodiments, the panorama shooting method further includes: after receiving a playback instruction of the target object, playing a plurality of images to generate an image of the eye of the sky.

For example, the target object can be a user, and the user can send a playback instruction to the drone through the terminal device. The drone can include a display device. After receiving the user's playback instruction, the display device of the drone can play multiple images to generate the sky. The process of creating an eye image, you can also play multiple images on the terminal device to generate the eye of the sky.

In the embodiment of the present disclosure, the generation process of the eye of the sky image can be dynamically displayed, which can enhance user interest and improve user experience.

It should be noted that multiple images captured by drones can also be used to generate images in asteroid mode.

The panorama shooting method of the embodiment of the present disclosure adjusts the device parameters of the movable platform based on the shooting instructions, shoots multiple images, can automatically and quickly generate the image of the eye of the sky, does not require other software processing in the later stage, is simple in operation, can reduce the difficulty of operation, and improve the User interest.

FIG. 8 is a flowchart of another panorama shooting method shown in an exemplary embodiment of the present disclosure.

On the basis of the panoramic photographing method shown in FIG. 2 , the panoramic photographing method shown in FIG. 8 may further include steps S81 - S83 .

In step S81, the sky area in the eye of the sky image is identified.

In some embodiments, regions of the sky in the Eye of the Sky image are identified based on grayscale values and/or gradient values of the Eye of the Sky image.

For example, the sky region in the eye of the sky image can be identified based on the gray value of the eye of the sky image; the sky region in the eye of the sky image can be identified based on the gradient value of the eye of the sky image; it can also be based on the eye of the sky image The grayscale and gradient values of , identify the sky regions in the Eye of the Sky image.

For example, an area in the eye of the sky image with a grayscale value at a preset threshold may be identified as a sky area, and the preset threshold may be, for example, a grayscale value of 204. Alternatively, an area with a small gradient in the image of the eye of the sky can be identified as a sky area, or, a flat area in the image of the eye of the sky can be identified as a sky area, and the gray value in the image of the sky can be identified as a preset threshold, Regions with small and flat gradients are identified as sky regions.

Those skilled in the art can also identify sky regions based on other values of the eye in the sky image.

In step S82, the sky area is replaced with a preset image, wherein the preset image includes one or more of two-dimensional code, target object information, and spherical panoramic image.

Wherein, the spherical panorama image is an image obtained by performing coordinate transformation on a photographed scene other than the sky in the image as a rotation center.

For example, the entire sky area can be replaced with a preset image, or a part of the sky area can be replaced with a preset image according to a preset ratio, and the preset ratio can be set according to user needs.

The preset image can be, for example, a two-dimensional code or a user-defined logo (logo), and the sky area is replaced with the user's two-dimensional code, which is convenient for social sharing. Other users can scan the two-dimensional code to obtain user information or other information.

The preset image may be, for example, target object information, and the target object information may be, for example, user photos, landscape photos, airplane photos, sunset photos, and the like.

FIG. 9 is another image of the eye of the sky according to an exemplary embodiment of the present disclosure.

In some embodiments, the preset image may be, for example, a spherical panoramic image. For example, a preset proportion of the sky area can be replaced with a spherical panoramic image, and the unreplaced sky area can be image-fused with an adjacent edge area of the spherical panoramic image.

As shown in FIG. 9 , for example, the sky area 301 in the eye of the sky image shown in FIG. 5 can be replaced with the spherical panoramic image shown in FIG. Image fusion. The sky area in the eye of the sky image and the sky area in the spherical panorama image can be connected to achieve a better effect of displaying the color of the real sky, without the need for complex sky patching operations, which reduces the difficulty of operation.

For example, you can obtain the coordinates corresponding to the sky area in the eye of the sky image, perform coordinate transformation on the preset image according to the coordinates corresponding to the sky area, and replace the sky area with the preset image, which can be embedded without distortion of the preset image. to the inside of the Eye of the Sky image.

In step S83, the coordinates corresponding to the sky area are processed through coordinate transformation to enlarge or reduce the sky area.

For example, the coordinates corresponding to the sky area in the eye of the sky image can be obtained, and the coordinates corresponding to the sky area can be processed. For example, the coordinates corresponding to the sky area can be multiplied by a coefficient matrix, and the sky area in the eye of the sky image can be enlarged or reduced.

The ratio of enlarging or reducing the sky area can be set according to the user's needs.

For example, according to user requirements, when performing coordinate mapping, the coordinates of the projection center point can be changed, or the coordinate points can be offset to realize the zoom-in or zoom-out function of the sky area in the Eye of the Sky image.

It should be noted that the embodiments of the present disclosure include, but are not limited to, enlarging or reducing the sky area in the Eye of the Sky image, and may also enlarge or reduce the shooting scene or other areas in the Eye of the Sky image.

The panorama shooting method provided by the embodiments of the present disclosure can replace the sky area in the eye of the sky image with a preset image, customize the image according to the user's personalized needs, facilitate social media sharing, increase user interest, and improve user experience.

FIG. 10 is a flowchart of another panorama shooting method shown in an exemplary embodiment of the present disclosure.

On the basis of the panoramic photographing method shown in FIG. 2 , the panoramic photographing method shown in FIG. 10 may further include S101 - S104 .

In step S101, a splicing process is performed on a plurality of images to obtain a spherical panoramic image.

For example, a spherical panorama image can be obtained by splicing multiple images captured by a drone according to an image splicing algorithm, and the image splicing algorithm can be, for example, an alpha fusion algorithm.

In step S102, when the edges of adjacent pictures in the spherical panoramic image are not aligned, the shooting position corresponding to the unaligned image area is acquired.

In this embodiment of the present disclosure, it can be determined whether the edges of adjacent pictures are misaligned according to the spherical panoramic image, or whether the edges of adjacent pictures are misaligned according to the banner panoramic image generated by expanding the spherical panoramic image.

When the edges of adjacent pictures in the spherical panorama image are not aligned, the coordinates of the unaligned image area in the spherical coordinate system can be directly determined, and the geographic location and direction of the drone can be directly determined according to the coordinates. , attitude, etc.

When the edges of adjacent pictures in the banner panoramic image are not aligned, the coordinates corresponding to the unaligned areas in the banner panoramic image can be mapped into the spherical coordinate system to obtain the coordinates in the spherical coordinate system corresponding to the unaligned areas. The coordinates can directly determine the geographic location, direction, attitude, etc. of the drone when shooting.

For example, the edges of adjacent stitched images can be detected to determine whether the edges are aligned, and when the edges of the adjacent stitched images are not aligned, the coordinates corresponding to the unaligned image areas are obtained.

For example, in the process of taking an image by a drone, the aircraft may shake, drift, etc. At this time, after the banner panoramic image is generated, the edges of the image taken at this time may be misaligned, and there may be gaps left blank. position to perform a supplementary shooting operation.

In step S103, a supplementary shooting position of the movable platform is determined based on the coordinates corresponding to the unaligned image areas.

For example, coordinate mapping can be performed on the coordinates corresponding to the unaligned image area to obtain the geographic location of the drone when shooting the area, and the geographic location can be used as a supplementary shooting location of the drone.

In step S104, based on the supplementary shooting position, a supplementary shot operation is performed to obtain a supplementary shot image.

The drone can perform a supplementary shooting operation according to the movement to the supplementary shooting position, and use the image captured at the supplementary shooting position as the supplementary shooting image.

After the supplementary shot image is obtained, the original image corresponding to the supplementary shot image in the original multiple images can be replaced with the supplementary shot image, and the replaced image set is stitched to obtain a stitched image with a smooth transition.

FIG. 11 is a flowchart of another panorama shooting method shown in an exemplary embodiment of the present disclosure.

On the basis of the panoramic photographing method shown in FIG. 2 , the panoramic photographing method shown in FIG. 11 may further include S111 - S114 .

In step S111, a splicing process is performed on a plurality of images to obtain a spherical panoramic image.

For example, multiple images captured by the drone can be stitched according to an image stitching algorithm to obtain a banner panoramic image. The image stitching algorithm can be, for example, an alpha fusion algorithm.

In step S112, when the target object in the spherical panoramic image has a distorted area, the coordinates corresponding to the distorted area are obtained.

In the embodiment of the present disclosure, whether the target object has a distorted area can be determined according to the spherical panoramic image, or whether the target object has a distorted area can be determined according to a banner panoramic image generated by expanding the spherical panoramic image.

When the target object in the spherical panoramic image has a distorted area, the coordinates of the distorted area in the spherical coordinate system can be directly determined, and according to the coordinates, the geographic location, direction, attitude, etc.

When the target object in the banner panoramic image has a distorted area, the coordinates corresponding to the distorted area in the banner panoramic image can be mapped to the spherical coordinate system, and the coordinates in the spherical coordinate system corresponding to the distorted area can be obtained, which can be directly determined according to the coordinates The geographic location, direction, and attitude of the drone when shooting. The target object may be, for example, a building, a mountain, a river, a bridge, or the like.

For example, a building in a panoramic image of a banner can be detected to determine whether the building has a distorted area. When there is a distorted area in the building, the coordinates corresponding to the distorted area can be obtained, and the coordinates corresponding to the building containing the distorted area can also be obtained. .

For example, in the process of taking an image by a drone, the aircraft may shake, drift, etc. At this time, after the image taken at this time generates a panoramic image of the banner, the target object in the image may be distorted. The entire target object is subjected to a supplementary shooting operation.

In step S113, based on the coordinates corresponding to the distorted area, a supplementary shooting position of the movable platform is determined.

For example, the coordinates corresponding to the distorted area can be mapped to obtain the geographic location of the drone when shooting the area, and the geographic location can be used as the supplementary shooting location of the drone.

In step S114, the number of supplementary images is determined based on the FOV of the target object and the movable platform.

For example, the number of supplementary images can be determined according to the size of the FOV (field of view) of the movable platform by the target object with distortion. Multiple catch-up images.

In step S115, based on the supplementary shooting position, a supplementary shot operation is performed to obtain a supplementary shot image.

After the supplementary shot image is obtained, the original shot image containing the distorted area or the image containing the target object can be replaced with the supplementary shot image, and the replaced image set can be stitched to obtain a stitched image with a smooth transition.

FIG. 12 is a flowchart of another panorama shooting method shown in an exemplary embodiment of the present disclosure.

The panorama photographing method shown in FIG. 12 may be performed, for example, before step S101 shown in FIG. 10 or step S111 shown in FIG. 11 . The panorama shooting method shown in FIG. 12 may include S121-S123.

In step S121, when the photographing mode of the movable platform is the night scene mode, the brightness of a plurality of images is acquired.

For example, when the light intensity is low, the drone can use the night scene mode to shoot. When the drone uses the night scene mode to shoot, before stitching the multiple images, you can verify whether the brightness of the multiple images is roughly equal.

In step S122, it is determined whether the brightness of the plurality of images is within a preset range.

For example, it can determine whether the brightness of the multiple images captured by the drone is within the preset range. When the brightness of the multiple images is within the preset range, the multiple images can be stitched to generate a banner panorama image or a sky image. eye image; when the brightness of the multiple images is not within the preset range, step S123 may be performed.

The preset range can be set as required.

In step S123, when the brightness of the multiple images is not within the preset range, the brightness of the multiple images is adjusted.

When the brightness of multiple images is not within the preset range, the drone can automatically adjust the brightness of the multiple images, so that the brightness of the adjusted multiple images is within the preset range, which is convenient for subsequent stitching processing, and can achieve better visual effects. Nice panoramic image.

FIG. 13 is a flowchart of another panorama shooting method shown in an exemplary embodiment of the present disclosure.

The panorama shooting method shown in FIG. 13 may include S131-S136.

In step S131, a photographing instruction is received.

For example, drones can receive shooting instructions sent by users.

In step S132, a plurality of images are captured.

For example, the drone can take multiple images according to the shooting instructions.

In step S133, a panoramic image of the banner is generated based on the alpha fusion algorithm.

For example, based on the alpha fusion algorithm, multiple images can be stitched to generate a panoramic image of the banner.

In step S134, polar coordinate transformation is performed on the panoramic image of the banner to generate an image of the eye of the sky.

For example, the polar coordinates of the banner panorama image can be changed to generate the eye of the sky image; the banner panorama image or the eye of the sky image can also be processed according to the preset mode selected by the user to generate the image of the preset mode.

The image in the preset mode may be, for example, an image in an asteroid mode, or may be a panoramic image in other modes.

For example, the user's QR code, custom logo or other images can also be embedded into the eye of the sky image according to the user's needs.

In step S135, the animation of the generation process of the eye of the sky image is played.

For example, the animation of the generation process of the eye of the sky image can be shown to the user, so that the user can watch the generation process of the eye of the sky image more intuitively.

In step S136, the sky eye image is edited, and the edited sky eye image is released.

For example, according to the user's operation instruction, the image of the eye of the sky can be edited, and the edited image of the eye of the sky can be sent to the terminal device for publication.

For example, you can zoom in or zoom out the area in the eye of the sky image.

As will be appreciated by one skilled in the art, various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention can be embodied in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, which may be collectively referred to herein as implementations "Module" or "System".

14 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

As shown in FIG. 14 , the electronic device 1400 may include: a processor 1410 and a memory 1420, and the memory 1420 may be used to store executable instructions of the processor 1410; wherein the processor 1410 may be configured to execute the executable instructions by executing the instructions The following process: obtaining a shooting instruction; adjusting the posture of the movable platform based on the shooting instruction, shooting multiple images, and performing coordinate transformation according to the multiple images to generate an eye-to-sky image, and the image of the eye to the sky is based on the image in the image. The sky area is an image obtained by coordinate transformation of the center of rotation.

In an exemplary embodiment of the present disclosure, in the eye-of-the-sky image, the sky area is located in the center of the picture, and the shooting scene outside the sky is located around the picture. In an exemplary embodiment of the present disclosure, the performing coordinate transformation according to the multiple images to generate the eye image of the sky includes: using the spherical coordinate system corresponding to the movable platform as a reference, converting the multiple images Mapping to the spherical coordinate system to generate a spherical panorama; according to the conversion relationship between the spherical coordinate system and the polar coordinate system, mapping the spherical panorama to the polar coordinate system to generate the eye image of the sky.

In an exemplary embodiment of the present disclosure, the performing coordinate transformation according to the multiple images to generate the eye image of the sky includes: using the spherical coordinate system corresponding to the movable platform as a reference, converting the multiple images Map to the spherical coordinate system to generate a spherical panorama; take the preset position on the spherical coordinate system as the center, expand the spherical panorama to obtain a banner image; according to the plane coordinate system and polar coordinates where the banner image is located The transformation relationship of the system is used to map the spherical panorama to the polar coordinate system to generate the eye image of the sky.

In an exemplary embodiment of the present disclosure, the device is further configured to: after receiving a play instruction of the target object, play the plurality of images to generate the image in the eye of the sky mode.

In an exemplary embodiment of the present disclosure, the device is further configured to: identify a sky area in the eye of the sky image; replace the sky area with a preset image, wherein the preset image includes two One or more of the dimensional code, target object information, and spherical panoramic image, wherein the spherical panoramic image is an image obtained by coordinate transformation with the shooting scene other than the sky in the image as the rotation center.

In an exemplary embodiment of the present disclosure, identifying the sky region in the eye-to-sky image includes: identifying, based on grayscale values and/or gradient values of the eye-to-sky image, identifying the sky region in the eye-to-sky image sky area.

In an exemplary embodiment of the present disclosure, the preset image is a spherical panorama image, and the replacing the sky area with a preset image includes: replacing a preset scale area in the sky area with the A spherical panorama image, and image fusion is performed on the unreplaced sky region and the edge region adjacent to the spherical panorama image.

In an exemplary embodiment of the present disclosure, after generating the spherical panoramic image, the device is further configured to: when the edges of the spherical panoramic image are not aligned, obtain coordinates corresponding to the unaligned image areas; The coordinates corresponding to the unaligned image areas are used to determine a supplementary shooting position of the movable platform; based on the supplementary shooting position, a supplementary shooting operation is performed to obtain a supplementary shooting image.

In an exemplary embodiment of the present disclosure, after generating the spherical panoramic image, the device is further configured to: when the target object in the spherical panoramic image has a distorted area, obtain the coordinates corresponding to the distorted area; The coordinates corresponding to the distorted area are used to determine a supplementary shooting position of the movable platform; based on the supplementary shooting position, a supplementary shooting operation is performed to obtain a supplementary shooting image.

In an exemplary embodiment of the present disclosure, before performing the supplementary shot operation, the device is further configured to: determine the number of supplementary shots based on the target object and the field of view FOV of the movable platform.

In an exemplary embodiment of the present disclosure, before performing the stitching process on the multiple images, the device is further configured to: when the photographing mode of the movable platform is a night scene mode, acquire the multiple images judging whether the brightness of the multiple images is within the preset range; when the brightness of the multiple images is not within the preset range, adjust the brightness of the multiple images.

Referring to FIG. 15, a program product 1500 for implementing the above method according to an embodiment of the present disclosure is described, which can adopt a portable compact disk read only memory (CD-ROM) and include program codes, and can be stored in a terminal device, For example running on a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

A computer readable signal medium may include a propagated data signal in baseband or as part of a carrier wave with readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium can also be any readable medium, other than a readable storage medium, that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages—such as Java, C++, etc., as well as conventional procedural Programming Language - such as the "C" language or similar programming language. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (eg, using an Internet service provider business via an Internet connection).

It should be noted that although several modules or units of the apparatus for action performance are mentioned in the above detailed description, this division is not mandatory. Indeed, according to embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above may be further divided into multiple modules or units to be embodied.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solutions of the embodiments of the present disclosure.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common general knowledge or techniques in the technical field not disclosed by this disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the appended claims.

Claims

A panorama shooting method, comprising:

Get shooting instructions;

Adjust the posture of the movable platform based on the shooting instruction, and shoot multiple images;

Performing coordinate transformation according to the plurality of images to generate an eye-to-sky image, where the eye-to-sky image is an image obtained by performing coordinate transformation on the sky area in the image as a rotation center.
The method according to claim 1, wherein the sky area in the eye of the sky image is located in the center of the picture, and the shooting scene outside the sky is located around the picture.
The method according to claim 1 or 2, wherein the generating an eye-to-sky image by performing coordinate transformation according to the plurality of images comprises:

Using the spherical coordinate system corresponding to the movable platform as a benchmark, map the plurality of images to the spherical coordinate system to generate a spherical panorama;

According to the conversion relationship between the spherical coordinate system and the polar coordinate system, the spherical panorama image is mapped to the polar coordinate system to generate the eye image of the sky.
The method according to claim 1 or 2, wherein the generating an eye-to-sky image by performing coordinate transformation according to the plurality of images comprises:

Using the spherical coordinate system corresponding to the movable platform as a benchmark, map the plurality of images to the spherical coordinate system to generate a spherical panorama;

Taking the preset position on the spherical coordinate system as the center, expanding the spherical panorama to obtain a banner image;

According to the transformation relationship between the plane coordinate system where the banner image is located and the polar coordinate system, the spherical panorama is mapped to the polar coordinate system to generate the eye image of the sky.
The method of claim 1, further comprising:

After receiving the play instruction of the target object, the process of generating the image of the eye of the sky by playing the plurality of images.
The method according to claim 1, wherein the method further comprises:

identifying a sky region in the eye of the sky image;

Replace the sky area with a preset image, wherein the preset image includes one or more of a two-dimensional code, target object information, and a spherical panorama image, wherein the spherical panorama image is in the image The scene other than the sky is the image obtained by the coordinate transformation of the rotation center.
The method according to claim 6, wherein the identifying the sky area in the eye of the sky image comprises:

Based on the grayscale values and/or gradient values of the eye-to-sky image, a sky region in the eye-to-sky image is identified.
The method according to claim 6, wherein the preset image is a spherical panoramic image, and the replacing the sky area with the preset image comprises:

Replacing an area with a preset proportion in the sky area with the spherical panoramic image, and performing image fusion on the unreplaced sky area and an edge area adjacent to the spherical panoramic image.
The method according to claim 3 or 4, characterized in that, after generating the spherical panoramic image, further comprising:

When the edges of adjacent images in the spherical panoramic image are not aligned, obtain the coordinates corresponding to the unaligned image areas;

determining a supplementary shooting position of the movable platform based on the coordinates corresponding to the unaligned image areas;

Based on the supplementary shot position, a supplementary shot operation is performed to obtain a supplementary shot image.
The method according to claim 3 or 4, characterized in that, after generating the spherical panoramic image, further comprising:

When the target object in the spherical panoramic image has a distorted area, obtain the coordinates corresponding to the distorted area;

determining a supplementary shooting position of the movable platform based on the coordinates corresponding to the twisted area;

Based on the supplementary shot position, a supplementary shot operation is performed to obtain a supplementary shot image.
The method according to claim 10, wherein before performing the supplementary shooting operation, further comprising:

Based on the target object and the field of view FOV of the movable platform, the number of supplementary images is determined.
The method according to claim 1, characterized in that, before performing coordinate transformation according to the plurality of images to generate the eye of the sky image, the method further comprises:

When the photographing mode of the movable platform is a night scene mode, acquiring the brightness of the multiple images;

judging whether the brightness of the plurality of images is within a preset range;

When the brightness of the plurality of images is not within the preset range, the brightness of the plurality of images is adjusted.
An electronic device, comprising:

processor; and

a memory for storing executable instructions for the processor;

Wherein, the processor is configured to perform the following processes by executing the executable instructions:

Get shooting instructions;

Adjust the posture of the movable platform based on the shooting instruction, and shoot multiple images;

Performing coordinate transformation according to the plurality of images to generate an eye-to-sky image, where the eye-to-sky image is an image obtained by performing coordinate transformation on the sky area in the image as a rotation center.
The device according to claim 13, wherein in the eye of the sky image, the sky area is located in the center of the picture, and the shooting scene outside the sky is located around the picture.
The device according to claim 13 or 14, wherein the performing coordinate transformation according to the plurality of images to generate the eye of the sky image comprises:

Using the spherical coordinate system corresponding to the movable platform as a benchmark, map the plurality of images to the spherical coordinate system to generate a spherical panorama;

According to the conversion relationship between the spherical coordinate system and the polar coordinate system, the spherical panorama image is mapped to the polar coordinate system to generate the eye image of the sky.
The device according to claim 13 or 14, wherein the performing coordinate transformation according to the plurality of images to generate the eye of the sky image comprises:

Using the spherical coordinate system corresponding to the movable platform as a benchmark, map the plurality of images to the spherical coordinate system to generate a spherical panorama;

Taking the preset position on the spherical coordinate system as the center, expanding the spherical panorama to obtain a banner image;

According to the transformation relationship between the plane coordinate system where the banner image is located and the polar coordinate system, the spherical panorama is mapped to the polar coordinate system to generate the eye image of the sky.
The device of claim 13, wherein the processor is further configured to:

After receiving the play instruction of the target object, the process of generating the image of the eye of the sky by playing the plurality of images.
The device of claim 13, wherein the processor is further configured to:

identifying a sky region in the eye of the sky image;

Replace the sky area with a preset image, wherein the preset image includes one or more of a two-dimensional code, target object information, and a spherical panoramic image, wherein the spherical panoramic image is the image in the image. The scene other than the sky is the image obtained by the coordinate transformation of the rotation center.
The device according to claim 18, wherein the identifying the sky area in the eye of the sky image comprises:

Based on the grayscale values and/or gradient values of the eye-to-sky image, a sky region in the eye-to-sky image is identified.
The device according to claim 18, wherein the preset image is a spherical panoramic image, and the replacing the sky area with the preset image comprises:

Replacing an area with a preset proportion in the sky area with the spherical panoramic image, and performing image fusion on the unreplaced sky area and an edge area adjacent to the spherical panoramic image.
The device according to claim 15 or 16, characterized in that after generating the spherical panoramic image, it further comprises:

When the edges of adjacent images in the spherical panoramic image are not aligned, obtain the coordinates corresponding to the unaligned image areas;

determining a supplementary shooting position of the movable platform based on the coordinates corresponding to the unaligned image areas;

Based on the supplementary shot position, a supplementary shot operation is performed to obtain a supplementary shot image.
The device according to claim 15 or 16, characterized in that after generating the spherical panoramic image, it further comprises:

When the target object in the spherical panoramic image has a distorted area, obtain the coordinates corresponding to the distorted area;

determining a supplementary shooting position of the movable platform based on the coordinates corresponding to the twisted area;

Based on the supplementary shot position, a supplementary shot operation is performed to obtain a supplementary shot image.
The device according to claim 22, further comprising:

Based on the target object and the field of view FOV of the movable platform, the number of supplementary images is determined.
The device according to claim 13, characterized in that, before performing coordinate transformation according to the plurality of images to generate the eye of the sky image, further comprising:

When the photographing mode of the movable platform is a night scene mode, acquiring the brightness of the multiple images;

judging whether the brightness of the plurality of images is within a preset range;

When the brightness of the plurality of images is not within the preset range, the brightness of the plurality of images is adjusted.
A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the method of any one of claims 1-12 is implemented.