WO2023115390A1

WO2023115390A1 - Image processing method and device, movable platform, control terminal, and system

Info

Publication number: WO2023115390A1
Application number: PCT/CN2021/140483
Authority: WO
Inventors: 封旭阳
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2021-12-22
Filing date: 2021-12-22
Publication date: 2023-06-29

Abstract

An image processing method and device, a movable platform, a control terminal, and a system. A first photographing device and a second photographing device are mounted on the movable platform. An image acquired by the first photographing device is used for obtaining depth information, that is, the first photographing device is a photographing device for obstacle avoidance; an image acquired by the second photographing device is used for displaying to a user. Moreover, an image to be displayed on a user interface comprises at least part of image content acquired by the first photographing device and all image content acquired by the second photographing device. In this way, in addition to presenting all the image content acquired by the second photographing device, the user interface additionally presents at least part of the image content acquired by the photographing device for obstacle avoidance. Therefore, the field of view which can be obtained by a user from a user interface is broadened, the view finding and lens pose can be conveniently adjusted during photographing, and the photographing difficulty is reduced.

Description

Image processing method, device, mobile platform, control terminal and system

technical field

The present application relates to the field of image technology, in particular to an image processing method, device, mobile platform, control terminal and system.

Background technique

Movable platforms such as drones, unmanned vehicles, and robots have been widely used in many fields. Take UAVs as an example. At present, UAVs can be equipped with shooting devices, and users can get new shooting angles by using UAVs to shoot. However, in the scene of using drones to shoot, the framing and operation of the lens are often limited by the field of view of the shooting device, which increases the difficulty of shooting. It takes multiple shots to get a satisfactory photo, which greatly reduces the user's shooting experience.

Contents of the invention

In view of this, one of the purposes of this application is to provide an image processing method, device, mobile platform, control terminal and system, so as to effectively reduce the difficulty of shooting.

In order to achieve the above technical effects, the embodiment of the present application discloses the following technical solutions:

In a first aspect, an image processing method is provided, the method comprising:

Acquiring an image to be displayed, wherein the image to be displayed includes at least part of the image content captured by the first camera mounted on the movable platform, and all image content collected by the second camera mounted on the movable platform, the first A photographing device is used to collect images for obtaining depth information, and the second photographing device is used to collect images displayed to users;

The image to be displayed is displayed through a user interface.

In a second aspect, an image processing device is provided, including:

processor;

memory for storing processor-executable instructions;

Wherein, when the processor invokes the executable instruction, the operation of the first aspect above is implemented.

In a third aspect, a mobile platform is provided, including:

body;

a power assembly, used to drive the movable platform to move in space;

The first photographing device is used to collect and obtain images of depth information;

The second shooting device is used to collect images displayed to the user;

processor;

memory for storing processor-executable instructions;

In a fourth aspect, a control terminal is provided, which communicates with a movable platform, and the control terminal includes:

a communication module for communicating with the movable platform;

processor;

memory for storing processor-executable instructions;

Wherein, when the processor invokes the executable instruction, the operation of the first aspect above is realized.

In a fifth aspect, an image processing system is provided, the system comprising:

control terminal;

A movable platform communicating with the control terminal, the movable platform is equipped with a first photographing device and a second photographing device; the movable platform includes a processor and a memory, and the memory is used to store processor-executable instructions ; implementing the operation in the first aspect above when the processor calls the executable instruction.

According to a sixth aspect, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned steps in the first aspect are implemented.

In a seventh aspect, a computer-readable storage medium is provided, and several computer instructions are stored on the computer-readable storage medium, and when the computer instructions are executed, the method of the above-mentioned first aspect is executed.

The present application provides an image processing method, device, movable platform, control terminal and system, wherein the movable platform is equipped with a first photographing device and a second photographing device, and the images collected by the first photographing device are used to acquire Depth information, that is, the first shooting device is a shooting device for obstacle avoidance, and the image collected by the second shooting device is used to show the user, that is, the second shooting device is used to provide the user with the first mobile platform Angle of shooting device. The image to be displayed displayed on the user interface includes at least part of the image content captured by the first shooting device and all image content collected by the second shooting device. In this way, in addition to presenting all the image content collected by the second shooting device, the user interface also presents at least part of the image content collected by the shooting device used for obstacle avoidance. Thereby, the field of view that the user can obtain from the user interface is widened, and it is helpful to adjust the viewfinder and/or the pose of the lens during shooting, and reduces the difficulty of shooting.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 is an application scene of photographing by a mobile platform in the related art.

Fig. 2 is a flowchart of an image processing method according to an embodiment of the present application.

FIG. 3A is a schematic diagram of an image to be displayed according to an embodiment of the present application.

Fig. 3B is a schematic diagram of an image to be displayed according to another embodiment of the present application.

Fig. 3C is a schematic diagram of an image to be displayed according to another embodiment of the present application.

Fig. 3D is a schematic diagram of an image to be displayed according to another embodiment of the present application.

Fig. 4A is a schematic diagram of a user interface according to an embodiment of the present application.

Fig. 4B is a schematic diagram of an image to be displayed according to another embodiment of the present application.

Fig. 5 is a schematic diagram of a user interface according to another embodiment of the present application.

Fig. 6A is a schematic diagram showing a first image and a second image according to an embodiment of the present application.

Fig. 6B is a schematic diagram of a first image and a second image according to another embodiment of the present application.

Fig. 6C is a schematic diagram of a first image and a second image according to another embodiment of the present application.

Fig. 7 is a schematic diagram of a superimposed image according to an embodiment of the present application.

Fig. 8 is a schematic diagram of a spliced image according to an embodiment of the present application.

Fig. 9 is a flowchart of an image processing method according to another embodiment of the present application.

Fig. 10 is a schematic diagram of a first image shown according to an embodiment of the present application.

Fig. 11 is a flowchart of an image processing method according to another embodiment of the present application.

Fig. 12 is a structural diagram of an image processing device according to an embodiment of the present application.

Fig. 13 is a structural diagram of a movable platform according to an embodiment of the present application.

Fig. 14 is a structural diagram of a control terminal according to an embodiment of the present application.

Fig. 15 is a structural diagram of an image processing system according to an embodiment of the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

A mobile platform refers to any device that can move, and may include, but is not limited to, land vehicles, water vehicles, air vehicles, and other types of motorized vehicles. As an example, the movable platform may be a UAV (Unmanned Aerial Vehicle, UAV), an unmanned vehicle, a robot, and the like. Mobile platforms have been widely used in many fields. Take UAVs as an example. At present, UAVs can be equipped with shooting devices, and users can get new shooting angles by using UAVs to shoot.

The shooting process often involves complex camera movements, composition and framing. In traditional shooting scenarios, for example, when a user shoots through a mobile terminal such as a mobile phone, the field of view obtained by the human eye is often larger than the range of the live view (liveview) displayed on the mobile phone shooting interface, so the user can rely on the understanding of the surrounding environment Come and plan ahead for good luck mirrors, composition and framing, etc.

In some scenes shot by UAV, the user can use the control terminal used to control the UAV to control the UAV to fly to a longer distance and take pictures. The computer transmits to the control terminal, and the control terminal displays the received image on the liveview screen. In this case, the user's field of vision is limited to the liveview screen displayed on the control terminal, and the user cannot obtain environmental information other than the liveview screen, nor can he know the surrounding environment of the drone. On the other hand, the limited battery life of UAVs also restricts users from using UAVs to make more explorations in the air in advance. When the shooting range is large, or there are many moving objects in the shooting range, the user cannot predict the moving objects that will enter the screen only through the liveview screen displayed on the control terminal, let alone predict in advance what will be displayed on the screen after changing the viewing direction. Therefore, the framing of shooting and the operation of the lens are greatly restricted, which increases the difficulty of shooting. Users need to constantly adjust the viewfinder range and lens pose, and take multiple shots to get satisfactory photos, which greatly reduces the user's shooting experience.

As shown in FIG. 1 , taking a photo of a racing car 120 running at high speed as an example, since the user's field of vision is limited to the liveview screen 110 displayed on the control terminal, the user cannot predict how long the racing car 120 will enter the liveview screen 110 after. Due to the lack of prediction, when the racing car 120 enters the liveview screen 110, it is too late to control the drone to move the mirror, and it is impossible to capture the image or video of the complete driving process.

In some related technologies, pre-exploration and understanding of the shooting environment can be realized by planning the route and composition in advance. For example, the key points in the route are marked in advance, and the composition is carried out in advance on the key points. However, such a method has poor flexibility and is not suitable for shooting sports scenes, such as the above-mentioned shooting scene of a high-speed racing car. Moreover, it takes more time to plan in advance, and it also brings additional power consumption to the drone.

For this reason, the application proposes a kind of image processing method, comprises the steps as shown in Figure 2:

Step 210: Acquiring an image to be displayed, wherein the image to be displayed includes at least part of the image content captured by the first camera mounted on the movable platform, and all image content collected by the second camera mounted on the movable platform;

Step 220: Display the image to be displayed through a user interface.

The movable platform is equipped with a first shooting device and a second shooting device. Wherein, the first photographing device is used for capturing images with depth information, so that the depth information can be used for obstacle avoidance. Specifically, the image captured by the first photographing device can be used to acquire depth information of an object, such as an obstacle.

In some embodiments, the first photographing device may adopt a binocular vision system. The principle of the binocular vision system is to use two cameras to form a binocular. Based on the principle of parallax and use imaging equipment to obtain two images of the measured object from different positions, the three-dimensional geometry of the object can be obtained by calculating the position deviation between the corresponding points of the image. Information, that is, using two cameras to form binoculars to complete the depth information perception of the scene in a certain direction.

In some other embodiments, the first shooting device may be a monocular camera, and the camera may collect multiple images at different positions, and use changes of the same object in the multiple images to determine the depth information of the measured object. Through the perceived depth information of the measured object, the obstacle avoidance function of the movable platform can be guaranteed, so that the movable platform can move safely.

Wherein, in order to improve the safety of the movement of the movable platform, the movable platform can be equipped with a plurality of first photographing devices with different orientations at different positions for realizing the obstacle avoidance function. Multiple first photographing devices with different orientations can realize obstacle avoidance in more directions.

The second shooting device is used to capture images displayed to the user, and these images may be liveview images, that is, the second shooting device may be a shooting device for capturing images and displaying a real-time view to the user, such as a camera. Of course, the image captured by the second shooting device can also be stored in the memory of the movable platform, wherein the liveview image and the image stored in the memory of the movable platform can be the collection result of the same object at the same time, but the images of the two The parameters can be different, for example, due to the problem of limited transmission bandwidth, the resolution of the liveview image can be lower than the image stored in the memory of the removable platform. In related technologies, the user interface displays image content captured by the second shooting device.

At least part of the content captured by the first shooting device can be obtained through the first image captured by the first shooting device; all the content of the image captured by the second shooting device can be obtained through the second image captured by the second shooting device, or through the first image get.

The image processing method provided in this embodiment can be applied to a mobile platform, where the image to be displayed is acquired by the mobile platform, and the image to be displayed is displayed through a user interface. Wherein, if the movable platform is provided with a user interface, the image to be displayed is displayed on the user interface of the movable platform; The image to be displayed is sent to the control terminal through the communication link, and the image to be displayed is displayed on the user interface through the control terminal.

The control terminal can be a ground control terminal that can be operated by the user, such as a remote controller, a mobile phone installed with mobile platform control software, a personal computer (Personal Computer, PC), a tablet computer, or a wearable device and the like. If the control terminal is provided with a user interface, such as a control terminal with a screen remote control or a mobile phone, the image to be displayed can be displayed on the user interface of the control terminal; if the control terminal is not provided with a user interface, such as a non-screen remote control, Then, it is possible to establish a communication connection between the control terminal and other electronic devices, and display the image to be displayed on the user interface of the other electronic devices, wherein the other electronic devices can be portable terminals with screens such as mobile phones and tablet computers.

The image processing method provided in this embodiment can also be applied in a control terminal. The control terminal communicates with the movable platform, and the control terminal can directly or indirectly receive the image to be displayed sent by the movable platform, and display the image to be displayed on the user interface. The control terminal may also receive at least part of the image content captured by the first photographing device and all image content captured by the second photographing device sent by the movable platform. After the image to be displayed is acquired through image processing, the image to be displayed is displayed on the user interface.

If the control terminal is provided with a user interface, such as a control terminal with a screen remote control or a mobile phone, the image to be displayed can be displayed on the user interface of the control terminal; if the control terminal is not provided with a user interface, such as a non-screen remote control, Then, it is possible to establish a communication connection between the control terminal and other electronic devices, and display the image to be displayed on the user interface of the other electronic devices, wherein the other electronic devices can be portable terminals with screens such as mobile phones and tablet computers.

In this embodiment, the image to be displayed displayed on the user interface includes at least part of image content captured by the first shooting device and all image content captured by the second shooting device. The so-called "at least part of the image content captured by the first camera device" may be part of the image content captured by the first camera device, or all the image content captured by the first camera device. In this way, in addition to presenting all the image content collected by the second shooting device, the user interface also presents at least part of the image content collected by the first shooting device for obstacle avoidance. Compared with the user interface in the above-mentioned related art that only displays the image content collected by the second shooting device, this embodiment displays more image content on the user interface for the user's reference or prediction, thereby broadening the user's ability to The range of field of view that can be obtained in the user interface helps to adjust the framing and/or the pose of the lens during shooting, reducing the difficulty of shooting. This provides users with more environmental reference information and assists users in creating with more degrees of freedom.

On the other hand, in the related art, the images captured by the first shooting device are only used to obtain depth information to realize the obstacle avoidance function, but in this embodiment, the first shooting device is also used to obtain additional image content, so that in Broaden the user's field of vision and reduce the difficulty of shooting without increasing the cost of the mobile platform.

Wherein, adjusting the framing and/or the pose of the lens by means of additional image content of the first photographing device is to adjust the viewfinder range/or the pose of the lens of the second photographing device.

Referring to FIGS. 3A-3D , in some embodiments, the image to be displayed 300 may include a first image area 310 and a second image area 320 . The first image area 310 may include at least part of the image content captured by the first camera; the second image area 320 may include all image content captured by the second camera.

The positional relationship between the first image area 310 and the second image area 320 may include one or more of the following positional relationships:

In some embodiments, as shown in FIG. 3A , the first image area 310 and the second image area 320 may be located on two sides of the image to be displayed 300 . As an example, the first image area 310 and the second image area 320 may be separated left and right, or may be separated up and down on both sides of the image 300 to be displayed. Of course, the first image area 310 and the second image area 320 may also be separated on both sides of the image to be displayed 300 in other ways, which is not limited in this application.

In some other embodiments, as mentioned above, the movable platform may be equipped with multiple first photographing devices with different orientations at different positions for realizing the obstacle avoidance function. Image content captured by different first shooting devices may be displayed in different first regions. Thus, the number of the first image areas 310 may be two or more. Taking two first image areas as an example, as shown in FIG. 3B , the two first image areas 310 may be located on two sides of the second image area 320 . As an example, the two first image areas 310 may be separated left and right, or may be separated up and down on both sides of the second image area 320 . Of course, the two first image areas 310 may also be separated on both sides of the second image area 320 in other ways, which is not limited in this application.

In some embodiments, as shown in FIG. 3C , the first image area 310 may surround the periphery of the second image area 320 . The first image area 310 may include a plurality of first image areas 310 surrounding the second image area 320 . Wherein, the image content displayed in the plurality of first image areas may be continuous image content or discontinuous image content.

In some embodiments, as shown in FIG. 3D , the size of the first image area 310 is larger than the size of the second image area 320 , and the first image area 310 includes the second image area 320 .

In addition, the position of the second image area in the image to be displayed may be determined by one or more of the following methods:

Way 1: Determine the position of the second image area in the image to be displayed according to the first preset parameter. For example, the first preset parameter can be the coordinates of the feature point of the second image area in the image to be displayed, and the feature point can be the center point, vertex, etc. of the second image area; the first preset parameter can also be the second image The location of the region in the image to be displayed. Certainly, the first preset parameter may also be other parameters capable of determining the position of the second image area in the image to be displayed, which is not limited in this application.

Mode 2: Determine the position of the second image area in the image to be displayed according to the position information carried in the first user instruction. The position of the second image area in the image to be displayed can be adjusted according to the wishes of the user, and the user can change the position of the second image area in the image to be displayed by inputting a first user instruction carrying position information. The position information can be the coordinates of the feature points of the second image area in the image to be displayed, and the feature points can be the center point, vertex, etc. of the second image area; the position information can also be the position of the second image area in the image to be displayed area. Of course, the location information may also be other parameters capable of determining the location of the second image area in the image to be displayed, which is not limited in this application.

In some embodiments, the position of the second image area in the image to be displayed may be determined by default according to the first preset parameter, and then the position of the second image area in the image to be displayed is adjusted according to the position information carried in the first user instruction.

Mode 3: Determine the position of the second image area in the image to be displayed based on the first obstacle avoidance strategy of the movable platform. As an example, the first obstacle avoidance strategy may be based on the orientation and number of obstacles, correspondingly adjusting the position of the second image area in the image to be displayed. For example, if there are more obstacles in the left front of the movable platform than in the right front, then the second image area can be set on the right side of the image to be displayed, and the first image area can be set on the right side of the image area to be displayed. Left side, to show the user the environment in front of the movable platform, which helps the user better control the movement of the movable platform when shooting, avoiding collisions with obstacles, so as to better realize the framing and/or lens alignment Pose adjustment.

In some embodiments, referring to FIG. 3D , the second image area 320 may be located in a central area of the image to be displayed 300 .

In order to allow the user to better distinguish the first image area and the second image area in the image to be displayed, in some embodiments, the first image area and the second image area may have differentiated marks on the user interface. As shown in FIG. 4A , the user interface 400 includes a first window 410 and a second window 420 . The first window 410 and the second window 420 can be respectively arranged on the user interface 400 (as shown in FIG. 4A ), and the second window 420 can also be inside the first window 410 (not shown in the figure). The first window 410 displays a first image area 430 of the image to be displayed, and the second window displays a second image area 440 of the image to be displayed. In this way, the first image area and the second image area are displayed with differential marks on the user interface.

In some other embodiments, the first image area and the second image area may have differential marks on the images to be displayed. For example, different marks may be added to the image to be displayed to distinguish between the first image area and the second image area.

As an example, the images displayed in the first image area and the second image area are respectively set with different image parameters, such as different contrast, brightness, color, etc., so as to realize differentiated identification. For example, one of the image areas can display a grayscale image and the other image area can display a color image for differentiated identification.

As an example, the junction of the first image area and the second image area is provided with a first specific mark, so as to realize differentiated marks. As shown in FIG. 4B, a dotted line mark is provided at the junction of the first image area 430 and the second image area 440, indicating that the image area within the dotted line frame is the second image area 440, and the image area outside the dotted line frame is the first image area. 430.

As an example, the second image area may be provided with a second specific logo, so as to realize differentiated logos. For example, the second specific identification may be a text identification and/or a graphic identification.

The first image area and the second image area may be differentially marked according to one or more combinations of the above examples, so that the user can distinguish the two image areas more clearly.

In addition, the size of the second image area in the image to be displayed may be determined by one or more of the following methods:

Way 1: Determine the size of the second image area in the image to be displayed according to the second preset parameter. As an example, the second preset parameter may be the length and/or width of the second image area. As an example, the second preset parameter may be the area of the second image region. As an example, the second preset parameter may be the ratio of the size parameter of the second image area to the size parameter of the first image area, the image to be displayed or the user interface, wherein the size parameter may include one of length, width, or area or more.

It can be understood that the image to be displayed can be displayed on the entire user interface (full screen mode), and the user interface can also be divided into multiple areas or windows, wherein one area or window is used to display the image to be displayed, and other areas or windows are used to display Movable platform parameters or other functional virtual buttons. Since the second image area includes the real-time view image collected by the second shooting device, in the shooting scene, in order to ensure that the user can clearly see the real-time view, the area of the second image area and the user interface or the image to be displayed can be set to meet a certain requirement. proportional relationship.

Way 2: Determine the size of the second image area in the image to be displayed according to the size information carried in the second user instruction. The size of the second image area in the image to be displayed can be adjusted according to the wishes of the user, and the user can change the size of the second image area in the image to be displayed by inputting a second user instruction carrying size information. The size information may include one or more of the length, width, area, and size ratio of the second image area.

In some embodiments, the size of the second image area in the image to be displayed may be determined according to the second preset parameter by default, and then the size of the second image area in the image to be displayed is adjusted according to the size information carried in the second user instruction.

In some embodiments, the above-mentioned first user instruction and the second user instruction may be the same user instruction, and the user instruction carries position information and size information at the same time. The position of the second image area in the image to be displayed can be determined according to the position information, and the size of the second image area in the image to be displayed can be determined according to the size information. The first user instruction and the second user instruction may also be two different user instructions. As shown in FIG. 5 , the user interface 520 only displays a part of the image to be displayed 510 (left figure) including the second image area 530, then the first user instruction can be a drag instruction, and the dragging start point and the dragging instruction are used according to the drag instruction. The moving end point determines the movement vector of the second image area 530, and moves the image to be displayed 510 based on the movement vector, so that the position of the second image area moves to a position expected by the user. The second user instruction may be a zoom instruction, and the zoom ratio of the image to be displayed 510 may be determined according to the zoom instruction. After the image to be displayed 510 is moved and scaled, the image displayed on the user interface 520 is shown in the right figure of FIG. Getting more environmental information is helpful for predicting during shooting.

Mode 3: Determine the size of the second image area in the image to be displayed based on the second obstacle avoidance strategy of the movable platform. As an example, the second obstacle avoidance strategy may be based on the orientation and number of obstacles, correspondingly adjusting the size of the second image area in the image to be displayed. For example, if there are many obstacles around the movable platform, the size of the second image area in the image to be displayed can be smaller, and the first image area is correspondingly larger, so that the user can see more obstacle information, so as to avoid The movable platform collides with obstacles during shooting.

Based on the above-mentioned embodiments, it is possible to determine the positional relationship between the first image area and the second image area in the image to be displayed, differentiated display, and the position and size of the second image area in the image to be displayed. In this way, based on the two images The image to be displayed can be determined by one or more of the positional relationship of the regions, the position of the second image region, and the size of the second image.

The image to be displayed includes at least part of image content captured by the first shooting device and all image content captured by the second shooting device. In some embodiments, the image to be displayed may be obtained by using the first image captured by the first shooting device, that is, the first image includes all image content captured by the first shooting device and all image content captured by the second shooting device . For example, the first photographing device mounted on the movable platform may be a fisheye camera or other cameras that can cover the photographing area of the second photographing device. As shown in FIG. 6A , the left image is a first image 610 captured by a first camera, and the right image is a second image 620 captured by a second camera. It can be seen that since the shooting area of the first shooting device covers the shooting area of the second shooting device, the first image 610 contains all the image content captured by the second shooting device, that is, all the image content of the second image 620 . Therefore, the image to be displayed can be acquired directly by using the first image.

In some other embodiments, the image to be displayed may be acquired by using the first image captured by the first shooting device and the second image captured by the second shooting device. The first image may at least partially overlap the second image, so-called "at least partial overlap", as shown in Figure 6A, the first image 610 includes all image content of the second image 620; it may also be as shown in Figure 6B , the first image 610 and the second image 620 include the same image content 630 . In addition, as shown in FIG. 6C , the first image 610 and the second image 620 may also be adjacent to each other.

In some embodiments, the image to be displayed is acquired based on the image content relationship between the image content captured by the first shooting device and the image content captured by the second shooting device. Or in other words, the image to be displayed is acquired based on the image content relationship between the first image and the second image. Image content relationships may include at least partial overlap, adjacency, or discontinuity.

In some embodiments, the image content relationship may be determined based on the relative pose relationship between the first camera and the second camera, the field of view of the first camera, and the field of view of the second camera. Based on the relative pose relationship of the two shooting devices, and their respective field of view sizes, it can be determined whether the field of view of the first shooting device covers the field of view of the second shooting device, or whether it is consistent with the field of view of the second shooting device. The field areas partially overlap or are adjacent, so that the image content relationship between the first image and the second image can be judged.

In some embodiments, the second photographing device can be installed on the movable platform through the pan-tilt, and the relative pose relationship between the first photographing device and the second photographing device can be based on the relative pose between the first photographing device and the pan-tilt , and the attitude of the gimbal to determine.

In some embodiments, the second shooting device can be fixedly installed on the movable platform, so the relative pose relationship between the first shooting device and the second shooting device can be calibrated in advance.

In some other embodiments, image recognition technology may also be used to determine whether the first image includes the image content of the second image, or whether the first image and the second image have the same image content. For the specific image recognition technology, reference may be made to relevant technical records, and this application will not discuss it here.

In some embodiments, the image to be displayed is obtained by performing one or more of superposition processing, splicing processing, clipping processing, image parameter adjustment processing, and dedistortion processing on at least one of the first image and the second image .

In some embodiments, if the first image 610 and the second image 620 at least partially overlap, the first image and the second image may be superimposed.

As an example, as shown in FIG. 6A, the first image 610 includes all the image content of the second image 620, then the second image can be superimposed on the first image based on the position of the image content of the second image 620 in the first image 610. On the image 610 , a superimposed image as shown in FIG. 4B is obtained, wherein the second image is displayed in the second image area 440 ; the first image is displayed in the first image area 430 and the second image area 440 .

As an example, as shown in FIG. 6B, the first image 610 and the second image 620 include the same image content 630, then based on the positions of the same image content 630 in the first image 610 and the second image 620 respectively, the second The first image 610 and the second image 620 are superimposed to obtain the superimposed image as shown in FIG. 7 .

Wherein, when performing the above-mentioned superposition processing, pixel fusion may be performed on the overlapped portion of the first image 610 and the second image 620 to obtain an image with better quality, and the fused image may be displayed as a part of the image to be displayed.

As an example, as shown in FIG. 6C , if the first image 610 is adjacent to the second image 620 , the first image and the second image may be spliced to obtain a spliced image as shown in FIG. 8 .

In some embodiments, the clipping process can be performed on the first image, and the clipping process includes steps as shown in FIG. 9:

Step 910: Obtain the first display size of the image to be displayed and/or the second display size corresponding to the image content of the second image in the image to be displayed;

Step 920: Crop the first image based on the first display size and/or the second display size.

Wherein, the first display size and the second display size may be preset size parameters, or may be user-adjustable size parameters.

As an example, the first display size of the image to be displayed may be obtained, and the image to be displayed that satisfies the first display size may be clipped from the first image.

As an example, the second display size corresponding to the image content of the second image in the image to be displayed may be the size of the second image area in the image to be displayed in the above embodiment. According to the second display size, an image to be displayed that is larger than the second display size can be cut out from the first image.

In some embodiments, when cutting out the image to be displayed from the first image, in addition to considering the first display size and/or the second display size, the corresponding image content of the second image in the image to be displayed may also be considered position for clipping.

The edge areas of the image are often distorted, or a few pixels at the edge of the image are blurred. In some embodiments, edge regions of the first image and/or the second image may be cropped to remove distorted or blurred regions. The edge area can be a preset image area, and those skilled in the art can set the size of the edge area according to actual needs, for example, several pixels.

As mentioned above, the first photographing device may be a camera for realizing the obstacle avoidance function, and the second photographing device may be a camera for capturing images and presenting a real-time view to the user. In some cases, since the first shooting device and the second shooting device are used to implement different functions, the images captured by them, that is, the image parameters of the first image and the second image will also be different. The image parameters may include one or more of brightness, contrast, resolution, super-resolution, and distortion. For example, since the first image captured by the first photographing device is used to obtain depth information, its resolution may be smaller than that of the second image displayed to the user.

In order to improve viewing experience and effects, in some embodiments, image parameter adjustment processing may be performed on the first image and/or the second image. Adjust the image parameters of the first image and/or the second image, so that the image parameters of the first image and the second image are consistent.

As an example, an image parameter of the other of the first image and the second image may be adjusted according to an image parameter of the other of the first image and the second image. For example, the image parameters of the first image are adjusted according to the image parameters of the second image, so that the image parameters of the first image are aligned with the image parameters of the second image.

As an example, the image parameters of the second image and the image parameters of the first image may be respectively adjusted to preset image parameters.

In some embodiments, de-distortion processing may be performed on the first image and/or the second image, so as to eliminate imaging distortion caused by inconsistent magnifications at different positions of the optical lens. Those skilled in the art may select a correction algorithm to perform de-distortion processing on the first image and/or the second image according to actual needs, and this application does not discuss further here.

In this embodiment, the image to be displayed can be obtained by performing one or more of superposition processing, splicing processing, clipping processing, image parameter adjustment processing, and dedistortion processing on the first image and/or the second image.

In some embodiments, if the first image includes all the image content of the second image, then the first image may not be subjected to superposition processing, splicing processing, cropping processing, image parameter adjustment processing or de-distortion processing, but directly as the image to be displayed. image. It can be understood that when there are multiple first shooting devices, the first image may be obtained by splicing the images captured by the multiple first shooting devices.

In some scenarios, if the movable platform performs shooting tasks while moving, or there is a moving object in the shooting area of the movable platform, the first shooting device and the second shooting device can be controlled to expose synchronously, so that the first image and the second The images are images captured by the first shooting device and the second shooting device at the same time, so as to avoid recording the states of the same object at different times in the same image.

As mentioned above, the movable platform can be equipped with a plurality of first photographing devices with different orientations at different positions for realizing the obstacle avoidance function. Multiple first shooting devices with different orientations can realize obstacle avoidance in more directions, and the images collected by multiple first shooting devices with different orientations can be spliced to obtain an image with a larger field of view as shown in Figure 10 .

In some embodiments, the first image may be an image captured by one of the first shooting devices, or may be an image spliced from images captured by at least two first shooting devices. In a case where there are multiple first photographing devices, the first image may be acquired in various ways as required. For example, all the images collected by the first shooting device may be selected, or part of the images collected by the first shooting device may be selected for computing power, or the first shooting device whose orientation is the same as that of the second shooting device may be selected, and so on.

In some embodiments, based on the motion state of the movable platform, the first photographing device used to acquire the first image may be determined from multiple first photographing devices.

In some embodiments, the motion state of the movable platform may include one or more of the motion direction, the motion speed, or the relative motion relationship between the movable platform and the shooting target.

As an example, the first photographing device for acquiring the first image may be determined based on the moving direction of the movable platform. For example, when the movable platform is moving forward, a first photographing device facing the moving direction (forward) may be used to acquire a first image, so that the acquired first image includes environmental information in the moving direction of the movable platform.

As an example, the first photographing device for acquiring the first image may be determined based on the moving direction and moving speed of the movable platform. For example, when the movable platform is moving forward and the moving speed is fast, in addition to the first photographing device facing the direction of movement (front), the first photographing device facing the left and right sides can also be used to capture a wider field of view. Big first image. That is, when the speed of the movable platform is fast, the first image with a relatively large field of view can be acquired; otherwise, the first image with a relatively small field of view can be acquired.

As an example, the first photographing device for acquiring the first image may be determined based on the relative motion relationship between the movable platform and the photographing target. For example, the photographing target may be a racing car driving at high speed, if the photographing target enters the photographing range of the first photographing device and the second photographing device from the left. Then, the first photographing device facing the photographing target (left side) can be used to acquire the first image, so that the acquired first image meets the user's photographing requirements.

In this embodiment, the image to be displayed displayed on the user interface includes at least part of image content captured by the first shooting device and all image content captured by the second shooting device. In this way, in addition to presenting all the image content collected by the second shooting device, the user interface also presents at least part of the image content collected by the first shooting device for obstacle avoidance. Compared with the user interface in the related art that only displays the image content collected by the second shooting device, this embodiment displays more image content on the user interface for the user's reference or prediction, thereby broadening the user's understanding from the user. The range of field of view that can be obtained in the interface helps to adjust the framing and the pose of the lens during shooting, reducing the difficulty of shooting. This provides users with more environmental reference information and assists users in creating with more degrees of freedom.

In addition, the present application also provides an image processing method, including the steps shown in Figure 11:

Step 1110: Based on the motion state of the movable platform, determine a target first photographing device for capturing the first image from a plurality of first photographing devices;

Wherein, the motion state of the movable platform includes one or more of the motion direction, the motion speed, or the relative motion relationship between the movable platform and the shooting target.

Step 1120: Obtain the first image captured by the target first shooting device and the second image captured by the second shooting device;

Wherein, the first image and the second image are respectively images captured by the first photographing device and the second photographing device at the same moment. The second shooting device is installed on the movable platform through the platform.

Step 1130: Adjust the image parameters of the first image according to the image parameters of the second image, so that the image parameters of the first image are aligned with the image parameters of the second image;

Wherein, the image parameter may be one or more of brightness, contrast, resolution, super-resolution, and distortion.

Step 1140: Determine the image content relationship between the first image and the second image based on the pre-calibrated relative pose of the first shooting device and the pan/tilt, and the posture of the pan/tilt;

Step 1151: If the first image includes all the image content of the second image, based on the position of the image content of the second image in the first image, superimpose the second image on the first image to obtain a processed image;

Step 1152: If the first image and the second image contain the same image content, based on the positions of the same image content in the first image and the second image respectively, superimpose the first image and the second image to obtain the processed after the image;

Step 1153: If the first image is adjacent to the second image, then splicing the first image and the second image to obtain a processed image;

Step 1160: Based on the first display size of the image to be displayed, the second display size and position corresponding to the image content of the second image in the image to be displayed, crop the image to be displayed from the processed image;

Wherein, the image area to be displayed includes a first image area and a second image area. The first image area includes at least part of the image content of the first image and the second image area includes the entire image content of the second image.

1170: Perform differential identification on the first image area and the second image area;

1180: Display the image to be displayed through the user interface.

For the specific implementation manner, refer to the above embodiments, and the present application will not repeat them here.

Wherein, step 1110 may be performed by the mobile platform, and steps 1120-1180 may be performed by the mobile platform, or may be performed by a control terminal communicatively connected with the mobile platform.

Based on the image processing method described in any of the above embodiments, the present application also provides a schematic structural diagram of an image processing device as shown in FIG. 12 . As shown in FIG. 12 , at the hardware level, the image processing device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required by other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and runs it, so as to realize the image processing method described in any of the above embodiments.

Based on the image processing method described in any of the above embodiments, the present application also provides a schematic structural diagram of a movable platform as shown in FIG. 13 . As shown in Figure 13, at the hardware level, the mobile platform includes a fuselage, a power assembly, a first camera, a second camera, a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course it may also include Hardware required by other businesses. Wherein, the power assembly is used to drive the movable platform to move in space; the first photographing device is used to collect images for obtaining depth information; the second photographing device is used to collect images displayed to users. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the image processing method described in any of the above embodiments.

Based on the image processing method described in any of the above embodiments, the present application also provides a schematic structural diagram of a control terminal as shown in FIG. 14 , where the control terminal communicates with a movable platform. As shown in Figure 14, at the hardware level, the control terminal includes a communication module, a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required by other services. The communication module is used for communicating with the movable platform. The processor reads the corresponding computer program from the non-volatile memory into the memory and runs it, so as to realize the image processing method described in any of the above embodiments.

Based on the image processing method described in any of the above embodiments, the present application also provides a schematic structural diagram of an image processing system as shown in FIG. 15 . As shown in Figure 15, at the hardware level, the image processing system includes a control terminal as shown in Figure 14, and a movable platform communicating with the control terminal as shown in Figure 13, the movable platform is equipped with a first camera and a second The photographing device; the movable platform includes a processor and a memory, and the memory is used to store executable instructions of the processor; when the processor invokes the executable instructions, an image processing method described in any of the above embodiments is implemented.

Based on the image processing method described in any of the above embodiments, the present application also provides a computer program product, including a computer program, when the computer program is executed by a processor, it can be used to perform the image processing method described in any of the above embodiments method.

Based on the image processing method described in any of the above embodiments, the present application also provides a computer storage medium, where a computer program is stored in the storage medium, and when the computer program is executed by a processor, it can be used to perform an image processing method described in any of the above embodiments. an image processing method.

As for the device embodiment, since it basically corresponds to the method embodiment, for related parts, please refer to the part description of the method embodiment. The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without creative effort.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. The term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements but also other elements not expressly listed elements, or also elements inherent in such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The methods and devices provided by the embodiments of the present application have been described in detail above. The principles and implementation methods of the present application have been explained by using specific examples in this paper. The descriptions of the above embodiments are only used to help understand the methods and methods of the present application. core idea; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be construed as limiting the application .

Claims

An image processing method, characterized in that the method comprises:

Acquiring an image to be displayed, wherein the image to be displayed includes at least part of the image content captured by the first camera mounted on the movable platform, and all image content collected by the second camera mounted on the movable platform, the first A photographing device is used to collect images for obtaining depth information, and the second photographing device is used to collect images displayed to users;

The image to be displayed is displayed through a user interface.
The method according to claim 1, wherein the image to be displayed includes a first image area and a second image area, the first image area includes at least part of the image content captured by the first photographing device, the The second image area includes all image content captured by the second shooting device.
The method according to claim 2, wherein the positional relationship between the first image area and the second image area includes at least one of the following:

The first image area and the second image area are separated on both sides of the image to be displayed;

The first image area includes two, and the two first image areas are separated on both sides of the second image area;

The first image area surrounds the periphery of the second image area;

The size of the first image area is larger than the size of the second image area, and the first image area includes the second image area.
The method according to claim 2 or 3, wherein the position of the second image area in the image to be displayed is determined by at least one of the following methods:

determining a position of the second image area in the image to be displayed according to a first preset parameter;

determining the position of the second image area in the image to be displayed according to the position information carried in the first user instruction;

The position of the second image area in the image to be displayed is determined based on the first obstacle avoidance strategy of the movable platform.
The method according to claim 4, wherein the second image area is located in a central area of the image to be displayed.
The method according to any one of claims 2 to 5, wherein the first image area and the second image area have differential marks on the image to be displayed.
The method according to any one of claims 2 to 5, wherein the first image area and the second image area have differentiated marks on the user interface.
The method according to claim 6 or 7, wherein a first specific mark is provided at the junction of the first image area and the second image area; and/or,

The second image area is provided with a second specific mark.
The method according to any one of claims 2 to 8, wherein the size of the second image area in the image to be displayed is determined by at least one of the following methods:

determining the size of the second image area in the image to be displayed according to a second preset parameter;

determining the size of the second image area in the image to be displayed according to the size information carried in the second user instruction;

Determine the size of the second image area in the image to be displayed based on the second obstacle avoidance strategy of the movable platform.
The method according to any one of claims 1 to 9, wherein the image to be displayed is obtained by using a first image collected by the first photographing device, and the first image includes the first All image content of the second image captured by the second shooting device.
The method according to any one of claims 1 to 9, wherein the image to be displayed is obtained by using the first image and a second image captured by the second shooting device.
The method of claim 11, wherein the first image at least partially overlaps the second image; or,

The first image is adjacent to the second image.
The method according to any one of claims 10 to 12, wherein the image to be displayed is performed on at least one of the first image and the second image by superposition processing, splicing processing, and clipping obtained after one or more of processing, image parameter adjustment processing or de-distortion processing.
The method according to claim 13, wherein the process of adjusting the image parameters comprises:

Adjusting the image parameters of the first image and/or the second image, so that the image parameters of the first image and the second image are consistent.
The method according to claim 14, wherein the adjusting image parameters of the first image and/or the second image comprises:

adjusting an image parameter of the other of the first image and the second image based on an image parameter of the other of the first image and the second image; or

Adjusting the image parameters of the second image and the image parameters of the first image to preset image parameters respectively.
The method according to claim 13, wherein the image parameters include one or more of the following: brightness, contrast, and super-resolution.
The method according to claim 13, wherein the clipping process comprises:

Acquiring a first display size of the image to be displayed and/or a second display size corresponding to the image content of the second image in the image to be displayed;

Crop the first image based on the first display size and/or the second display size.
The method according to any one of claims 10 to 17, wherein the first image and the second image are respectively captured by the first shooting device and the second shooting device at the same time image.
The method according to any one of claims 10 to 18, wherein there are a plurality of first photographing devices, and the first image comprises: an image captured by one of the first photographing devices, or obtained by An image obtained by splicing at least two images captured by the first photographing device.
The method according to claim 19, wherein the first photographing device used to acquire the first image is determined from a plurality of first photographing devices based on the motion state of the movable platform. .
The method according to claim 20, wherein the motion state includes one or more of a motion direction, a motion speed, or a relative motion relationship between the movable platform and the shooting target.
The method according to any one of claims 1 to 21, wherein the image to be displayed is based on the difference between the content of the image captured by the first shooting device and the content of the image captured by the second shooting device The image content relationship is obtained.
The method according to claim 22, wherein the image content relationship is based on the relative pose relationship between the first camera and the second camera, the field of view of the first camera, and The field of view area of the second photographing device is determined.
The method according to claim 23, wherein the second photographing device is installed on the movable platform through a platform, and the relative pose relationship between the first photographing device and the second photographing device is based on The relative pose between the first photographing device and the pan/tilt is determined.
An image processing device, characterized in that it comprises:

processor;

memory for storing processor-executable instructions;

Wherein, when the processor invokes the executable instruction, the operations of the method of any one of claims 1-24 are implemented.
A mobile platform, characterized in that it comprises:

body;

a power assembly, used to drive the movable platform to move in space;

The first photographing device is used to collect and obtain images of depth information;

The second shooting device is used to collect images displayed to the user;

processor;

memory for storing processor-executable instructions;

Wherein, when the processor invokes the executable instruction, the operations of the method of any one of claims 1-24 are implemented.
A control terminal, which communicates with a movable platform, is characterized in that the control terminal includes:

a communication module for communicating with the movable platform;

processor;

memory for storing processor-executable instructions;

Wherein, when the processor invokes the executable instruction, the operations of the method of any one of claims 1-24 are implemented.
An image processing system, characterized in that the system includes:

control terminal;

A movable platform communicating with the control terminal, the movable platform is equipped with a first photographing device and a second photographing device; the movable platform includes a processor and a memory, and the memory is used to store processor-executable instructions ; when the processor invokes the executable instruction, the operation of the method of any one of claims 1-24 is implemented.
A computer program product, comprising a computer program, characterized in that, when the computer program is executed by a processor, the steps of the method in any one of claims 1-24 are implemented.
A computer-readable storage medium, characterized in that several computer instructions are stored on the computer-readable storage medium, and when the computer instructions are executed, the method described in any one of claims 1-24 is performed.