WO2023077421A1

WO2023077421A1 - Movable platform control method and apparatus, and movable platform and storage medium

Info

Publication number: WO2023077421A1
Application number: PCT/CN2021/128983
Authority: WO
Inventors: 李号; 徐佐腾; 郭阳阳
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2021-11-05
Filing date: 2021-11-05
Publication date: 2023-05-11
Also published as: CN117837160A

Abstract

A movable platform control method and apparatus, and a movable platform and a storage medium, which can control a movable platform to safely move in a space. In the movable platform, a first sensor and a second sensor respectively have a local field of view, which overlaps with a third sensor, wherein the local field of view is used for observing scenery in a corresponding direction. The control method comprises: acquiring first brightness information of a first local field of view of a third sensor and second brightness information of a second local field of view of the third sensor (S201); determining a first exposure parameter of the third sensor at least on the basis of the first brightness information, controlling the third sensor to collect a first image under the first exposure parameter, and acquiring, on the basis of the first image and an image collected by a first sensor, depth information of scenery in a first direction (S202); determining a second exposure parameter of the third sensor at least on the basis of the second brightness information, controlling the third sensor to collect a second image under the second exposure parameter, and acquiring, on the basis of the second image and an image collected by a second sensor, depth information of scenery in a second direction (S203); and controlling, according to the depth information, a movable platform to move in a space (S204).

Description

Control method and device of movable platform, movable platform and storage medium

technical field

The present application relates to the technical field of movable platforms, and in particular, relates to a method and device for controlling a movable platform, a movable platform, and a computer-readable storage medium.

Background technique

As technology develops, mobile platforms such as drones, autonomous vehicles, unmanned logistics vehicles or autonomous cleaning equipment are increasingly being used. Usually, the mobile platform is equipped with various sensors, which can collect data from the surrounding environment, and the mobile platform can control its own movement based on the data collected by the sensors. How to control the movable platform to move safely in space has been a technical issue that has been concerned in this field.

Contents of the invention

In order to solve the above technical problem of how to control the safe movement of the movable platform, the present application provides a method and device for controlling the movable platform, the movable platform and a computer-readable storage medium.

In the first aspect, the embodiment of the present application provides a method for controlling a mobile platform;

Wherein, the movable platform includes a first sensor, a second sensor and a third sensor; the first sensor and the third sensor have a first partial field of view that overlaps; the second sensor and the third sensor Having overlapping second partial fields of view; the first partial field of view is used to observe the scene in the first direction of the movable platform; the second partial field of view is used to observe the scene in the second direction of the movable platform;

The methods include:

acquiring first brightness information of the first partial field of view and second brightness information of the second partial field of view of the third sensor;

determining a first exposure parameter of the third sensor based at least on the first brightness information; controlling the third sensor to capture a first image under the first exposure parameter; based on the first image and the first Obtain the depth information of the scene in the first direction from the image collected by the sensor;

determining a second exposure parameter of the third sensor based at least on the second brightness information; controlling the third sensor to capture a second image under the second exposure parameter; based on the second image and the second The image collected by the sensor is used to acquire the depth information of the scene in the second direction; and the movable platform is controlled to move in space according to the depth information.

In a second aspect, the embodiment of the present application provides a control device for a movable platform;

The device includes a processor, a memory, and a computer program stored on the memory that can be executed by the processor. When the processor executes the computer program, the method for controlling the mobile platform described in the first aspect above is implemented. .

In a third aspect, the embodiment of the present application provides a mobile platform;

The mobile platform includes a processor, a memory, and a computer program stored on the memory that can be executed by the processor. When the processor executes the computer program, the above-mentioned mobile platform described in the first aspect is implemented. Control Method.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a number of computer instructions are stored, and when the computer instructions are executed, the mobile platform described in the aforementioned first aspect is implemented. The steps of the control method.

In the embodiment of the present application, the third sensor can form a binocular vision system with the first sensor and the second sensor respectively, and the first partial field of view and the second partial field of view respectively observe different first and second directions. Therefore, The first brightness information of the first partial field of view can be obtained for the first partial field of view alone, and the first exposure parameter can be determined based on this. Therefore, the first image captured by the third sensor under the first exposure parameter can be compared with the first partial field of view. The brightness of the field of view is matched, and the quality of the first image is high, so that the depth information of the scene in the first direction can be obtained based on the first image and the image collected by the first sensor. Similarly, since the second brightness information of the second partial field of view can be obtained for the second partial field of view alone, and the second exposure parameter can be determined based on this, the second image collected by the third sensor under the second exposure parameter can be Matching the brightness of the second partial field of view, the quality of the second image is relatively high, so that the depth information of the scene in the second direction can be obtained based on the second image and the image collected by the second sensor. Furthermore, since the movable platform can obtain reliable depth information of the scene in the first direction and depth information of the scene in the second direction based on the above processing, it can safely control itself to move in space.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

Fig. 1A is a schematic diagram showing the installation structure of a sensor on a movable platform and the field of view of the sensor according to an embodiment of the present application;

Fig. 1B is a schematic diagram showing the field of view of two sensors on a movable platform according to an embodiment of the present application;

Fig. 2 is a schematic flowchart of a method for controlling a mobile platform according to an embodiment of the present application;

Fig. 3 is a schematic flowchart of some steps in a method for controlling a mobile platform according to an embodiment of the present application;

Fig. 4 is a schematic flowchart of some steps in another method for controlling a mobile platform according to an embodiment of the present application;

Fig. 5 is a schematic flowchart of a method for acquiring an image by a sensor on a movable platform according to an embodiment of the present application;

Fig. 6 is a schematic diagram of a sequence diagram in a method for controlling a mobile platform according to an embodiment of the present application;

Fig. 7 is a schematic flowchart of a method for outputting images by a sensor on a movable platform according to an embodiment of the present application;

Fig. 8 is a schematic diagram of a first state in a mobile platform control method according to an embodiment of the present application;

Fig. 9 is a schematic diagram of a second state in a mobile platform control method according to an embodiment of the present application;

Fig. 10 is a schematic block diagram of a control device for a movable platform shown according to an embodiment of the present application;

Fig. 11 is a schematic block diagram showing a hardware structure of a device in a mobile platform according to an embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described The embodiments are only some of the embodiments of the present application, but not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in this application shall fall within the protection scope of this application.

It should be noted that in other embodiments, the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or less steps than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step in other embodiments describe.

In order to control the mobile platform to move safely in space, the mobile platform can observe the information of the scene in the space through its own sensors, these sensors include laser radar, millimeter wave radar, visual sensor, infrared sensor or TOF ( Time of flight, flight time) sensors and so on.

Taking the vision sensor as an example, in some scenarios, a binocular vision system can be used. The principle of the binocular vision system uses two cameras to form a binocular, based on the parallax principle and using imaging equipment to obtain two images of the measured object from different positions, and obtain the three-dimensional geometric information of the object by calculating the position deviation between the corresponding points of the image . That is to use two cameras to form binoculars to complete the depth information perception of the scene in a certain direction, so as to ensure the obstacle avoidance function of the movable platform through the perceived depth information of the scene, so that the movable platform can move safely. Based on the limitation of the field of view of ordinary cameras, when using a binocular vision system to obtain depth information in four directions, two binocular cameras are required for each direction, that is, two cameras are configured in each direction, and the The field of view overlaps, and the binocular cameras in each direction are independently controlled. Therefore, the movable platform in the related art usually uses at least eight independently controllable cameras.

In some scenarios, mobile platforms such as drones or automatic cleaning equipment have requirements for miniaturization and low cost. Therefore, the inventors of the present application hope to meet this requirement by reducing the number of visual sensors. Based on this, in the solution of this embodiment, a movable platform is designed to be equipped with a sensor with a large field of view. One sensor can overlap the field of view of at least two sensors, that is, one sensor can take care of at least two directions. Therefore, one sensor can form a binocular vision system with at least two sensors, thereby reducing the number of vision sensors on the movable platform.

As shown in FIG. 1A , it is a schematic diagram of a sensor mounted on a mobile platform according to an exemplary embodiment of the present application. In the embodiment shown in Fig. 1A, as an example, the platform body of the movable platform is a rectangle, and four sensors are mounted on the four corners of the rectangle, and the field angles of each sensor are 180°, and one The sensor is a binocular vision system with two sensors respectively. It can be understood that in practical applications, the configuration of the movable platform has various forms, and the field of view, quantity, and mounting position of the sensors carried by it can be implemented in various ways, and one sensor can also be composed of two or more sensors. Multiple binocular vision systems; for example, can be comprehensively designed according to factors such as the configuration of the movable platform, the field of view of the sensor, and the direction that the movable platform needs to observe, which is not limited in this embodiment.

As shown in FIG. 1A , the four sensors mounted on the movable platform are sensor C, sensor D, sensor E and sensor F. Taking the sensor C as an example, the sensor C has an angle of view of 180°, and its field of view includes the field of view jointly formed by the area C1 and the area C2. Similarly, the fields of view of the other three sensors are shown in FIG. 1A .

The sensor C and the sensor D form a binocular vision system. The area where the area C1 in the field of view of the sensor C intersects with the area D1 in the field of view of the sensor D, that is, the field of view where the two overlap, is illustrated by a diagonal line in FIG. 1A .

The sensor C and the sensor F form a binocular vision system. The area where the area C2 in the field of view of the sensor C intersects with the area F2 in the field of view of the sensor F, that is, the field of view where the two overlap, is illustrated by a diagonal line in FIG. 1A .

It can be seen that the sensor C can form a binocular vision system with the sensor D and the sensor F respectively; that is, in the image collected by the sensor C and the image collected by the sensor D, the parts facing the same direction respectively can be used for binocular vision processing; Parts of the image collected by the sensor C and the image collected by the sensor F facing the same direction can be used for binocular vision processing. That is to say, the sensor C can take into account both directions, so the image obtained by the sensor can be divided into two parts (the specific division method can be determined according to the needs, for example, the example shown in FIG. 1A can be divided into left and right parts equally) where One part forms a binocular image with a part of the image collected by the sensor D, and the other part forms a binocular image with a part of the image collected by the sensor F.

In this way, by designing a sensor with a larger field of view mounted on the movable platform, one sensor and at least two sensors respectively have overlapping fields of view, so the number and cost of sensors mounted on the movable platform can be significantly reduced.

Although the above design saves the cost and power consumption of the mobile platform, there are still other technical problems to be solved. Specifically, one sensor forms binocular vision with at least two sensors, that is, an image output by the sensor can be processed with images collected by other sensors for binocular vision, and how the image is collected will affect the subsequent binocular vision. Vision processing, which in turn affects the safe movement of the movable platform.

For example, the sensor needs to consider how to set appropriate exposure parameters when capturing images. In the related art, an automatic exposure (Auto Exposure, AE) algorithm is used to automatically calculate and adjust exposure parameters, so that the midtone of the scene to be photographed is as close as possible to the midtone of the obtained image. If the exposure of the sensor is not appropriate, the collected image will be overexposed or underexposed, and the overexposure or underexposure of the image will lead to the loss of image information; and the movable platform needs to use the image collected by the sensor to obtain the image of the scene in the field of view. The loss of depth information and image information will cause the movable platform to fail to obtain rich depth information, which may lead to the failure of the movable platform to successfully avoid obstacles.

Generally speaking, exposure parameters include ideal exposure time, exposure time, analog gain and digital gain, and the relationship between the above parameters is shown in the following formula:

icit＝cit*a_gain*d_gain (1)

Among them, icit refers to the ideal exposure time, cit refers to the exposure time, a_gain refers to the analog gain, and d_gain refers to the digital gain.

In some scenes, the sensor can determine the appropriate exposure parameters by sensing the brightness information of the environment in the field of view and using the above formula. Therefore, the sensor can accurately perceive the brightness information of the environment in the field of view, which will affect the accurate determination of the exposure parameters.

After the light in the environment is reflected by the scenery in the environment, it enters the photosensitive element in the sensor through the lens of the sensor. The photosensitive element perceives the light and forms an image. The brightness information in the environment can be determined through the brightness information of the image. Light intensity represents brightness. Different photosensitive elements have different perception abilities to light intensity. The perception ability of photosensitive elements to light intensity is the brightness dynamic range of the sensor. scope. In some scenes, the luminance dynamic range of the actual environment where the movable platform is located may be very large, while the luminance dynamic range of the sensor is always limited, and it is impossible to capture the brightest and darkest objects in nature at the same time; and, Due to cost considerations, if it is desired to mount a sensor with a small dynamic range of brightness on a mobile platform, it will face many problems in practical applications.

For example, due to the large field of view of the sensor, the sensor can collect a wide range of images; therefore, in some scenarios, in the range observed by the sensor, the environmental information in different areas may change greatly, for example, the light intensity The dynamic range is larger. Still taking FIG. 1A as an example, assume that the distance between sensor C and sensor D on the movable platform faces strong light, such as facing the sun; the distance between sensor C and sensor F faces the woods, and the ambient brightness of the woods is obviously low. Therefore, controlling which exposure parameters the sensor C uses to collect images will affect the quality of the collected images and the subsequent binocular vision processing.

The field of view of the sensor carried on the movable platform in the related art is relatively small, and one sensor only forms a binocular vision system with one sensor, and at least eight cameras are carried on the movable platform; therefore, in the related art, due to the small field of view of the sensor , basically will not face the problem of large changes in the environmental information in the field of view due to the large field of view.

In addition, due to the small field of view of the sensor and the small change in environmental information within the field of view, the automatic exposure algorithm usually adopts the idea of global average exposure processing. The input of the algorithm is usually the average brightness information of the image, that is, the image collected by the sensor The image information is used to calculate the average brightness information, which is estimated or used as the average brightness information of the environment within the field of view, and the exposure parameter is determined by using the average brightness information of the environment within the field of view. In the above example scene, there is both strong light and low brightness within the field of view of sensor C. If the exposure processing idea based on global average is adopted, the average brightness information is calculated from the collected images to determine the exposure parameters. This will lead to the problem that the quality of the collected image is poor, and the image may fail to match both the strong light environment in the field of view and the low brightness environment in the field of view. However, in the image collected by sensor C, it is necessary to segment a part of the image collected by sensor D for binocular vision processing. The segmented part of the image does not match the brightness of the actual environment, resulting in underexposure or overexposure of the image, so it will not be possible Obtain reliable scene information. In the same way, in the image collected by sensor C, another part needs to be segmented and processed with the image collected by sensor F for binocular vision. The key to the safe movement of the platform will eventually lead to greater security risks for the mobile platform during the movement.

Another solution can be to optimize the quality of the sensor and increase the dynamic range of the ambient brightness that the sensor can perceive, but currently the price of sensors that can perceive a larger dynamic range is relatively high, so this solution will lead to an increase in the manufacturing cost of the mobile platform. Increase.

Therefore, an embodiment of the present application proposes a mobile platform control solution to solve the above problems. The mobile platform in this embodiment may refer to any mobile device. Wherein, the movable platform may include but not limited to land vehicles, water vehicles, air vehicles and other types of motor vehicles. As an example, the movable platform may include a passenger vehicle and/or an unmanned aerial vehicle (Unmanned Aerial Vehicle, UAV), etc., and the movement of the movable platform may include flying.

The movable platform is equipped with a sensor; the sensor in this embodiment refers to a visual sensor that can collect images. The visual sensor has a certain field of view and can overlap with the field of view of at least two sensors. As an example, it can be a wide-angle camera Or a fisheye camera etc. In practical applications, the number of sensors mounted on the movable platform may not be limited to three; moreover, one sensor may have overlapping fields of view with at least two sensors; in practical applications, it may be configured as required.

In the subsequent embodiments, three sensors are taken as an example to illustrate the specific process of applying the mobile platform control method according to the embodiment of the present application when one sensor has overlapping fields of view with two sensors respectively, and the overlapping fields of view face different directions. It can be understood that in the case where the movable platform is equipped with more than three sensors, as long as there are three of them, the design of the three sensors conforms to the principle of "one sensor has overlapping fields of view with two sensors respectively, and the overlapping fields of view face different directions" in this embodiment. feature, the solution provided in this embodiment can be applied. For example, in the embodiment shown in Figure 1A, there are four sensors mounted on the movable platform, and any three of the four sensors conform to the above-mentioned design, so the scheme of this embodiment can be applied to any sensor for exposure processing to collect images .

In other scenarios, a sensor not only has overlapping fields of view with two sensors, but also has overlapping fields of view with multiple sensors. Among the multiple overlapping fields of view, two of them may face different directions, or both may face different directions; it is understandable , in this case, the solution provided by this embodiment can still be applied.

As an example, the movable platform includes a first sensor, a second sensor and a third sensor. Still taking the example shown in FIG. 1A, the first sensor, the second sensor and the third sensor can be sensor D, sensor F and sensor C respectively. .

The fields of view of the first sensor and the third sensor overlap, that is, the intersection area of C1 and D1 shown in FIG. 1A , that is, the fields of view of the first sensor and the third sensor overlap. The first sensor and the third sensor have a first partial field of view that overlaps. In some examples, the first partial field of view may refer to the field of view that the first sensor overlaps with the third sensor; in other examples , the first partial field of view may also refer to the part of the field of view that the first sensor overlaps with the third sensor; A partial field of view may select a part of the field of view where the first sensor overlaps with the third sensor, for example, remove the central area of the edge, etc., thereby reducing the influence of image distortion. As shown in FIG. 1B, it is the field of view of the sensor C and the field of view of the sensor D shown in this embodiment, wherein the field of view of the sensor C and the field of view of the sensor D have overlapped fields of view, and in the field of view that both overlap, There is a partial field of view, that is, the first sensor and the third sensor in this embodiment have overlapping first partial fields of view, and the first partial field of view is used to observe the scene in the first direction of the movable platform.

Similarly, the field of view of the second sensor overlaps with that of the third sensor, that is, the intersection area of F2 and C2 shown in FIG. 1A , that is, the field of view of the second sensor overlaps with the third sensor. The second sensor and the third sensor have a second partial field of view that overlaps. The second partial field of view may refer to the field of view overlapped by the second sensor and the third sensor, or may refer to the field of view of the second sensor and the third sensor. A part of the overlapping field of view of the third sensor, the second partial field of view is used to observe the scene in the second direction of the movable platform. Wherein, the first direction and the second direction in this embodiment adopt a coordinate system (body system) fixed on the movable platform, such as a three-dimensional orthogonal rectangular coordinate system fixed on the movable platform following the right-hand rule, where The origin is at the movable platform or at the center of mass on the movable platform.

As an example, the field of view ranges of the first partial field of view and the second partial field of view may be different, and there may be no overlapped range between the two, of course, there may also be a partially overlapped range.

Through the above-mentioned design of the sensors carried on the movable platform, correspondingly, as shown in FIG. 2 , it is a flowchart of a control method for a movable platform according to an exemplary embodiment of the present application, and the method may include the following steps:

Step S201: Obtain first brightness information of the first partial field of view and second brightness information of the second partial field of view of the third sensor.

Step S202: Determine a first exposure parameter of the third sensor based at least on the first brightness information; control the third sensor to capture a first image under the first exposure parameter; based on the first image and the The image collected by the first sensor is used to obtain the depth information of the scene in the first direction.

Step S203: Determine a second exposure parameter of the third sensor based at least on the second brightness information; control the third sensor to acquire a second image under the second exposure parameter; based on the second image and the The image collected by the second sensor is used to obtain the depth information of the scene in the second direction.

Step S204: Control the movable platform to move in space according to the depth information.

In this embodiment, the third sensor forms a binocular vision system with the first sensor and the second sensor respectively, and the first partial field of view and the second partial field of view respectively observe different first directions and second directions. The first partial field of view, acquire the first brightness information of the first partial field of view, and determine the first exposure parameter based on this, so the first image collected by the third sensor under the first exposure parameter can be compared with the brightness of the first partial field of view Matching, the quality of the first image is relatively high, so that the depth information of the scene in the first direction can be obtained based on the first image and the image collected by the first sensor. Similarly, since the second brightness information of the second partial field of view can be obtained for the second partial field of view alone, and the second exposure parameter can be determined based on this, the second image collected by the third sensor under the second exposure parameter can be Matching the brightness of the second partial field of view, the quality of the second image is relatively high, so that the depth information of the scene in the second direction can be obtained based on the second image and the image collected by the second sensor. Furthermore, since the movable platform can obtain reliable depth information of the scene in the first direction and depth information of the scene in the second direction based on the above processing, it can safely control itself to move in space.

In practical applications, there are many ways to obtain brightness information. As an example, one or more images can be collected by a sensor, and the brightness information of the environment can be estimated according to the brightness information of the images; the first partial field of view in this embodiment For the first brightness information, since the field of view of the first sensor and the third sensor are covered, the area corresponding to the first partial field of view can be determined from the image collected by the first sensor and/or the third sensor, and by counting the area The image brightness information of the image is used to determine the first brightness information of the first partial field of view. Similarly, the second luminance information of the second partial field of view in this embodiment can be determined from the images collected by the second sensor and/or the third sensor because the field of view ranges of the second sensor and the third sensor are covered. For the region corresponding to the second partial field of view, the second brightness information of the second partial field of view is determined by counting the image brightness information of the region.

In some examples, the determining the first exposure parameter of the third sensor based at least on the first brightness information may be determining the first exposure parameter of the third sensor based only on the first brightness information, or may be combined with other information, such as other environmental parameters in the first partial field of view, or environmental parameters in other areas within the field of vision other than the first partial field of view, or data collected by other sensors on the movable platform, or data that can be Flight status information of the mobile platform, etc. Similarly, the determining of the second exposure parameter of the third sensor based at least on the second brightness information may be to determine the second exposure parameter of the third sensor based only on the second brightness information, or may be combined with other information as required To determine, for example, other environmental parameters in the second partial field of view or environmental parameters in other areas within the field of vision other than the second partial field of view, or the data collected by other sensors on the movable platform, or the flight of the movable platform status information and more.

In practical applications, step S202 and step S203 are not necessarily executed in the order shown and described in the embodiment shown in FIG. 2 . In addition, in the process of controlling the movement of the movable platform in space, in some time periods, the movable platform can only perform step S202 and then control its own movement; in other time periods, the movable platform can only perform step S203 and then control move by itself; or, the movable platform can control itself to move after steps S202 and S203 are executed.

In practical applications, the mobile platform can be configured with multiple schemes for setting the exposure parameters of the sensor, such as the aforementioned global average exposure scheme, and the scheme in this embodiment can be used as one of them. Therefore, the execution timing of the scheme in this embodiment Can be determined according to needs. For example, when the movable platform is in a scene with a small dynamic range of ambient brightness, the movable platform can adopt other schemes for setting exposure parameters; when the movable platform recognizes that it is in a scene with a large dynamic range of ambient brightness, it can trigger execution The scheme of this embodiment. For example, the movable platform compares the image information of different areas in the image collected by the sensor, and determines that the brightness information of different areas is quite different, such as the first brightness information of the first partial field of view and the second The second luminance information difference of the local field of view satisfies the set condition, which can trigger the execution of the solution of this embodiment; wherein, the set condition refers to a condition representing a large difference in image luminance information, which can be set as required, for example, both The brightness value is greater than the set brightness threshold and so on.

It can be seen from the above embodiments that the solution of this embodiment involves the exposure control of the third sensor. This solution does not adopt the idea of global average exposure, but determines the exposure parameters based on the brightness information of the local field of view. Therefore, this embodiment does not Affected by the brightness of other partial fields of view, it can ensure that the image information of the scene in the partial field of view is collected. In addition, since the sensor can be controlled to collect images at the first exposure parameter and the second exposure parameter respectively, the solution of this embodiment can also significantly reduce the influence of the large dynamic range of light intensity in different areas of the field of view, and affect the perception of the large dynamic range by the sensor. The performance requirements are not high, so that the mobile platform can be equipped with lower-cost sensors, and there is no need to choose hardware with a larger dynamic range, so that the image quality collected by the sensor can be guaranteed without increasing the hardware cost.

It can be seen from the above embodiments that the movable platform determines two sets of exposure parameters of the sensor respectively according to the first brightness information of the first partial field of view and the second brightness information of the second partial field of view to collect different images. The above-mentioned embodiment does not limit the timing of determining the exposure parameters, the timing of controlling the sensor to collect images, the timing of controlling the movement of the movable platform based on the collected images, and the execution timing of each step; it can be understood that in practical applications, Design according to needs, for example, considering one or more factors such as the type of the movable platform, the way of moving the movable platform, and the application scenarios faced. These possible different treatments are all within the scope of the solution of this application. Next, through different scenarios and requirements, two embodiments are provided to illustrate the control scheme of the mobile platform of the present application.

In some examples, considering that the main requirement of the movable platform is to move in a certain direction in space, based on this, an embodiment considering the moving direction of the movable platform is provided. As an example, the image quality of the scene in the moving direction of the movable platform can be given priority, and the exposure parameters of the movable platform can be set based on the scenery in the moving direction of the movable platform, so as to give priority to the obstacle avoidance of the movable platform in the moving direction Function.

As an example, when the movable platform moves forward, the image quality of the forward field of view is the most important, which determines whether the functions such as obstacle avoidance and circumvention of the movable platform can be realized normally; when moving to the left, the image quality of the left field of view Image quality is the most important; when moving toward the upper left 45 degrees, it is necessary to take into account both left and forward vision directions; direction. The same goes for other moving directions of the movable platform, which will not be repeated here.

Therefore, when determining the exposure parameter, it may be determined based on at least the brightness information and the moving direction.

As an example, taking the determination of the first exposure parameter as an example, as shown in FIG. 3 , it is a flow chart of determining the first exposure by the mobile platform according to an embodiment shown in this embodiment, which may include the following steps:

Step S301: Obtain the moving direction of the movable platform.

Step S302: Determine a first exposure parameter of the third sensor based at least on the first brightness information and the moving direction of the movable platform.

Among them, the moving direction of the movable platform can be obtained in various ways. As an example, the movable platform can be equipped with a variety of sensors, such as inertial sensors such as accelerometers and angular velocity meters. direction of movement.

Wherein, how to determine the first exposure parameter based on the first brightness information and the moving direction can be implemented in various ways according to actual needs, which is not limited in this embodiment. For example, the first brightness information corresponds to the first direction, and the weight of the first brightness information when determining the first exposure parameter can be determined according to the relationship between the moving direction and the first direction. Of course, the specific weight can be flexibly selected according to needs.

For example, the weight of the first brightness information may be determined according to the relative orientation relationship between the moving direction of the movable platform and the first direction of the third sensor, wherein the weight reflects the first brightness information for the current moving direction of the movable platform. The importance of brightness information in the field of view in the first direction when the three sensors collect images.

As an example, the relative orientation relationship between the moving direction of the movable platform and the first direction of the third sensor can be divided into two situations: coincident with the first direction and deviation from the first direction. In this embodiment, priority is given to the moving direction of the movable platform. The closer the first direction and the second direction are to the moving direction of the movable platform, the higher the importance of the scenery in this direction is, and it can be used when collecting images. Prioritize image quality in that direction. For example, if the moving direction of the movable platform coincides with the first direction, the first exposure parameter of the third sensor may be determined only based on the first brightness information, that is, other directions within the field of view of the third sensor may not be considered. In another case, if the moving direction of the movable platform deviates from the first direction, that is, it does not completely fall into the first direction, the first exposure parameter can be determined by considering the second brightness information in the second direction; When the direction deviates from the first direction, relative to the second direction, the moving direction of the movable platform is biased towards the first direction, therefore, the weight of the first brightness information of the third sensor is greater than the weight of the second brightness information, based on which it can be determined The first exposure parameter is obtained.

Since the first exposure parameter takes into account the moving direction of the movable platform, the image quality of the scene in the moving direction of the movable platform can be given priority, and the exposure parameters of the movable platform are set based on the brightness information in the moving direction, and the moving The rich image information in the direction enables the movable platform to obtain the depth information of the scene in the moving direction, so as to give priority to the obstacle avoidance function of the movable platform in the moving direction, making the movement of the movable platform safer.

The foregoing embodiment is described by taking the determination of the first exposure parameter as an example, and the same is true for the determination process of the second exposure parameter during specific implementation. For example, the moving direction of the movable platform may be obtained, and the second exposure parameter of the third sensor may be determined based at least on the second brightness information and the moving direction of the movable platform; in some examples, if the moving direction falls within the In a second direction, determining a second exposure parameter of the third sensor based on the second brightness information; if the moving direction is biased toward the second direction relative to the first direction, based on the first brightness information and the second brightness information to determine the first exposure parameter; wherein, the weight of the second brightness information is greater than the weight of the first brightness information.

The above embodiment gives priority to the images of the scene in the moving direction of the movable platform, reduces the influence of ambient brightness information in other non-moving directions, and solves the problem that the field of view of the sensors is coupled to each other when the dynamic range of parameters in the environment is large. The problem that the mobile platform cannot work normally; there is no need to use a sensor that can perceive a large dynamic range, and the cost is low; at the same time, the image quality of the scene collected by the sensor is guaranteed, and the image frame rate perceived by the sensor will not be reduced.

Next, some implementation processes of determining exposure parameters are described through some embodiments.

In this embodiment, taking the movable platform as an unmanned aerial vehicle as an example, theta (θ) represents the angle of the moving direction of the unmanned aerial vehicle, wherein, theta=0° represents forward flight; theta=90° represents leftward flight; theta=-90° indicates rightward flight; theta=180° (or -180°) indicates backward flight.

In this embodiment, the image collected by the sensor corresponds to the first direction and the second direction, that is, the collected image can be divided into two parts corresponding to different directions. In this embodiment, the weight sum of the two parts in the image is 1 as an example Be explained. When the movable platform moves forward or backward, its moving angle is 0° or 180° (-180°), the weight of the image information of area 1 obtained by the third sensor is 1, and the weight of image information of area 2 is is 0. Similarly, when the movable platform moves left or right, the weight of the image information in sensor area 1 is 0, and the weight of image information in area 2 is 1. For example, when the movable platform moves forward, taking sensor C as an example, the weight of area 1 corresponding to the forward direction in the collected image is 1, and the weight of area 2 corresponding to the left direction in the collected image is 0, that is, the first brightness information is determined only based on the part of the area 1 corresponding to the forward direction in the image. If the movable platform moves to the left, similarly, the second brightness information is determined only based on the part of the region 2 corresponding to the left direction in the image.

If the movable platform moves toward the left front, it may be determined based on the moving direction, based on the first brightness information and the second brightness information. In this embodiment, for the convenience of calculation, the weights of the images of the two regions of the third sensor of the movable platform may have a set function relationship with the moving angle of the movable platform. For example, the calculation method of the weight w _{area_1} of the image of the third sensor area 1 is shown in formula (2):

w _{area_1} = cos ² (theta) (2)

Wherein, w _{area_1} refers to the weight of the image of the third sensor area 1, and theta refers to the moving angle of the movable platform.

The calculation method of the weight w _{area_2} of the image in area 2 is shown in formula (3):

w _{area_2} = sin ² (theta) (3)

Wherein, w _{area_2} refers to the weight of the image of the third sensor area 2, and theta refers to the moving angle of the movable platform.

Through the above calculation, the weight of the first brightness information and the weight of the second brightness information can be obtained, and then the average brightness information of the image of the third sensor can be obtained through weighted calculation, and the automatic exposure algorithm can be executed to determine the first exposure of the third sensor parameter.

Optionally, the first exposure parameters include exposure time and analog gain, and the exposure time and analog gain of the third sensor are calculated by using an automatic exposure algorithm according to the calculated average brightness information of the image.

After the sensor of the movable platform acquires the image, the image can be input into the perception algorithm of the subsequent stage. In some examples, some perception algorithms that use binocular vision images as input have certain requirements on binocular images, such as semi-global matching and visual odometer, etc., have certain requirements on binocular images, such as the brightness of the left and right images cannot The difference is too much or the exposure time of the left and right images cannot be too different, etc. However, since only one luminance information of two adjacent sensors is the same, its weight and the other luminance information and its weight may be very different, which will cause the exposure parameters of the two adjacent sensors to be different, resulting in the left and right images. The difference in brightness cannot be too much or the exposure time of the left and right images cannot be too different.

Therefore, after the first exposure parameter of the third sensor is determined in the scheme of this embodiment, the first exposure parameter of the third sensor can also be adjusted based on the first exposure parameter of the first sensor. For example, the adjustment method can be determined based on the requirements of the perception algorithm for binocular images, and the specific requirements can be determined according to the actual perception algorithm used. Alternatively, a difference threshold can also be preset as required, and if the difference between the first exposure parameter of the first sensor and the first exposure parameter of the third sensor exceeds the difference threshold, the first exposure parameter of the third sensor is adjusted, Make the difference between the first exposure parameter of the first sensor and the first exposure parameter of the third sensor less than or equal to the difference threshold, wherein the difference threshold can be flexibly set according to needs, for example, it can be an empirical value.

In some examples, the adjustment method may include: reducing the difference between the first exposure parameter of the first sensor and the first exposure parameter of the third sensor, so as to ensure that the difference between the image captured by the first sensor and the image captured by the third sensor is Satisfy preset brightness conditions and/or preset image content conditions to meet the requirements of post-stage perception algorithms.

In some embodiments, the value range of the exposure parameter of the third sensor can be determined according to the first exposure parameter of the first sensor and the first exposure parameter of the third sensor, and the value range of the exposure parameter of the third sensor can be controlled within the range of the exposure parameter. value range. The exposure parameters of two adjacent sensors are controlled within the value range of the exposure parameter in the flying direction of the movable platform, and the difference between the two is controlled to be smaller than the value range of the exposure parameter in the flying direction of the movable platform. Ensure that the brightness of the left and right images is similar or the exposure time of the left and right images is similar.

Based on the determined value range of the exposure parameter of the third sensor, the manner of limiting the first exposure parameter of the third sensor may be that when the first exposure parameter of the third sensor is within the value range of the exposure parameter, the third sensor’s The first exposure parameter remains unchanged; or, when the first exposure parameter of the third sensor exceeds the value range of the exposure parameter, the first exposure parameter of the third sensor is adjusted to a boundary value of the value range of the exposure parameter. Adjusting the first exposure parameter of the third sensor to be the boundary value of the value range of the exposure parameter includes two types: If the first exposure parameter of the third sensor is greater than the maximum value of the value range of the exposure parameter, set the first exposure parameter of the third sensor is the maximum value of the value range of the exposure parameter; if the first exposure parameter of the third sensor is smaller than the minimum value of the value range of the exposure parameter, set the first exposure parameter of the third sensor to the minimum value of the value range of the exposure parameter.

In some embodiments, the value range of the exposure parameter is jointly determined based on the first exposure parameter of the first sensor and its weight, and the first exposure parameter of the third sensor and its weight. Wherein, because the scene information in the moving direction is more important, if the moving direction of the movable platform is biased towards the first direction relative to the second direction, the scene in the first direction will have a greater influence on the scene in the moving direction than The scene in the second direction, so the weight of the first exposure parameter of the third sensor can be controlled to be greater than the weight of the first exposure parameter of the first sensor. Through the above adjustment scheme, based on the obtained exposure parameters, the captured image controls the movement of the movable platform. Mobile is safer.

There are many ways to determine the value range of the exposure parameter. In some embodiments, in order to determine the exposure parameter more quickly, in consideration of the moving direction, a parameter can be defined first, which represents the moving direction of the movable platform. The exposure parameter is a virtual exposure parameter, not an actual physical parameter, which can be obtained by weighting the exposure parameters of two adjacent sensors determined based on the moving direction. Therefore, the boundary value of the value range of the exposure parameter is a multiple of the exposure parameter in the moving direction of the movable platform. Wherein, the multiple of the exposure parameter in the moving direction of the movable platform may be a value determined according to the actual use scene and experience of the movable platform.

An optional specific embodiment is provided below. In this optional embodiment, take a movable platform with four sensors as an example, and the four sensors are respectively located on the front left, front right, rear right and rear left. When the movable platform is moving forward, the exposure parameters in the moving direction of the movable platform are obtained by weighting the exposure parameters of the front left and front right sensors; when the movable platform is flying 30 degrees to the left, the clockwise direction is preferred , the two sensors determined according to the direction of movement are front right and front left respectively; when the movable platform flies forward and left 45 degrees, the clockwise direction takes priority, and the two sensors determined according to the direction of movement are front right and rear left . The method for determining the sensor is the same when the moving direction of the movable platform is other directions, and details are not described here. It can be clearly obtained that no matter which direction the movable platform moves, two adjacent sensors on the left and right can be determined according to the moving direction.

Optionally, the weights of the exposure parameters of the left and right adjacent sensors can be determined according to the moving direction of the movable platform. An optional embodiment is provided below. In this optional embodiment, the weights of the exposure parameters of the two sensors and the moving angle of the movable platform may have a set functional relationship. For example, if the moving angle of the movable platform satisfies: 45°<|theta|≤135°, the calculation methods of the weights w _left and w _right of the exposure parameters of two adjacent sensors can be as shown in formula (4) and formula (5) Show:

w _left = sin ² (theta-45) (4)

w _right = cos ² (theta-45) (5)

Among them, w _left refers to the weight of the exposure parameters of the sensors adjacent to the left of the moving direction of the movable platform, w _right refers to the weight of the exposure parameters of the sensors adjacent to the right of the moving direction of the movable platform, and theta refers to is the movement angle of the movable platform.

If the moving angle of the movable platform does not satisfy: 45°<|theta|≤135°, the calculation methods of the weights w _left and w _right of the exposure parameters of two adjacent sensors are shown in formula (6) and formula (7):

w _left = cos ² (theta-45) (6)

w _right = sin ² (theta-45) (7)

Among the two sensors, the closer to the moving direction of the movable platform, the greater the weight of the exposure parameter. Applying the above weight calculation method, when the moving angle theta=0, the left and right adjacent sensors in the moving direction of the movable platform are the front left and the front right. At this time, the weights of the exposure parameters of the two sensors are both 0.5; When the moving angle theta=46, the left and right adjacent sensors in the moving direction of the movable platform are rear left and front left, and their weights are 0.0003 and 0.9997 respectively.

In some embodiments, exposure parameters include ideal exposure time and exposure time. If the determined ideal exposure time and exposure time of the third sensor are not within the value range of the exposure parameter, adjust the ideal exposure time and exposure time of the third sensor to be a boundary value of the value range of the exposure parameter.

Optionally, the exposure parameters also include analog gain and digital gain. Based on the adjusted exposure time and the ideal exposure time, the analog gain and the digital gain of the third sensor are adjusted.

In this embodiment, the exposure parameters are determined by taking into account the moving direction of the movable platform, and the scene where the movable platform may be in is facing a large change during the moving process. Based on this, this embodiment can also obtain the The change of the moving speed, as an example, the moving speed can be obtained by an inertial measurement unit mounted on a movable platform. In addition, the change of the moving speed is positively correlated with any of the following: the acquisition frequency of the moving direction of the movable platform, the update frequency of the exposure parameter or the update frequency of the weight of the brightness information, that is, the movement of the movable platform The faster the speed changes, the correspondingly increase the acquisition frequency of the moving direction of the movable platform, the update frequency of the exposure parameters or the update frequency of the weight of the brightness information, so that the movable platform can update the exposure parameters faster to adapt to the movable platform It may be possible to face rapid changes in the scene under high-speed movement, so as to provide safe control of the movable platform. It can be understood that various motion parameters of the movable platform can be determined based on any one of the following coordinate systems: sensor coordinate system (sensor system), body coordinate system (body system), local coordinate system (local system), global coordinate system (global system) and so on. Certainly, there are many methods for obtaining the moving speed and moving direction of the movable platform, and the moving platform may also obtain the moving speed and moving direction of the movable platform based on other coordinate systems and methods, which are not limited in this application.

Next, an embodiment is used to illustrate the overall process of the above-mentioned method steps:

Take a UAV with four fisheye cameras installed as an example, as shown in Figure 1A. The symbols are labeled: C, D, E and F. At the same time, each fisheye camera is divided into left and right parts from the middle, and the part located in the front and rear directions of the drone is named area 1, and the part located in the left and right directions of the drone is named area 2. The drone looks around in four directions and is divided into eight fields of vision, which are denoted by symbols: C1 and C2, D1 and D2, E1 and E2, and F1 and F2.

Obviously, C1 and D1 form the front-view binocular of the drone, D2 and E2 form the right-view binocular of the drone, E1 and F1 form the rear-view binocular of the drone, and F2 and C2 form the left-view binocular of the drone. As binocular. Meanwhile, the exposure times of C1 and C2, D1 and D2, E1 and E2, F1 and F2 are the same.

Firstly, it shows the relationship between the flight angle (theta) and the flying direction of the drone. 0° means the drone is flying forward, 90° means the drone is flying to the left, and 180° (or -180°) means no one. The drone is flying forward, and -90° means the drone is flying forward. The unit of the flying speed of the UAV is m/s.

The ideal exposure time of the fisheye camera of the drone is recorded as icit, the exposure time is recorded as cit, the analog gain is recorded as a_gain, the brightness of the first area is recorded as Lum _{area_1} , and the brightness of the second area is recorded as Lum _{area_2} . Use icit _c to represent the ideal exposure time of fisheye camera C, use cit _c to represent the exposure time of fisheye camera C, and the exposure parameters of other fisheye cameras are the same.

The flow of the above-mentioned method steps is described below through a fisheye camera on the above-mentioned drone:

Taking fisheye camera C as an example, it is obvious that the exposure parameters of C1 and C2 are the same. When there is a large difference in the parameters of the scene in the field of view captured by C1 and C2, the dynamics that exceed the perception of fisheye camera C Range, at least one of the two images is overexposed or underexposed. In this case, the fisheye camera cannot take into account the quality of the two images.

First, the current flight speed and flight direction of the UAV based on the body coordinate system (body system) can be calculated through the information of the inertial measurement unit of the UAV. Wherein, the body coordinate system (body system) refers to a three-dimensional orthogonal rectangular coordinate system fixed on the aircraft or aircraft following the right-hand rule, and its origin is located at the center of mass of the aircraft or aircraft.

If the flying speed of the UAV is lower than the set threshold, the flying direction of the UAV remains the same as the previous frame and will not be updated. Otherwise, update the flight direction of the drone.

Then, determine the flight angle of the drone according to the flight direction of the drone, and calculate the weight of the brightness of the two images obtained by the fisheye camera C. The calculation method is as follows:

The weight calculation method of the image in area 1 is shown in formula (8):

w _{area_1} = cos ² (theta) (8)

Among them, w _{area_1} refers to the weight of the image of area 1 (ie C1) of the fisheye camera C, and theta refers to the flight angle of the drone.

The weight calculation method of the image in area 2 is shown in formula (9):

w _{area_2} = sin ² (theta) (9)

Among them, w _{area_2} refers to the weight of the image of area 2 (ie C2) of the fisheye camera C, and theta refers to the flight angle of the drone.

Then, calculate the average brightness of the image of fisheye camera C according to the brightness and weight of the two images of fisheye camera C above. The average brightness of the left and right images of the fisheye camera C is weighted to obtain the average brightness Lum _avg of the input of the final automatic exposure algorithm, as shown in formula (10):

Lum _avg ＝w _{area_1} *Lum _{area_1} +w _{area_2} *Lum _{area_2} (10)

Among them, Lum _avg refers to the average brightness of fisheye camera C, w _{area_1} refers to the weight of the image of area 1C1 of fisheye camera C, Lum _{area_1} refers to the brightness of the image of area 1C1 of fisheye camera C, w _{area_2} refers to the weight of the image of the area 2C2 of the fisheye camera C, and Lum _{area_2} refers to the brightness of the image of the area 2C2 of the fisheye camera C.

After obtaining the average brightness of the image of the fisheye camera C, execute the automatic exposure algorithm to calculate the exposure time and analog gain of the fisheye camera C. The exposure time and analog gain calculation methods of other sensors are the same, and will not be repeated here.

Through the above calculation process, the exposure time and simulation gain of the four fisheye cameras on the UAV are obtained. Then, the exposure time and analog gain calculated above are further restricted. First, determine the two fisheye cameras needed to calculate the exposure parameters in the flight direction of the drone according to the flight direction of the drone. For example, when the angle theta=0, the two adjacent cameras in the flight direction of the drone are C and D; when theta=46, the two adjacent cameras in the flight direction of the drone are F and C.

After the two fisheye cameras are determined, the weights of the exposure parameters of the two fisheye cameras are determined according to the flying direction of the drone. Let w _left represent the weight of the exposure parameters of the fisheye camera adjacent to the left side of the UAV's flight direction; w _right represents the weight of the exposure parameters of the fisheye camera adjacent to the right side of the UAV's flight direction. Its calculation method is:

If the flight angle of the UAV satisfies: 45°<|theta|≤135°, the calculation method of the weights w _left and w _right of the exposure parameters of the two adjacent fisheye cameras on the left and right of the UAV flight direction is as shown in formula (11) and formula (12):

w _left = sin ² (theta-45) (11)

w _right = cos ² (theta-45) (12)

Among them, w _left refers to the weight of the exposure parameters of the fisheye camera adjacent to the left of the flight direction of the drone, and w _right refers to the weight of the exposure parameters of the fisheye camera adjacent to the right of the flight direction of the drone. Weight, theta refers to the flying angle of the drone.

If the flight angle of the UAV does not satisfy: 45°<|theta|≤135°, the calculation method of the weight w _left and w _right of the exposure parameters of two adjacent fisheye cameras on the left and right of the UAV flight direction is as follows: formula (13 ) and formula (14):

w _left = cos ² (theta-45) (13)

w _right = sin ² (theta-45) (14)

According to the exposure parameters and weights of the two adjacent fisheye cameras on the left and right of the flight direction of the UAV determined by the above calculation method, the exposure parameters in the flight direction of the UAV are calculated. The ideal exposure time in the flying direction of the UAV is recorded as icit _{fly_dir} , and the exposure time is recorded as cit _{fly_dir} . The calculation methods of the above two parameters icit _{fly_dir} and cit _{fly_dir} are shown in formula (15) and formula (16):

icit _{fly_dir} = w _left *icit _left +w _right *icit _right (15)

cit _{fly_dir} = w _left *cit _left +w _right *cit _right (16)

Among them, icit _{fly_dir} refers to the ideal exposure time in the flying direction of the drone, cit _{fly_dir} refers to the exposure time in the flying direction of the drone, and w _left refers to the fisheye adjacent to the left of the flying direction of the drone The weight of the exposure parameters of the camera, w _right refers to the weight of the exposure parameters of the fisheye camera adjacent to the right of the drone's flight direction, and icit _left refers to the weight of the fisheye camera adjacent to the left of the drone's flight direction Ideal exposure time, icit _right refers to the ideal exposure time of the fisheye camera adjacent to the right of the drone's flight direction, cit _left refers to the exposure time of the fisheye camera adjacent to the left of the drone's flight direction, cit _right refers to the exposure time of the fisheye camera adjacent to the right of the flying direction of the drone.

After calculating the exposure parameters in the flight direction of the UAV, use the exposure parameters in the flight direction of the UAV to limit the exposure parameters of each fisheye camera on the UAV. In order to ensure that the brightness difference between the two images of the fisheye camera is within the allowable range, according to the calculated ideal exposure time icit _{fly_dir} in the flight direction of the drone, the ideal exposure time of each fisheye camera on the drone is limited. The method of is shown in formula (17):

icit _x = min(icit _{fly_dir} *ratio _icit , icit _x ), x∈{C,D,E,F} (17)

Among them, x refers to any one of the four fisheye cameras C, D, E and F on the drone, icit _x refers to the ideal exposure time of the fisheye camera x on the drone, icit _{fly_dir} refers to the The ideal exposure time in the direction of human-machine flight, ratio _icit refers to a parameter set according to experience.

Then, in order to ensure that the content difference between the two images of the fisheye camera is within the allowable range, the exposure time in the flying direction of the drone is used to limit the exposure time of each fisheye camera on the drone. According to the above calculation, the exposure time cit _{fly_dir} in the flying direction of the UAV is obtained, and the method of limiting each fisheye camera on the UAV is shown in formula (18):

cit _x = min(cit _{fly_dir} *ratio _{cit_up} ,max(cit _{fly_dir} *ratio _{cit_low} ,icit _x )), x∈{C,D,E,F} (18)

Among them, x refers to any one of the four fisheye cameras C, D, E and F on the drone, cit _x refers to the exposure time of the fisheye camera x on the drone, and cit _{fly_dir} refers to the The exposure time in the flight direction of the aircraft, ratio _{cit_up} refers to a parameter set according to experience, ratio _{cit_low} refers to another parameter set according to experience, and the parameter value of ratio _{cit_up} is greater than the parameter value of ratio _{cit_low} .

Finally, according to the ideal exposure time and exposure time after each fisheye camera limit on the drone, the analog gain and digital gain of each fisheye camera were adjusted respectively. After setting and adjusting the exposure parameters of the fisheye camera based on the above scheme, it is not necessary to use a fisheye camera that can perceive a large dynamic range. Various tasks such as obstacle avoidance can be performed normally.

It can be seen from the above embodiment that in this embodiment, a sensor with a high dynamic range may not be used, and a perception camera of a general consumer product may meet the requirements, thereby reducing the cost of the movable platform. When calculating the exposure time of each sensor, the moving state of the movable platform, the brightness of each area in each sensor, etc. are considered comprehensively, so that the imaging brightness of the final movable platform in all directions is in the optimal state under various constraints. , so as to improve the effect of the movable platform obstacle avoidance, tracking and other functions; when the movable platform is in a maneuvering state, such as sudden braking, sudden change of direction, this embodiment can well ensure that the exposure of each sensor is appropriate; when the movable platform In a stable state, such as when hovering, there will be no unstable critical states such as jumps in this embodiment.

In the above embodiments, the exposure parameters are determined in consideration of the moving direction. As another optional way to implement step S202 and/or step S203, it may also be implemented in other ways without introducing the moving direction. For example, under the condition that the field of view of the sensors is coupled to each other and it is impossible to look around in four directions at the same time, the sensor can be controlled to perform time-division multiplexing, focusing on different field of view ranges under different time slices, so as to ensure that the images in different directions acquired by the sensor are consistent. It meets the quality requirements, so that the mobile platform can use high-quality images for depth information perception, and it is safer to control the movement of the mobile platform based on the obtained depth information. To achieve this goal, the exposure parameters of the movable platform sensor can be controlled to switch under the exposure parameters corresponding to the brightness information obtained based on different fields of view. As an example, as shown in FIG. A schematic flowchart of some steps in a method for controlling a movable platform may include the following steps:

Step S401: Determine a first exposure parameter of the third sensor based on the first brightness information; determine a second exposure parameter of the third sensor based on the second brightness information.

Step S402: Control the exposure parameter of the third sensor to switch between the first exposure parameter and the second exposure parameter.

In this embodiment, the brightness information of the scene in the two directions of the movable platform can be obtained respectively, and two sets of exposure parameters of the third sensor can be determined accordingly. Then control the exposure parameter of the third sensor to continuously switch between the two sets of exposure parameters; wherein, when the exposure parameter of the third sensor is the first exposure parameter, the third sensor obtains the image of the scene in the first direction of the movable platform; When the exposure parameter of the third sensor is the second exposure parameter, the third sensor acquires an image of the scene in the second direction of the movable platform. Since the images of the scene in the two directions are acquired separately, there is no influence on each other.

Since the exposure parameter of the third sensor controlling the movable platform is switched between the first exposure parameter and the second exposure parameter, for the third sensor, when the brightness information in the first direction and the second direction differ greatly , and the luminance information in the two directions can also be obtained independently, without being affected by the ambient luminance information in the other direction. Therefore, the movable platform can obtain accurate depth information based on the above scheme, and the movable platform can move safely. This solution does not need to use a sensor capable of sensing a large dynamic range, and the cost is low; at the same time, it solves the problem of being unable to look around in four directions at the same time due to the mutual coupling of the sensor field of view when the dynamic range is large in the environment; guarantees mobility The image quality of the scenery in all directions acquired by the platform sensor enables the normal operation of perception algorithms such as obstacle avoidance in all directions of the movable platform.

In this embodiment, there are multiple ways to control the third sensor to switch between the first exposure parameter and the second exposure parameter, which can be configured according to needs in practical applications, which is not limited in this embodiment. For example, the number of frames whose exposure parameter of the third sensor is the first exposure parameter may be the same as or different from the number of frames whose exposure parameter of the third sensor is the second exposure parameter. Optionally, the influence of the moving direction of the movable platform can also be increased, for example, when the movable platform moves forward, increase the number of frames when the exposure parameter of the third sensor is the first exposure parameter, so that the exposure parameter of the third sensor The number of frames when the first exposure parameter is greater than the number of frames when the exposure parameter of the third sensor is the second exposure parameter.

In some embodiments, the time when the third sensor collects a round of images of the first partial field of view and images of the second partial field of view is set as an image collection period of the third sensor. In the case that the image acquisition cycle of the third sensor satisfies the preset condition, the exposure parameter of the third sensor is controlled to switch between the first exposure parameter and the second exposure parameter. Wherein, the preset condition that the image acquisition period of the third sensor satisfies may be dividing the image acquisition period into two time slices: configuring the exposure parameter of the third sensor under the first time slice as the first exposure parameter , the first exposure parameter is used to control the third sensor to acquire the image of the first partial field of view; the exposure parameter of the third sensor under the second time slice is configured as the second exposure parameter, and the second exposure parameter is used to control the first The triple sensor acquires images of the second partial field of view.

Therefore, if the current image acquisition period belongs to the first time slice, the third sensor is controlled to acquire the image of the scene in the first direction under the first exposure parameter; if the current image acquisition period belongs to the second time slice, the third sensor is controlled to The image of the scene in the second direction is collected under the second exposure parameter.

In some embodiments, the durations of the first time slice and the second time slice of the image acquisition cycle of the third sensor are both equal to the duration of acquiring one frame of image by the third sensor. At this time, one of the image sequences acquired by the third sensor in the first time slice and the image sequence acquired in the second time slice is an odd-numbered frame sequence, and the other is an even-numbered frame sequence.

In general, each sensor of the movable platform corresponds to a physical channel and executes the same set of automatic exposure algorithms. After implementing the above solution, the sensors on the movable platform are configured with different exposure parameters under different time slices, which can be understood as dividing each sensor on the movable platform into multiples. Correspondingly, each physical channel also becomes multiple virtual channels, and the automatic exposure algorithms executed by each virtual channel are independent of each other.

In some embodiments, a physical channel corresponding to the third sensor can be divided into two virtual channels according to the two time slices divided by the third sensor; Images are acquired independently between the channels, as shown in Figure 5. The steps of acquiring an image based on the above method include:

Step S501: Calculate the number of the virtual channel to which each frame of image in the image sequence acquired by the third sensor belongs.

Step S502: Set the exposure parameter of the third sensor as the exposure parameter corresponding to the virtual channel number.

Step S503: The movable platform acquires an image based on the exposure parameters set by the third sensor.

In some embodiments, in addition to calculating the number of the virtual channel to which the image of the current frame belongs when calculating the frame corresponding to the current image, the number of the virtual channel to which the image of the next frame belongs is also calculated, and corresponding to the number of the virtual channel to which the image of the next frame belongs The exposure parameters are stored.

Optionally, the exposure parameter corresponding to the virtual channel number to which the next frame of image belongs is stored in a register of the third sensor on the movable platform.

Optionally, all image sequences on the movable platform share and synchronize an image frame number sequence, so that the frame number sequence of the image sequence acquired by the third sensor is the same as the frame number sequence of the image sequence output by the third sensor.

Next, an embodiment is used to illustrate the flow of the method in steps S501 to S503:

In the case that the time slice is divided into thirds, the third sensor of the movable platform executes the automatic exposure algorithm and sets the timing diagram of the register, as shown in FIG. 6 . Taking the frame number of the current image as 5 as an example, the flow of the method includes:

Calculate the current image frame number frame _idx to execute the virtual channel number vchn _{aec_calc} corresponding to the automatic exposure algorithm, and the calculation method is shown in formula (19):

vchn _{aec_calc} = frame _idx %vchn _num (19)

Among them, vchn _{aec_calc} refers to the virtual channel number to which the frame number of the current image belongs, frame _idx refers to the frame number of the current image, and vchn _num refers to the number of virtual channels.

The virtual channel number vchn _{aec_calc} =2 corresponding to the current image frame number is obtained. The exposure parameter of the third sensor is set to be the corresponding exposure parameter when the virtual channel number is 2, and then the third sensor executes the automatic exposure algorithm to obtain the scenery corresponding to the direction of the movable platform,

At the same time, calculate the virtual channel number vchn _{aec_set} corresponding to the next frame of the current image frame number, and the calculation method is shown in formula (20):

vchn _{aec_set} = (frame _idx +1)% vchn _num (20)

Among them, vchn _{aec_set} refers to the virtual channel number of the next frame of the current image frame, frame _idx refers to the frame number of the current image, and vchn _num refers to the number of virtual channels.

Obtain the virtual channel number vchn _{aec_set} =0 corresponding to the next frame of the current image frame number, and set its corresponding exposure parameters into the register.

For example, before acquiring the image of the next frame, read the exposure parameter corresponding to the virtual channel number of the next frame image temporarily stored in the current frame image, and set it in the register of the sensor; After that, the virtual channel number of the next frame image calculated based on the image of the next frame is the same as the virtual channel number set in the register of the fisheye camera above.

Since the exposure parameters of the third sensor switch between the two sets of exposure parameters, the sequence of images acquired by the third sensor may fluctuate and jump when the surrounding environment changes greatly. However, the image sequence acquired by the third sensor under the same exposure parameters is smooth and uniform. After acquiring the images of the scene in all directions of the movable platform, it is necessary to correctly divide the images under different exposure parameters from the image sequence acquired by the third sensor. The images under different exposure parameters can be correctly divided from the image sequence acquired by the third sensor according to the corresponding relationship between the frame number of the above image sequence and the time slice or virtual channel. The step of dividing the image sequence of the third sensor includes: obtaining the image sequence output by the third sensor, each image in the image sequence corresponds to an image frame number; obtaining the corresponding relationship between the image frame number and the time slice; using the image frame number and the corresponding relationship , acquire the first image acquired under the first exposure parameter from the image sequence.

Another embodiment is provided below for illustration. This embodiment involves steps S401 to S403, and steps S501 to S503, as shown in FIG. 7. In this embodiment, the movable platform includes the following steps:

Step S701: According to the corresponding relationship between the exposure parameters of the third sensor and the scenes in different directions of the movable platform, determine the virtual channel number corresponding to the algorithm.

Step S702: Calculate the number of the virtual channel to which each frame of image in the image sequence output by the third sensor belongs.

Step S703: Input the image in the image sequence with the same virtual channel number corresponding to the algorithm into the algorithm.

In the mobile platform surround-view perception solution, the embodiment of the present application does not need to use a camera with a very high dynamic range, and the sensors of general consumer products can meet the requirements, thereby greatly reducing the cost; when calculating the exposure time of each sensor In this process, the moving state of the movable platform, the brightness of each area in each sensor, etc. are considered comprehensively, so that the final imaging brightness of the movable platform in all directions is in the optimal state under various constraints, thereby improving the avoidance of the movable platform. Obstacles, tracking and other functions; when the movable platform is in a maneuvering state, such as sudden braking, sudden change of direction, this application can well ensure that the exposure of each sensor is appropriate; when the movable platform is in a stable state, such as hovering , there will be no unstable critical states such as jumps in this application.

For the control method of the mobile platform in this embodiment, this specification also provides some other embodiments of the control method of the mobile platform.

For the embodiment described in FIG. 2, first, the first state of the movable platform is defined as the first partial field of view of the third sensor is blocked, and the second partial field of view is not blocked, as shown in FIG. 8; the second state of the movable platform The state is that the first partial field of view of the third sensor is not blocked, and the second partial field of view is not blocked, as shown in FIG. 9 . In practical applications, there are various ways of blocking, for example, a black tape blocking object can be used to cover a part of the sensor lens area, or other blocking objects such as baffles can block the partial field of view of the sensor.

If the movable platform uses the aforementioned embodiment shown in Figure 2, when the third sensor on the movable platform is in the first state and the second state, the recognition results of the scene in the second direction obtained based on the images respectively collected by the third sensor unanimous. That is, no matter whether the movable platform is in the first state or the second state, when the movable platform moves in space, the movable platform has correct recognition results for the scene in the second direction.

If the solution in the related art is adopted, the first partial field of view of the movable platform is blocked, and the automatic exposure algorithm of the sensor in the movable platform will inevitably appear abnormal, resulting in the failure of the obstacle avoidance function of the movable platform.

Especially, for low-cost sensors with a small dynamic range, the mobile platforms of this embodiment can all be safely controlled to move. For example, in the first state, the light intensity reflected by the scene in the second partial field of view of the third sensor on the movable platform may be set to be higher than the maximum light intensity sensed by the third sensor. As an example, the second partial field of view of the third sensor of the movable platform may be directed toward strong light, and since the first partial field of view is blocked, the third sensor is in an extreme high-dynamic scene; When moving in space, it has correct recognition results for the scenes in the second direction.

Optionally, the correct identification result of the scene by the movable platform includes any of the following: output information of the recognized scene in the user interface; the movable platform successfully avoids obstacles; The correct recognition rate of the scene within is higher than the preset first threshold. It can be understood that other expressions also belong to the protection scope of the present application, which is not limited in the present application.

Optionally, identifying the scene includes identifying depth information and texture features of the scene, which is not limited in this application.

For the embodiment shown in FIG. 3 , firstly, the first state is defined as the first partial field of view of the third sensor is blocked, and the second partial field of view is not blocked, as shown in FIG. 8 .

When the third sensor on the movable platform is in the first state, when the movable platform moves in the first direction, the recognition result of the scene in the second direction based on the image collected by the third sensor is the same as the recognition result of the scene in the second direction when the movable platform moves in the second direction. When moving, the recognition results of the scene in the second direction obtained based on the image collected by the third sensor are inconsistent. For example, when the movable platform moves towards the first direction in the first state, the recognition result of the scenery in the second direction is unstable; correct recognition result.

Optionally, the recognition result of the scene by the movable platform is unstable, including any of the following: outputting a prompt message in the user interface prompting that the recognition result is unstable or has no recognition result; Obstacles: the correct recognition rate of the movable platform for the scene in the partial field of view is lower than the preset second threshold. It can be understood that other expressions also belong to the protection scope of the present application, which is not limited in the present application.

For the embodiment shown in FIG. 4 , firstly, the first state is defined as the first partial field of view of the third sensor is blocked, and the second partial field of view is not blocked, as shown in FIG. 8 .

When the third sensor on the movable platform is in the first state, and when the movable platform moves in the first direction, the recognition result of the scene in the second direction obtained based on the image collected by the third sensor is the same as the recognition result of the scene in the second direction when the movable platform moves in the second direction. When moving, the recognition results of the scene in the second direction obtained based on the image collected by the third sensor are consistent. For example, when the movable platform moves in the first direction and the second direction respectively in the first state, the movable platform has correct recognition results for the scenes in the second direction.

Optionally, the correct recognition result of the scene includes any of the following: outputting the information of the recognized scene in the user interface; the movable platform successfully avoids obstacles; The correct recognition rate is higher than the preset first threshold. It can be understood that other forms of expression are also included, which also belong to the protection scope of this application, and this application does not limit it.

Regarding the embodiment shown in FIG. 4 , another embodiment is provided below, and this embodiment does not require a movable platform to move. Firstly, the first state is defined as that the first partial field of view of the third sensor is blocked, and the second partial field of view is not blocked, as shown in FIG. 8 .

In this embodiment, when the third sensor on the movable platform is in the first state, the scene in the second direction obtained based on the image collected by the third sensor has a correct recognition result.

It can be seen from the above embodiments that it can be understood that the movable platform can be integrated based on the depth information obtained by the two sensors, and then form the above method and steps with the third sensor to solve the exposure problem caused by the mutual coupling of multiple sensor fields of view. Therefore, the present application is not limited to the problem between two sensors, and is also applicable to solving the exposure problem caused by the coupling of multiple sensors, and this application will not describe it in detail.

The present application also provides a control device for a movable platform, the movable platform includes a first sensor, a second sensor and a third sensor, the first sensor and the third sensor have overlapping first partial fields of view , the second sensor and the third sensor have a second partial field of view that overlaps, the first partial field of view is used to observe the scene in the first direction of the movable platform; the second partial field of view is used to observe The scene in the second direction of the movable platform is shown in FIG. 10 .

The device includes a processor, a memory, and a computer program stored on the memory that can be executed by the processor. When the processor executes the computer program, the following methods are implemented:

determining a second exposure parameter of the third sensor based at least on the second brightness information; controlling the third sensor to capture a second image under the second exposure parameter; based on the second image and the second acquire the depth information of the scene in the second direction from the image collected by the sensor;

The movable platform is controlled to move in space according to the depth information.

In some examples, when the third sensor is in the first state and the second state, the depth information of the scene in the second direction obtained based on the images respectively collected by the third sensor is consistent;

Wherein, the first state includes: the first partial field of view of the third sensor is blocked, and the second partial field of view is not blocked;

The second state includes: the first partial field of view of the third sensor is not blocked, and the second partial field of view is not blocked.

In some examples, the light intensity reflected by the scene in the second partial field of view is higher than the maximum light intensity sensed by the third sensor.

In some examples, the method further includes: acquiring the moving direction of the movable platform;

The first exposure parameter of the third sensor is determined based at least on the first brightness information of the third sensor and the moving direction.

In some examples, when the third sensor is in the first state, when the movable platform moves toward the first direction, the depth of the scene in the second direction obtained based on the image collected by the third sensor The information is inconsistent with the depth information of the scene in the second direction obtained based on the image collected by the third sensor when the movable platform moves in the second direction;

Wherein, the first state includes: the first partial field of view of the third sensor is blocked, and the second partial field of view is not blocked.

In some examples, the determining the first exposure parameter of the third sensor based at least on the first brightness information includes:

If the moving direction falls into the first direction, determining a first exposure parameter of the third sensor based on the first brightness information;

If the moving direction is biased toward the first direction relative to the second direction, the first exposure parameter is determined based on the first brightness information and the second brightness information; wherein, the first brightness information The weight of is greater than the weight of the second brightness information.

In some examples, the first exposure parameters include exposure time and analog gain;

The determining the first exposure parameter of the third sensor based at least on the first brightness information includes:

An exposure time and an analog gain of the third sensor are calculated by using an automatic exposure algorithm at least according to the first brightness information.

In some examples, after determining the first exposure parameter of the third sensor, the method further includes:

Based on the first exposure parameter of the first sensor, the first exposure parameter of the third sensor is adjusted.

In some examples, the first exposure parameter of the third sensor is adjusted in the following manner:

reducing the difference between the first exposure parameter of the first sensor and the first exposure parameter of the third sensor, so that the image captured by the first sensor and the image captured by the third sensor meet a preset brightness condition and/or Preset image content conditions.

In some examples, the adjusting the first exposure parameter of the third sensor includes:

According to the first exposure parameter of the first sensor and the first exposure parameter of the third sensor, determine the value range of the exposure parameter of the third sensor, and control the first exposure parameter of the third sensor in the within the range of exposure parameters.

In some examples, the value range of the exposure parameter is determined based on the first exposure parameter of the first sensor and its weight, and the first exposure parameter of the third sensor and its weight;

Wherein, the weight of the first exposure parameter of the third sensor is greater than the weight of the first exposure parameter of the first sensor.

In some examples, the exposure parameters include ideal exposure time and exposure time;

The adjusting the first exposure parameter of the third sensor based on the value range of the exposure parameter includes:

If the determined ideal exposure time and exposure time of the third sensor are not within the value range of the exposure parameter, adjusting the ideal exposure time and exposure time of the third sensor to a boundary value of the value range of the exposure parameter.

In some examples, the exposure parameters also include: analog gain and digital gain;

The adjusting the exposure parameter of the third sensor based on the value range of the exposure parameter further includes:

Adjusting the analog gain and digital gain of the third sensor based on the adjusted exposure time and the ideal exposure time.

In some examples, the method further includes: acquiring a change in the moving speed of the movable platform; wherein, the change in the moving speed is positively correlated with any of the following:

The acquisition frequency of the moving direction of the movable platform, the update frequency of the exposure parameters or the update frequency of the weight of the brightness information.

In some examples, the method further includes: controlling an exposure parameter of a third sensor to switch between the first exposure parameter and the second exposure parameter.

In some examples, when the third sensor is in the first state, when the movable platform moves toward the first direction, the depth of the scene in the second direction obtained based on the image collected by the third sensor The information is consistent with the depth information of the scene in the second direction obtained based on the image collected by the third sensor when the movable platform moves in the second direction;

In some examples, the controlling the exposure parameter of the third sensor to switch between the first exposure parameter and the second exposure parameter is performed when the image acquisition cycle of the third sensor satisfies a preset condition .

In some examples, the controlling the exposure parameter of the third sensor to switch between the first exposure parameter and the second exposure parameter includes:

If the current image acquisition period belongs to the first time slice, controlling the third sensor to acquire the first image under the first exposure parameters;

If the current image acquisition period belongs to the second time slice, controlling the third sensor to acquire a second image under the second exposure parameters.

In some examples, the duration of the time slice is the duration of one frame of image collected by the third sensor.

In some examples, the method also includes:

Obtain an image sequence output by the third sensor, where each image in the image sequence corresponds to an image frame number;

Obtain the correspondence between the image frame number and the time slice;

Using the image frame number and the corresponding relationship, the first image collected under the first exposure parameter is acquired from the image sequence.

In some examples, the first sensor, the second sensor and the third sensor are fisheye cameras.

In some examples, the movable platform includes a drone.

In some embodiments, the functions or modules included in the device provided by the embodiments of the present application can be used to execute the methods described in the above method embodiments, and its specific implementation can refer to the descriptions of the above method embodiments. For brevity, here No longer.

As shown in FIG. 11 , the embodiment of the present application also provides a movable platform 1100 , including: the movable platform includes a first sensor 1101 , a second sensor 1102 and a third sensor 1103 ;

The first sensor 1101 and the third sensor 1103 have overlapping first partial fields of view;

The second sensor 1102 and the third sensor 1103 have overlapping second partial fields of view;

The first partial field of view is used to observe the scene in the first direction of the movable platform 1100; the second partial field of view is used to observe the scene in the second direction of the movable platform 1100;

The removable platform also includes a processor 1104, a memory 1105, and a computer program stored on the memory that can be executed by the processor.

Wherein, the movable platform also includes a power system 1106 for driving the movable platform to move in space.

When the processor executes the computer program, the following methods are implemented:

In some examples, the processor further executes: acquiring a moving direction of the movable platform;

In some examples, the processor adjusts the first exposure parameter of the third sensor by:

In some examples, the processor further executes: acquiring a change in the moving speed of the movable platform; wherein, the change in the moving speed is positively correlated with any of the following:

In some examples, the processor further executes: controlling an exposure parameter of a third sensor to switch between the first exposure parameter and the second exposure parameter.

In some examples, the processor also performs:

Obtain the correspondence between the image frame number and the time slice;

In some examples, the movable platform includes a drone.

In practical applications, the movable platform may also include other hardware as required. For example, the device may include: a processor, memory, input/output interfaces, communication interfaces, and a bus. The processor, the memory, the input/output interface and the communication interface are connected to each other within the device through the bus.

The processor can be implemented by a general-purpose CPU (Central Processing Unit, central processing unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute related programs , so as to realize the technical solutions provided by the embodiments of this specification. The processor can also include a graphics card, and the graphics card can be an Nvidia titan X graphics card or a 1080Ti graphics card.

The memory can be implemented in the form of ROM (Read Only Memory, read-only memory), RAM (Random Access Memory, random access memory), static storage device, and dynamic storage device. The memory 1105 can store operating systems and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1105 and invoked by the processor for execution.

The input/output interface is used to connect the input/output module to realize information input and output. The input/output/module can be configured in the device as a component (not shown in the figure), or can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, and the like.

The communication interface is used to connect the communication module to realize the communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.), and can also realize communication through wireless means (such as mobile network, WIFI, Bluetooth, etc.).

A bus includes a pathway that carries information between various components of a device such as processors, memory, input/output interfaces, and communication interfaces.

It should be noted that although the above device only shows a processor, a memory, an input/output interface, a communication interface, and a bus, in a specific implementation process, the device may also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above-mentioned device may only include components necessary to implement the solutions of the embodiments of this specification, and does not necessarily include all the components shown in the figure.

The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method for controlling the mobile platform described in any of the foregoing embodiments are implemented.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for computers include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.

It can be known from the above description of the implementation manners that those skilled in the art can clearly understand that the embodiments of this specification can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the essence of the technical solutions of the embodiments of this specification or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in storage media, such as ROM/RAM, A magnetic disk, an optical disk, etc., include several instructions to enable a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this specification.

The systems, devices, modules, or units described in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementing device is a computer, which may take the form of a personal computer, laptop computer, cellular phone, camera phone, smart phone, personal digital assistant, media player, navigation device, e-mail device, game control device, etc. desktops, tablets, wearables, or any combination of these.

Each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, as for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for relevant parts, please refer to part of the description of the method embodiment. The device embodiments described above are only illustrative, and the modules described as separate components may or may not be physically separated, and the functions of each module may be integrated in the same or multiple software and/or hardware implementations. Part or all of the modules can also be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without creative effort.

The above is only the specific implementation of the embodiment of this specification. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the embodiment of this specification, some improvements and modifications can also be made. These Improvements and modifications should also be regarded as the scope of protection of the embodiments of this specification.

Claims

A method for controlling a movable platform, characterized in that the movable platform includes a first sensor, a second sensor and a third sensor;

the first sensor and the third sensor have overlapping first partial fields of view;

the second sensor has a coincident second partial field of view with the third sensor;

The first partial field of view is used to observe the scene in the first direction of the movable platform; the second partial field of view is used to observe the scene in the second direction of the movable platform;

The methods include:

acquiring first brightness information of the first partial field of view and second brightness information of the second partial field of view of the third sensor;

determining a first exposure parameter of the third sensor based at least on the first brightness information; controlling the third sensor to capture a first image under the first exposure parameter; based on the first image and the first Obtain the depth information of the scene in the first direction from the image collected by the sensor;

determining a second exposure parameter of the third sensor based at least on the second brightness information; controlling the third sensor to capture a second image under the second exposure parameter; based on the second image and the second acquire the depth information of the scene in the second direction from the image collected by the sensor;

The movable platform is controlled to move in space according to the depth information.
The method according to claim 1, wherein when the third sensor is in the first state and the second state, the depth of the scene in the second direction obtained based on the images respectively collected by the third sensor consistent information;

Wherein, the first state includes: the first partial field of view of the third sensor is blocked, and the second partial field of view is not blocked;

The second state includes: the first partial field of view of the third sensor is not blocked, and the second partial field of view is not blocked.
The method according to claim 2, characterized in that the light intensity reflected by the scene in the second partial field of view is higher than the maximum light intensity sensed by the third sensor.
The method according to claim 1, further comprising:

Acquiring the moving direction of the movable platform;

The first exposure parameter of the third sensor is determined based on at least the first brightness information and the moving direction.
The method according to claim 4, wherein when the third sensor is in the first state, when the movable platform moves toward the first direction, the obtained information based on the image collected by the third sensor The depth information of the scene in the second direction is inconsistent with the depth information of the scene in the second direction obtained based on the image collected by the third sensor when the movable platform moves in the second direction;

Wherein, the first state includes: the first partial field of view of the third sensor is blocked, and the second partial field of view is not blocked.
The method according to claim 4, wherein the determining the first exposure parameter of the third sensor based at least on the first brightness information comprises:

If the moving direction falls into the first direction, determining a first exposure parameter of the third sensor based on the first brightness information;

If the moving direction is biased toward the first direction relative to the second direction, the first exposure parameter is determined based on the first brightness information and the second brightness information; wherein, the first brightness information The weight of is greater than the weight of the second brightness information.
The method according to claim 4, wherein the first exposure parameters include exposure time and analog gain;

The determining the first exposure parameter of the third sensor based at least on the first brightness information includes:

An exposure time and an analog gain of the third sensor are calculated by using an automatic exposure algorithm at least according to the first brightness information.
The method according to claim 6, wherein after determining the first exposure parameter of the third sensor, the method further comprises:

Based on the first exposure parameter of the first sensor, the first exposure parameter of the third sensor is adjusted.
The method according to claim 8, characterized in that the first exposure parameter of the third sensor is adjusted in the following manner:

reducing the difference between the first exposure parameter of the first sensor and the first exposure parameter of the third sensor, so that the image captured by the first sensor and the image captured by the third sensor meet a preset brightness condition and/or Preset image content conditions.
The method according to claim 8, wherein the adjusting the first exposure parameter of the third sensor comprises:

According to the first exposure parameter of the first sensor and the first exposure parameter of the third sensor, determine the value range of the exposure parameter of the third sensor, and control the first exposure parameter of the third sensor in the within the range of exposure parameters.
The method according to claim 10, wherein the value range of the exposure parameter is determined based on the first exposure parameter of the first sensor and its weight, and the first exposure parameter of the third sensor and its weight;

Wherein, the weight of the first exposure parameter of the third sensor is greater than the weight of the first exposure parameter of the first sensor.
The method according to claim 10, wherein the exposure parameters include ideal exposure time and exposure time;

The adjusting the first exposure parameter of the third sensor based on the value range of the exposure parameter includes:

If the determined ideal exposure time and exposure time of the third sensor are not within the value range of the exposure parameter, adjusting the ideal exposure time and exposure time of the third sensor to a boundary value of the value range of the exposure parameter.
The method according to claim 12, wherein the exposure parameters further comprise: analog gain and digital gain;

The adjusting the exposure parameter of the third sensor based on the value range of the exposure parameter further includes:

Adjusting the analog gain and digital gain of the third sensor based on the adjusted exposure time and the ideal exposure time.
The method according to any one of claims 4 to 13, further comprising:

Acquiring the change of the moving speed of the movable platform; wherein, the change of the moving speed is positively correlated with any of the following:

The acquisition frequency of the moving direction of the movable platform, the update frequency of the exposure parameters or the update frequency of the weight of the brightness information.
The method according to claim 1, further comprising:

controlling the exposure parameter of the third sensor to switch between the first exposure parameter and the second exposure parameter.
The method according to claim 15, wherein when the third sensor is in the first state, when the movable platform moves toward the first direction, the obtained information obtained based on the image collected by the third sensor The depth information of the scene in the second direction is consistent with the depth information of the scene in the second direction obtained based on the image collected by the third sensor when the movable platform moves in the second direction;

Wherein, the first state includes: the first partial field of view of the third sensor is blocked, and the second partial field of view is not blocked.
The method according to claim 15, wherein the controlling the exposure parameter of the third sensor to switch between the first exposure parameter and the second exposure parameter is during the image acquisition of the third sensor The cycle is executed when the preset conditions are met.
The method according to claim 17, wherein the controlling the exposure parameter of the third sensor to switch between the first exposure parameter and the second exposure parameter comprises:

If the current image acquisition period belongs to the first time slice, controlling the third sensor to acquire the first image under the first exposure parameters;

If the current image acquisition period belongs to the second time slice, controlling the third sensor to acquire a second image under the second exposure parameters.
The method according to claim 18, wherein the duration of the time slice is the duration of one frame of image collected by the third sensor.
The method according to claim 18, further comprising:

Obtain an image sequence output by the third sensor, where each image in the image sequence corresponds to an image frame number;

Obtain the correspondence between the image frame number and the time slice;

Using the image frame number and the corresponding relationship, the first image collected under the first exposure parameter is acquired from the image sequence.
The method according to claim 1, wherein the first sensor, the second sensor and the third sensor are fisheye cameras.
The method of claim 1, wherein the movable platform comprises a drone.
A control device for a movable platform, characterized in that the movable platform includes a first sensor, a second sensor and a third sensor;

the first sensor and the third sensor have overlapping first partial fields of view;

the second sensor has a coincident second partial field of view with the third sensor;

The first partial field of view is used to observe the scene in the first direction of the movable platform; the second partial field of view is used to observe the scene in the second direction of the movable platform;

The device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, and the processor implements the method according to any one of claims 1 to 22 when executing the computer program.
A movable platform, characterized in that the movable platform includes a first sensor, a second sensor and a third sensor;

the first sensor and the third sensor have overlapping first partial fields of view;

the second sensor has a coincident second partial field of view with the third sensor;

The first partial field of view is used to observe the scene in the first direction of the movable platform; the second partial field of view is used to observe the scene in the second direction of the movable platform;

The mobile platform further includes a processor, a memory, and a computer program stored on the memory that can be executed by the processor. When the processor executes the computer program, the invention described in any one of claims 1 to 22 is realized. method.
A computer-readable storage medium, characterized in that several computer instructions are stored on the computer-readable storage medium, and when the computer instructions are executed, the method for controlling the mobile platform according to any one of claims 1 to 22 is realized A step of.