WO2023272633A1

WO2023272633A1 - Unmanned aerial vehicle control method, unmanned aerial vehicle, flight system, and storage medium

Info

Publication number: WO2023272633A1
Application number: PCT/CN2021/103776
Authority: WO
Inventors: 聂谷洪; 李鑫超; 王栋
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2021-06-30
Filing date: 2021-06-30
Publication date: 2023-01-05
Also published as: CN116724281A

Abstract

An unmanned aerial vehicle control method, an unmanned aerial vehicle, a flight system, and a storage medium. The method comprises: collecting an image by means of a visual sensor mounted on an unmanned aerial vehicle (S101); when it is recognized that the image comprises a target of a preset type, determining a target control strategy according to the current working condition of the unmanned aerial vehicle (S102); and controlling, according to the target control strategy, the unmanned aerial vehicle to perform a corresponding operation (S103).

Description

Control method of unmanned aerial vehicle, unmanned aerial vehicle, flight system and storage medium

technical field

The present application relates to the technical field of flight control, and in particular to a method for controlling an unmanned aerial vehicle, an unmanned aerial vehicle, a flight system, and a storage medium.

Background technique

UAVs, such as plant protection UAVs, have higher and higher requirements for their flight operations, especially flight safety requirements. Flight safety includes the safety of operators and the safety of UAVs. However, the UAVs currently on the market are only equipped with radar devices as sensing devices. Since the radar devices cannot cover all scenarios of UAV flight, it will inevitably bring about flight safety problems.

Contents of the invention

Embodiments of the present application provide a method for controlling an unmanned aerial vehicle, an unmanned aerial vehicle, a flight system, and a storage medium, so as to improve flight safety of the unmanned aerial vehicle.

In the first aspect, the embodiment of the present application provides a method for controlling a drone, the method comprising:

Collect images through the visual sensor on the drone;

When it is recognized that the image includes a preset type of target, a target control strategy is determined according to the current working condition of the drone;

The UAV is controlled to perform corresponding operations according to the target control strategy.

In the second aspect, the embodiment of the present application also provides a drone, which includes a visual sensor, a radar device, a memory, and a processor;

the vision sensor is used to capture images and the radar device uses radar data;

The memory is used to store computer programs;

The processor is configured to execute the computer program and implement the following steps when executing the computer program:

Collect images through the visual sensor on the drone;

In the third aspect, the embodiment of the present application also provides a flight system, the flight system includes the drone and the control terminal according to any one of the embodiments of the present application, and the control terminal user controls the drone flight.

In the fourth aspect, the embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor realizes the implementation of the present application. The steps of the control method of any one of the drones provided by the example.

The control method of the drone, the drone, the flight system, and the storage medium disclosed in the embodiments of the present application collect images through the visual sensor carried on the drone; when it is recognized that the image includes a target of a preset type, according to The current working condition of the UAV determines the target control strategy; the UAV is controlled to perform corresponding operations according to the target control strategy. It can be applied to scenes that cannot be covered by radar devices, thereby improving the flight safety of drones.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application. Ordinary technicians can also obtain other drawings based on these drawings on the premise of not paying creative work.

Fig. 1 is a schematic structural diagram of an unmanned aerial vehicle provided by an embodiment of the present application;

Fig. 2 is a schematic block diagram of the flight control system of the unmanned aerial vehicle provided by the embodiment of the present application;

Fig. 3 is a schematic structural diagram of a flight system provided by an embodiment of the present application;

Fig. 4 is a schematic flowchart of the steps of a method for controlling a drone provided in an embodiment of the present application;

Fig. 5 is a schematic diagram of the effect of displaying alarm information provided by the embodiment of the present application;

Fig. 6 is a schematic diagram of the output recognition target provided by the embodiment of the present application;

Fig. 7 is a schematic diagram of the effect of another output recognition target provided by the embodiment of the present application;

Fig. 8 is a schematic diagram of the effect of displaying images provided by the embodiment of the present application;

FIG. 9 is a schematic flowchart of steps of another method for controlling a drone provided in an embodiment of the present application;

Fig. 10 is a schematic block diagram of an unmanned aerial vehicle provided by an embodiment of the present application.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

It should also be understood that the terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural referents unless the context clearly dictates otherwise.

It should also be further understood that the term "and/or" used in the description of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations .

The flow charts shown in the drawings are just illustrations, and do not necessarily include all contents and operations/steps, nor must they be performed in the order described. For example, some operations/steps can be decomposed, combined or partly combined, so the actual order of execution may be changed according to the actual situation.

At present, more and more attention is paid to the flight safety of drones, especially the safety of pedestrians such as operators. For example, plant protection drones (plant protection machines). The load of the plant protection machine is becoming heavier and heavier, and the lethality of the plant protection machine will become stronger and stronger, which may cause injuries to the workers, and the requirements for safety will become higher and higher.

However, at present, UAVs use radar devices to collect information about obstacles, and perform obstacle-avoiding flight based on the information of obstacles. These scenarios are the ones that cannot be covered by radar devices.

Exemplarily, for example, when the UAV is in the take-off phase, because the flying height of the UAV is lower than that of pedestrians, the detection by the radar device may fail, and the blades of the UAV may injure people. For another example, during the operation of the UAV, if there are objects with inclined surfaces such as tombstones, the detection ability of the radar device is weak or the detection effect is not good, and it cannot avoid obstacles well, and there may be collisions. possible.

In addition, the inventor also found that during the operation of the UAV, such as thatch, insect swarms, flying birds and other scenes that should not avoid obstacles appearing next to the UAV, the radar obstacle avoidance may be triggered by mistake, resulting in the termination of the operation of the UAV or the operation of the UAV. The path is re-planned, thereby affecting the operation effect and operation efficiency.

To this end, the embodiments of the present application provide a control method for a UAV, a UAV, a flight system, and a storage medium, so as to improve the flight safety of the UAV, and at the same time improve the efficiency of the UAV in performing operations. work efficiency.

Some implementations of the present application will be described in detail below in conjunction with the accompanying drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

Please refer to Figure 1 and Figure 2, Figure 1 shows the structure of a drone 100 provided by the embodiment of the present application, Figure 2 shows the structural framework of the flight control system of the drone 100 provided by the embodiment of the present application . As shown in FIGS. 1 and 2 , the UAV 100 may include a frame 10 , a power system 11 , a control system 12 and a radar device 20 .

In the embodiment of the present application, the drone 100 is also equipped with a visual sensor for taking pictures of the surrounding images of the drone. The visual sensor can also be called a shooting device, such as a camera, or a binocular camera, etc. .

Airframe 10 may include a fuselage and undercarriages (also referred to as landing gear). The fuselage may include a center frame and one or more arms connected to the center frame, and the one or more arms extend radially from the center frame. The tripod is connected with the fuselage and is used for supporting when the UAV 100 lands.

The radar device 20 can be installed on the UAV, specifically, it can be installed on the rack 10 of the UAV 100, and is used to measure the surrounding environment of the UAV 100 during the flight of the UAV 100, such as obstacles, etc. , to ensure flight safety. In the embodiment of the present application, the vision sensor can also be installed on the frame of the UAV 100 .

The radar device 20 and the visual sensor are all connected in communication with the control system 12. The radar device 20 transmits the collected observation data to the control system 12 for processing by the control system 12. The visual sensor transmits the collected images to the control system 12. The control system 12 performs the processing.

It should be noted that the drone 100 may include two or more tripods, and the radar device 20 is mounted on one of the tripods. The radar device 20 can also be mounted on other positions of the drone 100, which is not specifically limited.

The radar device 20 mainly includes a radio frequency front-end module and a signal processing module. The radio frequency front-end module may include a transmitting antenna and a receiving antenna. The transmitting antenna is used to send signals to the target, and the receiving antenna is used to receive signals reflected by the target. The signal processing module is responsible for generating Modulate the signal and process and analyze the collected intermediate frequency signal, where the targets are buildings, iron towers, crops, etc.

The power system 11 may include one or more electronic governors (referred to as ESCs for short), one or more propellers and one or more motors corresponding to the one or more propellers, wherein the motors are connected between the electronic governor and the Between the propellers, the motor and the propeller are arranged on the arm of the UAV 100; the electronic governor is used to receive the drive signal generated by the control system, and provide drive current to the motor according to the drive signal to control the speed of the motor.

The motor is used to drive the propeller to rotate, so as to provide power for the flight of the UAV 100, and the power enables the UAV 100 to realize one or more degrees of freedom of movement. In some embodiments, drone 100 may rotate about one or more axes of rotation. For example, the above-mentioned rotation axes may include a roll axis, a yaw axis and a pitch axis. It should be understood that the motor may be a DC motor or a permanent magnet synchronous motor. Alternatively, the motor can be a brushless motor or a brushed motor.

Control system 12 may include a controller and a sensing system. The controller is used to control the flight of the UAV 100, for example, the flight of the UAV 100 can be controlled according to the attitude information measured by the sensor system. It should be understood that the controller can control the UAV 100 according to pre-programmed instructions. The sensing system is used to measure the attitude information of the UAV 100, that is, the position information and state information of the UAV 100 in space, such as three-dimensional position, three-dimensional angle, three-dimensional velocity, three-dimensional acceleration and three-dimensional angular velocity, etc.

The sensing system may include, for example, at least one of sensors such as a gyroscope, an ultrasonic sensor, an electronic compass, an inertial measurement unit (Inertial Measurement Unit, IMU), a visual sensor, a global navigation satellite system, and a barometer. For example, the global navigation satellite system may be the Global Positioning System (GPS).

A controller may include one or more processors and memory. The processor may be, for example, a micro-controller unit (Micro-controller Unit, MCU), a central processing unit (Central Processing Unit, CPU), or a digital signal processor (Digital Signal Processor, DSP), etc. The memory can be a Flash chip, a read-only memory (ROM, Read-Only Memory) disk, an optical disk, a U disk or a mobile hard disk.

In some embodiments, the drone 100 may be a plant protection drone, which includes a spraying system for spraying pesticides or fertilizers on crops.

The UAV 100 may include a rotor-type UAV, such as a quad-rotor UAV, a six-rotor UAV, an octo-rotor UAV, or a fixed-wing UAV, or a combination of a rotor-type and a fixed-wing unmanned aerial vehicle. The combination of machines is not limited here.

As shown in FIG. 3 , FIG. 3 shows a structure of a flight system provided by an embodiment of the present application, and the flight system includes a drone 100 and a control terminal 200 . The control terminal 200 is located at the ground end of the flight system and can communicate with the UAV 100 in a wireless manner for remote control of the UAV 100 .

Wherein, the control terminal 200 also includes a terminal device 201, such as a mobile phone or a tablet computer, etc., which is used to display the control interface of the UAV 100 and the images collected by the visual sensor, and can also display the radar collected by the radar device. data, and recognition results of targets around the UAV 100, etc.

It should be understood that the above naming of the components of the drone 100 is only for the purpose of identification, and should not be construed as limiting the embodiments of this specification. The unmanned aerial vehicle 100, specifically the controller of the control system of the unmanned aerial vehicle 100, can be used to implement any one of the control methods for the unmanned aerial vehicle provided in the embodiments of the present application, so as to improve the flight safety of the unmanned aerial vehicle and pedestrian safety.

Exemplary, for example, the controller is used to: collect images through the visual sensor carried on the UAV; when it recognizes that the image includes a preset type of target, determine the target control strategy according to the current working condition of the UAV; according to the target The control strategy controls the UAV to perform corresponding operations, such as stopping the propeller (that is, controlling the UAV to stop driving the propeller to rotate), locking in a certain flight position, or continuing to perform preset operations, etc. As a result, the flight safety of the drone can be improved.

For ease of understanding, based on the drone or the flight system of the drone in the above-mentioned embodiments, the method for controlling the drone provided in the embodiment of the present application will be described in detail below.

Please refer to FIG. 4 . FIG. 4 is a schematic flowchart of steps of a method for controlling a drone provided in an embodiment of the present application.

As shown in FIG. 4 , the method for controlling the drone includes steps S101 to S103.

S101, collecting images through the visual sensor carried on the drone;

S102. When it is recognized that the image includes a preset type of target, determine a target control strategy according to the current working condition of the drone;

S103. Control the UAV to perform corresponding operations according to the target control strategy.

In some scenarios, such as scenarios that cannot be covered by radar devices, the visual sensor mounted on the UAV collects images and performs target recognition on the images. The working condition of the man-machine determines the target control strategy, which is the strategy used to control the UAV, and controls the UAV to perform corresponding operations according to the target control strategy to improve the safety of the UAV. including pedestrian safety.

The corresponding operation is specifically a specific strategy in the target control strategy, for example, controlling the drone to stop propellers, controlling the landing of the drone, controlling the hovering of the drone, continuing to fly, sending out an alarm message, and so on.

The default type of target is for the drone to detect through its on-board radar device, but it may result in a target with poor detection effect. For example, pedestrian targets around the drone during the take-off or landing phase, or objects with inclined surfaces such as tombstones during the flight.

The working conditions of the UAV include the flight state and/or environmental information of the UAV. The flight state is, for example, in the state of takeoff preparation, landing preparation state, hovering state, flight state, and task execution state. (For example, inspection, plant protection, etc.), etc., the environment information such as the target (such as obstacles) that can be included in the surrounding environment and the distance from the current drone to the target.

In some embodiments, in order to improve the flight safety of the UAV, the preset types of targets in the scene that the radar device cannot cover can be divided into a preset first type and a preset second type, wherein the preset first The type is a threatening obstacle that affects the flight safety of the UAV, and the second default type is a non-threatening obstacle that does not affect the flight safety of the UAV. In this way, it is convenient to determine the target control strategy according to the current working conditions of the UAV in different preset types, so as to improve the flight safety of the UAV.

It should be noted that for different preset types, even when the current working conditions of the UAV are the same, the determined target control strategies may be different; the same preset type, the current working conditions of the UAV are not the same. At the same time, the determined target control strategies may also be different.

Exemplarily, for example, the first target of the preset first type includes one or more of pedestrians and objects with inclined surfaces (such as tombstones), and the second target of the second preset type includes birds, insects, and grass one or more of .

It should be noted that for objects with sloping surfaces, the detection of radar devices may fail, which may cause the drone to hit objects with sloping surfaces, such as tombstones in farmland, thereby reducing the drone's flight safety issues.

For this reason, the visual sensor carried by the drone can be used to collect images. When it is recognized that the image includes an object with an inclined surface, according to the current flight information of the drone (flight height, flight speed and flight direction, etc.) A target control strategy is determined, and the UAV is controlled to perform corresponding operations according to the target control strategy. For example, the target control strategy may be to control the drone to stop, and the brake can specifically control the drone to hover at a certain position to avoid hitting the object.

For the target control strategy of an object with an inclined surface, the image including the object can also be sent to the control terminal of the drone (such as a mobile phone or FPV) for display, so that the operator can judge the drone according to the size of the object in the displayed image The distance from the target, the size of the object in the image is the number of pixels occupied by the object in the image, so it can be determined whether the drone is close to the object, and then the control terminal of the drone triggers the brake command, and the drone After receiving the stop command, the UAV is controlled to stop in response to the stop command. As a result, the flight safety of the drone can be improved.

It can be understood that the preset first type and the preset second type may also include other similar objects, which are not limited here.

In some embodiments, the preset type includes a preset first type, and target recognition is performed on the images collected by the visual sensor. According to the working status of the drone and the distance from the current drone to the first target, the target control strategy is determined, and the drone is controlled to perform corresponding operations according to the target control strategy to improve flight safety.

Exemplarily, for example, when it is recognized that the image includes a preset first type of first target, and the working state of the UAV is the take-off preparation state and the distance to the current first target is less than the first preset threshold , control the UAV to stop the propellers; or, when the working state of the UAV is the take-off preparation state and the distance to the current first target is greater than or equal to the first preset threshold, control the UAV to propel.

For example, when a pedestrian is detected during the take-off preparation phase, the human body frame is displayed on the display screen of the remote control terminal, and the distance of the pedestrian is obtained through a distance sensor (such as radar), and it is judged whether to stop the propeller according to the distance, and through the remote control terminal Make an alarm prompt. The alarm prompt can be realized by displaying the object category, etc., for example, a pedestrian is detected.

It should be noted that the current distance to the first target is the current distance from the UAV to the first target, which can be measured by a radar device, and of course can also be obtained by measuring a visual sensor.

Exemplarily, for another example, when it is recognized that the image includes a preset first type of first target, and the drone is in a landing preparation state and the current distance to the first target is greater than the second preset threshold, control the drone The plane landed.

Exemplarily, for another example, when it is recognized that the image includes a preset first type of first target, and the UAV is in a landing preparation state and the current distance to the first target is less than or equal to the second preset threshold, the control The drone hovers.

Exemplarily, for another example, when it is recognized that the image includes a preset first type of first target, and the UAV is in a landing preparation state and the current distance to the first target is less than or equal to the second preset threshold, the control The drone hovers and sends an alarm message to the drone's control terminal to prompt the user. The warning message is used to prompt the operator that the UAV is not suitable for landing at present.

Exemplarily, for another example, when it is recognized that the image includes a preset first type of first target, and the UAV is in a landing preparation state and the current distance to the first target is less than or equal to the second preset threshold, the control The drone flies in a direction away from the first target, and controls the drone to land until the distance is greater than a second preset threshold.

Exemplarily, for another example, when it is recognized that the image includes a preset first type of first target, and the UAV is in a low-altitude hovering state, control the UAV to hover at the current hovering position, or control the drone to hover at the current hovering position. The human-machine hovers at the current hovering position and sends an alarm message to the control terminal of the drone. Wherein, the low-altitude hovering state is that the hovering height is less than or equal to the preset hovering height.

It should be noted that the first preset threshold and the second preset threshold may be set according to actual applications, or may be set by a user, as long as the security requirements are met. Wherein, the first preset threshold and the second preset threshold may be the same or different.

Exemplarily, as shown in FIG. 5 , the warning information can be sent to the terminal device 201 of the control terminal 200 of the UAV 100 for display, so that the operating user can view the warning information and know the current situation of the UAV.

For example, the first target is the user, and the warning information is used to remind the user to stay away from the drone, so that the drone can be controlled to propel when the current distance to the first target is greater than or equal to the first preset threshold. Alternatively, the warning information is used to prompt the operator that the drone is not suitable for landing and so on. In this way, not only the flight safety of the drone can be improved, but also the experience of the operator can be improved.

In some embodiments, the preset type includes a preset second type, and the target recognition is performed on the image collected by the visual sensor. When the second target of the preset second type is recognized in the image, according to the In the preset operation state, the UAV is controlled to continue flying and perform preset operations according to the current flight route, wherein the preset operation includes at least one of the following: cruising operation and spraying operation. In this way, not only the flight safety of the drone can be improved, but also the operation efficiency can be improved.

In some embodiments, since the preset second type will not affect the flight safety of the UAV, the control method can also delete the target tracking track collected by the radar device carried by the UAV on the second target.

In some embodiments, in order to further improve the flight safety of the UAV, the UAV is controlled to perform corresponding operations according to the target control strategy. Specifically, the radar data collected by the radar device and the image collected by the visual sensor can be obtained, and the radar is fused. The data and images determine the obstacle information around the drone, and control the flight of the drone according to the obstacle information, thereby improving the safety of the drone's obstacle avoidance flight.

In an embodiment of the present application, the obstacle information includes at least one of the following: category information, distance information, and confidence information, the category information is used to indicate the category of the obstacle, the distance information is the distance from the drone to the obstacle, and the confidence The degree information is a probability for specifying the type of the obstacle.

In some embodiments, the radar data and images are fused to determine the obstacle information around the drone. Specifically, the obstacles around the drone and the category information and confidence information of the obstacles can be determined according to the image, and then determined according to the radar data. The distance information from the obstacle to the drone is fused with the obstacle category information and distance information to obtain the obstacle information of the obstacle.

Among them, according to the images, the obstacles existing around the drone and the category information and confidence information of the obstacles can be determined. Specifically, target detection can be performed on the image to obtain the first detection result, and multi-target tracking detection can be performed on the image to obtain the second detection result. , and then fuse the first detection result and the second detection result to obtain the obstacles around the UAV and the category information and confidence information of the obstacles. Through target detection and multi-target tracking detection, the category information and confidence information of obstacles can be identified more accurately to ensure the safety of UAV flight.

Specifically, the image is subjected to target detection to obtain the first detection result, and the image may be recognized based on a pre-trained target detection model and the category recognition frame and confidence information of the recognized obstacle are output, wherein the target detection model It is obtained based on convolutional neural network training.

Exemplarily, for example, as shown in Figure 6, the image is recognized based on the pre-trained target detection model, such as identifying pedestrians around the drone, and the target recognition frame of the pedestrian and the corresponding confidence information of 99% can be output, specifically The probability that the obstacle is a pedestrian is 99%. For example, as shown in 7, the image is recognized based on the pre-trained target detection model. For example, if a flock of birds appears around the drone, the target recognition frame of the flock of birds and the corresponding confidence information of 98% can be output, specifically indicating obstacles. The probability of being a flying bird is 98%.

It should be noted that the convolutional neural network is a type of neural network, and its neurons can respond to surrounding units within a part of the coverage area. It is suitable for image processing and contains a large number of convolution operations. Mathematically, a layer (a convolutional layer) in a convolutional neural network receives an activation tensor and then performs a convolution operation on it with the convolutional weights in the layer. In this way, the category information of obstacles and the corresponding confidence information can be accurately identified.

Specifically, the second detection result is obtained by performing multi-target tracking detection on the image, and the image can be recognized based on a pre-trained multi-target tracker to obtain the obstacle category identification frame and the obstacle's running track.

In some embodiments, the second detection result is obtained by performing multi-target tracking detection on the image. Specifically, the image features corresponding to the obstacles identified by the target detection model can be obtained, and based on the pre-trained multi-target tracker, according to the image The feature recognizes the image, and obtains the category recognition frame of the obstacle and the running track of the obstacle. Wherein, the image features include at least one of the following: color information and texture information. As a result, the type of obstacle can be identified quickly and accurately.

It should be noted that the movement trend of the target (obstacle) can be further judged according to the movement trajectory, and the target control strategy can be determined in combination with the working conditions of the UAV, so as to improve the flight safety of the UAV.

In some embodiments, in order to let the operator of the drone know the flight situation of the drone and avoid misoperation by the operator, the image collected by the visual sensor can also be sent to the control terminal of the drone for display, wherein, The image at least includes a category recognition frame of the obstacle.

Exemplarily, as shown in FIG. 8 , the drone 100 can send the image collected by the visual sensor to the terminal 201 (such as a mobile phone) on the control terminal 200 of the drone for display, wherein the displayed image includes at least obstacles For example, the probability of being a flying bird and being a flying bird is 98%, so that the operator can know the flight status of the drone and the information of the obstacles encountered.

In some embodiments, the drone 100 can also send the detection result of the image detection to the control terminal of the drone for display. Specifically, the detection result can be displayed in real time on the control interface displayed by the control terminal. The detection result The first detection result and/or the second detection result, or data fused with the first detection result and the second detection result, may be used to indicate the position of the target relative to the drone. This can prevent the operator from misoperation, so as to improve the flight safety of the drone.

In some embodiments, in order to improve the flight safety of the UAV and save the power of the UAV, when it is determined that the UAV is in the target flight state, the visual sensor carried by the UAV can be turned on to perform the wireless operation provided by the application. Man-machine control method, wherein the target flight state includes at least one of the take-off preparation state, landing preparation state, hovering state, flight state, and task execution state (such as spraying operations, etc.), and may not be turned on in other flight states Vision sensors and radar units for flight control. In turn, the flight safety of the UAV is improved, and the power of the UAV is saved at the same time.

In some embodiments, in order to improve the flight safety of the UAV while saving the power of the UAV, when collecting images through the visual sensor, the acquisition frame rate of the visual sensor can also be reduced to a preset frame rate, so as to The preset frame rate controls the vision sensor to collect images. The preset frame rate can determine that the collected images can be used to identify preset types of targets, and at the same time, it is lower than the normal frame rate of collection, thereby achieving the purpose of power saving.

The control method of the UAV provided by the above-mentioned embodiments can include preset types of targets in the image collected by the visual sensor in a scene that cannot be covered by the radar device, and then determine the corresponding target control strategy in combination with the working conditions of the UAV. Control the UAV, thereby providing the flight safety of the UAV, and at the same time improving the operating efficiency of the UAV.

Please refer to FIG. 9 . FIG. 9 shows a flow of steps of another method for controlling the flight of a drone provided by an embodiment of the present application.

As shown in FIG. 9 , the control method of the drone includes steps S201 to S205.

S201. Collect images through the visual sensor carried on the drone.

S202. When it is recognized that the image includes a preset type of target, determine a target control strategy according to the current working condition of the drone.

Among them, the target control strategy is to use the fusion data of the radar device and the visual sensor carried by the UAV to perform corresponding operations, such as flying for obstacle avoidance.

S203. Acquire radar data collected by the radar device and images collected by the vision sensor.

S204. Perform target detection on the image to obtain a first detection result, and perform multi-target tracking detection on the image to obtain a second detection result.

Specifically, based on a pre-trained target detection model, the image can be recognized to obtain a first detection result, wherein the first detection result includes the category recognition frame and confidence information of the recognized obstacle, and the target detection model is Based on convolutional neural network training.

Specifically, based on a pre-trained multi-target tracker, the image is recognized to obtain a second detection result, wherein the second detection result includes the category recognition frame of the recognized obstacle and the running track of the obstacle .

S205. Fuse the first detection result and the second detection result to obtain obstacle information of obstacles around the drone.

Wherein, the obstacle information includes at least one of the following: category information, distance information and confidence information.

S206. Control the drone to fly according to the obstacle information.

Exemplarily, the UAV can be controlled to perform obstacle avoidance flight according to the obstacle information, or the UAV can be controlled to stop propellers, or the UAV can be controlled to hover, etc.

The control method of the UAV provided by the above embodiment can, in a scene where the radar device carried by the UAV cannot cover, the image collected by the visual sensor includes a preset type of target, and then determine the corresponding target in combination with the working condition of the UAV. The target control strategy of the UAV is to determine the obstacle information around the UAV through two fusions, and then control the UAV according to the obstacle information, which can improve the flight safety of the UAV and improve the safety of the UAV. machine operating efficiency.

Please refer to FIG. 10 . FIG. 10 is a schematic block diagram of an unmanned aerial vehicle provided by an embodiment of the present application. As shown in FIG. 10 , the UAV 300 also includes at least one or more processors 301 , memory 302 , radar 303 and a camera 304 , and the camera 304 is a visual sensor.

The processor 301 may be, for example, a micro-controller unit (Micro-controller Unit, MCU), a central processing unit (Central Processing Unit, CPU), or a digital signal processor (Digital Signal Processor, DSP), etc.

The memory 302 can be a Flash chip, a read-only memory (ROM, Read-Only Memory) disk, an optical disk, a U disk, or a mobile hard disk.

The radar 303 is used to scan the operation area to obtain radar data around the drone, and the photographing device 304 is used to take images of the surrounding environment of the drone.

Wherein, the memory 302 is used to store a computer program; the processor 301 is used to execute the computer program and when executing the computer program, execute the control method of any one of the drones provided in the embodiments of the present application, so as to Improve the flight safety of drones.

Exemplarily, the processor is configured to execute the computer program and implement the following steps when executing the computer program:

The image is collected by the visual sensor carried on the UAV; when it is recognized that the image includes a preset type of target, the target control strategy is determined according to the current working condition of the UAV; according to the target control strategy control The drone performs corresponding operations.

In some embodiments, the preset type includes at least one of the following: a preset first type and a preset second type; wherein, the preset first type is a threat obstacle that affects drone flight safety, The second preset type is a non-threatening obstacle that does not affect the flight safety of the UAV.

In some embodiments, the preset first type of first target includes one or more of pedestrians and objects with inclined surfaces, and the preset second type of second target includes birds, insects, and thatch One or more of them, objects with inclined surfaces such as tombstones, etc.

In some embodiments, the preset type includes a preset first type, and the method further includes:

When it is recognized that the image includes a first target of the preset first type, target control is determined according to the current working state of the drone and the current distance from the drone to the first target Strategy.

In some embodiments, when the UAV is in a takeoff preparation state and the distance to the current first target is less than a first preset threshold, the UAV is controlled to stop propellers; or, the UAV is in takeoff In a ready state and when the distance to the current first target is greater than or equal to a first preset threshold, the UAV is controlled to stop propellers.

In some embodiments, when the UAV is in a landing preparation state and the distance to the current first target is greater than a second preset threshold, the UAV is controlled to land.

In some embodiments, when the UAV is in a landing preparation state and the distance to the current first target is less than or equal to a second preset threshold, the UAV is controlled to hover, or the UAV is controlled to hover. The drone hovers and sends a warning message to the control terminal of the drone to prompt the user, or controls the drone to fly in a direction away from the first target until the distance is greater than the second preset When the threshold is reached, the UAV is controlled to land.

In some embodiments, the UAV is in a low-altitude hovering state, and the UAV is controlled to hover at the current hovering position, or, the UAV is controlled to hover at the current hovering position and an alarm message is sent To the control terminal of the drone; wherein, the low-altitude hovering state is that the hovering height is less than or equal to the preset hovering height.

In some embodiments, the preset type includes a preset second type, and the processor is configured to: when identifying a second target of the preset second type in the image, according to the In the state of the preset job being executed, the UAV is controlled to continue flying and execute the preset job according to the current flight route.

In some embodiments, the processor is further configured to: delete the target tracking track collected by the radar device mounted on the drone with respect to the second target.

In some embodiments, the preset operation includes at least one of the following: cruising operation and spraying operation.

In some embodiments, controlling the UAV to perform corresponding operations according to the target control strategy includes:

Obtain the radar data collected by the radar device and the image collected by the visual sensor; fuse the radar data and the image to determine the obstacle information around the UAV, and control the UAV according to the obstacle information Man-machine flight.

In some embodiments, the obstacle information includes at least one of the following: category information, distance information, and confidence information; wherein the category information is used to indicate the category of the obstacle, and the distance information is the The distance from the aircraft to the obstacle, and the confidence information is the probability used to determine the category of the obstacle.

In some embodiments, the merging the radar data and the image to determine obstacle information around the UAV includes:

Determining the obstacles around the UAV and the category information and confidence information of the obstacles according to the image; determining the distance information from the obstacles to the UAV according to the radar data; fusing all the information The obstacle information of the obstacle is obtained from the category information and the distance information of the obstacle.

In some embodiments, the determination of the obstacles existing around the drone and the category information and confidence information of the obstacles according to the image includes:

Perform target detection on the image to obtain a first detection result; perform multi-target tracking detection on the image to obtain a second detection result; fuse the first detection result and the second detection result to obtain the surrounding area of the drone. The obstacle and the category information and confidence information of the obstacle.

In some embodiments, performing target detection on the image to obtain a first detection result includes:

Based on the pre-trained target detection model, the image is recognized, and the category recognition frame and confidence information of the recognized obstacle are output, wherein the target detection model is obtained based on convolutional neural network training.

In some embodiments, performing multi-target tracking detection on the image to obtain a second detection result includes:

Based on the pre-trained multi-target tracker, the image is recognized to obtain the category identification frame of the obstacle and the running track of the obstacle.

Obtain the image features corresponding to the obstacles identified by the target detection model; based on the pre-trained multi-target tracker, identify the image according to the image features, and obtain the category identification frame of the obstacle and the The trajectory of obstacles.

In some embodiments, the image features include at least one of the following: color information, texture information.

In some embodiments, the processor is further configured to: send the image collected by the visual sensor to the control terminal of the drone for display, wherein the image at least includes a category identification frame of an obstacle.

Embodiments of the present application also provide a computer-readable storage medium, the computer-readable storage medium stores a computer program, the computer program includes program instructions, and the processor executes the program instructions to implement the above implementation The steps of any one of the control methods of the unmanned aerial vehicle provided by the example.

Wherein, the computer-readable storage medium may be an internal storage unit of the drone described in any of the foregoing embodiments, such as the storage or internal memory of the drone. The computer-readable storage medium can also be an external storage device of the drone, such as a plug-in hard disk equipped on the drone, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, flash memory card (Flash Card), etc.

The above is only a specific embodiment of the application, but the scope of protection of the application is not limited thereto. Any person familiar with the technical field can easily think of various equivalents within the scope of the technology disclosed in the application. Modifications or replacements, these modifications or replacements shall be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

A control method for an unmanned aerial vehicle, characterized in that the method comprises:

Collect images through the visual sensor on the drone;

When it is recognized that the image includes a preset type of target, a target control strategy is determined according to the current working condition of the drone;

The UAV is controlled to perform corresponding operations according to the target control strategy.
The method according to claim 1, wherein the preset type includes at least one of the following: a preset first type and a preset second type;

Wherein, the preset first type is a threatening obstacle that affects the flying safety of the UAV, and the preset second type is a non-threatening obstacle that does not affect the flying safety of the UAV.
The method according to claim 2, wherein the first object of the preset first type includes one or more of pedestrians and objects with inclined surfaces, and the second object of the second preset type Targets include one or more of birds, insects and thatch.
The method according to claim 1, wherein the preset type includes a preset first type, and the method further includes:

When it is recognized that the image includes a first target of the preset first type, target control is determined according to the current working state of the drone and the current distance from the drone to the first target Strategy.
The method according to claim 4, characterized in that, when the UAV is in a take-off preparation state and the distance of the current first target is less than a first preset threshold, the UAV is controlled to stop propellers;

Alternatively, when the UAV is in a ready state for take-off and the distance to the current first target is greater than or equal to a first preset threshold, the UAV is controlled to propel.
The method according to claim 4, wherein the drone is controlled to land when the drone is in a landing preparation state and the distance to the current first target is greater than a second preset threshold.
The method according to claim 4, wherein the drone is controlled to hover when the drone is in a landing preparation state and the distance to the current first target is less than or equal to a second preset threshold, Or, control the UAV to hover and send a warning message to the control terminal of the UAV to prompt the user, or, control the UAV to fly in a direction away from the first target until the distance When it is greater than the second preset threshold, the drone is controlled to land.
The method according to claim 4, wherein the UAV is in a low-altitude hovering state, and the UAV is controlled to hover at the current hovering position, or, the UAV is controlled to hover at the current hovering position. Hover over the position and send an alarm message to the control terminal of the drone;

Wherein, the low-altitude hovering state is that the hovering height is less than or equal to the preset hovering height.
The method according to claim 1, wherein the preset type includes a preset second type, and the method comprises:

When it is recognized that the image includes a second target of the preset second type, according to the preset operation state performed by the drone, the drone is controlled to continue flying according to the current flight route and execute the preset Operation.
The method according to claim 9, characterized in that the method further comprises:

deleting the target tracking track of the second target collected by the radar device mounted on the drone.
The method according to claim 9, wherein the preset operation includes at least one of the following: cruising operation and spraying operation.
The method according to claim 1, wherein the controlling the UAV to perform corresponding operations according to the target control strategy includes:

acquiring radar data collected by the radar device and images collected by the visual sensor;

The radar data and the image are fused to determine obstacle information around the UAV, and the UAV is controlled to fly according to the obstacle information.
The method according to claim 12, wherein the obstacle information includes at least one of the following: category information, distance information, and confidence information;

Wherein, the category information is used to indicate the category of the obstacle, the distance information is the distance from the UAV to the obstacle, and the confidence level information is the probability for determining the category of the obstacle.
The method according to claim 12, wherein said fusing said radar data and said image to determine obstacle information around said drone comprises:

Determining obstacles existing around the drone and category information and confidence information of the obstacles according to the image;

determining distance information from the obstacle to the UAV according to the radar data;

The obstacle information of the obstacle is obtained by fusing the category information and distance information of the obstacle.
The method according to claim 14, wherein the determination of obstacles existing around the drone and category information and confidence information of the obstacles according to the image includes:

performing target detection on the image to obtain a first detection result;

performing multi-target tracking detection on the image to obtain a second detection result;

The first detection result and the second detection result are fused to obtain obstacles around the drone and category information and confidence information of the obstacles.
The method according to claim 15, wherein said performing target detection on said image to obtain a first detection result comprises:

Based on the pre-trained target detection model, the image is recognized, and the category recognition frame and confidence information of the recognized obstacle are output, wherein the target detection model is obtained based on convolutional neural network training.
The method according to claim 15, wherein said performing multi-target tracking detection on said image to obtain a second detection result comprises:

Based on the pre-trained multi-target tracker, the image is recognized to obtain the category identification frame of the obstacle and the running track of the obstacle.
The method according to claim 15, wherein said performing multi-target tracking detection on said image to obtain a second detection result comprises:

Obtain image features corresponding to obstacles identified by the target detection model;

Based on a pre-trained multi-target tracker, the image is identified according to the image features, and the category identification frame of the obstacle and the running track of the obstacle are obtained.
The method according to claim 18, wherein the image features include at least one of the following: color information, texture information.
The method according to claim 1, further comprising:

The image collected by the visual sensor is sent to the control terminal of the UAV for display, wherein the image at least includes an obstacle category identification frame.
An unmanned aerial vehicle, characterized in that the unmanned aerial vehicle includes a visual sensor, a radar device, a memory and a processor;

the vision sensor is used to capture images and the radar device uses radar data;

The memory is used to store computer programs;

The processor is configured to execute the computer program and implement the following steps when executing the computer program:

Collect images through the visual sensor on the drone;

When it is recognized that the image includes a preset type of target, a target control strategy is determined according to the current working condition of the drone;

The UAV is controlled to perform corresponding operations according to the target control strategy.
The drone according to claim 21, wherein the preset type includes at least one of the following: a preset first type and a preset second type;

Wherein, the preset first type is a threatening obstacle that affects the flying safety of the UAV, and the preset second type is a non-threatening obstacle that does not affect the flying safety of the UAV.
The unmanned aerial vehicle according to claim 22, wherein the first target of the preset first type includes one or more of pedestrians and objects with inclined surfaces, and the preset target of the second type The second target includes one or more of birds, insects and thatch.
The unmanned aerial vehicle according to claim 21, wherein the preset type includes a preset first type, and the processor is also used for:

When it is recognized that the image includes a first target of the preset first type, target control is determined according to the current working state of the drone and the current distance from the drone to the first target Strategy.
The unmanned aerial vehicle according to claim 24, characterized in that, when the unmanned aerial vehicle is in a take-off preparation state and the distance of the current first target is less than a first preset threshold, the unmanned aerial vehicle is controlled to stop propellers;

Alternatively, when the UAV is in a ready state for take-off and the distance to the current first target is greater than or equal to a first preset threshold, the UAV is controlled to propel.
The unmanned aerial vehicle according to claim 24, wherein the unmanned aerial vehicle is controlled to land when the unmanned aerial vehicle is in a landing preparation state and the distance to the current first target is greater than a second preset threshold.
The unmanned aerial vehicle according to claim 24, characterized in that, when the unmanned aerial vehicle is in a landing preparation state and the distance of the current first target is less than or equal to a second preset threshold, the unmanned aerial vehicle is controlled to hover Stop, or, control the drone to hover and send a warning message to the control terminal of the drone to prompt the user, or, control the drone to fly in a direction away from the first target until the When the distance is greater than the second preset threshold, the drone is controlled to land.
The unmanned aerial vehicle according to claim 24, wherein the unmanned aerial vehicle is in a low-altitude hovering state, and the unmanned aerial vehicle is controlled to hover at the current hovering position, or the unmanned aerial vehicle is controlled to hover At the current hovering position and send an alarm message to the control terminal of the drone;

Wherein, the low-altitude hovering state is that the hovering height is less than or equal to the preset hovering height.
The drone according to claim 21, wherein the preset type includes a preset second type, and the processor is used for:

When it is recognized that the image includes a second target of the preset second type, according to the preset operation state performed by the drone, the drone is controlled to continue flying according to the current flight route and execute the preset Operation.
The unmanned aerial vehicle according to claim 29, wherein the processor is also used for:

deleting the target tracking track of the second target collected by the radar device mounted on the drone.
The unmanned aerial vehicle according to claim 29, wherein the preset operation includes at least one of the following: cruising operation and spraying operation.
The unmanned aerial vehicle according to claim 21, wherein the controlling the unmanned aerial vehicle to perform corresponding operations according to the target control strategy includes:

acquiring radar data collected by the radar device and images collected by the visual sensor;

The radar data and the image are fused to determine obstacle information around the UAV, and the UAV is controlled to fly according to the obstacle information.
The unmanned aerial vehicle according to claim 32, wherein the obstacle information includes at least one of the following: category information, distance information and confidence information;

Wherein, the category information is used to indicate the category of the obstacle, the distance information is the distance from the UAV to the obstacle, and the confidence level information is the probability for determining the category of the obstacle.
The unmanned aerial vehicle according to claim 32, wherein said fusing said radar data and said image to determine obstacle information around said unmanned aerial vehicle comprises:

Determining obstacles existing around the drone and category information and confidence information of the obstacles according to the image;

determining distance information from the obstacle to the UAV according to the radar data;

The obstacle information of the obstacle is obtained by fusing the category information and distance information of the obstacle.
The unmanned aerial vehicle according to claim 34, wherein the determination of the obstacles existing around the unmanned aerial vehicle and the category information and confidence information of the obstacles according to the image includes:

performing target detection on the image to obtain a first detection result;

performing multi-target tracking detection on the image to obtain a second detection result;

The first detection result and the second detection result are fused to obtain obstacles around the drone and category information and confidence information of the obstacles.
The unmanned aerial vehicle according to claim 35, wherein said performing target detection on said image to obtain a first detection result comprises:

Based on the pre-trained target detection model, the image is recognized, and the category recognition frame and confidence information of the recognized obstacle are output, wherein the target detection model is obtained based on convolutional neural network training.
The unmanned aerial vehicle according to claim 35, wherein said performing multi-target tracking detection on said image to obtain a second detection result comprises:

Based on the pre-trained multi-target tracker, the image is recognized to obtain the category identification frame of the obstacle and the running track of the obstacle.
The unmanned aerial vehicle according to claim 35, wherein said performing multi-target tracking detection on said image to obtain a second detection result comprises:

Obtain image features corresponding to obstacles identified by the target detection model;

Based on a pre-trained multi-target tracker, the image is identified according to the image features, and the category identification frame of the obstacle and the running track of the obstacle are obtained.
The drone according to claim 38, wherein the image features include at least one of the following: color information, texture information.
The unmanned aerial vehicle according to claim 21, wherein the processor is also used for:

The image collected by the visual sensor is sent to the control terminal of the UAV for display, wherein the image at least includes an obstacle category identification frame.
A flight system, characterized in that the flight system comprises the unmanned aerial vehicle according to any one of claims 21-40 and a control terminal, and the user of the control terminal controls the flight of the unmanned aerial vehicle.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor realizes the following: The steps of the control method of the unmanned aerial vehicle.