DE102015224796A1 - Method and control unit for detecting a possible collision of an unmanned aerial vehicle with an object - Google Patents

Abstract

The invention relates to a method (900) for detecting a possible future collision (706) of an unmanned aerial vehicle (100) with an object (118), the method (900) comprising a step (902) of searching, a step (904) of Mapping and a step (906) of comparing. In the step (902) of the search, at least one object (118) in an image information (114) read by a camera (112) of the aircraft (100) is made using an optical feature (120) in an image sequence of at least two individual images (402 ) of the image information (114). In step (904) of assigning, at least object information (124) is assigned to the at least one object (118) using the optical feature (120). In step (906) of the comparing, the at least one object information (124) is compared with trajectory information (132) of the aircraft (100) to determine a possible future collision location (140) and a possible future collision time (142) of the possible future collision (706 ) to recognize.

Description

State of the art

The invention is based on a device or a method according to the preamble of the independent claims. The subject of the present invention is also a computer program.

In an unmanned aerial vehicle, which may also be referred to as a drone, in autonomous operation there is a risk that the aircraft collides with an object or obstacle which is in a planned trajectory of the aircraft or enters the trajectory.

Disclosure of the invention

Against this background, with the approach presented here, a method for detecting a possible future collision of an unmanned aerial vehicle with an object, furthermore a control device which uses this method, and finally a corresponding computer program according to the main claims are presented. The measures listed in the dependent claims advantageous refinements and improvements of the independent claim device are possible.

A method is provided for detecting a possible future collision of an unmanned aerial vehicle with an object, the method comprising the following steps:
Searching for at least one object in an image information read by a camera of the aircraft using an optical feature in an image sequence of at least two frames of the image information;
Associating at least one object information with the at least one object using the optical feature; and
Comparing the at least one object information with a trajectory information of the aircraft to detect a possible future collision location and a possible future collision time of the possible future collision.

A collision can be understood to mean an event in which the aircraft and the object come so close to each other that there is a high probability of contact between the aircraft and the object. In particular, a future potential collision can be detected if a safety distance between the aircraft and the object would fall below. Image information may be understood to be a data record of image data recorded by a camera of the aircraft, wherein the image information contains individual images at different acquisition times or an image sequence consisting of individual images, which images a movement of the object in the vicinity of the aircraft. An optical feature may be, for example, an optical flow, pattern or structure. The optical flow may be, for example, a difference vector or a set of difference vectors representing a displacement of corresponding pixels in the two frames each captured by the camera of the aircraft. Such mutually corresponding pixels in the two individual images can be, for example, corners or edges of objects imaged in the individual images, the position of the corners or edges in the first of the individual images being detected and compared with a position of these corners or edges in the second of the individual images in order to determine a difference vector from the different position of the corners or edges in the individual images. Object information may be a parameter that characterizes expansion and / or movement of the object. For example, such a parameter as object information may be a relative speed and / or a relative distance of the object with respect to the aircraft or may designate a size or extent of the object. Trajectory information may represent a planned future space requirement of the aircraft, for example in the form of an aircraft trajectory, which will most likely take the aircraft on its flight.

An image area of the image information may be recognized as an object if pixels in the image area have an optical mark equal to the optical feature within a tolerance range. For example, an optical tag may also be an optical flow, a pattern, or a structure. For example, an equality of the optical mark with the optical feature which is present within a tolerance range can be understood as meaning that the difference vectors of pixels in this region do not increase more than, for example, 20 percent, in particular not more than 10 percent, in their direction and their magnitude. specifically not to differ by more than 5 percent. For example, an object can be detected based on a substantially matching speed and direction of the pixels representing the object (for example in the image information).

The search for the object can also be done by other methods. The optical flow is a variant. Likewise, a classifier cascade or a pattern recognition, "structure from motion", which is suitable in particular for monocameras and / or a recognition of previously found, known and / or mapped objects be used. Under previously known, known and / or mapped objects mapped objects can be understood.

The optical flow results from a relative movement between a camera and imaged objects. It does not matter whether the camera moves towards the objects due to its own movement or whether the objects move towards the camera. Just as an operator of an aircraft can detect a potential obstacle or collision object due to optical flow between at least two images of the camera, a collision avoidance system may evaluate flow information about the optical flow to identify the obstacle or collision object.

If an extrapolated instantaneous trajectory violates a safety area around the obstacle or collision object, a possible future collision can be detected.

From the optical flow of the pixels representing the recognized object, speed information, position information and direction information of the object can be determined as object information. The optical flow is in this case, as already briefly outlined above, for example composed of displacement vectors having a length representing a velocity, a direction-representing orientation in space and an end point representing a position.

An image area recognized as an object can be compared with mapped objects to confirm the object. As a result, the approach presented here can be robust.

The speed information, the position information and the direction information of the object can be compared with an airspeed contained in the trajectory information, a vehicle position contained in the trajectory information and / or an aircraft direction included in the trajectory information, for example, to determine the possible future collision location and the possible future collision location detect future collision time. From the speed information, the position information and / or the direction information of the object, an expected movement trajectory of the object can be calculated or extrapolated. From the airspeed, the vehicle position and / or the direction of flight an expected flight trajectory can be calculated or extrapolated. By comparing the flight trajectory with the motion trajectory, the possible future collision can be detected.

Thus, to avoid the collision, a picture or video of a camera aboard the aircraft can thus be very simply sent to an operator or a control device in the aircraft, so that the operator or the control unit can correct the trajectory of the aircraft if necessary.

The method may include a step of receiving at least one further trajectory information of another aircraft. The further trajectory information may include or contain the aforementioned information or parameters relating to the further aircraft, which have already been mentioned for the trajectory information of the aircraft. In this case, furthermore, the trajectory information can be compared with the further trajectory information in order to detect the possible future collision location and the possible future collision time. The further trajectory information represents a further trajectory and may comprise a further airspeed, a further vehicle position and a further direction of flight of the further aircraft. The further flight trajectory can be compared with the flight trajectory to detect the possible future collision.

The trajectory information may be determined using the optical flow. In addition to the objects to be detected, the picture information of the camera also contains an optical flow resulting from a proper movement of the aircraft. For example, in this case, the optical flow may be evaluated with respect to pixels associated with objects associated with the ground, such as trees or buildings. It is then very easy to obtain the trajectory information of the aircraft itself on the basis of the evaluation of the "relative movement" of the aircraft with these objects connected to the ground, which can not move by the fixed connection with the ground itself and therefore the relative movement by a movement of the Aircraft results against these objects. In particular, a direction of flight may correspond to a direction to a point of convergence of displacement vectors of an image background, in particular where such a point of convergence may represent, for example, the tree or the building.

The method may include a step of providing the trajectory information. The own trajectory information can be provided, for example, for another aircraft or a central coordination point. Thus, by an appropriate disclosure of the trajectory information to the other or an additional aircraft a possible conflict situation already be prevented before the other aircraft or an additional aircraft enters the detection range of the camera.

The method may include a step of determining an alternative trajectory for the aircraft using the trajectory information, the object information, the potential future collision location, and the possible future collision time. The aircraft may be controlled to the alternative trajectory to prevent the potential future collision. The trajectory or planned trajectory can be changed until a safety distance between the object and the aircraft is maintained and thereby the alternative trajectory is determined.

The method may include a warning step of providing a warning signal about the potential future collision for the object. A warning signal can be understood, for example, as an acoustic signal or an optical signal. In particular, a warning sound may be provided if the object is recognized as a bird. For example, the warning signal may be emitted or disembarked directly from the aircraft, for example, to scare off a bird flying on the aircraft.

This method can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a control unit.

The approach presented here also provides a control device for detecting a possible future collision of an unmanned aerial vehicle with an object, wherein the control device is designed to perform the steps of a variant of a method presented here in corresponding facilities to drive or implement. Also by this embodiment of the invention in the form of a control device, the object underlying the invention can be achieved quickly and efficiently.

In the present case, a control device can be understood as meaning an electrical device which processes sensor signals and outputs control and / or data signals in dependence thereon. The control unit may have an interface, which may be formed in hardware and / or software. In the case of a hardware-based embodiment, the interfaces can be part of a so-called system ASIC, for example, which contains various functions of the control unit. However, it is also possible that the interfaces are their own integrated circuits or at least partially consist of discrete components. In a software training, the interfaces may be software modules that are present, for example, on a microcontroller in addition to other software modules.

Furthermore, a collision avoidance system for an unmanned aerial vehicle is presented, the collision avoidance system having the following features:
a camera for providing image information;
a controller according to the approach presented here; and
an avoidance device for influencing the trajectory of the aircraft, in particular in response to a control signal provided by the control device.

Also of advantage is a computer program product or computer program with program code which can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and for carrying out, implementing and / or controlling the steps of the method according to one of the embodiments described above is used, especially when the program product or program is executed on a computer or a device.

Embodiments of the invention are illustrated in the drawings and explained in more detail in the following description. It shows:

1 a representation of an unmanned aerial vehicle with a control unit for detecting a possible future collision according to an embodiment;

2 an overview of a collision avoidance system according to an embodiment;

3 a representation of an unmanned aerial vehicle with a collision avoidance system according to an embodiment;

4 a representation of a sequence of steps for detecting a possible future collision according to an embodiment;

5 a representation of a modified trajectory to avoid a collision according to an embodiment;

6 a representation of an unmanned aerial vehicle before a possible future collision with a bird;

7 a representation of an autonomous flight of an unmanned aerial vehicle with a A collision avoidance system according to an embodiment;

8th an illustration of expansion stages of a collision avoidance system for an unmanned aerial vehicle according to an embodiment; and

9 a flowchart of a method for detecting a possible future collision according to an embodiment.

In the following description of favorable embodiments of the present invention, the same or similar reference numerals are used for the elements shown in the various figures and similar acting, wherein a repeated description of these elements is omitted.

1 shows a representation of an unmanned aerial vehicle 100 with a control unit 102 for detecting a possible future collision according to an embodiment. The aircraft 100 can as a drone 100 be designated. Here is the aircraft 100 a quadrocopter 100 , The control unit 102 has a search facility 104 , an allocation device 106 and a comparator 108 on. The control unit 102 is part of a collision avoidance system 110 (which in the following description synonymous as an eMSS system 110 , eMSS = expandable Modular Safety System, engl. expandable modular safety system) of the aircraft 100 , The control unit 102 can as a collision detection system 102 be designated. The aircraft 100 has at least one camera 112 for providing image information 114 on. The camera 112 is in a main flight direction 116 of the aircraft 100 aligned and forms one in front of the aircraft 100 lying environment of the aircraft 100 in the picture information 114 from. The picture information 114 consists of a sequence of frames that give a video or moving image of the environment.

The camera 112 forms an example here an object 118 in the picture information 114 from. In a relative movement between the camera 112 and the imaged object 118 It is displayed in the individual images in different places and with a different image size. In other words, in each case a displacement vector results between image coordinates, in each case one feature of the object in the individual images 118 is shown. An entirety of the displacement vectors is considered a variant of an optical feature 120 referred to, which can also be referred to herein as optical flow. The search for the object can also be done by other methods. The use of the optical flow is just a variant.

Likewise, a classifier cascade or pattern recognition, "structure from motion", which is particularly suitable for monocameras and / or recognition of mapped objects can be used. All of these exemplarily used approaches or combinations thereof are to be understood by the designation of an optical feature, wherein in the following description the variant using the optical flow is used.

The search facility 104 is designed to be the object 118 or objects in the image information 114 to search. The search engine reads this 104 the picture information 114 and the optical flow 120 one. It becomes an image area 122 the picture information 114 as the object 118 detected when pixels in the image area 122 a similar optical flux 120 exhibit. For example, the displacement vectors may have a similar length. Likewise, the displacement vectors may have a common vanishing point. The allocation device 106 is designed using the optical flow 120 Property information 124 to the object 118 or assign to the objects. This will be a speed information 126 , a position information 128 and direction information 130 of the object 118 from the optical river 120 the, the recognized object 118 determined pixels. The comparison device 108 is designed to be the object information 124 with a trajectory information 132 of the aircraft 100 to compare. Thereby the speed information becomes 126 , the position information 128 and the direction information 130 of the object 118 at an airspeed 134 , a vehicle position 136 and a direction of flight 138 of the aircraft 100 compared to a possible future collision location 140 and a possible future collision time 142 to recognize the possible future collision.

The object information 124 and the trajectory information 132 are extrapolated. A possible future collision can then be detected if the extrapolated object information 124 and the extrapolated trajectory information 132 indicate a future fall below a safety margin.

The possible future collision location 140 and the possible future collision time 142 be sent to an alternate device 144 of the collision avoidance system 110 forwarded. The evasive device 144 is designed using trajectory information 132 , of possible future collision location 140 and the possible future collision time 142 a future trajectory of the aircraft 100 to influence to avoid the collision.

2 shows an overview of a collision avoidance system 110 according to an embodiment. The collision avoidance system 110 corresponds essentially to the collision avoidance system in 1 , Here is the camera 112 an RGB signal 114 ready. Should it be with the camera 112 can not be an RGB camera, it can be at the signal 114 also act to a signal that is not designed as an RGB signal. The search facility 104 and the allocator 106 for object recognition and object classification are summarized here. As in 1 become the speed information 126 , the position information 128 and the direction information 130 for the comparison device 108 provided. In the comparison device 108 be this information with the airspeed 134 , the vehicle position 136 and the direction of flight 138 the trajectory information 132 of the aircraft 100 compared to the possible future collision location 140 and the possible future collision time 142 to obtain. Here is the backup device 144 designed to directly setpoint speeds 200 the rotors of the quadrocopter 100 to affect the quadrocopter 100 to dissuade his collision course. For this purpose a change of the flight trajectory and a speed change of the rotors are calculated by an algorithm.

In one embodiment, the aircraft 100 a basic sensor system 202 on, which is an actual location information 204 about an attitude of the aircraft 100 for the comparison device 108 provides. In this case, the position information includes 204 a position, orientation and speed of the aircraft 100 , The basic sensors 202 may include, for example, radar, ultrasound and / or infrared.

The location information 204 may also represent a position, orientation, and speed of the object.

The location information 204 is in the comparator 108 used to the possible future collision location 140 and the possible future collision time 142 to determine.

In other words, in the comparator 108 an algorithm for matching the object information 124 from the camera with the sensor data 204 executed. This will be the trajectory of the aircraft 100 and motion data of the object, wherein the motion data once about the object information 124 and once about the sensor data 204 be matched. The trajectory of the aircraft 100 is checked for speed, position, direction and position. The object is also checked for speed, position, direction and / or position. It will be a time-to-collision 142 and a position-of-collision 140 calculated.

In one embodiment, the aircraft 100 a map module 206 on. The map module 206 is trained geographic data 208 for the search facility 104 and the allocator 106 provide. For example, the card module receives 206 Card data over a wireless data connection, such as cellular. Using the geographic data 208 becomes the picture information 114 for detecting the environment of the aircraft 100 used. In this case, for example, an inverse perspective can be generated which corresponds to an equalized view of the environment.

In one embodiment, in the search engine 104 and the allocator 106 carried out a three-dimensional measurement of the detected objects, from which geographic data of the environment can be generated again. 3 shows a representation of an unmanned aerial vehicle 100 with a collision avoidance system 110 according to an embodiment. The aircraft 100 or the drone 100 is like in the 1 and 2 a rotorcraft, especially a multicopter 100 , The aircraft 100 can also be a fixed-wing aircraft. In a first stage, the drone points 100 here a sensor 300 to capture the environment. The sensor 300 like in the 1 and 2 a camera 112 be. Likewise, the sensor 300 be a laser sensor.

In an expanded stage, the drone points 100 a radar 302 which is also designed to capture the environment. This shows the radar 302 a greater range than the sensor 300 on. The radar 302 is also independent of viewing conditions, as the radar waves can penetrate, for example, fog, clouds and smoke.

To capture a ground, the drone points 100 here are two downward ground detection sensors 304 on. The ground detection sensors 304 here are ultrasonic sensors 304 , To a spatial location of the drone 100 to detect are the ultrasonic sensors 304 on a fuselage of the drone 100 arranged diametrically opposite.

The drone 100 indicates in a further expansion stage a wireless transponder 306 on, who is communicating with a ground station 308 (which in the following description synonymous as a central coordinator, information service or Cloud zone coordinator can be called) maintains. The ground station monitors the airspace to detect potential conflicts between the drone 100 and other aircraft to detect and resolve. This is provided by a flight control module 310 the drone 100 over the transponder 306 scheduled flight information available. From the ground station 308 then clearances for the airspace are granted or withheld. Thus, a potential for conflict between several autonomous drones is greatly reduced.

In a further expansion stage, the drone points 100 a transponder 312 to communicate from drone to drone. About the transponder 312 Scheduled flight information is also provided and scheduled flight information received from other drones. The flight control module 310 Detects future conflicts between the involved drones and adjusts a planned flight path accordingly. Likewise, the other drones can adjust their flight paths when the drone 100 for example, has assigned a higher priority.

In the picture is the eMSS system 110 represented with possible components. The five advanced levels are marked differently.

4 FIG. 12 is an illustration of a sequence of steps for detecting a potential future collision in accordance with one embodiment. FIG. In this illustration is an aircraft 100 with a directional camera 112 , as in 1 shown. By a relative movement between the aircraft 100 and objects 118 in a detection area 400 the camera 112 results in an optical flow 120 between frames of the camera 112 ,

An example is a single image 402 with the optical flow 120 representing displacement vectors 404 shown. The displacement vectors 404 are here for a single moving object 118 in the single picture 402 shown. The object 118 Here is an illustrative road vehicle. The displacement vectors 404 of the object are similar. The displacement vectors 404 of the object 118 represent the optical flow 120 of the object 118 between the displayed single picture 402 and a previous frame. In this case, the displacement vectors 404 all have a similar length, which is a speed of the object 118 represents. Furthermore, the displacement vectors converge 404 all in a common vanishing point. The object 118 can therefore be recognized as an area in which the optical flow 120 is similar.

In a planned flight trajectory of the aircraft 100 are a tree 406 and a human 408 arranged. Analogous to the single picture shown 402 become the tree 406 and the human 408 by evaluating the optical flow 120 as objects 118 detected and a possible future collision location and a possible future collision time determined. To avoid the potential collision, approaches are appealing to the determining 410 designed for collision avoidance. These approaches 410 are already being used in road traffic. Here are the approaches 410 expanded by the third dimension or spatial direction, which offers a variety of ways to avoid the future collision.

The approach presented here enables autonomous flights without the driver being able to intervene. It becomes a redundant security system via camera 112 and basic sensors presented. For example, applications that are flying close to people 408 require as delivery drones enabled.

It will be a combination of approaches and driver assistance solutions with an unmanned aerial vehicle 100 proposed. For this purpose, known algorithms can be extended from driver assistance and new algorithms can be added. In 4 is the result of the applied driver assistance algorithm for collision avoidance on the road for implementation on the unmanned aerial vehicle 100 shown.

It schematically illustrates the function of the collision avoidance system for unmanned aerial vehicles. The copter 100 is with a camera 112 equipped, which receives a camera image in the direction of flight. Since most unmanned aerial vehicles are already equipped with cameras, for example for aerial photography or the first person view, ie images from the perspective of the flying object for the pilot, there are no additional costs for the sensors of the collision avoidance system.

In this case, an evaluation of the optical flow takes place 120 for object recognition. This is an approach from the driver assistance, for use in the quadrocopter 100 is adjusted.

As shown here on the lower left and upper right, pixels move with fast approaching objects 118 from a picture 402 to the next also fast, represented by the long vectors 404 , For distant objects 118 as they approach slowly, the pixels move slowly, so the algorithm produces short vectors 404 calculated. From this approach can objects 118 recognized and their distance or the Time-to-collision can be calculated. These data are taken with the tractors of the Kopter 100 compared. If a collision is predicted, a controller can change the trajectory of the unmanned aerial vehicle and directly access the speed controls of the rotors which directly affect airspeed and heading.

In an extension is the unmanned aerial vehicle 100 equipped with further basic sensors, such as radar, ultrasound, infrared and inertial sensors. The collision avoidance system can access all sensor signals. A reconciliation or fusion of sensor data is performed, providing a robust and low-dead-time estimation of the position, extent and state of motion of the detected objects 118 allows.

In one embodiment, the unmanned aerial vehicle 100 a 360 ° camera 112 on. The evaluation of the camera image always takes place at least in the direction of flight or in the direction of the known flight trajectory of the unmanned aerial vehicle 100 , The camera 112 can also be aligned according to the currently prevailing gravity. In this way, a simple attitude estimation of the unmanned aerial vehicle 100 possible in the world.

The evaluation of the camera image, that is, the calculation of the optical flow and the algorithm for object recognition including comparison with other sensor data and the change of the flight trajectory can be done on a system on chip. The collision avoidance system is thus dependent on the sensor set-up and is inexpensive.

5 shows a representation of a changed trajectory 500 to avoid a collision according to an embodiment. The representation essentially corresponds to the illustration in FIG 4 , The tree 406 is within a tolerance of a planned trajectory 502 or planned flight trajectory 502 , Using the approach presented here, the tree 406 and the human 408 as objects 118 been recognized. With the tree 406 threatens a possible future collision. Therefore, there is a change in the flight trajectory 502 in the area of the tree 406 amended. Here describes the changed trajectory 500 a bow over the tree 406 ,

Standing objects 406 can be detected by comparing the expected optical flux with the measured optical flux. This is possible knowing the position and the movement as well as the profile of the landscape. This information can be found in detailed 3-D maps. In a simple form enough already the rough altitude of the landscape. If contiguous image regions in which the measured flow differs in a comparable manner from the expected flow, the image region is considered as an object candidate further.

Calculation of 3-D information from a sequence of monaural images is possible using a technique such as "structure-from-motion". This can be an obstacle 406 roughly measured in 3-D. On the basis of the obstacle measurement is then again planning a 3-D evasion trajectory 500 possible.

Other criticality measures such as the time-to-brake or how much time the aircraft 100 is still available to avoid the collision by slowing down in the air, or the time-to-steer or how much time the aircraft 100 still available to go through a steering intervention 500 to avoid the collision will be charged. Falling below thresholds to the aforementioned Critikalitätsmaße is a situation with risk of collision. The unmanned aerial vehicle 100 can then slow down, stay in the air or its trajectory 502 change autonomously. With vertical wings it is also possible to stand still. Depending on the classified object type, the evasive maneuver may 500 be adjusted accordingly. For curious birds may sound a warning tone.

6 shows a representation of an unmanned aerial vehicle 100 before a possible future collision with a bird 600 , The bird 600 flies in the area of a planned trajectory 502 of the aircraft 100 , By comparing the planned trajectory 502 with an air trajectory 602 of the bird 600 it is recognized that a probability of collision decreases as the bird 600 in front of the aircraft 100 dodging. Dodge can be assisted by emitting an audible warning. For example, a warning call from a bird of prey can be played to the bird 600 to scare away.

When there is a moving, dynamic object 600 in the trajectory 502 of the unmanned aerial vehicle 100 is, the object becomes 600 detected as dynamic over the optical flow and the "time-to-collision" or the possible future collision time is calculated. Here, a so-called focus-of-expansion can be used. In the case of a moving camera, all stationary objects generate flux vectors that emerge from a point in the image, the focus of expansion. The flow vectors of dynamic objects 600 show by the object movement 602 a direction deviating from the focus of expansion. A threshold on the thus calculable directional error is used to detect dynamic objects 600 ,

Likewise, the algorithm can calculate if an object is 600 out of the trajectory 502 moved out of the unmanned aerial vehicle and thus no collision takes place. The condition of the unmanned aerial vehicle, that is its speed, attitude and trajectory are therefore passed to the algorithm. The unmanned aerial vehicle therefore requires sensor data of these sizes.

7 shows a representation of an autonomous flight of an unmanned aerial vehicle 100 with a collision avoidance system according to an embodiment. There is an available airspace in cloud zones 700 . 702 divided. Every cloud zone 700 . 702 is by a central coordinator 308 supervised. The task of the central coordinator 308 is the surveillance of the airspace. This can be aircraft 100 . 704 be monitored, at least the flight information for the central coordinator 308 provide. It is an early collision detection between flying aircraft 100 . 704 possible. These are releases for the flight in a cloud zone 700 . 702 granted. If a possible collision 706 Recognition becomes evasion trajectories 500 calculated for collision avoidance.

Here flies the first aircraft 100 in the first cloud zone 700 and shares his flight plan with the responsible cloud zone coordinator 308 , The second aircraft 704 wants in the first cloud zone 700 fly in and ask for permission. For the second aircraft shares 704 his flight plan with the responsible cloud zone coordinator 308 , The cloud zone coordinator 308 detects a possible collision 706 between both aircraft 100 . 704 , The cloud zone coordinator 308 transmits to the second aircraft 704 a new flight route 500 to the collision with the first aircraft 100 to avoid. The first aircraft 100 flies on undisturbed and recognizes an unexpected object 318 with the sensors on board. The first aircraft 100 calculates a new flight route 500 , Then the first aircraft flies 100 continue into the second cloud zone 702 and shares his flight plan with the new competent central coordinator 308 , The concept with the "cloud zones" 700 . 702 resembles the radio cells of a mobile network.

8th shows a conceptual diagram of expansion stages of a collision avoidance system 110 for an unmanned aerial vehicle according to an embodiment. The collision avoidance system 110 builds on an automated detection 800 a possible future collision with an object, such as in the 4 to 6 is described.

It becomes a system 110 which ensures a collision-free flight for unmanned aerial vehicles. The approach presented here ensures that no collision with other unmanned aerial vehicles and stationary or moving objects can occur. The system 110 uses object detection or cloud communication components such as sensors, a camera, object recognition algorithms, airspace monitoring services, and software that monitors airspace and resolves conflicts between unmanned aerial vehicles.

A description is given of the module levels of the eMSS system for early detection and avoidance of collisions.

In a first stage of development 800 can the collision avoidance system 110 as a Non Co-Operative Sensor System 800 be designated. Here, collisions are calculated by object recognition and distance measurement between unmanned aerial vehicle and obstacle. There is no communication with the obstacle. The avoidance trajectory is selected without an agreement with the other object on the trajectory, which could lead to a collision despite avoidance trajectory. Typical sensors for this are RGB-D cameras, ultrasonic sensors or lasers. Use cases include, for example, the monitoring of agricultural fields, indoor mapping and firefighting. The system is advantageous when the airspace is limited and known.

In a second expansion stage 802 can the collision avoidance system 110 as Cloud Based Traffic Control System 802 be designated. Here, the pre-programmed trajectory is shared with a central coordinator who issues the permission to fly for a specific airspace. If an unmanned aerial vehicle does not get permission, it waits at the airspace boundary until the flight path inside the airspace is clear again. Due to the waiting time, the flight times are extended and thus the range shortened. In one embodiment, the aircraft has a transponder for communication with the central coordinator. This second stage of development 802 can be used to monitor bridges, to monitor a factory site or to inspect oil platforms.

In a third stage of development 804 can the collision avoidance system 110 as Cloud Solves Conflict Situations in Real Time 804 be designated. Here, the pre-programmed trajectory is shared with a central coordinator, the central coordinator calculates in real time the evasion trajectories and transmits them to the affected aircraft in collision conflict, which can immediately implement the evasive maneuver. There are no waiting times here. A central coordinator required. The third expansion stage 804 can be delivered within Localities, to monitor construction sites or to monitor crowds of people.

In a fourth expansion stage 806 can the collision avoidance system 110 as UAV-to-UAV communication 806 be designated. In the event that the aircraft enters the cloud zone into an uncontrolled airspace, such as an area without a cloud coordinator, such as delivery docks in remote areas or sea rescue, an enhanced or modified approach is required to calculate conflicts between multiple flying objects. Here the pre-programmed trajectory is shared with other participants in the airspace. Subsequently, an agreement is made between the aircraft on the evasion trajectory. The fourth expansion stage 806 Can be used for delivery in remote locations or to monitor livestock.

In a fifth expansion stage 808 can the collision avoidance system 110 as radar / ACAS 808 be designated. In the last stage of construction 808 is a system for integrating the aircraft into civil airspace, where even manned aircraft fly, into the collision avoidance system 110 integrated. It is to be expected that unmanned aircraft will always have to avoid manned aircraft. For an installation of either a radar for the detection of other aircraft and their speed makes sense because the Doppler lane very quickly the type of aircraft and thus the most important information that is, the size can be detected. Alternatively, an ACAS system can be installed. The fifth expansion stage 808 can be used to monitor the border and to provide Internet in remote locations.

Since unmanned aerial vehicles can be used in very different applications, not all applications or applications have the same requirements for the system 110 for collision avoidance. Therefore, here is a comprehensive and modular security platform 110 proposed. At the same time, with the increasing complexity of the task that an unmanned aerial vehicle is required to perform, the complexity and range of the security system, which can be incorporated into aircraft and assist the aircraft during flight, increases.

If an unmanned aerial vehicle is used, for example, to determine fertilizer and water requirements on agricultural fields, then the requirements for the collision avoidance system must be met 110 rather low, because the airspace is limited and controllable. In other words, the number of flight participants is known and can be influenced by the property owner according to current legislation, as he can allow or reject flights over his property. In addition, possible obstacles in the trajectory or stationary objects such as power pylons or trees are generally known. There is a risk of collision due to unexpected flying objects, for example birds. Under such conditions is a system 110 for collision avoidance, which is based on sensors such as cameras, ultrasonic sensors or laser, sufficient. The sensors assume the object recognition and distance measurement. However, it can not be ensured that new collision conflicts are to be expected due to the avoidance trajectory since the trajectory of the unexpected object is unknown. Nevertheless, the risk of collisions is due to the "non-comparative sensor system" 800 considerably reduced. Here, therefore, is the exemplary application "Precision Farming" of the lowest module level 800 which requires only "non-cooperative" sensor systems and the least complexity of the security system 110 requires.

However, if an unmanned aircraft is used, for example, for parcel delivery, then the requirements for the system 110 increase. As an example, here is the use case "delivery" of the 3rd module stage 804 assigned to "cloud solves conflict situtations". An unmanned aerial vehicle should in this case fly within localities, that is, with overflight of humans. In addition, other flight participants, like other unmanned aerial vehicles expected in the airspace, which can not be known in advance. It is also expected that the structure within the airspace will constantly change, for example by adding buildings in construction areas. In order to meet these new challenges, in addition to the module 800 also a system 110 needed to solve the potential for conflict with other flight participants 100% risk-free. Since the flight plan is known from an unmanned aerial vehicle in advance, the flight plan can be shared with a central coordinator who can calculate and resolve potential conflicts in advance. That is, permission to fly and enter a defined airspace is only granted if the desired trajectory is free of other unmanned aerial vehicles. This system 110 is more reliable than the sensor-based system 800 and extends the subordinate module levels 800 . 802 ,

Both examples justify the benefits and advantages of a modular system 110 for collision avoidance for unmanned aerial vehicles. In the system presented here 110 Each level can contain the previous system level.

In other words, a security platform for unmanned aerial vehicles is presented. The security platform includes security features such as collision avoidance, reliable sensors and fail-safe components.

9 shows a flowchart of a method 900 for detecting a possible future collision according to an embodiment. The procedure 900 may be on an aircraft control unit, such as in 1 is shown executed. The procedure 900 has a step 902 of searching, a step 904 of assigning and a step 906 of comparing. In step 902 of searching, at least one object in an image information read by a camera of the aircraft is searched for. In this case, the at least one object is searched for using at least two individual images of the image information using an optical flow in an image sequence. In step 904 of assigning, at least one object information is assigned to the at least one object using the optical flow. In step 906 comparing the at least one object information with a trajectory information of the aircraft is compared to detect a possible future collision location and a possible future collision time of the possible future collision.

An automated collision avoidance for an unmanned aerial vehicle (UAV) is presented.

Unmanned aerial vehicles, such as quadrocopters, multicopters or UAVs (Unmanned Aerial Vehicles) will often be used in the future, as they have a very wide range of possible applications. When operating the aircraft safety, security and privacy are important, so that no collisions and crashes, no abuse of the drone and the protection of privacy are guaranteed.

Currently, for example, in Germany it is permitted to fly unmanned aerial vehicles only outside built-up areas, not over persons, and not autonomously, but only in visual contact and with the possibility of intervention for the driver. In addition, commercial experience and an ascent approval are required for commercial purposes. This is partly due to the fact that commercially available unmanned aircraft could fly autonomously, but have no concept for avoiding collisions.

The safety of unmanned aerial vehicles can be improved by using distance sensors such as infrared sensors, radar or ultrasound.

Conventional autopilot systems work on a detected object via the transmission of warning signals to the pilot and are thus dependent on the correct reaction of the pilot.

In order to meet future safety requirements, unmanned aerial vehicles can be equipped with the concept described here for automated collision avoidance.

The approach presented here comprises a camera-based concept for collision avoidance, based on the evaluation of a camera image by image processing algorithms. Almost all unmanned aircraft have already installed a camera or have a gimbal or a suspension for attaching cameras.

Based on a sequence of camera images, the optical flux can be calculated. This can be used to estimate the time to collision (time-to-collision) and to separate dynamic objects from the background.

Furthermore, an object classification can be made. In particular, specifically using the optical flow, objects classified as self-moving objects, such as birds, other unmanned aerial vehicles, hot-air balloons or paragliders can be classified.

A calculation of the inverse perspective or a so-called bird's eye view can take place if a height and position of the camera or of the aircraft is known. This allows a metric, distortion-free top view of the environment to be generated, which can be used in conjunction with a map for localization.

Further, using structure-from-motion or motion stereo, a three-dimensional extent of objects can be determined.

The aforementioned image processing algorithms can be calculated online using an embedded system (system on chip) on board the unmanned aerial vehicle and used for object movement detection.

The use of such embedded systems, as used for example in smartphones and tablets, resulting in low costs, since the processors have low unit costs, have low power loss and have a low weight. There is no active cooling necessary and the embedded systems can be powered from the battery of the unmanned aerial vehicle.

The approach presented here combines approaches from driver assistance, such as the evaluation of a camera image, the object recognition and the collision avoidance with a new technology, the unmanned aerial vehicle. However, the peculiarities of the unmanned aerial vehicle platform make it possible to use a number of algorithms that are not widely available or possible in driver assistance. In particular, in the case of a flying object, a calculation of a three-dimensional collision-avoiding trajectory takes place.

When applied to an unmanned aerial vehicle, the criticality measures known from driver assistance are extended to assess the situation. In particular, new aircraft-specific movement models are created. New dimensions are added, for example, an upward deflection occurs by applying the maximum voltage to the rotor motors. Dodging down, however, occurs by stopping or severely slowing down the rotors without the unmanned aerial vehicle falling.

The approach presented here allows for the autonomous flying of drones, as the collision avoidance for unmanned aerial vehicles adds a safety aspect that was previously not taken into account. Moving and unforeseen objects within the trajectory are detected and the collision avoided.

If an exemplary embodiment comprises a "and / or" link between a first feature and a second feature, then this is to be read so that the embodiment according to one embodiment, both the first feature and the second feature and according to another embodiment either only first feature or only the second feature.

Claims (14)

Procedure ( 900 ) for recognizing a possible future collision ( 706 ) of an unmanned aerial vehicle ( 100 ) with an object ( 118 ), the process ( 900 ) has the following steps: Search ( 902 ) of at least one object ( 118 ) in one, from a camera ( 112 ) of the aircraft ( 100 ) read image information ( 114 ) using an optical feature ( 120 ) in an image sequence of at least two individual images ( 402 ) of the image information ( 114 ); Assign ( 904 ) of at least one object information ( 124 ) to the at least one object ( 118 ) using the optical feature ( 120 ); and comparing ( 906 ) of the at least one object information ( 124 ) with trajectory information ( 132 ) of the aircraft ( 100 ) to a possible future collision location ( 140 ) and a possible future collision time ( 142 ) of the possible future collision ( 706 ) to recognize. Procedure ( 900 ) according to claim 1, wherein in step ( 902 ) of searching an image area ( 122 ) of the image information ( 114 ) as an object ( 118 ) is detected when pixels in the image area ( 122 ) have an optical characteristic equal to the optical feature and / or the optical flow within a tolerance range. Procedure ( 900 ) according to one of the preceding claims, wherein in step ( 902 ) of searching as an object ( 118 ) recognized image area ( 122 ) is compared with previously found, known and / or mapped objects to the object ( 118 ) to confirm. Procedure ( 900 ) according to one of the preceding claims, wherein in step ( 904 ) of the assignment from the optical feature ( 120 ) of the pixels that the detected object ( 118 ), as object information ( 124 ) a speed information ( 126 ), position information ( 128 ) and / or direction information ( 130 ) of the object ( 118 ) be determined. Procedure ( 900 ) according to claim 4, wherein in step ( 906 ) of comparing the speed information ( 126 ), the position information ( 128 ) and / or the direction information ( 130 ) of the object ( 118 ) with one in the trajectory information ( 132 ) contained airspeed ( 134 ), one in the trajectory information ( 132 ) vehicle position ( 136 ) and / or in the trajectory information ( 132 ) contained flight direction ( 138 ) of the aircraft ( 100 ) are compared to the possible future collision location ( 140 ) and the possible future collision time ( 142 ) to recognize. Procedure ( 900 ) according to any one of the preceding claims, comprising a step of receiving at least one further trajectory information of another aircraft ( 704 ), wherein in step ( 906 ) of comparison further the trajectory information ( 132 ) is compared with the further trajectory information to determine the possible future collision location ( 140 ) and the possible future collision time ( 142 ) to recognize. Procedure ( 900 ) according to one of the preceding claims, wherein in step ( 902 ) of searching the trajectory information ( 132 ) using the optical feature ( 120 ) is determined. Procedure ( 900 ) according to one of the preceding claims, comprising a step of providing the trajectory information ( 132 ). Procedure ( 900 ) according to one of the preceding claims, with a step of determining an alternative trajectory ( 500 ) for the aircraft ( 100 ) using the trajectory information ( 132 ), the object information ( 124 ), the possible future collision location ( 140 ) and the possible future collision time ( 142 ), the aircraft ( 100 ) on the alternative trajectory ( 500 ) to control the possible future collision ( 706 ) to prevent. Procedure ( 900 ) according to one of the preceding claims, with a step of warning, in which a warning signal about the possible future collision ( 706 ) for the object ( 118 ) provided. Control unit ( 102 ) for recognizing a possible future collision ( 706 ) of an unmanned aerial vehicle ( 100 ) with an object ( 118 ), whereby the control unit ( 102 ), the procedure ( 900 ) according to one of the preceding claims. Collision avoidance system ( 110 ) for an unmanned aerial vehicle ( 100 ), the collision avoidance system ( 110 ) has the following features: a camera ( 112 ) for providing image information ( 114 ); a control device ( 102) according to claim 11; and an evasive device ( 144 ) for influencing the trajectory ( 502 ) of the aircraft ( 100 ). Computer program adapted to perform the procedure ( 900 ) according to one of the preceding claims. A machine readable storage medium storing the computer program of claim 13.

Images