DE102022118915A1 - Autonomous flying robot with external cloud-based data processing - Google Patents

Abstract

The present invention relates to an arrangement for improving the autonomy of a flying robot, comprising a flying robot (2) with a camera system (21) for recording image data (BD) and/or image/depth data (BTD), a computing unit (22) for obtaining environmental information and/or odometry information from the image data and/or image/depth data (BTD) and possibly further data, as well as for generating flight control commands (FSK) taking into account the obstacle information, odometry information and possibly further information, a transceiver (23) for sending a Part or all of the image data and/or the image information and/or the odometry information to an external performance computer (3) for carrying out more complex data processing tasks, comprising further processing of the image data and/or environmental information and/or odometry information in order to calculate it based on situation and navigation specifications and pass it on to the flying robot (2) via a transceiver (33) of the performance computer (3), the computing unit (22) taking the situation into account for the navigation specifications when generating the flight control commands (FSK). The invention further relates to a corresponding method for improving the autonomy of a flying robot and a logistics method using the arrangement and/or application of the method.

Description

The present invention relates to an arrangement and a method for improving the autonomy of a compact flying robot, comprising a flying robot with a camera system for acquiring image data, a computing unit for carrying out simple data processing tasks, a transceiver for sending data or information to an external data processing platform, which can carry out more complex ones Data processing tasks are used to support the flying robot in situation detection and navigation. The invention further relates to a logistics method using the arrangement and application of the method according to the invention.

Autonomous flying robots, also known as drones, in the small to medium payload range of a few hundred grams to a few kilograms have been proposed and used for several years to transport goods, particularly time-critical goods. Such goods, such as important letters or medication, were traditionally sent by courier in order to reach their destination quickly. For technical reasons, flying robots capable of hovering are more suitable for short distances of a few kilometers, for example inner-city routes. However, drones with fixed wings can also be used on longer routes, for example overland routes of a few tens of kilometers or more.

Flying robots have so far only been used outdoors. Examples of the use of flying robots for outdoor transport can be found in the writings, for example US 9,567,081 B1 , US 9,650,136 B1 , US 2014/0217230 A1 , and US 10,730,621 B2 . Outdoor delivery of goods is also in the patent applications DE 10 2019 108256 .0, and DE 10 2021 130652.3 addressed by the applicant.

The reasons for this are, on the one hand, that the dimensions of previously known transport drones are unsuitable for safely moving through or entering a building. In order to fly through the doors designed for people or to move through rooms and corridors with sufficient safety distance on all sides, the spans / diameters of the propeller circle zones must be in the range of a maximum of 25 - 40cm. When used in warehouses with larger gates and clear aisles, larger drones in the size range of 60cm propeller circle diameter could also be used. With the size and the maximum tolerable turbulence caused by the propellers, the maximum flight mass is also limited to one to a few kilograms and the payload to one hundred to a few hundred grams.

On the other hand, previous delivery drones do not have sufficiently precise sensors to be able to navigate within a building. Finding your way around a building requires much more complex route planning and navigation than flying outside. Whereas in the latter, in the simple case, you only have to cover a straight route from A to B, and in more complicated cases you only have to fly through a series of waypoints, which is possible, for example, with the help of satellite-based navigation alone or in combination with inertial navigation, requires navigation by a building, the determination of the position on a plan or map (of the floors) of the building, which contains the relative position and size of the corridors, rooms and doors or other openings connecting them. When navigating through unknown buildings, simultaneous mapping and positioning (SLAM, simultaneous localization and mapping) must be carried out. In any case, a building-accessible delivery drone must also be capable of much more sophisticated obstacle detection and avoidance, since, unlike most cases in outdoor use, it cannot be swerved in any direction. In some cases, intended routes may become impassable for the drone due to obstacles, so the route through the building must be discarded and an alternative route must be found.

Obstacles are preferably also recorded on the map of the building required or to be created for navigation, which includes the position, size and relative position of the individual rooms, corridors and openings connecting them. Such cards require a comparatively large memory requirement, which exceeds the memory usually available in the flight controls of known flying robots. Furthermore, route planning on such a map is comparatively computationally intensive, which poses major problems for an autonomously navigating flying robot, especially in the case of a necessary ad-hoc re-planning of the route, and based on the data usually used on autonomous flying robots, which are suitable for building flight indicated above size and flight mass, existing computing capacity would lead to too great a delay for smooth and safe navigation through a building. The simplest solution of increasing the computing power available on board the drone is disadvantageous because it increases the energy requirement and thus leads to either a reduced payload or a higher take-off mass and may not be practical within the above-mentioned flight mass limits.

• Die Veröffentlichungsschrift einer US Patentanmeldung US 2019/0072979 A1 stellt ein System und ein Verfahren zur gemeinsamen Operation zwischen einem autonomen mobilen Roboter, insbesondere einer flugfähigen Drohne und einer intelligenten Umgebung, in welcher über ein Kommunikationsnetzwerk zur Verfügung gestellte Cloud-Dienste dem mobilen Roboter erlauben, im Inneren eines Gebäudes zu navigieren um etwa Güter abzuholen oder -liefern. Die verfügbaren Cloud-Dienste können einen Karteninformationensdienst, von dem der mobile Roboter Karten des Gebäude erfragen kann, und einen Gebäudeinfrastrukturdienst, der Auskunft über Abhol- oder Abladepunkte sowie Kundenpräferenzen geben kann, umfassen. Eine Unterscheidung der auf einem mobilen Roboter beim sicheren und zielgerichteten Bewegen innerhalb eines Gebäudes nötigen Datenverarbeitungsaufgaben nach Komplexitätsgrad und die Auslagerung komplexerer Datenverarbeitungsaufgaben an externe Datenverarbeitungsplattformen an die Cloud-Dienste wird dort aber nicht vorgeschlagen. Die im dort betrachteten Stand der Technik als bekannt bezeichnete zentrale Steuerung des mobilen Roboters wird eher im Gegenteil als zu unflexibel bei sich änderenden Umständen verworfen.

Flying robots intended for building flights are described in the following documents:

• The published specification of a US patent application US 2019/0072979 A1 presents a system and a method for joint operation between an autonomous mobile robot, in particular a flight-capable drone, and an intelligent environment in which cloud services provided via a communication network allow the mobile robot to navigate inside a building in order to pick up goods, for example or deliver. The available cloud services may include a map information service from which the mobile robot can query maps of the building, and a building infrastructure service which can provide information about pick-up or drop-off points as well as customer preferences. However, a distinction between the data processing tasks required on a mobile robot for safe and targeted movement within a building according to the level of complexity and the outsourcing of more complex data processing tasks to external data processing platforms to the cloud services is not suggested there. On the contrary, the central control of the mobile robot, which is described as known in the prior art considered there, is rejected as being too inflexible when circumstances change.

The international publication WO 2014/197067 A1 relates to a modular drone with a navigation unit that enables the drone to navigate within a warehouse, an RFID reader and a wireless transceiver via which the drone can receive route information provided by an external data processing infrastructure or a human pilot. The drone then navigates along the route to a destination using the navigation unit, reading RFID tags along the way and exchanging location information via the transmitter receiver. The drone operates completely autonomously and carries out all data processing tasks necessary for navigating the department store on-board.

In the document WO 2018/067544 A2 a system comprising a drone for flight within buildings, in particular warehouses, is proposed, in which a drone communicates with an external computer system which stores map and route information, which can be recorded by the drone by mapping the environment. The drone includes a navigation module for dynamic navigation and obstacle avoidance, with sonar, radar, image analysis of visual information, thermal images, orientation to signal beacons, laser altimeter and GPS proposed as sensors for environmental detection. However, outsourcing more complex data processing processes to the computer system is not considered.

The above-described, known flying robots designed for building flights have the problem of only being able to transport comparatively small payloads over only moderate distances, since the sensor and control components on board are extensive and correspondingly massive and therefore also have a high energy requirement.

Due to their associated complexity, these flying robots are also comparatively expensive to purchase.

There is therefore still the problem of enabling freight transport flying robots of suitable size and flight mass to safely and smoothly autonomously navigate within buildings. At the same time, these flying robots should be as cheap as possible to produce. It is the object of the present invention to provide such flying robots and navigation methods for them.

This task is solved by an arrangement according to claim 1, a method according to claim 8 and a use of the arrangement and application of the method according to claim 10.

The flying robot according to the arrangement according to the invention is here as main sensor system at least with a camera system comprising at least one camera for capturing image data, a first computing unit for processing the image data to obtain environmental or obstacle information and / or odometry information and a transmitter receiver for data exchange with an external data processing platform fitted. This external data processing platform is used to carry out more complex data processing tasks, the requirements of which would exceed the computing capacity available on the flying robot, which is necessarily compact for navigation within buildings.

The external data processing platform can be a dedicated computing device in the form of a performance computer, such as a stationary PC, a workstation or even a mainframe computer, which at least has a CPU with high computing capacity, i.e. 100 gigaflops/s or more, preferably 1 teraflop/s. s or more. The external performance computer can also be mobile, for example a portable notebook or laptop with appropriate computing capacity. Alternatively A larger autonomous flying robot, which is only designed for external use, but accompanies the smaller, building-accessible flying robot as far as possible, especially on the outdoor flight route, can also take on the role of the external data processing platform as a kind of mother ship.

In alternative embodiments of the invention, the data processing platform is not a single, physical performance computer, but rather a network of connected computers, known under the term “cloud”, to which any data and data processing processes can be outsourced. By running dedicated software, such a cloud is prepared to support the flying robot as described above and in more detail below. Such a cloud-based data processing platform can also be referred to as a virtual performance computer.

Regardless of whether the external data processing platform is implemented as a physical or virtual performance computer, data is exchanged between it and the flying robot via radio using its own transmission protocol, such as WiFi, ZigBee, Bluetooth or Z-Wave. No direct communication between the flying robot and the external data processing platform is necessary; instead, intermediate relay stations, such as WiFi routers or extenders, can be integrated into the communication path. In the context of the present invention, the only important thing is that data exchange with sufficiently low latency is possible between the flying robot and the external power computer. The latency should not exceed one hundred milliseconds, preferably 50 milliseconds, although this is also easily possible with the interposition of data forwarding stations.

The flying robot, the arrangement according to the invention, has suitable drives and control surfaces to control its orientation, direction of flight and flight speed and has means and devices to accommodate the payload to be transported and to hold it safely during the flight. For the latter, for example, a loading bay, a loading box or a transport fork can be used.

Furthermore, the flying robot has sensors for detecting the environment and preferably also for independently detecting the flight attitude, direction and speed, as well as data processing means for generating flight control commands from the recorded data, which determine the orientation, flight speed and flight direction of the flying robot during the flight . For this purpose, the flying robot can have a computing unit with at least one CPU. This CPU preferably has a low maximum power consumption of 15 watts or less or at least only an average maximum power consumption of between 15 to 30 watts, with the performance-specific computing power being 100 Mflops / W (megaflops per watt, 1 Mflops = 2^20 floating point operations per second ) or more, better at 500 megaflops per watt or more, or at 100 DMIPS or more, preferably 600 DMIPS or more (1 DMIPS = 1757 Dhrystone integer operations per second).

The advantage of the arrangement of the present invention is that, for the first time, under the given restrictions in terms of available computing capacity per watt and processor weight, it enables the implementation of a flying robot that can move safely and purposefully within buildings. This was not possible with the flying robots known from the prior art, which carry out all data processing tasks themselves, i.e. on-board. The flying robot according to the present invention can save weight and performance by outsourcing computing-intensive processes to the external data processing platform in the form of a dedicated, physical or a virtual, cloud-based computer and thus provide a high payload capacity despite the compact design necessary for building flight.

Preferred developments of the present invention, which can be implemented individually or in combination, provided they are not mutually exclusive, are set out in the dependent claims and will be presented individually below.

The flying robot preferably has a flight control or flight control unit, which converts the control commands generated by the computing unit into movements of the control surfaces and the drives of the flying robot. The CPU contained in the flight controller preferably has a low power consumption as defined above.

Both the computing unit and the flight control unit can use computer architectures other than the classic von Neumann, for example neuromorphic or hybrid architectures.

In order to keep the power requirements of the flying robot's sensors and data processing low, sensors and computing modules with the lowest possible power consumption are used. However, it should be noted that the data processing speed is sufficient for the flight robot with on-board, d. H are the internal data processing tasks. According to the invention, these data processing tasks preferably include, on the one hand, the execution of the drivers required by the individual components, as well as the processing of the signals detected by the sensors, as well as the regulation and control of the actuators of the flying robot as basic functions, which in all versions are carried out by the flying robot itself (on- board).

As somewhat higher functions, in some embodiments of the invention the flying robot also carries out obstacle and simple situation detection, which serves to quickly recognize obstacles that are recognized at short notice or that occur in the previously calculated flight path and to initiate an evasive and/or braking maneuver to avoid collisions. However, in other embodiments, particularly in the case of particularly compact and lightweight delivery drones according to the invention, these functions are also carried out by the external data processing platform. These embodiments are particularly suitable in the event that there is a low latency radio connection, approximately 10 ms or less, between the drone and the data processing platform.

According to the invention, more complex data processing tasks are outsourced to the external data processing platform. These further data processing tasks include, on the one hand, more detailed situation detection, which, for example, goes beyond the mere recognition of the existence of an obstacle and its approximate position and dimensions to a more precise recognition of the obstacle in terms of size, position, direction of movement and type (stationary, mobile, fragile/vulnerable, etc .) contains. Navigation also falls under this, at least when flying inside buildings / in building flight mode.

The task here can be to determine a current position on an already known map based on the image data recorded by the flying robot and/or obstacle and environmental information obtained, as well as the odometry information obtained or recorded by other sensors present on the aircraft, such as initial measuring units within the building. Alternatively, when flying through an unknown building, the surrounding information can also be obtained from the image data recorded by the flying robot. as well as the odometry information mentioned, the building is mapped, i.e. a map of the size and relative location/position of rooms and corridors, their windows and doors as well as obstacles is created. When mapping, the position on it is determined at the same time as the map of the building is created in a process called SLAM (simultaneous localization and mapping).

As a subtask of navigation, the external data processing platform can also calculate a preferred route from the current position of the flying robot to the desired destination within the building and quickly adapt this route to changing circumstances, such as blocked paths or locked doors. Detecting whether an obstacle that the flying robot has encountered can be flown around or makes the planned route impossible is also the task of route planning on the external data processing platform.

The significantly higher computing power of the external data processing platform according to the invention compared to the data processing means of the flying robot is predestined for this comparatively computationally intensive process. This then transmits situation and navigation information or specifications to the flying robot via radio, which the flying robot uses to generate suitable flight control commands to fly along the route calculated by the data processing platform.

Artificial intelligence methods, procedures and algorithms can be used for both navigation and mapping.

The external data processing platform preferably uses data that was entered when the delivery mission to be carried out by the flying robot was commissioned, as well as other data obtained from external sources.

The requests or inputs as part of the order can include the pickup point for the goods to be transported in a starting building, a destination point in a destination building, which can also be the starting building, a preferred route between the buildings, and, if applicable, delivery times and/or delivery mission priorities. In addition, when the mission is commissioned, the necessary information for finding the way within the respective buildings, for example maps of the buildings or individual floors with or without marking suitable entry or exit openings, can be provided. Alternatively, the external data processing platform can also be prepared to retrieve such information from other available sources such as the Internet or a cloud.

The external data processing platform also allows data from missions to be stored, for example the path of a mission from one specific start to a specific target building as well as the route or routes followed within the buildings and the time required for this. This enables a learning process between missions to optimize the flight route followed.

So that the technology used on the flying robot can remain small and light, in some embodiments it is provided that only the most necessary data processing tasks are on-board, i.e. H can be carried out in real time by the flying robot itself, without recourse to the external data processing platform. More complex data processing processes are therefore outsourced to the external data processing platform.

A safety-relevant function is immediate collision avoidance. To do this, the flying robot must be able to detect obstacles on the flight path and avoid them.

For safety reasons, this function is preferably not outsourced, but rather carried out onboard by the flying robot using its own data processing means. However, if a stable data connection with low latency is guaranteed between the drone and the external data processing platform, the image data analysis necessary to avoid collisions can also be carried out by the external data processing platform.

Obstacle detection is always carried out by evaluating the image data captured by the camera system using a suitable image analysis algorithm. In the case of onboard collision avoidance, this is carried out by the computing unit of the flying robot. Alternatively or additionally, dedicated image recognition hardware can also be used on the drone or the data processing platform, either in the form of an ASIC (application specific integrated circuit), which physically implements the image processing algorithm, or a correspondingly programmed FPGA (field programmable gate array), or a neuromorphic one Type of computer that implements image recognition based on neural networks directly as hardware. Such hardware can also be present in the external performance computer as an alternative or in addition to other existing freely programmable (general purpose) CPUs.

In the simplest case, the camera system of the flying robot only uses a single camera pointing in the usual direction of flight of the flying robot. However, with a single camera, depth information cannot be extracted directly from the image data, which makes the detection of image objects and in particular the measurement of the distance to detected objects more difficult. In principle, it is possible to calculate the distance to an object and its apparent change in size as it approaches or departs, taking into account the known flight speed. Alternatively or additionally, wave-shaped flight movements, in which closer objects experience greater relative changes in direction than more distant objects, can also allow or facilitate distance determination.

However, this assumes that the flight speed or trajectory is reliably determined in another way and/or that the absolute size of the object(s) is known with sufficient accuracy.

The former can be achieved in the arrangement according to the invention through the interaction of the flying robot and the external data processing platform by carrying out a more precise image or object recognition on the data processing platform, identifying objects present in the image and determining their size based on previously known values or values specified on external sources becomes. This information can then be used by the external data processing platform to determine the flight speed, and generally also the direction of flight, from the apparent change in size of the various objects detected in the image. If an impending collision is detected, the flying robot is instructed to change its trajectory accordingly. A flying robot that is autonomous in this way could also be referred to as a flying robot remotely controlled by an AI.

The disadvantage here is that these basic parameters of flight speed and direction are then only known to the flying robot with a latency essentially given by the data connection between the performance computer and the flying robot. In order to be able to reliably avoid or stop in a timely manner in front of an obstacle, the flying robot itself is preferably able to determine a rough, conservative approximation of these important flight parameters in a short time. With a pure image analysis of the image data recorded by a camera, this is not practical due to the processing capacity available on the flying robot.

In preferred embodiments of the invention, at least one depth camera is therefore used in the camera system of the flying robot. This depth camera, which is preferably integrated into the flying robot pointing in the usual direction of flight, allows depth data to be recorded directly, which enables an estimate of the distance of at least objects in the closer area, up to distances of a few meters. Then you can the existing distance and a detected apparent change in object size can also be determined directly by the flying robot, the flight speed or the direction of flight. When using just one camera, the orientation or pose of the flying robot results from the image captured by this camera and the positions of the surrounding objects that can be recognized from this.

In addition to a forward-facing depth camera, further cameras are present in further preferred embodiments. For example, it is advantageous to integrate at least one rear-facing camera, in particular a rear-facing stereo camera, into the flying robot. This allows flight direction and speed to be determined based on both approaching objects in the field of view of the front depth camera and receding objects in the field of vision of the rear-facing stereo camera. This achieves greater accuracy when determining the direction and speed of flight. The direction of flight can also be reversed without having to rotate around the vertical axis.

Even when using stereo and/or depth cameras, the camera system of the flying robot according to the invention can be implemented inexpensively, easily and with low power consumption. This is a significant advantage over the laser scanner (LI DAR) that is commonly used in drones. Although these have the advantage of being able to quickly capture a very accurate 3D image of the environment, they are comparatively large, expensive and have quite high energy consumption due to the fact that they are active sensors and scan the environment with an active laser beam. Through the exclusive use of (passive) cameras in the flying robot of the arrangement according to the invention, a significant reduction in size, weight reduction and reduction in the power consumption of the main navigation sensor system can be achieved. In order to maximize these advantages, preferred embodiments of the invention therefore provide flying robots that completely forego the use of laser scanners and work exclusively with cameras to detect the environment.

Another important aspect of navigation within a building is the determination of the distance traveled and, in general, taking into account the directional information, the route or trajectory traveled. This will be summarized here under the term odometry. Odometry in this sense can be carried out by the flying robot itself based on the obstacle and environmental information obtained by the computing unit. As described above, the flight speed and direction of flight are first determined from the image data captured by the camera(s) and these are then integrated over time in the course of the flight in order to determine the distance traveled and trajectory. If only a single (mono) camera is used, this is only possible using the external data processing platform due to the low computing capacity. If the flying robot itself should have the ability to obtain odometry data, at least the use of a stereo or depth camera is necessary so that, as described above, the flying robot can obtain odometry data based on the depth data of the image data from its comparatively low data processing capabilities .

In further preferred embodiments, the flying robot also has one or more inertial measurement units (IMU), which represent an independent sensor system for obtaining information about flight direction and speed. In combination with the information about flight direction and speed obtained from image data, this results in greater accuracy and advantageous redundancy. In order to further improve the accuracy in position determination and the determination of flight direction and speed, it is particularly proposed to provide the flying robot with two or more initial measuring units.

The computing unit of the flying robot can have a CPU with low power consumption (less than 15 W) and/or a CPU with medium power consumption (15 -30 watts). Alternatively, both types of CPU can be present in the computing unit, or a flying robot is used that has two computing units, one with a CPU of medium power consumption. Processors with full-load power consumption of over 30 watts are referred to here as high-performance processors (HP-CPU).

The method according to the invention for improving the autonomy of a compact flying robot includes, as essential features, the acquisition of image data by a camera system of the flying robot, the acquisition of obstacle and environmental information and / or odometry information from the image data and, if necessary, further data provided by other sensors Computing unit of the flying robot, sending part or all of the image data and/or obstacle and environmental information and/or odometry information to an external data processing platform by means of a transceiver, which further or more in-depth processing of the data and/or information received from the flying robot for the calculation of situation and navigation information and specifications or instructions, sending the situation and navigation specifications to the flying robot using a transceiver of the external data processing platform, relating to the generation of flight control commands the orientation, flight direction and flight speed of the flying robot from the environment and obstacle information and/or odometry information determined by the flying robot itself, as well as based on the situation and navigation specifications received from the external performance computer.

In preferred embodiments of the method according to the invention, navigation takes place in different modes, depending on whether the flying robot is inside or outside a building. In the outdoor area, navigation along waypoints or on a direct line from a starting building to a target building is used in a generally known manner. The position determination can be carried out using satellite navigation data, supplemented or replaced by inertial navigation. This outdoor or free flight mode generally does not require any recourse to the processing of image data determined by the camera system. In this respect, the flying robot can complete this section of the mission fully autonomously, even without data contact with the external performance computer. However, the capture and processing of the image data recorded by the camera system of the flying robot by the computing unit of the flying robot are preferably also active in free flight mode in order to at least ensure obstacle detection and collision avoidance. However, if there is a data connection to the external performance computer outside, navigation can also be carried out in free flight mode using environmental information obtained from the image data.

The drone of the arrangement according to the present invention finds the entry point into the target building either by setting an (end) waypoint of the outdoor area route at the corresponding point on the outer shell of the building, or by opening the entry window when it reaches the last waypoint near the building and restoring the data connection to the external data processing platform. Alternatively, within the scope of the present invention, it is also possible to equip the flying robot with the ability to systematically search the target building for suitable entry points.

When flying into the building, the drone automatically positions itself in the middle based on the side boundary of the entry point (a window, door or gate) in order to have the highest possible tolerance for errors and turbulence-related fluctuations during entry. Central positioning in both vertical and horizontal directions is also preferred when flying through rooms and/or corridors of the building. This positioning is only used to avoid obstacles or collisions.

Alternatively, the flying robot can also be prepared for this, or can switch to this in the course of a mission, for example based on the corresponding navigation specifications of the data processing platform, instead of a central positioning in the vertical and / or horizontal respect and also adopt a different positioning. For example, the external line computer can instruct the flying robot to fly closer to the ceiling of a room or hallway to reduce the likelihood of collisions with people walking around a building. The clear width in the upper area of corridors or rooms is also larger on average, as furniture is usually arranged in the lower area of the room. When flying through rooms or corridors with known obstacles in the ceiling area, the flying robot can position itself in the middle or in the lower area based on the instructions from the performance computer.

Stereo cameras, depth cameras, initial measuring units or transponder systems, for example UWB transponder systems, can be used to detect the entry opening into the target building.

An advantage of the method according to the invention and the use of the arrangement according to the invention is that when the data processing platform is accessed, the flight route and positions of the flying robot can be tracked worldwide, which means coordination of several systems as well as real-time adjustment of the functional parameters in the event of a change in the weather, cancellation of the delivery or makes something like that possible.

• Zunächst wird die Drohne am Startpunkt mit dem Transportgut, beispielsweise einem Medikament oder anderem medizinischen Gut oder einem Brief, beladen. Der Zielort kann dem Fluggerät anschließend durch bereits extern eingegebene Informationsdaten mitgeteilt werden, oder es erfolgt eine Eingabe durch einen Bediener über ein Computerterminal. Die Beladung der Drohne kann durch einen Belader oder automatisch erfolgen. Anschließend erfolgte eine grobe Routenplanung durch die externe Datenverarbeitungsplattform, dem die Drohne in der Zwischenzeit, sofern das Missionsprofil nicht bereits von der externen Datenverarbeitungsplattform heruntergeladen worden war, den Zielpunkt mitgeteilt hat. Der Bediener vor Ort oder ein Bediener der externen Datenverarbeitungsplattform gibt der Drohne sodann das Startkommando.

The present invention also includes a use of the arrangement according to the invention as part of a logistics process for delivering a transport item from a starting building to a room or to a destination point of a target building, which includes the following steps:

• First, the drone is loaded at the starting point with the transported goods, for example a medication or other medical item or a letter. The destination can then be given to the aircraft externally entered information data is communicated, or an entry is made by an operator via a computer terminal. The drone can be loaded by a loader or automatically. Rough route planning was then carried out by the external data processing platform, to which the drone communicated the destination point in the meantime, provided the mission profile had not already been downloaded from the external data processing platform. The on-site operator or an operator on the external data processing platform then gives the drone the take-off command.

The drone then starts and looks for a way out through the building, for example through an open window or a gate or door. The drone can either already know the route to the starting point when it flew in if it flew into the starting building itself from outside, or it could have been programmed. If the route from the starting room to a known excursion site is not known, the drone can also explore this route in an exploration mode according to the navigation in an unknown target building. Various exploration strategies can be used here. Publicly available knowledge about the starting building is also preferably included. For example, the drone can also have a priori knowledge of the starting room and its location relative to the building shell of the starting building, which helps with orientation and navigation in the building. The map of the building generated by the drone during an exploration is saved on the external data processing platform and is available there when the mission is repeated, either to fly the same route again or to try to optimize by finding a faster, alternative route.

As soon as the drone is outdoors, you can switch to free flight mode. The drone uses the known coordinates of the target building and flies over a sequence of waypoints previously determined by the external performance computer, which takes into account obstacles and/or flight route restrictions and danger areas in the area, such as wind turbines or flight or take-off/landing areas Airplanes or helicopters or larger delivery drones.

Once at the target building, the drone locates an entry opening and flies to it. The entry opening can either be known in advance or the drone can look for a suitable one based on known patterns. It is also in data connection with the external data processing platform, which supports the drone through available à priori knowledge about suitable and usually existing openings. When flying in, the drone positions itself in the middle based on the side boundaries of the entry point. From the entry point, it either navigates to the destination using a route determined on a known map of the building by the external data processing platform, or it explores the destination in the building using mapping and simultaneous positioning until a path to the destination point is found. The identification of the target point can either be done via a known relative position of the target point in comparison to the building shell, it is simply done via image recognition and identification of essential features of the target room, for example a room number or a special color scheme.

Further properties, advantages and features of the present invention will now be explained below using preferred exemplary embodiments with reference to the figures. These are only intended to illustrate the invention and not limit it in any way.

• 1: Eine schematische Darstellung der erfindungsgemäßen Anordnung gemäß einer ersten bevorzugten Ausführungsform.
• 2: Eine schematische Darstellung der erfindungsgemäßen Anordnung gemäß einer zweiten bevorzugten Ausführungsform.

Show it:

• 1 : A schematic representation of the arrangement according to the invention according to a first preferred embodiment.
• 2 : A schematic representation of the arrangement according to the invention according to a second preferred embodiment.

In 1 the schematic representation of a first preferred embodiment of an arrangement according to the invention consisting of a flying robot and an external data processing platform is shown.

The flying robot or drone 2 includes a camera system 21 and one or more inertial measuring units 25 as sensors. The camera system 21 includes a depth camera 211 directed to the front, in the direction of flight, and a stereo camera 212 directed to the rear, i.e., directed against the usual direction of flight. The image depth data BTD generated by the depth camera 211 and the image data BD generated by the rear stereo camera are passed on to the computing unit 22 comprising a CPU 221 with medium power consumption. In the context of the present invention, average power consumption means a maximum power consumption of between 15 and 30 watts.

The computing unit evaluates the image/depth data and image data BTD, BD received from the camera system 21 as well as that from the inertial measurement device(s). IMD inertial measurement data received from the units in order to obtain information from them. This evaluation can include a rough image analysis to obtain environmental information, such as the detection of objects captured in the camera's field of view and their distance and size. If an object is detected that is on the flight path, the computing unit 22 can calculate the flight control commands FSK for an evasion and/or braking maneuver and forward them to the flight control unit 24.

In addition, image analysis can also be used to obtain odometry information, i.e. distance traveled (1D) or trajectory (3D). The computing unit 22 can also obtain odometry information independently of the image data analysis by integrating the inertial measurement data IMD. By combining the odometry information determined in these two independent ways, higher accuracy and a certain level of redundancy can be achieved.

The computing unit passes on the environmental and odometry information obtained and/or the data BTD, BD, IMD received from the sensors in whole or in part to the performance computer 3 via the radio transceiver 23. There they are received by the transceiver 33 and forwarded to a computing unit 32. This can be a universal computing unit on which any software can be executed and include a high-performance CPU. Alternatively or additionally, the data processing tasks that are outsourced to the power computer for controlling or guiding the flying robot can also be implemented using dedicated hardware, in particular ASICs.

The computing unit 32 calculates, if necessary, situational and navigation information or specifications, in particular a target trajectory or at least a waypoint sequence, using locally available knowledge or publicly available knowledge, for example via the network interface 36 from the Internet or a cloud 4 be sent back to the flying robot 2 via the same data connection using the transceiver 33. The computing unit 22 of the flying robot 2 uses the situation and navigation information and the environmental and odometry information obtained from the image depth data BTD of the depth camera 211, the image data BD of the stereo camera 212 and the inertial measurement data IMD of the inertial measurement unit 25 to generate the flight control commands FSK for following the intended trajectory , which are passed on to a flight control computing unit 24 via a command interface. A distinction must be made here between two options: depending on the performance of the computing unit 22 of the flying robot and the latency of the radio connection between the flying robot 2 and the performance computer 3, the latter can either specify exactly the desired trajectory to be followed by the flying robot 2 or only a sequence of waypoints to be controlled. In the former case, the task of the computing unit 22 is simply to generate the flight control commands FSK necessary to follow the flight path. In the latter case, however, the computing unit 22 itself has to calculate a target trajectory based on the specified waypoints. This does not necessarily have to take all waypoints into account, but should at least extend to the next waypoint to be addressed. In both cases, this target trajectory would be deviated from to avoid collisions. After an evasive and/or braking maneuver, the flying robot 2 can return to the previously calculated target trajectory or it or the performance computer 3 calculate an adjusted target trajectory based on the actual position achieved after the evasive and/or braking maneuver.

The flight control 24 preferably includes a CPU with low power consumption 241. The flight control unit 24 implements the flight control commands FSK in movement of the control surfaces and drives of the flight robot 2. To ensure a stable flight even under the influence of wind and turbulence, the flight control can preferably directly access the inertial measurement data IMD from the inertial measurement units 25.

The use of a CPU with medium power consumption 221 in the computing unit 22 of the flying robot 2 is optional. For example, a CPU with low power consumption could also be used. Alternatively or additionally, other computer architectures could also be used, such as one or more neuromorphic chips, especially for image analysis.

2 shows a second preferred embodiment of the arrangement according to the invention. The structure of the drone 2 corresponds to that of the first embodiment. In contrast to the first embodiment, however, no performance computer in the form of a dedicated, physical device is used as an external data processing platform, but rather a cloud-based “virtual performance computer” in the form of software running in the cloud. If software is used to carry out the data processing tasks of the performance computer 3 of the first embodiment, it can also be used in the second embodiment, except that it is not on a separate, physical device, but in the cloud, i.e. one, possibly worldwide , distributed computer network is executed. Adjustments to the software structure may make sense here in order to enable faster execution on distributed computers.

In the second embodiment, the effort for the arrangement 1 according to the invention is reduced to producing the flying robot 2 and its software, as well as performance computer software running in the cloud.

The data processing tasks carried out by the drone 2 itself in the context of these preferred embodiments include executing the drivers and signal color processing, the regulation and control of the aircraft and preferably also obstacle detection and simple situation detection as well as collision avoidance. However, more complex processing tasks, such as complex situation recognition, navigation and exploring and mapping new environments, as well as any payload-related data processing processes, are carried out by the external performance computer 3. This allows the data processing hardware of the flying robot to be small, light and with low power consumption.

Reference symbol list

1

arrangement

2

Flying robots

21

Camera system

211

Depth camera

212

Stereo camera

22

Arithmetic unit of 2

221

Medium performance CPU

23

Transmitter-receiver

24

Flight control unit

25

Inertial measurement unit

3

Performance calculator

32

Arithmetic unit of 3

321, 322

High performance CPU

33

Transmitter receiver from 3

36

Network interface

4

Cloud

BD

Image data

BTD

Image data with depth data

IMD

Inertial measurement data

FSK

Flight control commands

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 9567081 B1 [0003]
• US 9650136 B1 [0003]
• US 20140217230 A1 [0003]
• US 10730621 B2 [0003]
• DE 102019108256 [0003]
• DE 1020211306523 [0003]
• US 20190072979 A1 [0007]
• WO 2014197067 A1 [0008]
• WO 2018067544 A2 [0009]

Claims (10)

Arrangement for improving the autonomy of a flying robot, comprehensive - a flying robot (2). o a camera system (21) for capturing image data (BD) and/or possibly further data, in particular image/depth data (BTD), o a computing unit (22) for carrying out simpler data processing tasks including obtaining environmental information and/or odometry information from the image data (BD) and optionally the further data, in particular image/depth data (BTD) and/or inertial measurement data (IMD) recorded by an inertial measurement unit. , as well as for generating flight control commands (FSK) taking into account the environmental information, odometry information and, if necessary, other information, o a transceiver (23) for sending part or all of the image data (BD), further data (BTD) and / or the environmental information and / or the odometry information - an external data processing platform (3, 4) for carrying out more complex data processing tasks including the further processing of the image data (BD), image/depth data (BTD) and/or environmental information and/or odometry information in order to calculate situation and navigation specifications from this and via a to pass on the transceiver (33) of the power computer (3) to the flying robot (2), - The computing unit (22) takes the situation and navigation specifications into account when generating the flight control commands (FSK). Arrangement according to Claim 1 , wherein the simpler data processing tasks carried out on the flying robot (2) - the regulation and control of the flying robot (2), - obstacle detection, and / or - simple situation detection and collision avoidance and where those carried out on the external data processing platform (3, 4). more complex data processing tasks - a complex situation detection, and / or - a position determination on an existing map or a mapping of the environment or a simultaneous mapping and position determination, and / or - the navigation of the flying robot (2) through its environment, and / or - if necessary include data processing related to existing payload. Arrangement according to one of the preceding claims, wherein the flying robot (2) comprises a flight control unit (24) for implementing the flight control commands (FSK) in movements of control surfaces and / or drives of the flying robot (2), the flight control unit (24) in particular having a CPU a maximum power consumption of 15 watts or less and a power-specific computing power of 100 Mflops/watt or more, preferably 500 Mflops/watt or more, or 100 DMIPS/watt or more, preferably 600 DMIPS/watt. Arrangement according to one of the preceding claims, wherein the flying robot (2) has one or more inertial measuring units (25) as further sensors for recording odometry data, which are taken into account by the computing unit (22) when obtaining the odometry information, in particular two or more inertial measuring units (25) are present. Arrangement according to one of the preceding claims, wherein the computing unit (22) has a CPU (221) for the comparatively computationally intensive extraction of the image information and/or odometry information from the image data (BD) and/or image/depth data (BTD) provided by the camera system (21). ) with a maximum power consumption between 15 and 30 watts, preferably between 15 and 25 watts, with a power-specific computing power of at least 100 Mflops/watt, preferably at least 500 Mflops/watt, or at least 100 DMIPS/watt, preferably at least 600 DMIPS/watt . Arrangement according to one of the preceding claims, wherein the camera system (21) comprises: - a depth camera (211) pointing in the direction of flight, which generates image/depth data (BTD) combining image data and depth data (TD), which allows a rapid approximation of the environment when obtaining the environmental information and/or odometry information, and/or - A camera pointing against the direction of flight, in particular a stereo camera (212). Arrangement according to one of the preceding claims, wherein the external data processing platform - a cloud (4), i.e. a distributed computer network for storing data and executing data processing tasks, or, - a performance computer (3) which, in particular when processing the image data, environmental information and/or the odometry information, uses an external information source (4), for example the Internet and/or a cloud (4). Method for improving the autonomy of a flying robot (2), comprising: - acquiring image data (BD, BTD) by a camera system (21) of the flying robot (2), - obtaining environmental information and / or Odometry information from the image data (BD) and possibly further data, in particular image/depth data (BTD) and/or inertial measurement data (IMD) by a computing unit (22) of the flying robot (2), - sending part or all of the image data (BD) and/or further data (BTD, IMD), environmental information and/or odometry information to an external data processing platform, in particular a performance computer (3) or a cloud (4), by means of a transceiver (23) of the flying robot (2), - further processing of the Image data (BD) and/or other data (BTD, IMD), environmental information and/or odometry information through the external data processing platform (3, 4) for calculating situation and navigation specifications, - sending the situation and navigation specifications to the flying robot (2) by means of a transceiver (33) of the external data processing platform (3, 4), and - by the computing unit (22), generation of flight control commands (FSK) relating to the orientation, direction of flight and flight speed of the flying robot (2), taking into account the obstacle information, odometry information and the situation and navigation specifications received from the external data processing platform (3, 4). Method according to the preceding claim, wherein when navigating the flying robot (2) a building flight mode is used when flying inside buildings and a free flight mode is used when flying outside buildings, the free flight mode in particular comprising a waypoint flight supported by a global satellite navigation system and / or in Building flight mode allows orientation and positioning based on an existing map or a map created simultaneously for exploration in real time. Using an arrangement according to one of the Claims 1 - 7 using the method according to one of the Claims 8 or 9 for transporting a payload, in particular comprising medical goods or letters, from a room in a starting building to a destination within a target building, the use comprising: a. Loading the flying robot (2) with the payload, and b. Transmitting a mission profile including a destination and a start command to the flying robot (2), whereupon it c. starts and, if necessary, looks for a way through the starting building to an exit opening and to the outside, in particular through an open window or a door, the flying robot (2) supported by the external data processing platform (3, 4) using the building flight mode using its sensors consisting of at least the camera system (21), but preferably one or more inertial measuring units (25), and a UWB transponder system navigates to the exit opening, and in particular the path taken is stored in the drone, or preferably in the external data processing platform (3, 4). so that the path can be optimized when repeated, i.e. after leaving the starting building, the flying robot (2) autonomously switches to a free flight mode and approaches the target building in waypoint flight supported by a global satellite-based navigation system and/or inertial navigation, e. after reaching the target building, independently controlling an entry opening, which is known to the arrangement (1) consisting of the flying robot (2) and external data processing platform (3, 4) from a previous mission or communicated as part of the mission profile or by the data processing platform (3, 4). from information available about the target building and communicated to the flying robot (2), f. Flying the flying robot (2) through the entry opening into the target building and navigating there, using its sensor system consisting of at least the camera system (21), but preferably further one or more inertial measuring units (25), and a UWB transponder system, to the target point within the target building, the flying robot (2) being located in particular based on the lateral boundary of the entry or exit opening and the walls within the start and/or target building, if possible positioned in the middle unless it is necessary to deviate from this to avoid obstacles.

Images