WO2020053215A1

WO2020053215A1 - System and method for radio-wave-based locating and coordinate transformation

Info

Publication number: WO2020053215A1
Application number: PCT/EP2019/074118
Authority: WO
Inventors: Francisco de Asís MOLINA MARTEL; Juri Sidorenko
Original assignee: Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.
Priority date: 2018-09-14
Filing date: 2019-09-10
Publication date: 2020-03-19
Also published as: DE102018215715A1

Abstract

The invention relates to a system (100), comprising a first and a second mobile device (101, 201). The first mobile device (101) has a first environment-sensing- and mapping device (102) for creating a virtual first local environment map (103) of a real environment (2101) in a first local coordinate system. The second mobile device (201) has a second environment-sensing- and mapping device (202) for creating a virtual second local environment map (203) of a real environment (2102) in a second local coordinate system. At least the first mobile device (101) has a radio-based locating device (104) for locating the second mobile device (201). The first mobile device (101) is also designed, on the basis of locating data captured by means of the radio-based locating device (104), to perform a coordinate transformation between the first local coordinate system and the second local coordinate system and to create a common global coordinate system.

Description

SYSTEM AND METHOD FOR RADIO WAVE-BASED LOCATION AND

COORDINATE TRANSFORMATION

description

The invention relates to a system and a method in which a radio wave-based location between two mobile devices generates location data that can be used for the purpose of coordinate transformation. With this coordinate transformation, the respective local coordinate systems of the respective mobile devices can be converted into a common global coordinate system. The mobile devices can, for example, create virtual local environment maps of a real environment, each mobile device having its own local coordinate system within which the respective local environment map is generated.

Such virtual environment maps can be generated, for example, from real environments that are initially unknown. For example, mobile robots can be used for this, which move in a real environment and thereby capture this real environment. The robot can then map the captured data and create a virtual map of the real environment. This automated process of capturing the real environment, creating a virtual environment map and simultaneously estimating one's own pose relative to the environment map is also referred to in robotics as SLAM (Simultaneous Localization and Mapping).

One of the basic skills of a SLAM-based mobile device is to be able to orient yourself, that is, to know what its environment looks like and where it is. If the absolute position of the SLAM-based mobile device in the real environment is known, a virtual environment map can be built up. For example, the current position of the SLAM-based mobile device relative to a topographical feature (eg wall) in the real environment can be detected, and the absolute position of the topographical feature can be determined therein using a position that is already known within its own local coordinate system . The recorded topographical feature can then be entered in the virtual map of the area.

However, at the beginning of a SLAM process, neither the virtual environment map itself nor the own pose (orientation and position) of the SLAM-based mobile is Device known within this environment map. Therefore, the environment map and pose must be estimated at the same time. SLAM is therefore a well-known chicken and egg problem.

Cooperative mapping or cooperative SLAM describes the use of several SLAM-based devices for creating a global map. The individual SLAM devices each create their own virtual local environment map, which in turn can then be combined to form a common virtual global environment map. In practice, however, the initial pose of the individual SLAM devices relative to one another is unknown, so it usually has to be waited until parts of the individual virtual local area maps overlap. It can therefore take valuable time before the individual SLAM devices can locate one another and a common virtual global environment map can be formed.

The simplest case occurs, at least in theory, on the assumption that the initial poses are known [9]. In practice, however, this information is usually not available. The initial relative poses of the SLAM devices are usually calculated by overlapping common features of the individual local virtual environment maps [6] [7] or by estimating the position of another SLAM device in your own local virtual environment map [8 ] The initial pose of the SLAM devices can be determined by the measured relative rotations and distances to each other.

The most relevant method for the present invention is the estimation of the initial relative poses as described in [1] [2] [3] [4]. It contains several mathematical proofs that the initial relative pose can be estimated on the basis of a certain number of distance measurements and the corresponding point pairs in your own coordinate systems. The two-dimensional case is dealt with in [1] and [2]. In [3] and [4] the three-dimensional case with six degrees of freedom. The theory was tested in [2] using two Pioneer II robots. The positions of the robots were measured by the number of revolutions of the wheels and the distance from one another by mounted cameras.

The cameras have determined the basic truth of the pose of both robots.

[5] describes a cooperative SLAM process in which two mobile robots are used. Each robot is equipped with a laser-based distance measuring system and a color camera. The laser-based distance measuring system measures the distance of the respective robot from topographic elements in the real environment. The Color camera has a pattern recognition, a colored cylinder is arranged on each of the two robots, which can be recognized by means of the pattern recognition. The distance and the orientation of the two robots to each other are estimated on the basis of the size and orientation of the cylinder recognized in the image. The problem here is that the two robots have to meet and have a direct line of sight to each other, which is why this publication [5] also bears the subtitle "Robot Rendezvous Case".

In principle, the previously known systems for performing SLAM work very reliably, and the interplay of individual SLAM components with cooperative SLAM can now be mastered well. However, these known systems have a very complex structure and can therefore only be produced with great effort, which makes them expensive and previously uneconomical for commercial use.

It would therefore be desirable to improve known methods and systems for creating virtual environment maps of a real environment in this regard.

A system having the features of claim 1 is therefore proposed. A method having the features of claim 15 is also proposed. Embodiments and further advantageous aspects of the system and the method are mentioned in the respective dependent claims.

The system according to the invention has a first mobile device and a second mobile device. The first mobile device has a first environment detection and mapping device for creating a virtual first local environment map of a real environment in a first local coordinate system. The real environment is therefore mapped virtually, whereby “virtual” is to be understood in the sense of “computer-generated”. The second mobile device has a second environment detection and mapping device for creating a virtual second local environment map of a real environment in a second local coordinate system. According to the invention, at least the first mobile device has a radio-based locating device for locating the second mobile device. Locations are generally methods that determine the spatial position of distant objects in relation to the observer. For example, a distance measurement between the first and the second mobile device can be carried out for this purpose in a certain direction. In the present case, the location can be used, for example, to determine the relative spatial pose between the first and the second mobile device. According to the invention, the first mobile device can be designed to carry out a coordinate transformation between the first and the second local coordinate system and to create a common global coordinate system based on location data acquired by means of the radio-based location device. In other words, the first mobile device can have a first local coordinate system and the second mobile device can have a second local coordinate system. The first device can now use a suitable coordinate transformation to convert the two local coordinate systems into a common global coordinate system. One of the two local coordinate systems can serve as a reference, ie, for example, the second coordinate system can be converted into the first coordinate system serving as a reference. In this case, the first coordinate system serving as a reference would be the common global coordinate system. However, it would also be conceivable for the first local coordinate system and the second local coordinate system to be converted into a third coordinate system by means of a suitable coordinate transformation. In this case, this third coordinate system would be the common global coordinate system. According to the invention, the coordinate transformation is based on the location data determined by means of the radio wave-based location. For example, one or more radio wave-based location processes (for example distance measurements) can be carried out in order to obtain a correspondingly large number of location data (for example measured distances). For example, a large number (for example five) of radio wave-based location processes (for example distance measurements) can be carried out, and the corresponding large number of individual location data (for example the distances measured in each case) obtained in this way can be stored together in one data record. The coordinate transformation can then be carried out on this same data record, which contains the corresponding large number of location data. Since, according to the invention, the locating method takes place by means of radio waves, there is no need for a direct line of sight between the individual mobile devices. The location data can be transmitted to the first mobile device via a transmission channel of the radio-based positioning device and / or via a separate transmission channel (for example WLAN, Bluetooth, etc.). In addition, the location data can optionally include a position and / or orientation associated with a location operation carried out in the space of the respective mobile device. For example, the second mobile device can transmit its local pose valid at the time of the locating process (ie within its own or second local coordinate system) to the first mobile device. According to one exemplary embodiment, the radio-based locating device, as mentioned at the beginning, can have a distance measuring device, and in this case the locating data can have at least one distance measured between the first mobile device and the second mobile device. This means that the recorded location data contain at least one measured distance between the first and the second mobile device. At least one distance measurement can be carried out for each location process, with one set of location data (for example a measured relative distance between the first and the second mobile device) which can be assigned to the location process, for example by means of time stamps, being recorded for each distance measurement. The location data can optionally include an associated position and / or orientation in the space of the respective mobile device for a measured distance.

In one embodiment, the system may be a cooperative localization and mapping (SLAM) system, the first mobile device being a first mobile SLAM device, and the second mobile device being a second mobile SLAM device . Accordingly, the first mobile device and the second mobile device are each designed to record and map their surroundings. The environmental detection and mapping are preferably carried out approximately at the same time.

According to one exemplary embodiment, the radio-based locating device can be an ultra-wideband locating device that is configured to carry out a location of the second mobile device using radio waves in the ultra-wideband. For this purpose, the radio-based locating device can have, for example, an ultra wide band or UWB (UWB) transceiver. The Uitra broadband covers frequencies in the range between 3.1 GHz and 10.6 GHz. One of the advantages is that extremely inexpensive and energy-efficient devices with the option of position determination and moderate data rates can be used. In addition, UWB waves can pass through objects such as walls. The use of UWB is therefore particularly advantageous in indoor applications compared to GPS-based systems. In comparison to conventional GPS-based systems, the mobile devices according to the invention can therefore also be used indoors, since they can also locate one another without any problems there, while GPS, however, is usually not (reliably) available indoors. According to one exemplary embodiment, the radio-based locating device can be designed to locate the second mobile device by means of time of arrival - ToA -

To carry out measurements or by means of Time Difference of Arrival - TDoA measurements, these methods allow reliable localization in real time.

According to one exemplary embodiment, the environment detection and mapping device can have at least one optical means, which is designed to topographically record the real environment by means of optical sensors and to virtually map optically recorded topographical environment data. The optical means can have an image sensor, for example. The optical means can have a camera, for example, and the real environment can be captured by means of image-based pattern recognition. Alternatively or additionally, the optical means can have an LI DAR (Light Detection and Ranging) and / or a LADAR (Laser Detection and Ranging) by means of which the real environment can be detected. LI DAR and LADAR have compared to conventional imaging methods, e.g. Cameras, the advantage that they can also be used in poor visibility or lighting conditions.

According to one exemplary embodiment, the environment detection and mapping device can have at least one radio-based means, which is designed to topographically record the real environment by means of radio waves and to virtually map recorded topographical environment data. The radio-based means can have, for example, a RADAR (Radio Detection and Ranging). Radio wave-based means have the advantage over optical means that they can be replaced even in very poor visibility conditions. For example, a radio-based means for detecting the surroundings can also be used in heavily smoky rooms during fires in order to detect the surroundings.

According to one exemplary embodiment, the first mobile device can be designed to determine the pose of the second mobile device relative to the first mobile device based on the coordinate transformation. In other words, the first mobile device can use the coordinate transformation to transmit the respective pose of the first and the second mobile device into a common coordinate system, namely into the common global coordinate system. For example, the second mobile device can transmit its local pose valid at the time of the location (ie within its own or second coordinate system) to the first mobile device. The two mobile devices can do this for pose-related data exchange with each other. This pose-related data can be part of the location data itself or can be transmitted as a separate data record.

According to one exemplary embodiment, the first mobile device and / or the second mobile device can be designed to combine the virtual first local environment map and the virtual second local environment map to form a virtual global environment map within the common global coordinate system. The second mobile device can, for example, transmit a part of its own or second virtual local area map that has already been created to the first mobile device. If, for example, the second mobile device has already created all or part of a virtual second local environment map at the time of the locating process, the part of the virtual second local environment map that has already been created can be transmitted to the first mobile device. For this purpose, the two mobile devices can exchange area map data with one another. This already created part of the virtual second local environment map can be part of the location data itself or can be transmitted as a separate data record. Based on the coordinate transformation, the first mobile device can then combine the first and the second virtual local environment map into a virtual global environment map within the global coordinate system. An advantage of the concept according to the invention is that the individual local environment maps do not have to overlap in order to be able to create a global environment map. The present invention advantageously combines an environmental detection device with a radio-based location device. This enables the construction of the mobile devices to be significantly simpler and less expensive than in the prior art.

According to one exemplary embodiment, at least the first mobile device can be augmented reality glasses that have an imaging device that is designed to display the virtual first local environment map and / or the virtual second local environment map and / or the virtual global environment map. This means that at least one of the virtual first local environment map, the virtual second local environment map and the virtual global environment map can be displayed with the augmented reality glasses. The wearer of the augmented reality glasses is shown the respective virtual environment map in the imaging device. The imaging device can display the respective virtual environment map in the field of view of the wearer. For this purpose, the imaging device can be partially transparent, for example, so that the wearer can see the real environment and, at the same time, the respective virtual one Area map sees. This can be advantageous for firefighters, for example, who enter a burning building with augmented reality glasses. The rooms of the burning building will usually be unknown to the firefighter. If the rooms are heavily used up, the fireman will not see anything and will have to rely on his sense of touch. The augmented reality glasses according to the invention, on the other hand, record the real environment and at the same time create a virtual map of the surroundings. The firefighter is shown the virtual map in augmented reality glasses. The fireman can see the room through the augmented reality glasses despite the strong visual impairment caused by smoke.

According to one exemplary embodiment, the second mobile device can be augmented reality glasses that have an imaging device that is configured to display the virtual first local environment map and / or the virtual second local environment map and / or the virtual global environment map. For example, the first and the second mobile device can each be augmented reality glasses that can cooperatively cooperate. Two augmented reality glasses can communicate with one another and, for example, exchange location data and / or position data and / or area map data and / or pose-related data and / or image data. According to a conceivable exemplary embodiment, the second augmented reality glasses can transmit a part of a second virtual local area map that has already been created to the first augmented reality glasses. The first augmented reality glasses can then combine this created part of the second virtual local environment map with its already created part of a first virtual local environment map to form a common virtual global environment map. If several firefighters equipped with augmented reality glasses are distributed in the building, the respective individual local virtual environment maps of the respective firefighters can be combined to form a global virtual environment map that spans the space and floor. The more virtual local environment maps are generated, the faster a virtual global environment map that is available for all augmented reality glasses can be generated.

According to one exemplary embodiment, the imaging device of the first and / or second augmented reality glasses can be configured around an object which is arranged in the real environment between the first and the second augmented reality glasses and which provides a visual connection between the first and the second augmented reality glasses hindered at least partially to hide in the virtual global environment map or not to show at all. If, for example, there is a wall between the two augmented reality glasses, the two augmented reality glasses do not have direct visual contact with one another. Nevertheless, every augmented reality glasses initially creates a virtual local map for itself. When the virtual local environment maps are merged into the virtual global environment map, the wall in the virtual global environment map can be hidden. This means that the available local environment maps are shown in the field of vision in the virtual global environment map, so that the wearer of the augmented reality glasses can look through the wall in the real room and recognize the space behind it.

According to one embodiment, the first mobile device can be configured to locate the second mobile device without an existing line of sight between the first and the second mobile device and / or around the first and the second local coordinate system without an existing line of sight between the first and the second to convert the second mobile device into the global coordinate system. In contrast to systems known from the prior art, no direct line of sight between the first and the second mobile device is necessary in the inventive concept. This makes the system according to the invention much more versatile.

According to an embodiment, a mobile device for use in a system described above is proposed.

The invention further relates to a corresponding method. According to the invention, topographical environmental features are recorded in a real environment. For example, these can be environmental features, such as walls, tables and the like, in a room. In addition, a virtual first local environment map is created based on the recorded topographical environmental features. This means that the captured real space is virtually mapped. The virtual first local environment map is created in a first local coordinate system. The method further comprises locating a mobile device, specifically by acquiring location data using radio waves. The mobile device located by means of radio waves has its own (or second) local coordinate system, in which the located mobile device can create a virtual second local area map. The second local Coordinate system can differ from the first local coordinate system. The method further comprises a step of performing a coordinate transformation between the first local coordinate system and the second local coordinate system in order to generate a common global coordinate system, based on the location data acquired by means of the radio waves.

The invention further relates to a computer program with a program code for performing the aforementioned method when the program runs on a computer,

Some exemplary embodiments are shown as examples in the drawing and are explained below. Show it:

1A shows a schematic overview of a system according to the invention in accordance with an exemplary embodiment,

1B is a schematic overview of a system according to the invention in accordance with a further exemplary embodiment,

2 shows a schematic overview of a system according to the invention in accordance with a further exemplary embodiment, and

Fig. 3 is a schematic block diagram of a method according to the invention according to an embodiment.

Exemplary embodiments are described in more detail below with reference to the figures, elements with the same or similar function being provided with the same reference symbols.

Method steps, which are shown in a block diagram and explained with reference to the same, can also be carried out in a different order than the depicted or described. In addition, method steps that relate to a specific feature of a device are interchangeable with this feature of the device, which also applies the other way round.

Some non-limiting examples of mobile devices according to the invention can be robots, for example. This can be, for example, autonomously moving vehicles of any kind, such as autonomously driving cars, drones, and the like. The concept according to the invention is described below using a non-limiting example of augmented reality glasses as a mobile device. Augmented Reality is also known as augmented virtuality or mixed reality. With augmented reality, virtual elements (eg virtual environment maps) are superimposed on the field of view of the real environment or are shown therein. Virtual reality glasses can also be mentioned as further non-limiting examples of mobile devices. Only virtual elements are displayed here.

FIG. 1A shows a schematic illustration of a system 100 according to the invention. The system 100 has a first mobile device 101 and a second mobile device 201.

The first mobile device 101 has a first environment detection and mapping device 102. The environment detection and mapping device 102 is configured to topographically capture a real environment and to create a virtual first local environment map of the captured real environment in a first local coordinate system 111.

The second mobile device 201 has a second environment detection and mapping device 202. The environment detection and mapping device 202 is configured to topographically capture a real environment and to create a virtual second local environment map of the captured real environment in a second local coordinate system 222. The first and the second local coordinate system 111, 222 can have different orientations to one another.

In the present case, a topographical recording of a real environment is understood to mean the recording of topographical environmental features in the surroundings. For example, walls, chairs, and a table may be present in a room, all of which are examples of topographical environmental features in that room. These topographical environmental features can be captured and virtually mapped, i.e. are shown in a virtual map of the area.

At least the first mobile device 101 has a radio-based locating device 104. The radio-based locating device 104 is configured to locate the second mobile device 201. For example, the first mobile device 101 can carry out a distance measurement in order to determine the relative distance D between the first mobile device 101 and the second mobile device 201. Location data is generated when the second mobile device 201 is located. This location data can include, for example, the determined distance between the two mobile devices 101, 201. The first mobile device 101 can be configured to carry out a coordinate transformation between the first and the second local coordinate system 111, 222 based on the location data acquired by means of the radio-based location device. The first mobile device 101 can combine the local coordinate systems 111, 222 to form a common global coordinate system 333. One of the two local coordinate systems 1 1 1, 222 can serve as a reference for creating the global coordinate system 333, so that in each case one of the two local coordinate systems is transferred to the other local coordinate system serving as a reference. The local coordinate system serving as reference then forms the common global coordinate system 333. Alternatively or additionally, the first local coordinate system 111 and the second local coordinate system 222 can be converted into a third coordinate system 333 by means of a suitable coordinate transformation. In this case, this third coordinate system 333 would be the common global coordinate system.

The coordinate transformation is based on the location data acquired by means of radio waves. As mentioned at the beginning, this can be one or more distance measurements between the first and the second mobile device 101, 201. For example, several distance measurements can be carried out at different times, the mobile devices 101, 201 having been able to move relative to one another at the respective different times, so that their relative distance to one another has changed.

According to the invention, the location, for example the distance measurement, is carried out between the mobile devices 101, 201 by means of radio waves. This allows highly precise location, e.g. Distance measurement, even without existing visual contact between the two mobile devices 101, 201.

The system 100 according to the invention can take on different advantageous configurations based on the creation of the global coordinate system 333, based on the coordinate transformation mentioned at the beginning. For example, the first mobile device 101 can calculate the pose of the second mobile device 201, specifically within the common global coordinate system 333. The first and the second mobile devices 101, 201 can thus be related to one another in the correct orientation. For this purpose, for example, the location data recorded by the first mobile device 101 can optionally contain additional information on the position, location and orientation of the second mobile device 201 in real space. So can for example a mutual localization of the two mobile devices 101,

201 are made possible. As an alternative or in addition, the respective local environment maps can, for example, be put together to form a common global environment map. This will be described below by way of example with reference to FIG. 1B.

1B shows an exemplary embodiment of a system 100 according to the invention, which is able to combine a virtual first local environment map 103 and a virtual second local environment map 203 to form a virtual common global environment map 303.

As mentioned at the beginning, the first mobile device 101 has its own or first, local coordinate system 111, within which the first virtual local environment map 103 is generated. The second mobile device 201 has its own or second, local coordinate system 222, within which the second virtual local environment map 203 is generated. A coordinate transformation between the first coordinate system 111 and the second coordinate system 222 is carried out for mutual localization or for joining the two virtual local environment maps 103, 203 into a common virtual global environment map 303. The location, for example by means of distance measurement, between the two mobile devices 101, 201 provides the information required for this in the form of location data. As mentioned at the beginning, the result of one or more distance measurements can be part of the location data recorded by the first mobile device 101, for example.

Based on the location data acquired by the first mobile device 101, the first mobile device 101 can therefore create a common global coordinate system 333. The virtual first local environment map 103 and the virtual second local environment map 203 are combined to form a virtual global environment map 303 within the common global coordinate system 333 in the correct orientation. For this purpose, the two mobile devices 101, 201 can, for example, exchange area map data with one another. For example, they can also exchange any position data from their corresponding local coordinate system 11 1, 222 with one another.

The number and the accuracy of the distance measurements determine the quality of the map merging. The more distance measurement results are available, the more precise it becomes the coordinate transformation and the joining of the two virtual local environment maps 103, 203 to the virtual global environment map 303. In two-dimensional space, five different distance measurements can be sufficient to be able to clearly determine the coordinate transformation. In three-dimensional space, six different distance measurements can be sufficient to uniquely determine the coordinate transformation with four degrees of freedom (3D translation and rotation around the gravitational vector). A higher number of distance measurements reduces the standard deviation and increases the accuracy.

The concept according to the invention is to be described below with reference to FIG. 2 using a non-limiting example of augmented reality glasses which can be worn in an unknown space when rescue workers are used. This can be, for example, a heavily smoky apartment 210 in the event of a building fire, with several firefighters staying in different rooms 210i, 2102 of apartment 210.

FIG. 2 schematically shows an apartment 210 with a wall 211 arranged therein, which divides the apartment 210 into a first room 210i and a second room 2102. On one side of wall 211, i.e. in the first room 210i there is a first mobile device 101. On the other side of the wall 211, i.e. in the second room 2102 there is a second mobile device 201.

The first mobile device 101 is augmented reality glasses that are equipped with an environment detection and mapping device 102 and a radio-based location device 104.

The environment detection and mapping device 102 can, for example, have a camera which uses image recognition methods to detect the real environment, i.e. the first room 210i. Alternatively or additionally, the environment detection and mapping device 102 may have an LI DAR and / or a RADAR. LIDAR and RADAR are advantageous compared to a camera because with these methods the real environment, i.e. the first room 210i can also be captured if there is no unobstructed view, for example if the real environment is so heavily used up that a camera has no or only a limited view.

In this example, the second mobile device 201 is also augmented reality glasses, which are equipped with an environment detection and mapping device 202 and a radio-based interface 204. The radio-based interface 204 can be a radio-based location device. Alternatively or additionally, the radio-based interface 204 can be a radio-based communication device, by means of which the second mobile device 102 can communicate with the first mobile device 101 and, for example, can exchange location data and / or area map data.

The environment detection and mapping device 202 of the second augmented reality glasses 201 can, for example, have a camera which records the real environment, ie the second room 210 ₂ , by means of image recognition methods. Alternatively or additionally, the environment detection and mapping device 202 of the second augmented reality glasses 201 can have a LIDAR and / or a RADAR. LI DAR and RADAR are advantageous compared to a camera, since with these methods the real environment, ie the second room 210 ₂ , can also be recorded when there is no unobstructed view, for example when the second room 210 _{2 is} so heavily used is that a camera has no or only a limited view.

The two augmented reality glasses 101, 201 can be SLAM-based devices, that is to say the detection of the real environment, or the respective room 210i, 210 ₂ , and the virtual mapping for creating a virtual local area map 103, 203 of the respective room 210i , 210 ₂ , can take place simultaneously. The system 100 according to the invention can in this case be a cooperative SLAM system in which the two SLAM-based augmented reality glasses 101, 201 cooperate. That is, at least one of the two augmented reality glasses 101, 201 can be the respective virtual one Assemble local environment maps 103, 203 into a common virtual global environment map 303. The virtual local environment maps 103, 203 of the respective rooms 210i, 210 ₂ can thus be combined to form a common virtual global environment map 303 of the entire apartment 210.

In the present exemplary embodiment, the radio-based locating device 104 of the first augmented reality glasses 101 has a UWB transceiver. The radio-based

Interface 204 of the second augmented reality glasses 201 can also have a UWB transceiver.

The UWB transceiver 104 of the first augmented reality glasses 101 can carry out radio-based location of the second augmented reality glasses 201. For this purpose, a distance measurement between the first augmented reality glasses 101 and the second Augmented reality glasses 201 are carried out, for example by means of time of flight (ToF), time of arrival (ToA) or time distance of arrival (TDoA).

The localization by means of UWB is particularly advantageous for applications as described here by way of example. The ultra broadband waves have only a low attenuation and can spread through solid objects, among other things. For example, the second augmented reality glasses 201 can be located through the wall 211. Although there is no direct line of sight between the two mobile devices (augmented reality glasses) 101, 201, the radio-based location can still be carried out using UWB.

The firefighter located in the first room 210i can thus locate the firefighter wearing the second augmented reality glasses 201 in the second room 210 ₂ using the first augmented reality glasses 101, although both firefighters cannot see through the wall 21 1 .

The concept according to the invention is therefore clearly advantageous over conventional systems in which the mobile devices have to make visual contact [5],

Based on the location data (eg distance) determined in the radio-based location (eg distance measurement), the first augmented reality glasses 101 can carry out a coordinate transformation and the second virtual local environment map 203 created by the second augmented reality glasses 201 in the local second coordinate system 222 transfer a common global coordinate system 333. Here, the first virtual local environment map 103 created in the local first coordinate system 111 and the second virtual local environment map 203 created in the local second coordinate system 222, in correct alignment with one another, can be combined to form the common global virtual environment map 303 in the common global coordinate system 333.

Both augmented reality glasses 101, 201 each have an imaging device. This can be, for example, a small display in which the virtual environment maps 103, 203, 303 can be shown. The display can be partially transparent, so that the wearer of the augmented reality glasses 101, 201 both sees the real environment, ie the respective room 210i, 210 ₂ , and at least one of the virtual environment maps 103, 203, 303 is shown simultaneously. An advantage of the invention is therefore that the individual augmented reality glasses 101, 201 can cooperate cooperatively and jointly create a virtual global environment map 303. Since the distances of the augmented reality glasses 101,

201 are determined with respect to one another, the individual virtual local environment maps 103, 203 can be put together in the correct orientation relative to one another, even without the local environment maps 103, 203 having to overlap. This considerably reduces the time required to create the global environment map 303 compared to conventional systems in which the local environment maps have to overlap in order to carry out a pattern recognition of features that are present together.

Another advantage of the invention is that the x-ray view can be made possible with the augmented reality glasses 101, 201. The fireman in the first room 210i is shown the virtual global map 303 in his (first) augmented reality glasses 101. If the fireman now turns towards the wall 211, then the augmented reality glasses 101 show him the virtual global environment map 303, or at least the virtual local environment map 203 of the second room 210 ₂ located behind it. The wall 211 cannot be displayed or at least partially hidden in the virtual global environment map 303.

In other words, the first fireman located in the first room 210i sees with his first augmented reality glasses 101 the virtual environment map 203 of the second room 210 ₂ , which was created using the second augmented reality glasses 201 of the second fireman located in the second room 210 ₂ . Here, too, there must be no direct line of sight between the two mobile devices (augmented reality glasses) 101, 201.

The exemplary embodiment described with reference to FIG. 2 with emergency services and augmented reality glasses 101, 201 is only to be understood as a non-limiting example. It would also be conceivable, for example, that the mobile devices are autonomously operating robots. This would preclude rescue workers from being endangered. It would also be conceivable that the mobile devices are vehicles or airplanes. For example, it can be unmanned or remote-controlled drones that can also be used indoors. Other examples provide that the mobile devices are mobile devices, such as, for example, mobile telecommunication devices (for example smartphones, wearables, notebooks, etc.), this form of mobile devices are usually equipped with cameras and radio-based location devices ex works.

In addition, the concept according to the invention was described using the non-limiting example of two mobile devices 101, 201. It is of course conceivable that any number of mobile devices 101, 201 can be used which can cooperatively interact with one another in the manner described here. Each of these mobile devices 101, 201 can advantageously be configured to create or display a virtual global environment map 303 using the concept according to the invention.

FIG. 3 shows a schematic block diagram of a method according to the invention, as can be carried out by the mobile devices 101, 201.

In block 301, topographical environmental features are recorded in a real environment and a virtual first local environment map 103 is created in a first local coordinate system 111.

In block 302, a mobile device 201 is located by detecting location data using radio waves, this mobile device 201 having a second local coordinate system 222. The mobile device 201 is configured to capture topographical environmental features in a real environment and to create a virtual second local environment map 203 in the second local coordinate system 222.

In block 303, a coordinate transformation between the first local coordinate system 1 1 1 and the second local coordinate system 222 is carried out in order to generate a common global coordinate system 333, based on the location data acquired by means of the radio waves.

The invention is to be summarized again in other words below, and the advantages thereof are briefly explained.

Previous research in the field of robotics has shown that the known SLAM problem can be solved by measuring the distance between two SLAM devices. In one exemplary embodiment of the invention, modern augmented reality glasses 101, 201 with integrated ultra-wideband (UWB) transceivers 104, 204 can be used for distance measurement by means of time of arrival (TOA). The concept according to the invention uses cooperative SLAM and the measured distances belong to at least two equivalent systems 101, 201 in order to estimate the relative pose to one another.

The present invention describes a system 100 that is intended to improve the use of cooperative SLAM. It can have at least one mobile device 101 with a SLAM-based optical sensor (e.g. optical camera, LIDAR, etc.) 102 and at least one integrated UWB transceiver 104. The transceiver 104 carries out distance measurements (e.g. by means of time of arrival or time difference of arrival) to at least one other integrated UWB transceiver 204 of an equivalent system (second mobile device) 201.

Example variants are similar systems, which may have additional sensors (e.g. IMU, magnetometer, barometer, etc.). These optional additional sensors can serve to increase the accuracy and robustness of the system 100. Exemplary embodiments of the present invention can be “augmented reality” glasses 101 with integrated optical SLAM sensors (eg cameras) 102 and a UWB transceiver 104 coupled to them, which operate a similar application. Such reality glasses 101 can, for example, have at least one camera 102 and an inertial measuring unit IMU Inertial Measurement Unit for SLAM.

With the concept according to the invention, the time efficiency and the reliability of cooperative SLAM systems can be increased, because immediate mutual localization and global mapping is possible with this method. This is particularly relevant for time-critical applications. With the concept according to the invention, a cooperative SLAM can also be realized if objects or walls 211 are located between the individual SLAM systems (mobile devices) 101, 201.

The concept according to the invention allows two SLAM-based devices 101, 201, which have at least one integrated optical sensor 102, 202 and at least one UWB transceiver 104, 204, to quickly locate one another, even if the devices 101, 201 have no extrinsic additional information feature.

Examples of applications of the concept according to the invention can be in the fire service, police, special operations (eg counter-terrorism) and the military in the form of “augmented reality” glasses 101, 201. In the field of robotics, for example, a more efficient 3D measurement of interiors in buildings or Outdoor areas, e.g. by means of unmanned aerial vehicles (UAVs), will be realized. The concept according to the invention is of great relevance for indoor use where GRS does not work well.

The above-described embodiments are merely illustrative of the principles of the present invention. It is to be understood that modifications and variations in the arrangements and details described herein will be apparent to those skilled in the art. Therefore, it is intended that the invention be limited only by the scope of the following claims and not by the specific details presented based on the description and explanation of the exemplary embodiments herein.

Although some aspects have been described in connection with a device, it goes without saying that these aspects also represent a description of the corresponding method, so that a block or a component of a device is also to be understood as a corresponding method step or as a feature of a method step. Analogously, aspects that have been described in connection with or as a method step also represent a description of a corresponding block or details or features of a corresponding device.

Some or all of the process steps can be performed by a hardware apparatus (or using a hardware apparatus) such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, some or more of the most important process steps can be performed by such an apparatus.

Depending on the specific implementation requirements, exemplary embodiments of the invention can be implemented in hardware or in software or at least partially in hardware or at least partially in software. The implementation can be carried out using a digital storage medium, for example a floppy disk, a DVD, a BluRay disc, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, a hard disk or another magnetic or optical Memory are carried out, on which electronically readable control signals are stored, which can interact with a programmable computer system or in such a way that the respective method is carried out. The digital storage medium can therefore be computer-readable.

Some exemplary embodiments according to the invention thus comprise a data carrier which has electronically readable control signals which are capable of using a co-programmable computer system such that one of the ^¬ be herein described methods is performed.

In general, exemplary embodiments of the present invention can be implemented as a computer program product with a program code, the program code being effective to carry out one of the methods when the computer program product runs on a computer.

The program code can, for example, also be stored on a machine-readable carrier.

Other exemplary embodiments include the computer program for performing one of the methods described herein, the computer program being stored on a machine-readable medium. In other words, an exemplary embodiment of the method according to the invention is thus a computer program which has a program code for performing one of the methods described here when the computer program runs on a computer.

A further exemplary embodiment of the method according to the invention is thus a data carrier (or a digital storage medium or a computer-readable medium) on which the computer program for carrying out one of the methods described herein is recorded. The data carrier or the digital storage medium or the computer-readable medium are typically tangible and / or non-volatile.

A further exemplary embodiment of the method according to the invention is thus a data stream or a sequence of signals which represents the computer program for performing one of the methods described herein. The data stream or the sequence of signals can, for example, be configured to be transferred via a data communication connection, for example via the Internet.

A further exemplary embodiment comprises a processing device, for example a computer or a programmable logic component, which is configured or adapted to carry out one of the methods described herein.

Another exemplary embodiment comprises a computer on which the computer program for performing one of the methods described herein is installed. A further exemplary embodiment according to the invention comprises a device or a system which is designed to transmit a computer program for performing at least one of the methods described herein to a receiver. The transmission can take place electronically or optically, for example. The receiver can be, for example, a computer, a mobile device, a storage device or a similar device. The device or the system can comprise, for example, a file server for transmitting the computer program to the recipient.

In some embodiments, a programmable logic device (for example a field programmable gate array, an FPGA) can be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, in some embodiments, the methods are performed by any hardware device. This can be a universally usable hardware such as a computer processor (CPU) or hardware specific to the method, such as an ASIC.

Swell:

[1] Zhou, X. S., Roumeliotis, S. 1 2007. Determining the Robot-to-

Robot Relative Pose Using Range-only Measurements, IEEE International Conference on Robotics and Automation.

[2] Zhou, X. S., Roumeliotis, S. 1 2008. Robot-to-Robot Relative

Pose Estimation From Range Measurements, IEEE Transactions

on Robotics, Val. 24, No. 6.

[3] Trawny, IM., Zhou, X. S., Zhou, K. X., Roumeliotis, S. 1., 2007.

30 Relative Pose Estimation from Distance-Only Measurements, Proceedings of the 2007 IEEE / RSJ International Conference on Intelligent Robots and Systems.

[4] Trawny, N., Zhou, X. S., Roumeliotis, S. 1., 2010. 30 Relative

Pose Estimation from Six Distances, Robotics: Science and

Systems V, Vol. 5, pp. 233-240.

[5] Zhou, X. S., Roumeliotis, S. 1., 2006. Multi-robot SLAM with Unknown Initial Correspondence: The Robot Rendezvous Case,

Proceedings of the 2006 IEEE / RSJ International Conference on

Intelligent robots and systems.

[6] Saeedi, S. et al, 2014. Map merging for multiple robots using Hough peak matching.

Robotics and Autonomous Systems, Val. 62, pp. 1408-1424.

[7] Birk, A., Carpin, S., 2006. Merging Occupancy Grid Maps From Multiple Robots.

Proceedings of the IEEE, Vol. 94, No. 7, pp. 1384-1397.

[8] Ko, J., Stewart, B., Fox, D, Konolige, K., Limketkai, B., 2003. A

Practical, decision-theoretical approach to multi-robot mapping

and Exploration, Proceedings of the 2003 IEEE / RSJ International

Conference on Intelligent Robots and Systems.

[9] Thrun, S., 2001. A probabilistic on-line mapping algorithm for teams of mobile robots, The International Journal of Robotics Research, Vol. 20, No. 5, pp. 335-363.

Claims

1 system (100) comprising: a first mobile device (101) with a first environment detection and mapping device (102) for creating a virtual first local environment map (103) of a real environment (210i) in a first local coordinate system (111) , a second mobile device (201) with a second environment detection and mapping device (202) for creating a virtual second local environment map (203) of a real environment (210 ₂ ) in a second local coordinate system (222), at least the first Mobile device (101) has a radio-based locating device (104) for locating the second mobile device (201), and wherein the first mobile device (101) is configured to perform a coordinate transformation between the location based on location data acquired by means of the radio-based locating device (104) first and second local coordinate system (1 11, 222) and a common global coor to create the dinate system (333).

2. The system (100) according to claim 1, wherein the radio-based locating device (104) has a distance measuring device, and wherein the locating data has at least one distance measured between the first mobile device (101) and the second mobile device (201).

3. System (100) according to claim 1 or 2, wherein the system (100) is a cooperative SLAM (Simultaneous Localization and Mapping) system, wherein the first mobile device (101) is a first mobile SLAM device and wherein the second mobile device (201) is a second mobile SLAM device.

4. The system (100) according to one of claims 1 to 3, wherein the radio-based locating device (104) is an ultra-wideband locating device that is configured to to locate the second mobile device (201) using radio waves in the ultra-wideband.

5. System (100) according to one of claims 1 to 4, wherein the radio-based locating device (104) is designed to locate the second mobile device (201) by means of time of arrival - ToA measurements or by means of time difference of arrival - Perform TDoA measurements.

6. System (100) according to one of claims 1 to 5, wherein the first and / or second environment detection and mapping device (102, 202) comprises at least one optical means, which is configured to the real environment (210i, 210 ₂ ) to be recorded topographically by means of optical sensors, and the first and / or second environment detection and mapping device (102, 202) is designed to map optically recorded topographical environment data in the virtual first and / or second map (103, 203) .

7. System (100) according to any one of claims 1 to 6, wherein the first and / or second environment detection and mapping device (102, 202) comprises at least one radio-based means, which is configured to the real environment (210i, 210 ₂ ) topographically using radio waves, and wherein the first and / or second environment detection and mapping device (102, 202) is designed to map acquired topographical environment data in the virtual first and / or second environment map (103, 203).

8. System (100) according to one of claims 1 to 7, wherein the first mobile device (101) is configured to determine the pose of the second mobile device (201) relative to the first mobile device (101) based on the coordinate transformation .

9. System (100) according to one of claims 1 to 8, wherein at least one of the first and second mobile devices (101, 201) is configured to base the virtual first local environment map (103) and the virtual second local environment map (203) on the coordinate transformation to a virtual global environment map (303) within the common global coordinate system (333).

10. The system (100) according to claim 9, wherein at least the first mobile device (101) comprises augmented reality glasses that have an imaging device that is configured to display the virtual first local environment map (103) and / or the virtual global one Show surrounding map (303).

1 1. System (100) according to one of claims 9 or 10, wherein the second mobile device (201) comprises augmented reality glasses that have an imaging device that is configured to the virtual second local environment map (203) and / or to display the virtual global environment map (303).

12. System (100) according to one of claims 10 or 11, wherein the imaging device is configured to an object arranged in the real environment (210i, 210 ₂ ) between the first and the second mobile device (101, 201) (211), which impedes a line of sight between the first and the second mobile device (101, 201), at least partially not to be displayed in the virtual global environment map (303).

13. System (100) according to any one of claims 1 to 12, wherein the first mobile device (101) is configured to the second mobile device (201) without an existing line of sight between the first and the second mobile device ( 101, 201) to locate and / or to transfer the first and the second local coordinate system (11 1, 222) into the global coordinate system (333) without an existing line of sight between the first and the second mobile device (101, 201).

14. Mobile device (101, 201) for use in a system (100) according to one of claims 1 to 13.

15. Procedure with the following steps:

Acquisition of topographical environmental features of a real environment (210i) and creation of a virtual first local environment map (103) in a first local coordinate system (111), Locating a mobile device (201) by acquiring location data using radio waves, the mobile device (201) having a second local coordinate system (222), and

Performing a coordinate transformation between the first local coordinate system (111) and the second local coordinate system (222) to generate a common global coordinate system (333) based on the location data acquired by means of the radio waves.

16. The method of claim 15, wherein the location of the second mobile device (201) includes determining a distance to this second mobile device (201), and wherein the location data have at least one such measured distance.

17. The method of claim 15 or 16, wherein the detection of topographical environmental features is done using an optical image capture.

18. The method according to any one of claims 15 to 17, wherein the radio-based location is done using radio waves in the ultra-wideband.

19. The method according to any one of claims 15 to 18, wherein the method further comprises the following step:

Creating a virtual global environment map (303) in the common global coordinate system (333) by combining the virtual first local environment map (103) created in the first local coordinate system (1 1 1) with a mobile device (201) located by the virtual second local environment map (203) created in the second local coordinate system (222).

20. The method according to claim 19, the method further comprising the step of: displaying the virtual first local environment map (103) and / or the virtual global environment map (303), and / or at least partially hiding an object (211) located in the real environment (21 Oi, 210 ₂ ) in the virtual global environment map (303).

21. Computer program with a program code for performing the method according to one of claims 15 to 20, when the program runs on a computer.