DE202020105557U1 - Internet of Things device - Google Patents

Abstract

Device for positioning an Internet of Things, IoT, device with augmented reality, AR, glasses, which has:
an access module which is designed such that it accesses an interior card (10);
a first determination module which is designed such that it determines an AR position (16) of the AR glasses on the interior map (10) and a direction of view (18) of the AR glasses on the basis of initial information of the AR glasses;
a receiving module which is designed such that it receives tracking information relating to the IoT device to be positioned from the AR glasses; and
a second determination module which is designed such that an IoT position of the IoT device (12) to be positioned is determined based on the determined AR position and viewing direction (18) in combination with the tracking information of the AR glasses.

Description

background

The present disclosure is based on the object of providing a device for positioning an Internet of Things (IoT) device with augmented reality (AR) glasses. Furthermore, the present invention is based on the object of providing a system for positioning IoT devices and a smart home system with such a system provided for positioning.

Nowadays, more and more domestic objects are being connected to each other and to the Internet. IoT devices are typically connected to the Internet through a home network gateway which is used to route messages between devices or from devices to devices outside of the home (via the Internet) over many different types of networks, such as: Bluetooth, ZigBee or WiFi, is responsible.

From a user's point of view, IoT devices are often controlled by at least one home automation system that is controlled both within the home (typically via WiFi) or from outside the home (essentially via the Internet). It is not uncommon for multiple IoT devices to belong to different home automation ecosystems and users sometimes have to switch back and forth between different home automation applications or use bridging applications that are usually complex in structure and do not provide all of the functionality of the original home automation applications.

Furthermore, as the number of home IoT devices increases, it is becoming more and more complicated to operate a single device from the home automation application.

A first solution, which was presented in the prior art, involves voice control systems for controlling the IoT devices.

In any case, the position of the IoT device is necessary for the correct addressing of the IoT device. A position display for IoT devices is now available to a certain extent in every single home automation system, but it is always limited to room categorization. For example, a connected incandescent lamp can be identified as part of the living room, but there is no information whatsoever about exactly where in the living room the lamp is located. And when there are several lamps, it becomes impossible to distinguish them unless the user has conveniently assigned them different names which the user must remember and which must be passed on to all users of the home automation system.

Furthermore, home automation systems usually offer very little information about the spatial conditions of the home; at most they know how many rooms there are. Only one IoT device has such knowledge these days: the connected vacuum cleaner robot, which typically scans the home to better define its suction path. However, although such a home map or indoor map is created, it is not shared with other devices or with the home automation system. In the future it is also foreseeable that other IoT devices, such as a video camera with ToF (Time-of-Flight) sensors will also be able to create such interior maps or at least provide assistance in creating them

Thus, for better control of IoT devices, exact positions of the IoT devices are required. The positioning, i.e. The initial determination of the location of a newly implemented IoT device and the subsequent location of already positioned IoT devices will be crucial for the efficient control of home automation systems.

The present disclosure is therefore based on the object of providing a device for positioning IoT devices.

overview

• ein Zugriffsmodul, das derart ausgebildet ist, dass es auf eine Innenraumkarte zugreift;
• ein erstes Bestimmungsmodul, das derart ausgebildet ist, dass es eine AR-Position der AR-Brille auf der Innenraumkarte und eine Blickrichtung der AR-Brille auf der Basis von Anfangsinformationen der AR-Brille bestimmt;
• ein erstes Empfangsmodul, das derart ausgebildet ist, dass es Nachverfolgungsinformationen bezüglich des zu positionierenden IoT-Geräts aus der AR-Brille empfängt; und
• ein zweites Bestimmungsmodul, das derart ausgebildet ist, dass eine IoT-Position des zu positionierenden IoT-Geräts anhand der bestimmten AR-Position und der Blickrichtung in Kombination mit den Nachverfolgungsinformationen der AR-Brille bestimmt.

In one aspect of the present disclosure, an apparatus for positioning an Internet-of-Things, IoT, device with AR glasses is provided, which has:

• an access module which is designed such that it accesses an interior map;
• a first determination module which is designed such that it determines an AR position of the AR glasses on the interior map and a viewing direction of the AR glasses on the basis of initial information of the AR glasses;
• a first receiving module which is designed such that it receives tracking information relating to the IoT device to be positioned from the AR glasses; and
• a second determination module which is designed such that an IoT position of the to positioning IoT device based on the determined AR position and viewing direction in combination with the tracking information of the AR glasses.

Augmented Reality is a real environment experience in which further information is provided with virtual elements, such as Graphics, videos, animations and the like are displayed. The virtual elements of the AR glasses are made available to the user via visual perception.

Furthermore, IoT devices include all types of small / personal devices that are connected to the Internet. These can be home devices such as

Incandescent lamps, televisions, vacuum cleaners and their docking stations, sound devices, smart TVs, kitchen appliances, and the like, include, but are not limited to, the present disclosure. The IoT devices can have different functionalities that can be controlled via the internet connection. A software application usually runs on a user's terminal device that provides connectivity for the functions of the IoT devices.

Furthermore, the interior map is a map of a home showing the relative positions between objects on an interior map. An interior map is thus accessed, which is used to determine the AR position and a viewing direction of the AR glasses on the interior map, the AR position and viewing direction of the AR glasses being based on the initial information of the AR glasses. The AR glasses preferably have at least one sensor for capturing the initial information. The at least one sensor can be a compass, a gyrometer and / or an accelerometer for determining a movement / rotation of the AR glasses and in particular an angle of rotation at least in the horizontal plane. Thus, by means of the sensor in the AR glasses, an angle between a first viewing direction and a second viewing direction can preferably be determined as initial information for determining the AR position on the interior map and the viewing direction of the AR glasses.

To determine the IoT position and an IoT device by the device, the AR glasses receive tracking information relating to the IoT device to be positioned. To acquire the tracking information, the AR glasses are preferably aimed at the IoT device by the user by looking in the direction of the IoT device to be positioned. The IoT position of the IoT device to be positioned is determined from the AR position and the viewing direction of the AR glasses in combination with the recorded tracking information relating to the IoT device to be positioned, which is recorded by the AR glasses.

The AR glasses are thus used to determine the IoT position of the IoT device on the interior map. However, so that the IoT position of the IoT device can be determined, the AR position and the viewing direction of the AR glasses on the interior map must first be determined using the initial information of the AR glasses. This makes it easier to position the IoT device and it is no longer necessary to manually position the IoT device on an interior map or to manually assign the position of the IoT device to the IoT device. This means that correctly positioned IoT devices can be addressed more easily during control. In particular, it would be possible to specify the IoT device directly on the interior map on the user's end device. Thus, for example, the same IoT devices (two light bulbs in one room) can be easily distinguished from one another and correctly addressed and controlled by the user.

• ein erstes Empfangssubmodul, das derart ausgebildet ist, dass es eine Benutzereingabe auf einem Endgerät empfängt, die die AR-Position der AR-Brille auf der Innenraumkarte und die Blickrichtung angibt.

The first determination module preferably has:

• a first receiving submodule, which is designed such that it receives a user input on a terminal that indicates the AR position of the AR glasses on the interior map and the viewing direction.

The AR position of the AR glasses on the interior map is thus provided by the user input. The user input can be a touch input or some other click or mark input of the position on the terminal. The AR position is preferably indicated directly on the interior map displayed on the terminal. Furthermore, the current viewing direction can be entered by pointing to an object in the user's current field of vision or by dragging on the terminal, starting from the AR position of the AR glasses in the direction of the viewing direction. The viewing direction is preferably indicated directly on the interior map displayed on the terminal. The AR position and viewing direction of the AR glasses are thus provided in a simple manner through the user input on the terminal, which can be used to determine the IoT position of the IoT device to be positioned.

• ein zweites Empfangssubmodul, das derart ausgebildet ist, dass es Nachverfolgungsinformationen bezüglich mindestens dreier Anker von der AR-Brille empfängt, wobei die Ankerpositionen der mindestens drei Anker auf der Innenraumkarte bekannt sind;
• ein erstes Bestimmungssubmodul, das derart ausgebildet ist, dass es Winkel zwischen jeweils zwei der mindestens drei Ankerpunkte anhand der jeweiligen Nachverfolgungsinformationen bestimmt; und
• ein zweites Bestimmungssubmodul, das derart ausgebildet ist, dass es die AR-Position und Blickrichtung der AR-Brille anhand der Winkel bestimmt.

The first determination module preferably has:

• a second receiving sub-module which is designed such that it receives tracking information relating to at least three anchors from the AR glasses, the anchor positions the at least three anchors on the interior map are known;
• a first determination sub-module which is designed such that it determines angles between two of the at least three anchor points on the basis of the respective tracking information; and
• a second determination sub-module which is designed such that it determines the AR position and viewing direction of the AR glasses on the basis of the angle.

The tracking information of the at least three anchors is preferably received from the AR glasses, the AR glasses remaining in the same position. The position of the AR glasses is thus established, with only the viewing direction of the AR glasses being changed in order to receive the tracking information of the at least three anchors. The anchors can be any point in the user's real environment, the position of the anchors on the interior map being known. By looking at one of these anchor points, tracking information is received from the AR glasses. Angles between the at least three anchors are preferably determined as tracking information received by the AR glasses. Thus, at least two angles are determined, i.e. a first angle between the first and second anchors and a second angle between the second and third anchors. These angles are preferably determined by the accelerometer or the gyrometer implemented in the AR glasses. The AR position and the viewing direction of the AR glasses can be determined on the basis of the specific angles.

• ein drittes Empfangssubmodul, das derart ausgebildet ist, dass es Nachverfolgungsinformationen bezüglich des ersten Ankers und mindestens eines zweiten Ankers aus der AR-Brille empfängt, wobei sich zumindest der erste Anker bewegt und Ankerpositionen des ersten Ankers und des mindestens eines zweiten Ankers auf der Innenraumkarte bekannt sind;
• ein drittes Bestimmungssubmodul, das derart ausgebildet ist, dass es einen zweiten Winkel zwischen dem ersten Anker und dem zweiten Anker zu einem zweiten Zeitpunkt bestimmt, wobei sich die Ankerposition zumindest des ersten Ankers zum zweiten Zeitpunkt von der Ankerposition zumindest des ersten Ankers zum ersten Zeitpunkt unterscheidet;
• ein viertes Bestimmungssubmodul, das derart ausgebildet ist, dass es die AR-Position und Blickrichtung der AR-Brille anhand des ersten Winkels und des zweiten Winkels bestimmt.

The first determination module preferably has:

• a third receiving sub-module which is designed such that it receives tracking information regarding the first anchor and at least one second anchor from the AR glasses, wherein at least the first anchor moves and anchor positions of the first anchor and the at least one second anchor are known on the interior map are;
• a third determination sub-module which is designed such that it determines a second angle between the first anchor and the second anchor at a second point in time, the anchor position of at least the first anchor at the second point in time differing from the anchor position of at least the first anchor at the first point in time ;
• a fourth determination sub-module which is designed such that it determines the AR position and viewing direction of the AR glasses on the basis of the first angle and the second angle.

Thus, the first anchor moves, the anchor position on the interior map being known. The tracking information of the first anchor can be used at a first point in time and a second point in time to determine a first angle and a second angle, wherein the AR position and line of sight of the AR glasses can be determined based on the angle. Thus, only two anchors are required, with at least one anchor moving with a known position on the interior map.

The first determination sub-module and / or the third determination sub-module and / or the fourth determination sub-module are preferably also designed such that they determine the direction / the sign of the angle between two of the anchors. Thus, initial information is recorded about whether the angle between two of the anchors is determined by turning the AR glasses from left to right or right to left. This additional information can be used to determine the position and viewing direction of the AR glasses. The direction (or mathematically the sign) of the angles is preferably determined on the basis of sensor data, e.g. Sensor data of the gyrometer, accelerometer or compass of the AR glasses.

At least one anchor preferably has an AR marking that can be recognized by the AR glasses for identifying the anchor. Automatic or at least semi-automatic recognition of the anchor can thus be simplified. The anchor is clearly identified by means of the AR marking. Therefore, the position of the anchor on the interior map can be assigned to the AR marking automatically recognized by the AR glasses.

At least one anchor is preferably identified by image recognition from the AR glasses. The automatic or at least semi-automatic recognition of the anchor can thus be simplified. Anchors can create unique domestic objects such as a television, refrigerator, stove or the like. These objects can be recognized by the AR glasses and then used as anchors to determine the position and the viewing direction of the AR glasses.

Compass information of the AR glasses is preferably determined by a compass information module, and the interior map is aligned in accordance with the compass information. The AR glasses preferably have a compass Generating compass information that can be used for an alignment of the interior map, which usually does not contain compass information. Then the interior map is oriented according to the compass information and the orientation of the interior map is determined for further use. A subsequent determination of the position and the viewing direction of the AR glasses can then be simplified by correctly aligning the interior map. As a result, the determination of the position and the viewing direction of the AR glasses can be made easier or the accuracy of the position and viewing direction can be increased by the additional compass information.

The first determination module preferably has: a sixth determination sub-module, which is designed such that it determines angles between at least two anchors and / or a direction of each angle between two of at least two anchors in accordance with the compass information.

The anchors are preferably provided by one or more IoT devices. Thus, the IoT devices are preferably marked with the AR markings. These AR markings can in particular be provided by the manufacturer of the IoT devices in order to facilitate the positioning of the IoT devices. In essentially the same way, the appearance of the IoT devices is known to the manufacturer of the IoT devices, whereby this information can be used by the image recognition for the automatic recognition of the IoT devices through the AR glasses.

The interior map is preferably provided by a vacuum cleaner and / or a surveillance camera. Preferably, the surveillance camera is any camera that can be a time-of-flight camera, a stereo camera or any other optical device that provides 3D information, e.g. LIDAR (light detecting and ranging) or the like.

• ein achtes Bestimmungssubmodul, das derart ausgebildet ist, dass es zumindest eine zweite Blickrichtung der AR-Brille von einer zweiten AR-Position zu dem zu positionierenden IoT-Gerät bestimmt;
• ein neuntes Bestimmungssubmodul, das derart ausgebildet ist, dass es anhand der ersten AR-Position, der ersten Blickrichtung, der zweiten AR-Position und der zweiten Blickrichtung die IoT-Position des zu positionierenden IoT-Geräts auf der Innenraumkarte bestimmt.

The second determination module preferably has: a seventh determination sub-module that is designed such that it determines a first viewing direction of the AR glasses from a first AR position to the IoT device to be positioned;

• an eighth determination sub-module which is designed in such a way that it determines at least one second viewing direction of the AR glasses from a second AR position to the IoT device to be positioned;
• a ninth determination sub-module which is designed such that it determines the IoT position of the IoT device to be positioned on the interior map using the first AR position, the first viewing direction, the second AR position and the second viewing direction.

A third viewing direction of the AR glasses from a third AR position to the IoT device to be positioned is preferably also used, in particular if no compass information is available. In this case, the IoT position of the IoT device to be positioned is determined analogously to the implementation of respective modules which are designed such that they determine the AR position of the AR glasses, as described above and below. Positioning an IoT device with an unknown position on the interior map is thus simplified by means of the AR glasses with a known AR position and line of sight. This IoT position of the IoT device can be assigned to the IoT device, used to control the IoT device and / or saved. When determining the IoT position of the IoT device, this IoT device can preferably be used as an anchor for further determining an AR position and a viewing angle of the AR glasses.

The second determination module preferably further comprises: a tenth determination sub-module for determining the first viewing direction and the second viewing direction based on the compass information.

In a further aspect, a positioning system is provided that includes the device described above and the AR glasses connected to the device. The device and the AR glasses are connected wirelessly or via a wire connection.

In a further aspect, a smart home system is provided that has a plurality of IoT devices and a positioning system as described above.

The present device thus simplifies the positioning of an IoT device by using AR glasses. First, the AR position and the direction of view of the AR glasses are determined on an interior map that is being accessed, and this information is used to determine the IoT position of the IoT device to be positioned. It is no longer necessary to manually specify the location of the IoT device. Using the known IoT position of the IoT device, the control of the IoT devices within a home automation application can be easily controlled and the identification of the individual IoT devices within the home automation application is facilitated by the known IoT position. Thus, even if the same or in If essentially the same IoT devices are present more than once in just one room, these IoT devices can be clearly distinguished from one another based on their exact position.

Figure list

• 1 ein erstes Beispiel gemäß der vorliegenden Offenbarung,
• 2 eine beispielhafte Innenraumkarte gemäß der vorliegenden Offenbarung,
• 3 ein detailliertes Beispiel gemäß der vorliegenden Offenbarung,
• 4 ein detailliertes Beispiel zum Bestimmen der AR-Position und Blickrichtung der AR-Brille,
• 5 ein detailliertes Beispiel gemäß der vorliegenden Offenbarung,
• 6 eine schematische Darstellung zum Bestimmen der AR-Position und Blickrichtung der AR-Brille gemäß der vorliegenden Offenbarung,
• 7 eine schematische Darstellung zur Berechnung der AR-Position der AR-Brille,
• 8 ein detailliertes Beispiel gemäß der vorliegenden Offenbarung,
• 9 ein detailliertes Beispiel gemäß der vorliegenden Offenbarung,
• 10 eine beispielhafte Darstellung zur Berechnung der IoT-Position des zu positionierenden IoT-Geräts, und
• 11 eine Datenverarbeitungseinrichtung gemäß der vorliegenden Offenbarung.

Reference will now be made, by way of example, to the accompanying drawings showing embodiments of the present application. In the figures show:

• 1 a first example according to the present disclosure,
• 2 an exemplary interior map in accordance with the present disclosure;
• 3 a detailed example according to the present disclosure,
• 4th a detailed example for determining the AR position and viewing direction of the AR glasses,
• 5 a detailed example according to the present disclosure,
• 6th a schematic representation for determining the AR position and viewing direction of the AR glasses according to the present disclosure,
• 7th a schematic representation for calculating the AR position of the AR glasses,
• 8th a detailed example according to the present disclosure,
• 9 a detailed example according to the present disclosure,
• 10 an exemplary representation for calculating the IoT position of the IoT device to be positioned, and
• 11 a data processing device according to the present disclosure.

Detailed description of the embodiments

The present application describes a device for determining the position of an IoT device on an interior map, which is referred to below as the IoT position. AR glasses are used to position the IoT device. To determine the IoT position of the IoT device, the position and viewing direction of the AR glasses must be determined on the interior map. The position of the AR glasses is referred to as the AR position in the following.

Some IoT devices are already present in home automation systems today that generate an indoor map to perform their tasks. However, these cards are not shared with other IoT devices and are typically not shared with the home automation system. Thus, the exact position of IoT devices within the home automation system is currently not known or is not used to control the IoT devices.

One of the IoT devices that is able to generate an interior map is a vacuum cleaner robot. In many home automation systems that include a vacuum cleaner robot, the robot is able to scan the home (typically with a SLAM laser) and map the home to accomplish its tasks. An example of an interior map 10 is in 2 shown. On the interior map 10 are a vacuum cleaner robot 12 and a docking station 14th shown. The interior map also shows the movement path of the vacuum cleaner robot as an example 10 that moves the robot hoover around objects while cleaning the entire area. Such a map is typically only used by the robot vacuum cleaner to define an optimal vacuum path in the home, providing users with a clear representation of the path the vacuum cleaner is following.

As the number of domestic IoT devices increases, it becomes advantageous that all IoT devices share such a map and position each of them on the map. This enables users to visually access their IoT devices through the home automation application instead of having to find the IoT device on a list of rooms or devices.

For example, the home contains several light bulbs, a set of speakers, a TV, and a vacuum cleaner, all of which can be operated from the home automation application. Operating incandescent lamps is usually a tedious process when there are several incandescent lamps in the home, since the user must first identify exactly in the application which incandescent lamp he wishes to operate.

Thus, by using a map showing the exact location of the IoT device such as a light bulb, indicates, possible to point to the desired IoT device and to click on it to activate the IoT device, i.e. switch it on.

The present disclosure provides a device which is designed such that it determines the IoT position of IoT devices by means of AR glasses. To determine the IoT position of the IoT devices on the interior map, however, the AR position and viewing direction of the AR glasses must first be determined on the interior map. As soon as this has been done, the timely tracking of the AR glasses can be carried out, as one or more sensors in the AR glasses are able to detect the movement of the AR glasses. Track glasses. Such a sensor in the AR glasses can be an accelerometer, a gyrometer, a compass or a video camera. Preferably more than one and more preferably all of these sensors are implemented in combination in the AR glasses.

• S01: Zugreifen auf eine Innenraumkarte;
• S02: Bestimmen einer AR-Position der AR-Brille auf der Innenraumkarte und einer Blickrichtung der AR-Brille auf der Basis von Anfangsinformationen der AR-Brille;
• S03: Empfangen von Nachverfolgungsinformationen bezüglich des zu positionierenden IoT-Geräts aus der AR-Brille; und
• S04: Bestimmen der IoT-Position des zu positionierenden IoT-Geräts anhand der bestimmten AR-Position und Blickrichtung in Kombination mit den Nachverfolgungsinformationen der AR-Brille.

A schematic in 1 Possible implementation shown includes:

• S01 : Accessing an interior map;
• S02 : Determining an AR position of the AR glasses on the interior map and a viewing direction of the AR glasses on the basis of initial information of the AR glasses;
• S03 : Receiving tracking information regarding the IoT device to be positioned from the AR glasses; and
• S04 : Determination of the IoT position of the IoT device to be positioned based on the determined AR position and viewing direction in combination with the tracking information of the AR glasses.

To determine the AR position and viewing direction of the AR glasses on the interior map, in a first solution, the user can enter the AR position of the AR glasses and the viewing direction directly into the interior map on a terminal. In this case, S021 the device receives the user input on a terminal, which indicates the AR position of the AR glasses on the interior map and the viewing direction. The terminal device is preferably the device according to the present invention or the terminal device for transmitting the user input into the device is connected to the device in a wireless or wired manner.

As in 4th shown showing the interior map 10 shows by way of example, the AR glasses are in the AR position 16 , which is indicated by "X" and in the direction of the vacuum cleaner robot 12 is directed. Thus, the user can on the interior map 10 the AR position on his device 16 and also the direction of view indicated by the arrow 18th is indicated by a user input, such as a click, touch and / or drag input. The AR position and the viewing direction of the AR glasses are therefore known. A change in the AR position and line of sight is tracked in real time by the sensors of the AR glasses, and a change in the AR position and / or line of sight of the AR glasses can be determined. As soon as the AR glasses have been located on the interior map in combination with the viewing direction, this information can be used to position an IoT device to determine the IoT position of the IoT device.

There is no need to determine an absolute position of the AR glasses. A relative position relative to the interior map is sufficient.

Although it has been described above that the indoor map generated by a robot vacuum cleaner is used, any other map can be used by any other device such as e.g. a surveillance camera or the like, can be used.

The device according to the present invention is adapted accordingly to what has been described above. The device provides respective modules for providing the respective functionalities.

• S022: Empfangen von Nachverfolgungsinformationen bezüglich mindestens dreier Anker aus der AR-Brille, wobei die Ankerpositionen der mindestens drei Anker auf der Innenraumkarte bekannt sind;
• S023: Bestimmen der Winkel zwischen jeweils zwei der mindestens drei Anker; und
• S024: Bestimmen der AR-Position und Blickrichtung anhand der Winkel.

A second solution for determining the AR position and viewing direction of the AR glasses, as shown schematically in 5 shown includes:

• S022 : Receiving tracking information regarding at least three anchors from the AR glasses, the anchor positions of the at least three anchors on the interior map being known;
• S023 : Determining the angles between any two of the at least three anchors; and
• S024 : Determination of the AR position and viewing direction based on the angle.

Thus, three anchors are used to determine the AR position and the viewing direction of the AR glasses, the position of the anchors on the interior map 10 is known. The three anchors of IoT devices can be provided with a known position. Especially if a vacuum cleaner robot 12 is present in the home automation system, the docking station 14th of the vacuum cleaner robot 12 with a known position on the interior map 10 used as an anchor; the vacuum cleaner robot 12 even with a known position on the interior map 10 can be used as an additional anchor. Thus, only one further anchor with a known position is required to simplify the determination of the AR position and the direction of view of the AR glasses.

The situation is in 6th shown schematically with a first anchor A, a second anchor B and a third anchor C, the positions of the anchors A, B, C on the interior map 10 are known. To determine the AR position of the AR glasses, a first angle θ _{AB is determined} between the anchor A and the anchor B, the possible positions of the AR glasses on a first arc 20th indicates. The determination of the angle θ _AB can take place by one or more sensors implemented in the AR glasses. Thus, the user first sets up the AR Glasses on anchor A, indicate the object of anchor A to identify the anchor so that the position of anchor A can be called up on the interior map. Then the user turns his head to point the AR glasses at anchor B, indicates the object of anchor B, so that the position of anchor B can be called up on the indoor map. By turning the head and directing the AR glasses from the first anchor A to the second anchor B, the AR glasses are rotated through the angle θ _AB , which is determined and used to determine the AR position and the angle of the AR glasses becomes. The user preferably remains in the same position.

The same implementation for determining the angle θ _AB as described above is carried out again for the anchor points B and C for determining the respective angle θ _BC . The angle θ _BC forms a second arc 22nd a possible position of the AR glasses, the interface I of the first arc 20th and the second arch 22nd indicates the AR position of the AR glasses. Using the AR position of the AR glasses, the viewing direction in the direction of one of the anchors A, B or C can be determined and the AR position and viewing direction of the AR glasses are thus completely determined.

1. 1.) Berücksichtigen eines Punkts H in 7 als Punkt auf dem Median des Liniensegments AB (Dies bedeutet, dass das Verbindungsliniensegment HM rechtwinklig zu dem Liniensegment AB ist und das Liniensegment AB in der Mitte schneidet. Somit wird ein gleichschenkliges Dreieck ΔABH erzeugt), auf dem der Winkel 24 (als >AHB bezeichnet - wobei das Liniensegment AH um den Punkt H geschwenkt wird, bis eine Überlappung mit dem Liniensegment HB erfolgt) gleich θ_AB ist. Dabei wird die Richtung des Winkels von dem Pfeil 24 angegeben und wird hier und im Folgenden entgegen dem Uhrzeigersinn als positive Richtung definiert.
2. 2.) Das Dreieck ΔHBM mit M in der Mitte von AB wird bestimmt, und der Abstand HM wird berechnet durch $$HM=\frac{1}{2}ABcot\frac{{\theta }_{AB}}{2}.$$
   
3. 3.) Bei dem Dreieck ΔHOP mit P in der Mitte des Abstands HB und dem Liniensegment OP rechtwinklig zu dem Liniensegment HB ist es möglich, den Abstand HO, der der Radius des ersten Bogens 20 ist, zu berechnen mit $$\text{cos}\frac{\mathrm{\theta }AB}{2}=\frac{HM}{HB}=\frac{HB/2}{HO},$$
   
   $$\iff \text{HO}=\frac{\text{HB}/2}{\text{cos}\frac{{\mathrm{\theta }}_{AB}}{2}}.$$
   
4. 4.) Der Abstand HM, der durch HO ausgedrückt wird, wird angegeben durch $$HM=2HOco{s}^2\frac{{\theta }_{AB}}{2}$$
   
   durch Verwenden der Beziehung HM = HB cos θ_AB aus 7.
5. 5.) Dann kann der Abstand OM berechnet werden durch $$OM=HM-HO,$$
   
   wobei das Einsetzen der Formeln für HM und HO ergibt $$OM=\frac{1}{2}ABcot{\theta }_{AB}.$$
   

7th shows schematically a situation with only two anchors for a simplified representation of the mathematical derivation for determining the first arc 20th . Determining the first arc 20th includes:

1. 1.) Consider a point H in 7th as a point on the median of the line segment AB (this means that the connecting line segment HM is perpendicular to the line segment AB and intersects the line segment AB in the middle. Thus, an isosceles triangle ΔABH is created) on which the angle 24 (referred to as> AHB - where line segment AH is panned around point H until it overlaps line segment HB) equals θ _AB . The direction of the angle is indicated by the arrow 24 and is defined here and below as a counterclockwise direction as the positive direction.
2. 2.) The triangle ΔHBM with M in the middle of AB is determined and the distance HM is calculated by $$H\mathrm{M.}=\frac{1}{2}\mathrm{A.}\mathrm{B.}cOt\frac{{\theta }_{\mathrm{A.}\mathrm{B.}}}{2}.$$
   
3. 3.) With the triangle ΔHOP with P in the middle of the distance HB and the line segment OP at right angles to the line segment HB, it is possible to use the distance HO, which is the radius of the first arc 20th is to be calculated with $$\text{cos}\frac{\mathrm{\theta }\mathrm{A.}\mathrm{B.}}{2}=\frac{H\mathrm{M.}}{H\mathrm{B.}}=\frac{H\mathrm{B.}/2}{HO},$$
   
   $$\iff \text{HO}=\frac{\text{HB}/2}{\text{cos}\frac{{\mathrm{\theta }}_{\mathrm{A.}\mathrm{B.}}}{2}}.$$
   
4. 4.) The distance HM expressed by HO is given by $$H\mathrm{M.}=2HOcO{s}^2\frac{{\theta }_{\mathrm{A.}\mathrm{B.}}}{2}$$
   
   by using the relationship HM = HB cos θ _AB 7th .
5. 5.) Then the distance OM can be calculated by $$O\mathrm{M.}=H\mathrm{M.}-HO,$$
   
   where inserting the formulas for HM and HO gives $$O\mathrm{M.}=\frac{1}{2}\mathrm{A.}\mathrm{B.}cOt{\theta }_{\mathrm{A.}\mathrm{B.}}.$$
   

The trigonometric identities sin 2θ = 2 sin θ cos θ and cos 2θ = 2 cos ² θ - 1 are used.

Thus, from knowing the length OM and knowing the point M, the position of O is known. Furthermore, the radius is HO for the first arc 20th which passes through points A, B and H is also known. Thus it is possible to use the quadratic equation for the first cycle 20th to calculate. The same implementation is used for anchor B and anchor C to compute the quadratic equation for the second cycle 22nd used as in 6th shown. Thus, the system becomes the two quadratic equations for the first arc 20th and the second arch 22nd solved, whereby two solutions are offered: B and the actual position of the AR glasses, indicated by the interface I of the first cycle 20th and the second cycle 22nd .

As in 7th shown, knowing only the value of θ _{AB there are} two possible first arcs 20th and 20 '. However, if the direction / sign of the angle θ _{AB is} known, the first arc becomes unique. As in 7th given, then if the angle with the direction is determined as + θ _AB , the arc becomes 20th determined as the first arch. When the angle with the direction is determined as -θ _AB , the arc 20 'becomes uniquely possible AR- Specify the positions of the AR glasses. Thus, the direction of the angle does not result in any ambiguity with regard to the first arc 20th and in essentially the same way also of the second sheet 22nd . Interface I therefore also offers a unique solution. The direction of the angles is preferably determined by one of the sensors in the AR glasses and / or on the basis of the compass information.

However, even if the directions of the angles between the anchors are not known, it is still possible to determine a solution for the AR position of the AR glasses. If the AR glasses are not positioned at anchor B or at least not too close and the angle> ABC between the anchor is greater than 0 ° and less than 180 °, the point of intersection I is the two arcs 20th , 22nd that is far from the anchor B, the position of the AR glasses.

Alternatively, a fourth anchor must be used to provide a unique solution to the location of the AR glasses on the interior map.

• S025: Empfangen von Nachverfolgungsinformationen bezüglich eines ersten Ankers und mindestens eines zweiten Ankers aus der AR-Brille, wobei sich zumindest der erste Anker bewegt und die Ankerpositionen des ersten Ankers und des zweiten Ankers auf der Innenraumkarte bekannt sind;
• S026: Bestimmen eines ersten Winkels zwischen dem ersten Anker und dem zweiten Anker zu einem ersten Zeitpunkt;
• S027: Bestimmen eines zweiten Winkels zwischen dem ersten Anker und dem zweiten Anker zu einem zweiten Zeitpunkt, wobei sich die Ankerposition zumindest des ersten Ankers zum zweiten Zeitpunkt von der Ankerposition zumindest des ersten Ankers zum ersten Zeitpunkt unterscheidet; und
• S028: Bestimmen der AR-Position und Blickrichtung der AR-Brille anhand des ersten Winkels und des zweiten Winkels.

Preferably, if one of the anchors from the vacuum cleaner robot 12 is provided that moves with a known position, the vacuum cleaner robot 12 can be used as a first anchor at a first point in time and as a second anchor at a second point in time, as described above. Thus, the determination of the AR position and the viewing direction of the AR glasses is in 8th shown and includes:

• S025 : Receiving tracking information relating to a first anchor and at least one second anchor from the AR glasses, wherein at least the first anchor is moving and the anchor positions of the first anchor and the second anchor are known on the interior map;
• S026 : Determining a first angle between the first anchor and the second anchor at a first point in time;
• S027 : Determining a second angle between the first anchor and the second anchor at a second point in time, wherein the anchor position of at least the first anchor at the second point in time differs from the anchor position of at least the first anchor at the first point in time; and
• S028 : Determination of the AR position and viewing direction of the AR glasses based on the first angle and the second angle.

In order to simplify the determination of the AR position and the viewing direction of the AR glasses, it is therefore sufficient that both the vacuum cleaner robot 12 that moves as well as the docking station 14th have a known position on the interior map.

In some embodiments, dedicated AR markers are used on at least one, more than one, or all of the home IoT devices. With the help of such AR markers, it becomes unnecessary for the user to identify which IoT devices he is currently looking at, as the AR marker provides this information and uniquely identifies the IoT device. As a result, the AR glasses detect the IoT devices simply by searching the home and identify them using the AR marker. Once this is done, the home automation system can determine the position of the AR glasses by mathematically calculating the same based on the IoT position of the IoT devices with the AR marker and the viewing angles between them that have been reported by the AR glasses, as described above. Alternatively, the IoT position of an individual IoT device can also be determined by viewing it from two different positions of the AR glasses if compass information is available, or three different positions of the AR glasses if only angle information (between the different viewing positions) is known , be calculated.

The obvious advantage of this solution is not only that it is fully automatic, but it can also be completely transparent to the user, provided that there are enough (at least 3) domestic IoT devices that the AR glasses can detect and their Positions on the map are known. For this solution it is only necessary to attach AR markers to the IoT devices. Such AR markers can be provided by the IoT device manufacturer or downloaded later.

In a further embodiment of this disclosure, the above-mentioned embodiments can also be used in real-time tracking of the AR glasses in the home in that the positioning of the AR glasses is always running in the background.

In a further embodiment, at least one, more than one or all IoT devices are identified by means of object recognition techniques. With some IoT devices, it is possible to use machine learning techniques and train the AR glasses software to recognize specific IoT devices. This has the obvious advantage that this is a fully automated and completely transparent solution.

In some embodiments, the AR glasses have a compass for capturing compass information. In a first implementation, the compass information is then linked to the interior map and provides an absolute alignment of the interior map relative to the compass information. For this purpose, when the AR position and viewing direction of the AR glasses are determined for the first time, the compass information is recorded relative to this position.

Then later, when the AR position and the line of sight of the AR glasses have been determined, the compass information can be used and synchronized with the aligned map of the home in order to better position the AR glasses and increase the accuracy. Furthermore, if compass information is available, only two anchor points are required to uniquely identify the AR position of the AR glasses. Using a third anchor can still lead to higher accuracy.

• S041: Bestimmen einer ersten Blickrichtung der AR-Brille von einer ersten AR-Position zu dem zu positionierenden IoT-Gerät;
• S042: Bestimmen einer zweiten Blickrichtung der AR-Brille von einer zweiten AR-Position zu dem zu positionierenden IoT-Gerät; und
• S043: Bestimmen anhand der ersten AR-Position, der ersten Blickrichtung, der zweiten AR-Position und der zweiten Blickrichtung die IoT-Position des zu positionierenden IoT-Gerätes auf der Innenraumkarte.

9 Figure 3 shows an embodiment for positioning the IoT device comprising:

• S041 : Determining a first viewing direction of the AR glasses from a first AR position to the IoT device to be positioned;
• S042 : Determining a second viewing direction of the AR glasses from a second AR position to the IoT device to be positioned; and
• S043 : Determine the IoT position of the IoT device to be positioned on the interior map based on the first AR position, the first viewing direction, the second AR position and the second viewing direction.

10 shows schematically the situation for an IoT device to be positioned 30th . Since the AR glasses have a known AR position and viewing direction, a first AR position becomes 32 with a compass a direction of view determined as angle α. Then a second AR position 34 a second line of sight in the direction of the IoT device to be positioned 30th certainly. Thus, based on the first AR position 32 , the second AR position 34 , the first viewing direction α and the second viewing direction β the IoT position of the IoT device 30th can be determined with a compass. This position can be assigned to the IoT device and saved for further use. When the IoT device 30th on the interior map 10 is positioned, the IoT device can 30th can be used as an anchor to determine the AR position and viewing direction of the AR glasses.

If compass information is not available, only the θ angle (the viewing angle between the two viewing positions) is known and a third viewing AR position is required to calculate the actual IoT position of the IoT device. This is the same process that is used to calculate the AR position of the AR glasses described above.

11 shows a device 110 that has a processor 112 having, the processor 112 with a memory 114 connected is. The memory 114 stores instructions to be used when executed by the processor 112 the respective modules of the device are implemented. Further is the processor 112 for receiving tracking information, viewing directions, compass information or the like from the AR glasses with a communication module 116 connected. Data can be transmitted wirelessly via WLAN, WIFI, GSM, 3G, 4G, 5G, Bluetooth, ZigBee or any other standard.

Alternatively, instead of the communication module 116 the processor 112 and the memory 114 of the device 110 connected directly to the AR glasses in the form of a single device.

Preferably the processor can be a general purpose processor, an ASIC, FPGA or any other type of processing unit.

Preferably the device is a terminal, e.g. a smartphone, tablet, laptop or any other type of device. In particular, the device also has a display for displaying the interior map. The IoT devices can then be displayed at their respective positions on the interior map. By interacting with the representation of the IoT devices on the end device, the respective domestic IoT devices are controlled in the home automation system.

Thus, in the present disclosure, real-time positioning and tracking of AR glasses worn by a user on an interior map are made possible. IoT devices can be positioned using the AR glasses. The precise IoT position of the IoT device makes it easier to control the IoT devices. In particular, an indoor map can be displayed on a terminal indicating the IoT devices on the indoor map, wherein the IoT positions of each IoT device are determined according to the present disclosure. The IoT devices can then be easily accessed via the position on the interior map.

Claims (14)

Device for positioning an Internet of Things, IoT, device with augmented reality, AR, glasses, comprising: an access module which is designed such that it accesses an interior map (10); a first determination module which is designed in such a way that there is an AR position (16) of the AR glasses on the interior map (10) and a viewing direction (18) determines the AR glasses based on initial information of the AR glasses; a receiving module which is designed such that it receives tracking information relating to the IoT device to be positioned from the AR glasses; and a second determination module which is designed such that an IoT position of the IoT device (12) to be positioned is determined on the basis of the determined AR position and viewing direction (18) in combination with the tracking information from the AR glasses. Device according to Claim 1 , characterized in that the initial information of the AR glasses comprises: a first receiving sub-module which is designed such that it receives a user input on a terminal device, the AR position (16) of the AR glasses on the interior map (10) and indicates the viewing direction (18). Device according to Claim 1 , characterized in that the first determination module comprises: a second receiving sub-module which is designed such that it receives tracking information regarding at least three anchors from the AR glasses, the anchor positions (A, B, C) of the at least three anchors on the interior map (10) are known; a first determination sub-module which is designed such that it determines angles between in each case two of the at least three anchor points from the respective tracking information; and a second determination sub-module which is designed such that it determines the AR position and viewing direction of the AR glasses on the basis of the angles. Device according to Claim 1 , characterized in that the first determination module comprises: a third receiving sub-module which is designed such that it receives tracking information regarding the first anchor and at least one second anchor from the AR glasses, wherein at least the first anchor moves and anchor positions of the first anchor and the at least one second anchor is known on the interior map; a third determination sub-module which is designed such that it determines a first angle between the first anchor and the second anchor at a first point in time; a fourth determination sub-module which is designed such that it determines a second angle between the first anchor and the second anchor at a second point in time, the anchor position of at least the first anchor at the second point in time differing from the anchor position of at least the first anchor at the first point in time ; a fifth determination sub-module which is designed such that it determines the AR position and viewing direction of the AR glasses on the basis of the first angle and the second angle. Device according to Claim 3 or 4th , characterized in that one or more of the first determination sub-module, the third determination sub-module and the fourth determination sub-module are further designed such that they determine the direction / the sign of an angle between two of the anchors. Device according to one of the Claims 3 to 5 , characterized in that at least one anchor has an AR marking recognizable for the AR glasses for identifying the anchor. Device according to one of the Claims 3 to 6th , characterized in that at least one anchor is recognized by image recognition from the AR glasses. Device according to one of the Claims 1 to 7th , characterized by a compass information module which is designed such that it determines compass information of the AR glasses, wherein the interior map is aligned according to the compass information. Device according to Claim 8 , characterized in that the first determination module has: a sixth determination sub-module which is designed such that it determines angles between at least two anchors and / or a direction of each angle between two of at least two anchors in accordance with the compass information. Device according to one of the Claims 3 to 9 , characterized in that the anchors are provided by one or more IoT devices. Device according to one of the Claims 1 to 10 , characterized in that the interior card (10) is provided by a vacuum cleaner (12) and / or a surveillance camera. Device according to one of the Claims 1 to 11 , characterized in that the second determination module has: a seventh determination sub-module which is designed such that it determines a first viewing direction of the AR glasses from a first AR position to the IoT device to be positioned; an eighth determination sub-module which is designed in such a way that it determines at least one second viewing direction of the AR glasses from a second AR position to the IoT device to be positioned; and a ninth determination sub-module which is designed such that it determines the IoT position of the IoT device to be positioned on the interior map using the first AR position, the first viewing direction, the second AR position and the second viewing direction. Positioning system that moves the device according to one of the Claims 1 to 12 and AR glasses connected to the device. Smart home system that employs a variety of IoT devices and a positioning system Claim 13 having.

Images