WO2023012215A1

WO2023012215A1 - Integration with intelligent objects in the surroundings of mobile devices

Info

Publication number: WO2023012215A1
Application number: PCT/EP2022/071813
Authority: WO
Inventors: Klaus David
Original assignee: Universität Kassel
Priority date: 2021-08-04
Filing date: 2022-08-03
Publication date: 2023-02-09
Also published as: DE102021120315A1

Abstract

The invention relates to a device (16) for integrating with an intelligent object (14) in the surroundings of a mobile device (10), comprising: an input interface (24) for receiving surroundings sensor data and alignment sensor data, wherein the surroundings sensor data comprises information on the position of the intelligent object relative to the mobile device, and the alignment data comprises information on the alignment of the mobile device relative to the surroundings; an alignment unit (26) for detecting whether the mobile device is aligned towards the intelligent object on the basis of the surroundings sensor data and the alignment sensor data; a communication unit (28) for establishing a communication connection with the intelligent object when the mobile device is aligned towards the intelligent object; and a user interface (30) for informing a user of the mobile device when the communication connection has been established. The invention additionally relates to a mobile device (19) and a method.

Description

University of Kassel

Interacting with smart objects around mobile devices

The present invention relates to a device for interacting with a smart object in an environment of a mobile device. The present invention further relates to a mobile device and a method for interacting with an intelligent object in an environment of the mobile device and a computer program product.

Cities are currently facing the challenge of including the digitization of society in urban development in order to make the city efficient and technologically advanced. In this context, the keyword "Smart City" summarizes approaches for applying progressive digitization in cities and communities. In particular, the equipping and upgrading of existing and new infrastructures with digital technology and the linking of previously separate infrastructures or their subsystems should be promoted.

Smartphones in particular, but also tablet computers, smart glasses, smart watches and other mobile devices are omnipresent and offer extensive possibilities for use in a smart city. Applications can be found, among other things, in the field of transport, for example when displaying current timetables for public transport, information, for example when guiding and informing tourists and Environmental protection, or environmental protection, for example in the recording and targeted avoidance of local pollution.

In this context, the term "Internet of things and services" is often interpreted to mean that sensors are embedded in intelligent objects within a smart city, which collect data and make it available for applications. An intelligent object is understood to mean in particular an object that is capable of communication in some way and with which an interaction in the sense of a reaction of the intelligent object to inputs, signals or queries is possible. Intelligent objects can be, for example, timetable displays at tram stops, traffic lights, street lights, irrigation systems, advertising pillars, information boards, sensors, stumbling blocks, restaurants (menu card, reservation), etc.

In this context, US 2020 073 482 A1 discloses a method and a system for detecting tactile interactions in augmented reality. Program actions may be initiated upon detection of a user's predefined gesture with a real-world object. Users can interact with their environment in augmented reality by detecting interaction with real objects using a combination of location and motion detection with identification using wearable sensors. Based on a detected user interaction with a real-world object, a predefined gesture may be identified and a program action associated with a corresponding target interaction with the real-world object performed. In some cases, the user's experience can be enhanced by providing haptic feedback on a tactile gesture and event.

A challenge in this context lies in simplifying the interaction of city dwellers or users of mobile devices with intelligent objects. For example, older people often have difficulties using modern technologies and using their user interfaces intuitively capture. Approaches to integrating digitization into everyday life in a smart city or in other contexts are only as good as their interface or the possibility of interaction with users in terms of their benefits.

Proceeding from this, the object of the present invention is to enable or simplify interaction with intelligent objects. In particular, an approach is to be created that enables an intuitively comprehensible interaction. Regardless of the type of intelligent object and the interaction option provided, a simple and easy-to-use interaction option should be created.

To solve this problem, the present invention relates in a first aspect to an apparatus for interacting with an intelligent object in an environment of a mobile device, comprising: an input interface for receiving environment sensor data and orientation sensor data, the environment sensor data relating to information on a relative position of the intelligent object to the mobile device and the orientation sensor data includes information about an orientation of the mobile device in relation to the environment; an alignment unit for detecting whether the mobile device is aligned with the smart object based on the environmental sensor data and the alignment sensor data; a communication unit to establish a communication link with the smart object when the mobile device points to the smart object; and a user interface for notifying a user of the mobile device when the communication link is established. In a further aspect, the present invention relates to a mobile device for interacting with an intelligent object in an environment of the mobile device, comprising: an environment sensor for detecting a relative position of the intelligent object; an orientation sensor for detecting an orientation of the mobile device; and an apparatus as described above.

Further aspects of the invention relate to a method designed according to the device and a computer program product with program code for performing the steps of the method when the program code is executed on a computer. In addition, one aspect of the invention relates to a storage medium on which a computer program is stored. The computer program, when executed on a computer, causes execution of the method described herein.

Preferred developments of the invention are described in the dependent claims. It goes without saying that the features mentioned above and those still to be explained below can be used not only in the combination specified in each case, but also in other combinations or on their own, without departing from the scope of the present invention. In particular, the method, the mobile device and the computer program product can be implemented in accordance with the configurations described for the device in the dependent claims.

According to the invention, it is provided that environmental sensor data on the one hand and alignment sensor data on the other hand are processed in the device. Information about a relative position of the intelligent object in relation to the mobile device is comprised by an environment sensor. The aim is to determine where the intelligent object is located. For this purpose in particular receive an absolute or relative position of the intelligent object. Information about how the mobile device is oriented in relation to the environment is received by an orientation sensor. In particular, therefore, a relative position of the mobile device in relation to the environment is received.

Based on the environmental sensor data and orientation sensor data, it can be detected whether the mobile device is aimed at the intelligent object. In particular, it can be determined whether a direction predefined for the mobile device corresponds to the direction in which the mobile object is located. The direction predefined for the mobile device can, for example, correspond to a direction orthogonal to a main display of the mobile device or also a direction along a longitudinal axis of the mobile device, depending on the mobile device or its orientation. In other words, the direction predefined for the mobile device can also be adapted or changed, if necessary, based on the alignment of the mobile device.

If it is determined that the mobile device is aimed at the intelligent object, a communication link is established with the intelligent object using a communication unit. In particular, a possibility is established to interact with the intelligent object. As soon as this possibility has been established and the communication connection has been established, the user of the mobile device is informed about this via a user interface of the device or a corresponding output unit of the mobile device. The user can then interact with the intelligent object in its environment.

In comparison to previous approaches of establishing a connection with intelligent objects, for example in a smart city environment, the inventively provided detection of the orientation and establishment of a communication connection depending on an orientation brings about a simplification. In other words, the initiation of communication becomes more intuitive and automatically matches the current context of the user. In comparison to approaches that require a button to be pressed or a touch display to be operated, direct input by the user can be dispensed with, for example. For the user there is an intuitive possibility of interaction with intelligent objects. The user can directly interact with the smart objects without any complex configuration. This simplifies the interaction between users and intelligent objects, which can lead to an improved acceptance of smart city concepts.

In a preferred embodiment, the device comprises a gesture unit for recognizing a gesture performed by a user of the mobile device based on the orientation sensor data. The gesture unit is preferably designed to recognize the gesture based on a machine learning approach, in particular based on a pre-trained artificial neural network. A gesture is understood to mean, in particular, a movement of the mobile device that is carried out by the user using the mobile device. For example, a user can swivel their smartphone and this movement can be detected using a sensor. This gesture can then be recognized. In particular, it is possible that a pre-trained artificial neural network or other machine learning approach is used to reliably and quickly recognize the gesture. A database can also be used for the recognition.

In a preferred refinement, the communication unit is designed to establish the communication connection on the condition that an activation gesture has previously been recognized. Additionally or alternatively, the communication unit is designed to end the communication connection when the communication connection is established and a deactivation gesture is recognized. In other words, it is possible for the establishment and termination of the communication connection to be made dependent on a gesture in addition to the detection of the alignment or the alignment of the mobile device with respect to the intelligent object. This can be achieved that em user of the mobile device only consciously establishes a communication link. Unintentionally interacting with smart objects is avoided. The activation and deactivation gestures can, for example, correspond to rotating and/or turning movements of a smartphone. The possibility of interaction is further simplified and the use is improved.

In a preferred refinement, the communication unit is designed to transmit a control command to the intelligent object when the communication connection is established and a control gesture is recognized. It is also possible for the actual interaction with the intelligent object to take place based on gesture recognition. A control gesture can be a gesture that causes a specific action or reaction of the intelligent object. For example, further information can be requested from the intelligent object by a swiping movement, or a display of the intelligent object can be changed on a display of the intelligent object. An intuitive and easy-to-use interaction option is provided through the use of gesture recognition and detection of a control gesture. It becomes possible for users of the device according to the invention to interact with intelligent objects in an intuitive manner.

In a preferred refinement, the communication unit is designed to receive interaction data with information on the user's options for interacting with the intelligent object. The user interface is designed to inform the user about the interaction options based on the interaction data received, preferably by means of an indication on a display. Advantageously, various options for interacting with the intelligent object are first received (interaction options). The user is then informed about the interaction options and can start the desired interaction. For example, available information can be displayed so that the user of the mobile device can decide what information he needs and wanted to query. It is also possible for control options to be displayed along with the appropriate user inputs, enabling the user to interact with and control the smart object. In this respect, the possibility of interaction is further improved and simplified.

In a preferred embodiment, the user interface is designed to output an acoustic, optical and/or haptic signal, in particular a haptic vibration signal, by means of a vibration actuator of the mobile device. The user interface can announce the establishment of the communication connection by a corresponding signal to the user. The use of a vibration actuator of the mobile device is particularly advantageous and intuitive. In other words, the mobile device can display a type of locking or make it tangible for the user when the communication connection has been established. Operation is further simplified and designed to be intuitive for the user.

In a preferred embodiment, the user interface is designed to visualize the relative position of the intelligent object and the orientation of the mobile device based on the environmental sensor data and the orientation sensor data, in particular by displaying a position of the intelligent object on a centered on a position of the user and based on the Orientation of the mobile device oriented map of the area. In particular, a display of the mobile device can be used for the visualization. It is indicated that there is a smart object in the vicinity of the user or mobile device. In addition, it is shown where, i.e. at which relative position, this intelligent object is located in relation to the position of the user. This enables the user to get an overview of available intelligent objects in his environment and to interact with them based on this. The result is a simple and intuitive interaction option. In a preferred refinement, the communication unit is designed to establish the communication connection when the mobile device is aimed at the intelligent object for a period of time that is above a predefined threshold value. To further avoid unintentionally established communication connections, it can be provided that a time threshold value has to be exceeded before the communication connection is established. In other words, the mobile device is pointed at a smart object for a certain period of time, which indicates to the user that a communication link should be established. Operating errors are avoided and the intuitiveness is further improved.

In a preferred refinement, the communication unit is designed to establish the communication connection with the intelligent object via a direct short-distance communication connection, in particular via Bluetooth Low Energy (BLE). Direct communication means that internet-based communication can be dispensed with. In this respect, use in an environment without a mobile data connection can also be considered. A direct connection is often more energy-efficient and less susceptible to manipulation.

In a preferred refinement, the alignment unit is designed to detect whether the mobile device is aligned with the intelligent object. This detection can be based on a machine learning approach, in particular based on a pre-trained artificial neural network. Additionally or alternatively, this detection can be based on a pointing direction predefined for a mobile device. The pointing direction corresponds to the direction specified for the mobile device, which corresponds to a type of device property. The pointing direction of a device is therefore defined. For example, a device is specified to have a pointing direction orthogonal to a display or along a longitudinal axis. This pointing direction is the basis for determining whether the mobile device is aimed at the smart object. A machine learning approach can be used for the detection of the orientation, the one reliable and efficiently calculable execution. In addition, a high degree of accuracy can be achieved.

In a preferred embodiment, the environmental data include position information of intelligent objects that were determined based on a current position of the mobile device, preferably by a database query based on a position determined by a satellite navigation sensor of the mobile device. Additionally or alternatively, the environmental sensor data includes image data from a camera sensor of the mobile device. In particular, it is possible that a satellite navigation system is used to determine the current position of the mobile device (GPS, GALILEO, GLONASS, Beidou, etc.). This position can then be displayed on a map, which also shows the positions of smart objects. The positions of the intelligent objects can, for example, be queried based on one's own position from an Internet database, taking into account a radius around one's own position. The user can then orient himself and, for example, control intelligent objects in the immediate vicinity in order to interact with them.

It is also possible for the environmental sensor data to include image data from a camera sensor. It is then also conceivable, for example, to use augmented reality approaches in order to superimpose the intelligent objects virtually in the image data.

In a preferred embodiment, the alignment sensor data includes position sensor data that is received from an acceleration sensor, yaw rate sensor, and/or magnetic field sensor of the mobile device. Additionally or alternatively, the orientation sensor data includes image data from a camera sensor of the mobile device. In particular, it is possible for the orientation to be determined based on position sensor data from an inertial sensor. This usually enables a reliable and efficiently calculable position estimate. Interaction of a device or a mobile device with an intelligent object is understood here to mean, in particular, an exchange of information. In particular, information can be received from the intelligent object and/or information can be transmitted to the intelligent object. In addition, it is possible that an interaction is understood to mean triggering an action on the mobile device and/or on the intelligent object by the respective communication partner. For example, a display can be effected and/or haptic, tactile, acoustic or visual feedback can be output.

The surroundings of a mobile device or of a user of a mobile device are understood to be their surroundings. For example, a radius can be variable.

An intelligent object is understood to mean, in particular, a communication-capable artefact. In particular, such an artefact can be embedded in an infrastructure object or in the environment. In other words, an intelligent object can be, in particular, a physical object that has a communication option or in which a communication option is embedded. In particular, an intelligent object can be a physical object of the infrastructure. For example, traffic lights, tram stops, sensors in green areas, embedded sensors in a historic building represent intelligent objects. At a tram stop, for example, current departure times can be displayed or a timetable can be requested. However, it is also possible for an intelligent object to exist only virtually and in this respect to be in the form of a digital object (digital artefact, digital item). To this extent, a digital or virtual intelligent object is provided at a specific position, with which a mobile device or a user of a mobile device can interact. A position is therefore linked to a communication option, with the communication then taking place, for example, with an Internet server. For example, a digital object can also be linked to a position in the vicinity of a tram stop certain display and interaction options. For example, a digital intelligent object can be provided in the middle of a public square, through which information about a builder of the square or a historical background, etc. can be obtained. It is also possible for a digital, intelligent object to be in the form of a digital noteboard or a digital pinboard, so that a (virtual) note can be attached and shared or a reaction to such can be made using a mobile device.

An interaction with an intelligent object is understood to mean, in particular, an exchange of information. It is possible, for example, for a control signal to be transmitted or received in order to trigger an action or reaction on the part of the intelligent object and/or the mobile device. It is also possible for a link or a reference to a downloadable app to be displayed, for example to enable access to further information and control options. In addition, retrieving current information or also transmitting a request signal is conceivable.

The invention is described and explained in more detail below using a few selected exemplary embodiments in connection with the accompanying drawings. Show it:

FIG. 1 shows a schematic representation of a mobile device according to the invention in interaction with an intelligent object in the vicinity of the mobile device;

Figure 2 is a schematic representation of an inventive

Apparatus for interacting with a smart object in a mobile device environment;

FIGS. 3 and 4 show a schematic representation of the establishment of a communication connection with an intelligent object and the control of the intelligent object; FIG. 5 shows a schematic representation of a gesture recognition based on a mobile device according to the invention; and

FIG. 6 shows a schematic representation of a device according to the invention

procedure.

A mobile device 10 according to the invention for interacting with an intelligent object 14 in the environment of the mobile device 10 is shown schematically in FIG. The representation is to be understood as a schematic plan view. It is shown that a user 12 is holding a mobile device 10, for example a smartphone or a tablet, in his hand. According to the invention, it is provided that a communication connection is established between the mobile device 10 and the intelligent object 14 when the mobile device 10 is aligned with the intelligent object 14 . Shown in FIG. 1 is a field of view 22 of the mobile device 10, which visualizes an area in which, for an intelligent object 14 located therein, it is assumed that the mobile device 10 is aligned with it.

The left-hand side in FIG. 1 shows that mobile device 10 is not aligned with intelligent object 14 . The right-hand side shows that the user 12 has turned together with the mobile device 10 until the mobile device 10 is aligned with the intelligent object 14 . When the user 12 has aligned the mobile device 10 with the intelligent object 14, a communication link is established between the mobile device 10 and the intelligent object 14 and the user 12 is informed of this, for example by the mobile device 10 vibrating.

By establishing a communication link between the mobile device 10 and the intelligent object 14, access to digital services can be made possible. Intelligent objects 14 in smart cities, for example, can be made easier to use. In the example shown, a traffic light request for pedestrians can be implemented, for example. The user 12 of the mobile device 10 directs the mobile device 10 towards the traffic light. As soon When mobile device 10 is aligned with the traffic light, a communication link is established and the user 12 is notified that the communication link has been established. The user can then request a green phase, for example. The application in a smart city described above is to be understood as an example. Also could offer a use of the approach in buildings to z. B. in a private apartment with kitchen appliances or other intelligent objects (artefacts) or in public areas (train station, office, etc.) to interact with intelligent objects.

To provide this function, the mobile device 10 in the exemplary embodiment shown comprises a device 16 for interacting with the intelligent object 14, an environment sensor 18 for detecting a relative position of the intelligent object, and an orientation sensor 20 for detecting an orientation of the mobile device 10.

The environment sensor 18 can in particular include a position sensor of a satellite navigation system. For example, a current absolute position of the mobile device 10 can be determined, which is then compared with positions of mobile intelligent objects 14 based on a database query in order to determine their relative positions (relative to the user's or mobile device's own position) based thereon.

The orientation sensor 20 can include, for example, an inertial sensor (acceleration and/or yaw rate sensor) and/or a magnetic field sensor. A position (for example in the form of an indication of the Euler angle) can be detected by such a sensor. In particular, this location can then be used in conjunction with the relative position to determine the alignment of the mobile device in relation to the intelligent object 14.

The device 16 according to the invention is shown schematically in FIG. The device 16 includes an input interface 24, a Alignment unit 26, a communication unit 28 and a user interface 30. Optionally, the device 16 also includes a gesture unit 32. The various interfaces and units can be implemented in hardware and/or software and can be designed individually or in combination. In particular, it is possible for the device 16 to correspond, so to speak, to a smartphone or another device with a corresponding app.

The environmental sensor data and the orientation sensor data are received via the input interface 24 . In particular, the input interface 24 can be connected to corresponding sensors of the mobile device for this purpose. However, it is also possible for the input interface 24 to receive the information from remotely located sensors, for example via a short-range, low-energy radio link. For example, it is possible for the device to be integrated into intelligent glasses (smart glasses) and for data to be received from a position sensor of a smartphone via the input interface 24 via Bluetooth.

The alignment unit 26 detects whether the mobile device is aligned with the intelligent object. For this purpose, the environmental sensor data and the alignment sensor data are processed. In this case, the data can be pre-processed and/or further data can be accessed. In particular, it is advantageous if a machine learning approach is used in the alignment unit 26 in order to detect whether the mobile device is aligned with the intelligent object. In particular, a pre-trained artificial neural network can be used that was trained based on corresponding training data at a previous point in time.

In addition or as an alternative, it is possible for a predefined pointing direction of the mobile device to be used to detect whether the mobile device is aimed at the intelligent object. This predefined pointing direction can in particular correspond to a direction specified for the mobile device, for which a deviation can then optionally be specified.

The communication unit 28 is designed to communicate with the intelligent object. In particular, a low-power, short-distance communication link can be established with the intelligent object. For example, a Bluetooth Low Energy (BLE) communication chip can be used, which is controlled in a corresponding manner via the communication unit 28 . It is also possible for the communication unit 28 to be designed to communicate via the internet, ie via a corresponding internet server and a mobile data connection. It goes without saying that the intelligent object itself is also capable of communication and to this extent can be addressed/reachable.

The user interface 30 is for informing the user of the mobile device when the communication link has been established. In particular, a display or another actuator of a mobile device can be controlled for this purpose. For example, it is possible that the establishment of the communication connection is indicated to the user by a kind of "snapping". This locking can be made detectable based on a vibration function or a vibration actuator of the mobile device 10 . In addition, a visual output can take place, for example, by marking a traffic light on a visualization of a map representation on the display of the mobile device 10 . Furthermore, additionally or alternatively, a signal tone can be emitted. The user 12 is thus shown with haptic, acoustic and/or visual feedback that the smartphone is locked and that an action in the sense of an interaction with the intelligent object 14 can take place. For example, a green phase of the traffic light can then be requested by gesture control or by pressing a button on a touch screen.

It is optionally possible for the user to also make a preselection via the user interface 30 with regard to the intelligent objects to be displayed Will be received. For example, the user can select or define categories of intelligent objects of interest in order to be able to interact with the environment in a more targeted manner and to find the intelligent objects or functionalities relevant to him.

The gesture unit 32 is for recognizing a gesture performed by a user of the mobile device. To do this, the alignment sensor data is processed. In particular, a gesture that the user of the mobile device performs with the mobile device can be recognized. For example, the user can nod, in the case of smart glasses, or perform a swipe or other movement with a smartphone or tablet.

The detection can then be used on the one hand as an additional prerequisite for establishing the communication link. In order to establish a communication connection, it may also be necessary to first perform an activation gesture in order to avoid unintentional connection establishments. Furthermore, it is possible for a control gesture to be recognized by means of the gesture unit 32 . In this respect, the user of the mobile device can remotely control an intelligent object using gesture control after the communication connection has been established.

In FIGS. 3 and 4, one possibility of establishing a communication link with an intelligent object and of controlling the intelligent object is shown schematically. In particular, a view of a display 34 of a smartphone or tablet or other mobile device is shown before (FIG. 3) and after (FIG. 4) the establishment of the communication connection.

Relative positions R1, R2, R3 of three intelligent objects in the area surrounding the mobile device are visualized at position M on a map of an area surrounding the mobile device. For example, thumbnails of the smart objects can be displayed. The map display is centered on the position M of the mobile device. In addition to the relative positions R1, R2 and R3, the alignment of the mobile device is also visualized since the representation moves with the rotation of the mobile device. The user can therefore rotate in relation to the intelligent objects, with the map display also rotating like a compass and the wind rose, thus visualizing the orientation.

As soon as the user points the mobile device in the direction of a smart object, a communication link is established with it. This may require a minimum time (for example 2 seconds) for which the mobile device must be aligned with the intelligent object.

Whether the mobile device is aligned with the intelligent object can be determined based on a pointing direction of the mobile device, for example. This pointing direction corresponds to a parameter set for the mobile device. This pointing direction can correspond to a longitudinal axis of the mobile device, for example. With reference to FIG. 1, the field of view of a mobile device can be defined, for example, based on an indication of a maximum deviation from this pointing direction, for example 5°.

In the exemplary embodiment shown, the user in FIG. 4 aligns the mobile device with the intelligent object at position R2, so that a communication connection can then be established with it. For example, the mobile device vibrates to indicate to the user that the connection has been established. In addition, after the communication connection has been established, interaction with the intelligent object can take place using a button 36 that is displayed for this purpose (FIG. 4).

In other words, the map view rotates like a compass rose so that the top of the mobile device display 34 shows the direction in which the mobile device is being held. This direction can be specified, for example, by specifying a number of degrees. In the example shown, the direction is at 70 degrees (Figure 3). If the mobile device is then rotated, for example to 115 degrees (Figure 4), a Pedestrian traffic light as an intelligent object in front of him. As soon as the mobile device is aligned with the traffic light, the snapping process described above takes place and the traffic light is marked. The smartphone vibrates, there is a beep and the image of a request button appears in the form of button 36. This shows the user with haptic, acoustic and visual feedback that the mobile device is locked and that an action or interaction with the intelligent object can be carried out.

It goes without saying that another interaction option can also be provided, for example by means of gesture control. For example, a green phase can be requested with a slight backward movement of the mobile device.

In this example, the mobile device is intended to be held approximately level until it points to a smart object. It can also be possible here for the mobile device to be rotated through 90 degrees, so that the real environment can then be shown on the display 34 by means of a camera arranged on the back of the mobile device. The pointing direction would then correspond to a direction perpendicular to the display 34. It is possible for the communication connection to be established by locking in the form of an augmented reality display. For example, the snapped smart object can be highlighted.

A gesture control provided according to the invention is shown schematically in FIG. For example, the gesture control can be used in connection with other applications. For example, a corresponding gesture control can be made available in a general smart city application for a city with networked buses, trams, museums and other information. The user 12 holds a mobile device 10 (in particular a smartphone or a tablet) in his hand and uses it to perform a movement. On the one hand, according to the invention, it can be provided that, in order to establish the communication connection, after the mobile device is aligned with the intelligent object, an activation gesture must first be recognized. For example, this can correspond to the swipe movement that is made. In other words, the activation gesture can represent the prerequisite for establishing a communication connection.

It is also possible that the actual control of the intelligent object takes place by means of gesture control. Thus, a control gesture may be recognized to interact with the smart object. For example, a green phase can be requested at a traffic light or a timetable can be requested from an intelligent tram stop.

It can be advantageous to first initiate a type of gesture control mode in order to then be able to carry out gesture recognition. It can also be advantageous if provision is made for a return from gesture control to a classic operation, for example by touching a touchscreen. This can reduce the risk of detecting an unintentional gesture. A relevant question in this context is how the user can arrive at the use of gesture control or initiate this gesture control mode.

One possibility for this can be activating a corresponding selection point on a touchscreen of the mobile device. A corresponding button is thus pressed, which puts the mobile device into a gesture control or recognition mode.

Alternatively, it is possible for the gesture control mode to be activated by a typical gesture (e.g., a right turn, like unlocking a lock with a key, or a click gesture, or an individually learned and individual gesture). Again, it's possible that reaching gesture control mode is through a kind of snapping of the mobile device is implemented by means of a vibration actuator etc. or in some other way. As soon as the gesture control mode is activated, the user's gestures are recognized and he can use this to interact with the intelligent object, for example. In a similar way, switching back from gesture control to classic operation can also take place.

A method according to the invention for interacting with an intelligent object in an environment of a mobile device is shown schematically in FIG. The method includes steps of receiving S10 environmental sensor data and orientation sensor data, detecting S12 whether the mobile device is aligned with the intelligent object, establishing S14 a communication link with the intelligent object, and notifying S16 a user of the mobile device when the communication link is established became. The method can be implemented, for example, as a smartphone app or as a tablet app and made available via a corresponding app store.

The invention has been comprehensively described and explained with reference to the drawings and the description. The description and explanation are intended to be exemplary and not limiting. The invention is not limited to the disclosed embodiments. Other embodiments or variations will become apparent to those skilled in the art upon use of the present invention upon a study of the drawings, the disclosure, and the claims that follow.

In the claims, the words "comprising" and "having" do not exclude the presence of other elements or steps. The undefined article "a" or "an" does not exclude the presence of a plural. A single element or unit can perform the functions of several of the units recited in the claims. An element, unit, device, and system may be partially or fully implemented in hardware and/or software. The mere naming of some measures in several different dependent patent claims is not to be understood that a combination of these measures cannot also be used to advantage. A computer program may be stored/distributed on a non-volatile medium, such as an optical memory or a solid state drive (SSD). A computer program may be distributed with and/or as part of hardware, for example via the Internet or via wired or wireless communication systems. Reference signs in the claims are not to be understood as restrictive.

Claims

patent claims

1. A device (16) for interacting with an intelligent object (14) in an environment of a mobile device (10), with: an input interface (24) for receiving environment sensor data and orientation sensor data, the environment sensor data containing information on a relative position of the intelligent object in include relation to the mobile device and the orientation sensor data includes information about an orientation of the mobile device in relation to the environment; an alignment unit (26) for detecting whether the mobile device is aligned with the intelligent object based on the environmental sensor data and the alignment sensor data; a communication unit (28) for establishing a communication link with the intelligent object when the mobile device points at the intelligent object; and a user interface (30) for informing a user of the mobile device when the communication link has been established.

2. Device (16) according to claim 1, having a gesture unit (32) for detecting a gesture performed by a user of the mobile device (10) based on the alignment sensor data, wherein the gesture unit is preferably designed to recognize the gesture based on a machine learning approach, in particular based on a pre-trained artificial neural network.

3. The device (16) as claimed in claim 2, wherein the communication unit (28) is designed to establish the communication connection on the condition that an activation gesture has previously been recognized; and/or is configured to end the communication connection when the communication connection is established and a deactivation gesture is recognized.

4. Device (16) according to one of claims 2 and 3, wherein the communication unit (28) is designed to transmit a control command to the intelligent object (14) when the communication link is established and a control gesture is recognized.

5. The device (16) according to any one of the preceding claims, wherein the communication unit (28) is designed to receive interaction data with information about interaction options of the user with the intelligent object (14); and the user interface (30) is designed to inform the user about the interaction options based on the interaction data received, preferably by means of an indication on a display.

6. Device (16) according to one of the preceding claims, wherein the user interface (30) is designed to output an acoustic, optical and / or haptic signal, in particular a haptic vibration signal by means of a vibration actuator of the mobile device (10).

7. The device (16) according to any one of the preceding claims, wherein the user interface (30) is designed to visualize the relative position of the intelligent object (14) and the orientation of the mobile device (10) based on the environmental sensor data and the orientation sensor data, in particular by a Displaying a location of the smart object on a map of the environment centered on a location of the user and oriented based on the orientation of the mobile device.

8. The device (16) according to any one of the preceding claims, wherein the communication unit (28) is designed to establish the communication connection when the mobile device (10) is aligned with the intelligent object (14) for a period of time in excess of a predefined threshold value.

9. Device (16) according to one of the preceding claims, wherein the communication unit (28) for establishing the communication connection with the intelligent object (14) is designed via a direct short-distance communication connection, in particular via Bluetooth Low Energy.

10. Device according to one of the preceding claims, wherein the alignment unit (26) for detecting whether the mobile device (10) is aligned to the intelligent object (14) based on a machine learning approach, in particular based on a pre-trained artificial neural network , is trained; and/or based on a pointing direction predefined for a mobile device.

11 . Apparatus (16) according to any one of the preceding claims, wherein the environmental sensor data

Position information of intelligent objects (14) which were determined based on a current position of the mobile device (10), preferably by a database query based on a position determined by a satellite navigation sensor of the mobile device; and or

Include image data of a camera sensor of the mobile device.

12. Device (16) according to any one of the preceding claims, wherein the orientation sensor data

Include position sensor data received from an acceleration, yaw rate and/or magnetic field sensor of the mobile device (10); and or

Include image data of a camera sensor of the mobile device.

26

A mobile device (10) for interacting with an intelligent object (14) in an environment of the mobile device, comprising: an environmental sensor (18) for detecting a relative position of the intelligent object; an orientation sensor (20) for detecting an orientation of the mobile device; and an apparatus (16) according to any one of the preceding claims.

14. A method for interacting with a smart object (14) in an environment of a mobile device (10), comprising:

receiving (S10) environment sensor data and orientation sensor data, the environment sensor data comprising information on a relative position of the intelligent object in relation to the mobile device and the orientation sensor data comprising information on an orientation of the mobile device in relation to the environment;

detecting (S12) whether the mobile device is facing the smart object (14) based on the environmental sensor data and the facing sensor data;

Establishing (S14) a communication link with the smart object when the mobile device points at the smart object; and

informing (S16) a user of the mobile device when the communication link has been established.

27

15. Computer program product with program code for performing the steps of the method according to claim 14 when the program code is executed on a computer.

28