DE102018201711A1 - ARRANGEMENT AND METHOD FOR PROVIDING INFORMATION IN A HEADBAND ADVANCED REALITY DEVICE - Google Patents

Abstract

The invention relates to a method for providing information in a head-worn augmented reality device (7), wherein the augmented reality device (7) comprises an optical detection device (22) and a first display device (20) connected thereto is arranged to display information objects (26) in a field of view (9) of a carrier (2) of the augmented reality device (7); and wherein in a vicinity of the augmented reality device (7) at least one predetermined optically recognizable feature (12) is associated with which at least one predetermined information object (11) is associated; the method comprising: - detecting the environment optically by the detection means (22) and generating detection data based thereon; - transmitting the detection data to at least one second display device (24) spatially separated from the augmented reality device (7); - recognizing the optically recognizable feature (12) based on the detection data; - Displaying at least one information object (26) associated with the optically recognizable feature (12) by means of the first display device (20). Furthermore, the invention relates to an arrangement, in particular for carrying out such a method.

Description

The invention relates to an arrangement and a method for providing information in a head-worn augmented reality device. In particular, the invention is directed to a method and an arrangement with an augmented reality device (eg augmented reality goggles) that recognize optically recognizable features (eg, a machine readable code) located in an environment and display associated information objects can be.

Advanced reality devices are available in various forms on the market. Under an augmented reality (hereinafter also ER and in English: "augmented reality" (AR)) can be generally understood that in a field of view of a carrier (ie a user) of an augmented reality device additional information objects are fade in, for example in the form of a text box. This can be done in particular in such a way that the information object is superimposed on the field of vision of the carrier (or, in other words, overlaid with its view of the reality or the environment). The information objects can be transparent or opaque from the perspective of the wearer.

Advanced reality devices are predominantly used in such a way that information objects are displayed predominantly or exclusively to the wearer of the device. However, ways to interactively exchange information and / or information objects with another person who is not the wearer of the device are limited.

There is a general need to allow people to exchange information, even over long distances. In particular, there is a need to be able to convey information about greater distances as comprehensively and realistically as possible. For example, it may happen that a person (in the following also: a prospective customer) is interested in a particular topic or a specific object. However, the item itself or a person who can explain that item or topic (hereafter also: a demonstrator or a presenter) may be at a distance that, for example, does not allow a direct verbal exchange of information between the persons. In other words, the prospect and the demonstrator may be spatially separate. In such cases, conventional means of communication have hitherto been used, for example in the form of a telephone connection, a video chat or the like. However, this is associated with individual disadvantages. In particular, it may prove difficult for the demonstrator to be able to respond flexibly and comprehensively to inquiries from the interested party.

An object of the present invention is therefore to facilitate the exchange of information between spatially separated persons.

This object is achieved by a method and an arrangement according to the attached independent claims. Advantageous developments are specified in the dependent claims. Furthermore, it is understood that the features mentioned in the introductory description can be provided individually or in any combination also in the case of the presently disclosed solution, unless stated otherwise or obvious.

The invention is generally based on the idea of providing a carrier of an augmented reality device with information associated with contents that another person is currently interested in. For this purpose, optically recognizable features in an environment of the carrier can be detected and / or analyzed or decoded, after which the carrier information objects can be displayed, which are associated with these features. Thus, as the wearer moves in the environment, he may receive specific information, and convey it to the other person verbally, in writing, and / or by image or video transmission, for example.

• – optisches Erfassen der Umgebung mittels der Erfassungseinrichtung und darauf basierendes Erzeugen von Erfassungsdaten;
• – Übertragen der Erfassungsdaten an wenigstens eine zweite Anzeigeeinrichtung, die optional räumlich getrennt von der erweiterte-Realität-Vorrichtung angeordnet ist (und/oder außerhalb eines Sichtfelds der Erfassungseinrichtung oder eines Blickfelds des Trägers);
• – Erkennen des optisch erkennbaren Merkmals anhand der Erfassungsdaten;
• – Anzeigen wenigstens eines Informationsobjekts, das mit dem optisch erkennbaren Merkmal assoziiert ist, mittels der ersten Anzeigeeinrichtung.

In more detail, a method of providing information to a head-worn augmented reality device is proposed, wherein the augmented reality device comprises an optical detection device and a first display device adapted to be mounted in a field of view of a wearer of the augmented reality device. Reality device display information objects; and wherein in a vicinity of the augmented reality device at least one predetermined optically recognizable feature is associated with which at least one predetermined information object is associated; the method comprising:

• - Optical detection of the environment by means of the detection device and based on generating detection data;
• - transferring the detection data to at least one second display device, which is optionally arranged spatially separate from the augmented reality device (and / or outside a field of view of the detection device or a field of view of the wearer);
• Detecting the optically recognizable feature based on the detection data;
• - Displaying at least one information object that is associated with the optically recognizable feature, by means of the first display device.

Preferably, the optically recognizable feature is attached or attachable to an object to be explained by the wearer, and the information objects serve to inform the wearer of the object.

The augmented reality device may be carried or wearable by a demonstrator who wishes to explain a content (eg, a teaching content) or an object (for example, a purchase object such as a car or real estate) to an interested party. The prospective customer, on the other hand, may have or view the second display device and be generally spatially separate from the demonstrator (i.e., not be on-site).

The augmented reality device may be in the form of augmented reality glasses. The augmented-reality eyeglasses may comprise a support frame for positioning on a wearer's head and / or an area positionable in front of at least one eye of a wearer. This area may include the display device. Preferably, the augmented-reality goggles are based on conventional variants, such as the market-available extended-reality type BT-2000 glasses from Seiko Epson Corporation, Suwa (Japan), which is an example of a so-called smart glass. However, the augmented-reality eyewear of the present invention can enable and provide the additional functions described below over such known variants.

The augmented reality device further comprises an optical detection device. This can generally be set up to visually capture information, objects, contents and / or the environment in general. In particular, the detection device can be set up to detect and preferably simultaneously detect both an optically recognizable feature, which is explained below, which is associated with information objects, and an environment of the augmented reality device. According to a variant, the optical detection device comprises a camera device or is realized as such.

The optical detection device may be oriented or alignable such that its field of view coincides with or at least partially overlaps a field of view of the wearer. In other words, the detection device can be set up to detect an area that is also located in and / or viewed by the wearer.

The display device of the augmented reality device may include, for example, a (preferably partially or fully transparent) screen or projection screen. In particular, the display device can be integrated in a region of the device which is arranged or can be arranged in the field of view of the wearer, for example in a see-through region and / or spectacle lens. In turn, transparent and / or opaque information objects can be superimposed on the display device, which can be overlaid with a field of view of the carrier.

The environment of the augmented reality device may be, for example, a sales room, a teaching room, a property, a private room or a workshop. The optically recognizable feature may in particular be designed such that it can be recognized by the detection device of the augmented reality device. Preferably, the optically recognizable feature is additionally or alternatively also recognizable by a person (in particular by a possible demonstrator or prospective customer). As described below, they may be general environmental or object features (in particular features already present there) that have been selected and / or defined in advance for optical detection, for example. Likewise, they may be features that have been specially made for optical recognition and / or positioned in the environment. Selecting, defining, making and / or positioning such features may be done by the extended reality device (or demonstrator) support.

According to the method, the environment is optically detected by means of the detection device. This may include, in particular, pictorial detection. According to a variant, the detection device detects environmental images, for example in the form of individual images, of a video and / or moving images. In particular, if the detection device comprises or is realized as a camera device, it can capture, as detection data, camera images from the environment, which in turn can be present individually or in the form of a video or as moving images.

The detection data may be generated by the detection device itself or by another suitable unit of the augmented reality device. Alternatively, the detection data may be generated by a separate device (ie, in particular, separate from the augmented reality device) to which the detection device transmits detected information or signals. The acquisition data may include image, video and / or moving image data or data from which images, videos and / or moving images can be generated. The data may be generally closed (ie as a record) or in the form of a Data stream (for example, a video stream) be transferable.

The second display device may be comprised of a conventional PC, a smartphone, a tablet or a comparable mobile device. The second display direction may be generally viewed by a person (for example, a prospective customer) that is spatially separate from the extended reality device's bearer. In particular, this person or the second display device may be located in a different room, a different building or a different geographical area than the carrier of the augmented reality device. In this way it can be ensured that the second display device is outside the field of view of the wearer.

The acquisition data obtained from the second display device may be displayed thereon, for example as a substantially continuous video and / or video stream. In general, transmitting the acquisition data may allow a person viewing the second display device to understand, view, and / or generally obtain information about the environment of the augmented reality device. In particular, if the detection area (or field of view) of the detection device at least partially coincides with or overlaps a field of view of the wearer, the second display device may facilitate tracing the wearer's gaze to the environment. In other words, a person viewing the second display device can view the environment of the augmented reality device from the perspective of the detection data from a perspective that corresponds to or at least approximates a perspective of the extended reality device's wearer. In summary, an observer of the second display device can thus be presented in a particularly comprehensive and realistic manner the environment of the augmented reality device, in particular in such a way that it can easily understand a position and / or perspective of the wearer of the augmented reality device.

According to the invention, a plurality of second display devices can also be provided, so that a so-called 1: n communication is made possible between the demonstrator and potential interested parties (a demonstrator with respect to n-possible interested parties, where n is a number greater than one).

Preferably, a communication exchange between an interested party viewing the second display device and a demonstrator carrying the augmented reality device is enabled not only by transmitting the detection data. Instead, a communication via text and / or audio input (that is, verbally), made possible by at least the prospective customer to the demonstrator. This can be done, for example, by means of a corresponding text and / or audio transmission device, which may comprise a keyboard and / or a microphone. The latter can be integrated into a computer unit (for example a PC), which also includes the second display device. The audio signals may generally be in the form of voice input by a user.

The augmented reality device preferably has at least one speaker for outputting voice or general audio signals to the carrier, in particular based on voice inputs of the prospective customer viewing the second display device. Any text inputs, however, can be displayed directly on the first display device. Also, text and / or audio input can be converted to other output formats rather than output directly to the demonstrator (for example, from text to audio or vice versa).

The augmented reality device preferably comprises an audio transmission device, for example comprising a microphone, in order to be able to detect and transmit voice or general audio signals of the carrier. These may in turn be transmitted to a loudspeaker positioned near or at the second display device.

In summary, it can thus be provided within the scope of the invention that transmission of audio signals and / or verbal communication from the interested party to the demonstrator (and / or vice versa) is also possible.

This means that the interested party can instruct the demonstrator, for example by text or voice, to move in the environment in a predetermined manner or in a predetermined direction. In particular, the prospect may instruct the demonstrator to move in an area containing an object and / or content of interest to him. The demonstrator can then use the generated and transmitted acquisition data to ensure that the interested party can view the object or the content in a realistic manner. For this purpose, the demonstrator can, for example, suitably align the augmented reality device or its own field of vision. Furthermore, the demonstrator can linguistically tell the interested party information about the object or content, which in turn can be made possible via an audio transmission.

In order to recognize the optically recognizable feature on the basis of the acquisition data, the augmented reality device can perform image recognition and / or have suitable image recognition software, as generally available on the market. In general, the optically recognizable feature can thus be recognized mathematically on the basis of or in or from the acquisition data. This is especially true if the latter are available as image or video data. In particular, the acquisition data may be evaluated substantially continuously, and preferably in real time, such that the augmented reality device may recognize the optically recognizable feature substantially independently (ie, without additional user input). In other words, an automatic detection of the optically recognizable feature by the detection means may be provided and / or an automatic determination of whether the optically detectable feature is located in the detected environment.

Alternatively, the carrier may properly align the augmented reality device such that the optically detectable feature is within a detection range of the detection device. The wearer may then enter an activation command, for example by voice or haptic, to initiate a recognition or recognition process. In other words, the recognition can also be initiated and / or triggered selectively and / or as required by the wearer.

The detection may generally include the augmented reality device detecting the presence, type, and / or any information encoded by the optically recognizable feature. For example, in the context of the recognition, the augmented reality device can recognize the presence and / or the type of QR code and preferably also decode it directly. In other words, the augmented reality device may generally determine whether and, preferably, which optical detectable feature is in the environment. This can in turn be done by evaluating the detection data, for example by image analysis software.

By using predetermined optically recognizable features, it can be understood that the nature and / or properties of these features are known and defined so that they can be searched for the features in the context of an image analysis.

Furthermore, a separate step of determining an information object associated with the optically recognizable feature may be provided. This can be done by querying a database, which can be stored, for example, on a memory device explained below and / or a server explained below. Thus, determining the information object may include sending a corresponding request and / or information that the optical feature has been detected to another device (for example, the server or the storage device). This step may be performed by a controller and or processor device of the augmented reality device.

The method further provides for at least one information object associated with the optically recognizable feature to be displayed by the first display device. This is preferably done as soon as possible or immediately after the recognition of the optically recognizable feature. The bearer of the augmented reality device can thus be quickly provided with relevant information, for example regarding an object or a content on which the optically recognizable feature is located.

The information object may be, for example, a text, a text box or a document, an audio or audio file, a video or video file, an image or an image file, a PDF document or a media attachment, a screenshot, a graphical user interface, a color space or the colorful design of a graphical user interface. In particular, the information objects can be provided as icons and / or as links which, when actuated, enable a link to further contents, for example by loading or playing back an audio, video or image file.

In general, the information objects may relate to objects and / or allow links to objects that allow information to be derived by a viewer.

The information objects that are displayed on the first display device for informing the carrier are preferably text or text fields. Information objects which are displayed on the second display device for viewing by the interested party can be present in particular as image, video or audio files (or as links to such files), as a PDF document or as a specially designed (in particular colored) user interface.

An advantage achievable with the described method is, for example, that a prospective customer can use the second display device to follow the environment of the augmented reality device in a realistic manner and in particular from the (human) perspective of the wearer. Thereupon can he expresses his interest in certain objects or contents in the environment, for example by means of text or voice input. Such objects or contents may have been marked in advance with an optically recognizable feature, which may also form a separate step of the method according to the invention. The wearer of the augmented reality device may then approach the object or content and initiate a recognition process of the optically recognizable feature or, especially if it is done automatically, await such a recognition process. If the optically recognizable feature has been detected, an information object associated therewith can be displayed as directly as possible in the field of vision of the wearer. For example, the carrier can pass on the information thus conveyed verbally to the interested party.

An application scenario of the described method is, for example, using the augmented reality device in a dealership where the carrier is a salesman. At the individual cars on a sales area of the dealership, pre-defined, visually recognizable features were placed in advance, for example in the form of QR codes. A geographically separate interested party contacts the seller via a communication network explained below, so that capture data (in particular, video data) can be displayed on a display device of a prospective customer's PC. The seller can present the individual cars to the customer and, if necessary, by recognizing the respective visually recognizable features, additional information can be displayed in order to inform the prospective customer as comprehensively as possible. For this, it is not necessary that the seller interrupts the customer conversation or makes time-consuming manual entries. Instead, the preferably even automatically activated detection of the optically recognizable feature on a car is sufficient, which, for example, can largely remain unnoticed from the point of view of the prospective customer. It will be understood that numerous other application scenarios are also envisioned in which the method disclosed herein may provide further advantages. Corresponding examples will be explained in more detail below.

It should be pointed out above that all of the preceding or following method steps can be executed by at least one computer device, processor device or general arithmetic unit, unless stated otherwise or obvious.

A further development of the method and the arrangement provides that an activation option is provided or can be provided by means of which the carrier of the augmented reality device can activate a display of the information object by means of the second display device. This can be done for example via a computing or processing unit of the augmented reality device or a server connected thereto. The activation option can be provided, for example, in the form of a virtual icon on a graphical user interface, wherein the user can initiate the display by means of the second display device by actuating the icon. Additionally or alternatively, the activation option can be provided by appropriately occupying a haptic or, in other words, contact or contact actuatable input element, wherein activation can be effected by actuating (for example pressing, tapping, shaking or moving) of the input element. Furthermore, the activation possibility can be provided by providing and / or releasing a voice input command. On a higher level, the provision of the activation option can take place promptly and preferably essentially immediately as soon as the information object is displayed.

By means of this variant, it is possible for the carrier to decide whether information is displayed not only to him but also to the prospective customer watching the second display device. The carrier can therefore display only some information objects, which for example serve solely to promote the sales success and / or the bridging of knowledge gaps, on its own display device. On the other hand, information that conveys the communication and the information exchange with the prospective customer can, so to speak, be released by the carrier, so that they are also displayed on the second display device.

In principle, various classes of information objects can thus be provided, namely a first class, which are displayed only on the first display device for viewing by the wearer, and a second class, which, if necessary, is also displayed on the second display device after being activated by the wearer can be.

According to a further variant of the method and the arrangement, it is provided that a plurality of information objects are determined, which are associated, for example, with the optically recognizable feature or a plurality thereof, wherein the information objects are displayed or displayable by means of the first display device, and wherein an activation possibility is provided only for a part of the information objects. Consequently, a first part (or a first class) of information objects that can only be displayed on the first display device and a second part (or a second part) exist. a second class) of information objects, which can also be displayed on the second display device.

In a development of the method and the arrangement, there is also another class of information objects which can be displayed or displayed at least temporarily only on the second display device. In detail, it is possible to determine a plurality of information objects which are associated, for example, with the optically recognizable feature or a plurality thereof, wherein at least one of the information objects is at least temporarily displayed exclusively on the second display device (ie not additionally also on the first display device). , In particular, the display on the second display device can take place independently of an activation input of the carrier of the augmented reality device. As promptly and substantially immediately after the optically recognizable feature has been detected, it is thus possible to display at least one information object associated therewith on the second display device. This information object can be displayed on the second display device in addition to the acquisition data (in particular if present as a video stream and / or displayed), for example below or above the acquisition data or directly into any video window.

This makes it possible for the prospective customer to be displayed additionally suitable information objects for a currently relevant object or a currently relevant content, which for example enable more comprehensive information and / or can generally adapt the presentation content of the second display device to the currently relevant object or content. The interested party can, as it were, be attuned to the currently relevant object or content and, if necessary, look at additional information (for example in the form of image files or PDF documents). Since the corresponding information objects are not necessarily required by the carrier and may be targeted to the interested party, they are preferably displayed only on the second display device.

According to the invention, however, it can also be provided that demand-based display of these information objects can also be activated on the first display device. The carrier thus specifically request the corresponding information objects, for example, if the interested party asks for it. This can in turn be activated by means of a corresponding command input by the carrier.

In a further development of the method and the arrangement, the augmented reality device and / or the second display device are connected to a storage device in which at least one information object is stored, wherein the information object can be read out of the storage device in order to store it on the first and / or the second display device. The storage device may be comprised by or implemented as a server. The storage device can contain a database in which the information objects can be stored and in particular are linked to the optically recognizable feature. Depending on which optically recognizable feature was detected, then the information object associated therewith can be read from the memory device.

In the context of the present disclosure, a server can be understood as a computer program and / or a computer device (for example a hardware server), the server being able to access other computer programs or computer devices (and / or processor units), for example data about them or establish a communication connection.

In the case of a plurality of associated information objects, all information objects can be stored on the storage device or the server and read therefrom. Optionally, you can also be stored in temporary storage devices that are connected to the first or the second display device. This may, for example, relate to a memory device of a personal computer or a mobile device, which comprise the second display device. However, the actual display of the read or loaded information objects may be selectively activated by the carrier in the above manner.

The described prior readout or loading of information objects can also be carried out independently of an actual recognition of an optically recognizable feature. For example, at the beginning of the process, all information objects can be loaded in the described manner, which are linked to the visually recognizable feature in the environment.

In other words, all information objects that might be to be displayed by recognizing an optically recognizable feature associated therewith in the course of the procedure may be pre-loaded. This increases the speed with which the corresponding information objects can subsequently be displayed on the first and / or the second display device. In particular, the information objects must are not first read individually and often over longer communication channels as soon as an optically recognizable feature has been detected, let alone sent from the augmented reality device or a memory unit linked thereto to the second display device. Instead, the information exchange may occur substantially in real time and the load on a communication network between the augmented reality device and the second display device may be kept low.

Preferably, however, the information objects are generally stored centrally on a storage device or a server, to which the extended reality device and / or the second display device can access by means of real-time communication. This increases data security, since the information objects can be managed centrally on the storage device or the server.

• – einen maschinenlesbaren Code;
• – einen QR-Code (Quick-Response-Code);
• – eine per Texterkennung erkennbare Zeichenfolge, insbesondere per OCR (Optical Character-Recognition);
• – ein per Bilderkennung erkennbares Merkmal;
• – ein per Mustererkennung (Pattern Matching) erkennbares Merkmal;
• – ein per Template-Matching erkennbares Merkmal.

The optically recognizable feature may be provided in the form of any of the following examples or at least include one of the following:

• A machine-readable code;
• - a QR code (quick response code);
• - A recognizable by text recognition string, in particular by OCR (Optical Character Recognition);
• A feature recognizable by image recognition;
• A feature recognizable by pattern matching;
• A feature recognizable by template matching.

The machine-readable code (for example in the form of a QR code or a bar code) can be recognized and decoded by suitable recognition or evaluation software. In a known manner, such codes can receive identification information (for example in the form of a link) which enable a link to other contents. In the present case, by recognizing and / or decoding a corresponding code, a link to the information object can thus be provided and / or the information can be conveyed, which information object is linked to the code. Thereupon, this information object can be loaded or read out (for example, from the described server or storage device) and displayed by means of one of the display devices. In principle, however, it is also possible that an information object (for example in the form of a text) is coded directly in such a machine-readable code.

An advantage of QR codes is that they can be created with little effort (for example, by free software) and / or can be printed out using conventional printers. They can then be placed on the relevant content or objects. A demonstrator can thus create and place visually identifiable features without significant costs or equipment expense. The QR code may generally be defined according to any standard, for example, as a micro QR code or secure QR code.

For image recognition, pattern recognition and / or template matching, predefined contours, edge or color gradients, patterns, image templates or other optical properties can be predefined as visually recognizable features (for example by the demonstrator). These can then be determined by means of conventional image evaluation algorithms on the basis of the detection data of the detection data or detected therein.

For example, text recognition can be used to automatically detect and read in vehicle identification numbers applied to the windshield of a vehicle.

According to a further embodiment of the method and the arrangement, the detection data and / or at least one information object displayed on the second display device can be recorded, for example over a period in which both the first and the second display device is activated and / or in which detection data is output to the second display device.

The simultaneous activation of the display devices, and in particular any data transfer therebetween, indicates that a communication process or so-called communication session is taking place between the demonstrator and a prospective customer. A recording can generally be understood as a storage of the corresponding data. In particular, the acquisition data can be stored as a video file. The information objects may be stored separately therefrom or else in such a way that they can be overlaid with the acquisition data (or the video displayed thereon). In particular, the display contents of the first and / or the second display device, including the information objects displayed thereon, can be recordable.

These variants allow the recorded data to be subsequently made available to the interested party in order to understand it again. In other words, even in an offline mode, the demonstration that has taken place can be considered again. This can also be done by interested parties who are not directly connected to the actual But would like to retrieve the recorded content, for example, in an Internet-accessible video portal. Also in this way, a previously mentioned 1: n communication is made possible.

In particular, it may be provided in this context to create a protocol that can be displayed and / or operated by means of a graphical user interface. The protocol may preferably summarize, based on the recorded acquisition data and / or displayed information objects, time-ordered information about the course and / or the events of an established communication between the carrier and the prospective customer. By way of example, video sequences, audio recordings, displayed information objects, exchanged text messages or voice messages can be marked or displayed along a time axis and arranged according to their respectively associated times.

In general, the augmented reality device and the second display device may be at least indirectly connected or connectable via a real-time communication system (or communication network), in particular wherein the communication system is adapted for communication based on a WebRTC protocol. The communication system may be at least partially wired and / or at least partially wireless. Furthermore, the communication system may include a memory device and / or a server, wherein the memory device and / or the server may be connected to both the augmented reality device and the second display device.

The Web Real-Time Communication (WebRTC) protocol refers to an open standard promoted by the World Wide Web Consor- tium (W3C) that defines a collection of communication protocols and programming interfaces (APIs). This makes it possible to enable real-time communication via computer-computer connections by means of web servers. Correspondingly connectable computers can be provided both by the augmented reality device, for example in the form of a processor, and in connection with the second display device (for example in the form of a PC or mobile device comprising the second display device).

According to a further embodiment of the method and the arrangement, the augmented reality device is connected to a server via a (for example local) non-public communication network, the server being accessible via a (for example global) public communication network and in particular a communication connection enabled with the second display device. The communication networks may in turn be at least partially wired and / or at least partially wireless.

The non-public communication network may be an IP-based network. In general, the non-public communication network may be implemented as a WLAN or Ethernet network and / or the public communication network as the Internet. The non-public communication network can be connectable to the public communication network via an access device (for example in the form of a (eg DSL) modem). The augmented reality device, if connected to the non-public communication network, can automatically and preferably cyclically transmit and / or receive data to and from a server. The server may be connected to both the non-public and the public communications network, but in principle may be located outside the public communications network (i.e., physically separate therefrom).

In particular, the augmented-reality device may transmit uniquely identifying data and / or features (or, in other words, uniquely identifying identifiers) to the server so that a tunnel connection (or generally a secure connection) may be established. The data or features may be, for example, a MAC address of the augmented reality device. Furthermore, the data and / or features can be transmitted by the access device from the non-public communication network to the public communication network and / or the server.

The server may associate the transmitted indicative data and / or features with a user account and thus allow external access to the augmented reality device by that user. In general, the server may also allow other non-public communication network-integrated computer-based devices (eg, other personal computers or mobile devices) to access the augmented reality device, particularly after allowing access based on the described user account.

Furthermore, it can be provided that clearly characterizing data and / or features are coded by means of a machine-readable code (for example a QR code). This code can be accessed using the advanced reality device in the can be recognized and decoded above, whereupon a user account associated with the augmented reality device can be identified and / or a tunnel connection to the server can be established automatically.

The establishment of a tunnel connection can generally be a prerequisite for enabling the described communication processes between a demonstrator and interested parties, in particular the recognition of further machine-readable codes and loading and / or exchanging of information objects based thereon.

As described, possible application scenarios of the method include a real estate sales talk, an investment sales talk, a machine sales talk, a medical technology sales talk, or a sporting goods sales talk, each of which may be executed in real time or online. Likewise, a bearer of the augmented reality device may notify a prospective customer, and preferably several prospects (see the above 1: n communication) about a particular topic, object, or content. In particular, in this way, a training in real estate, equipment, machinery, medical devices, sporting goods or the like can be performed in real time.

• – die Umgebung mittels der Erfassungseinrichtung zu erfassen und darauf basierend Erfassungsdaten zu erzeugen;
• – die Erfassungsdaten an die zweite Anzeigeeinrichtung zu übertragen oder auszugeben;
• – das optisch erkennbare Merkmal anhand der Erfassungsdaten zu erkennen; und
• – wenigstens ein Informationsobjekt, das mit dem optisch erkennbaren Merkmal assoziiert ist, mittels der ersten Anzeigeeinrichtung anzuzeigen.

The invention further relates to an arrangement comprising: a head-worn augmented reality device comprising an optical detection device and a first display device configured to display information objects in a field of view of a carrier of the augmented reality device; at least one predetermined optically recognizable feature disposed in an environment of the augmented reality device and associated with the at least one predetermined information object; and
at least one second display device positioned or positionable spatially separate from the augmented reality device;
the arrangement being arranged:

• To detect the environment by means of the detection device and to generate detection data based thereon;
• To transmit or output the detection data to the second display device;
• - to recognize the optically recognizable feature based on the detection data; and
• - Display at least one information object associated with the optically recognizable feature by means of the first display device.

Preferably, the optically recognizable feature is attached or attachable to an object to be explained by the wearer, and the information objects serve to inform the wearer of the object.

The apparatus may comprise any further development and any further feature to provide or carry out all of the above or below steps, operating conditions and functions. In particular, the device may be configured to perform a method according to any of the preceding or following aspects. Furthermore, the arrangement for carrying out the listed measures may comprise suitable computing, control and / or processor devices. These may be provided in the augmented reality device and / or in a device comprising the second display device.

Embodiments of the invention will be explained below with reference to the attached schematic figures. Characteristics which match each other in their type and / or function can be provided with the same reference numerals throughout the execution form. They show:

1 a schematic diagram of an inventive arrangement according to a first embodiment, which carries out a method according to the invention;

2A - 2 B schematic views of information objects, by means of the first and second display device of the arrangement 1 are displayed; and

3 a schematic diagram of an inventive arrangement according to a second embodiment, which carries out a method according to the invention.

In 1 is an arrangement according to the invention 1 which shows a variant of a method according to the invention. The order 1 may generally provide or be realized as an audiovisual reality perception real-time communication system.

The order 1 includes an augmented reality device 7 in the form of augmented reality glasses (hereinafter referred to as ER glasses 7 designated). This is done by one person 2 worn, which in the case shown is a demonstrator, the one explained below 8th an object 10 to explain and sell.

The person 2 is in a sales room 4 which spatially spaced to a place of interest 8th is, for example, several kilometers apart. The ER glasses 7 comprises a first display device 20 that are at par carrier 2 located and by means of the well-known information objects in a field of view 9 of the carrier 2 can be displayed. The field of vision 9 is otherwise not obstructed or obscured, so the wearer 2 the real environment within the showroom 4 can look at.

Furthermore, the ER includes glasses 7 a detection device 22 in the form of a camera. This captures the environment of ER glasses 7 optically or imaginatively and generates acquisition data optionally in the form of image and / or video data. In the case shown, video data is generated which, in the manner described below, is available to the interested party 8th be made available in real time. A field of view of the camera is essentially identical to the field of view 9 of the carrier 2 What about a suitable positioning and / or orientation of the camera within the ER glasses 7 is possible.

In the sales room is also an optional computer device 13 arranged. This includes a router (not shown separately) to provide a non-public local communication network that is WLAN-based. Consequently, between the ER glasses 7 , which likewise comprises a processor unit in a known manner, and the computer device 13 a Wi-Fi connection can be established for data exchange. It should be noted that all related to the computer setup 13 explained functions according to the invention of ER glasses 7 can be done by yourself. The computer device 13 is therefore optional.

The computer device 13 is also connected to a public global communication network in the form of the Internet 15 via the schematically indicated connection 16 connected. Consequently, data can be obtained from the ER glasses 7 about the computer setup 13 to the internet 15 be transmitted.

The prospective 8th is located in a private room 5 and considers a second display device 24 that is part of a traditional PC 6 or alternatively, a mobile device, not shown, such as a smartphone. The computer 6 is also via a schematically indicated connection 17 with the Internet 15 connected. The Internet 15 itself can be provided via a server not shown separately or have, over which the PC 6 and the computer device 13 (and thus indirectly also the ER glasses 7 ) can communicate with each other. All communication can take place in real time and preferably in accordance with a WebRTC protocol.

In the context of the present disclosure and also detached from the further details of the embodiments shown, the space 4 in which is the carrier 2 located, a sales room 4 for example be vehicles. It can also be a sales room 4 to deal with other objects or generally a workshop, a presentation room, a teaching room or a training room. The space 5 in which the prospect 8th may, according to other variants of the invention also be a classroom or a training room in which also several interested parties 8th can be located. The space 4 of the carrier 2 can therefore generally be used as a demonstration room and the room 5 of the prospective customer 8th can generally be referred to as an interested party, audience or customer space.

In the example shown, it is provided that detection data of the detection device 22 in the form of real-time video data from the ER glasses 7 to the second display device 24 be transmitted. The prospective 8th thus can the environment within the salesroom 4 from the (human) perspective of the wearer 2 what makes a particularly realistic information transfer possible. In addition to the video connection, there is also an audio connection, so the carrier 2 and the prospect 8th communicate verbally with each other. Alternatively or additionally, the interested party 8th Text messages via a keyboard of the PC 6 enter that on the first display device 20 the ER glasses 7 are displayed.

On the server, not shown, further information objects are stored, which with a machine-readable code explained below 11 are associated. At the beginning of a communication connection or, in other words, communication session, all information objects that are provided with machine-readable codes 11 inside the sales room 4 in a memory device of the computer device 13 and the PC 6 loaded. These can therefore be called up and displayed very quickly as soon as a corresponding command is issued. Alternatively, the corresponding information objects 26 be kept centrally in the server and only as needed to the ER glasses 7 or the second display device 24 be issued. Both variants, however, allow the information objects 26 not only from the ER glasses 7 or the PC 6 must be generated and / or sent, which limits the communication effort and increases the achievable communication speed accordingly.

At the machine-readable code 11 , the object to be explained 10 is arranged (in the case shown on a painting), it is a QR code. This was made in advance by the seller 2 created and printed by conventional software and on the object 10 positioned. More precisely, the QR code is on a label imprinted on a frame of the object 10 is appropriate.

The QR code generally provides an optically recognizable feature 12 that is from both the carrier 2 as well as from the detection device 22 (or within the detection data generated therefrom) can be detected. In the example shown, the detection data of the detection means 22 in the form of image and / or video data continuously from the computer device 13 (or the ER glasses 7 itself) evaluated for the presence of a corresponding optically recognizable feature 12 to investigate. For this purpose, you can fall back on an ordinary image analysis software. Furthermore, in advance information was deposited, which visually recognizable features 12 currently in the sales room 4 located and / or according to which characteristics 12 currently being searched.

Alternatively or additionally, the outline of the painting and / or motifs within the painting could have been deposited as an original or as a pattern, the latter again being visually recognizable features 12 form. Such features can also be determined and recognized within the framework of image recognition, pattern recognition and / or template matching.

The QR code includes a link, a reference, an identifying indication, and / or a link to an information object associated with the QR code 26 , The information objects 26 were already in the above-described manner in the computer device 13 and the PC 6 loaded or are centrally available from the server, not shown. As soon as the QR code has been recognized and decoded, which in turn can take place via a conventional decoding software, an associated information object is subsequently produced 26 on the display device 20 the ER glasses 7 displayed. These are, for example, textual information that may relate to a historical background of the painting, a possible acceptable price range or other relevant information. This information can initially only the carrier 2 are displayed.

Preferably, the ER glasses 7 and / or their detection device 22 recognize and / or decode the QR code directly yourself. For this purpose by means of the ER glasses 7 (and in particular any processor means thereof) suitable programs (or apps) are executed.

A possible ad content of the display device 20 is in 2A shown. Here, one first recognizes the outline of the display device 20 holding a glass of ER glasses 7 matches. If nothing is displayed in it, the display device 20 essentially transparent. It can be seen at the bottom of the display 20 three information objects arranged side by side 26 in the form of individual text fields. More specifically, with the decoded QR code, a plurality of information objects 26 linked in the form of individual text fields.

The carrier 2 can read through these text boxes during the ongoing sales pitch and the corresponding information to the interested parties 8th pass on. In addition, an activation option is provided, for example in the form of a key assignment of a hand-held input device, not shown, by means of which the carrier 2 displaying at least one of the information objects 26 also on the second display device 24 can activate. In other words, the carrier 2 thus selectively and user-controlled determine whether and which of the information objects 26 the prospective customer 8th are displayed.

It can also be provided that not for all of the information objects 26 a corresponding activation option exists, so that some of the information objects 26 in advance as only for the wearer 2 can be defined.

If the carrier 2 displaying one of the information objects also on the second display device 24 activated, this information object can 26 additionally or alternatively to the representation at the lower edge also enlarged within the first display device 20 are displayed.

In 2 B is the second display device 24 including the contents and information objects displayed therein 26 shown. You first recognize a video window 28 in which the detection data of the detection device 22 the ER glasses 7 be displayed as a real-time video stream. You can also get one at the bottom of the video window 28 superimposed enlarged information object 26 recognize in the form of a text field. This is an information object 26 that from the carrier 2 for display on the second display device 24 has been activated.

In addition, you can see three lower additional information objects 26 , which at the bottom of the second display device 24 and outside the video window 28 are positioned. These are information objects 26 which are displayed automatically immediately after recognition of the QR code, and only on the second display device 24 and not on the first display device 20 the ER glasses 7 , These information objects 26 can in the form of image data present and additional views of the object 10 present or otherwise convey information content.

Optionally, the carrier 2 the ER glasses 7 also cause (for example, by an appropriate key press, a voice command or the like) that all or at least selected of the initially exclusively on the second display device 24 displayed information objects 26 also on the first display device 20 are displayed. This may be relevant, for example, if the prospect 8th to one of the information objects 26 a concrete question.

By means of the arrangement shown 1 and the method which can be carried out thereby thus the carrier 2 the spatially separated interested parties 8th comprehensively about the object 10 Inform and also the flow of information by selectively displaying and / or not displaying information objects 26 Taxes. Furthermore, by recognizing the QR code, he can also quickly obtain information himself to the interested party 8th the object 10 be able to explain comprehensively. Since the communication takes place in particular when applying an optional WebRTC protocol in real time and the information objects 26 thanks to pre-loading or can be quickly accessed and displayed directly from a central server, the sales pitch can take place substantially or completely uninterrupted.

In this context, the use of a QR code provides an additional advantage, since it can be decoded very quickly. The carrier 2 So does not have to interrupt the sales pitch, but can timely and essentially immediately after detecting the QR code, the desired information about the object 10 receive.

The acquisition data and / or all displayed information objects 26 can be recorded during the sales pitch. In particular, the content of the second display device 24 and / or at least the video window 28 with those under control of the vehicle 2 displayed information objects 26 recorded, for example in the form of a video file. This can then be sent and / or retrieved if necessary, so that even persons not involved in the conversation can understand the information exchanged.

In 3 is an arrangement 1 , which carries out a method according to the invention, shown according to a further embodiment. The above explained basic principle for the exchange of information as well as the possible display contents according to the 2A - 2 B are identical and will not be explained again.

Instead, the embodiment differs 3 essentially in terms of its network and / or communication architecture from the previous example. In detail you can see the salesroom again 4 , the object arranged therein 10 with the machine-readable code 11 as well as the demonstrator 2 who has the ER glasses 7 wearing.

The ER glasses 7 is wireless (eg via WLAN) with an IP-based, non-public, local network 18 , ie, connected to a Local Area Network (located in a company). For the duration of the connection, the ER glasses 7 assigned an IP address, which can be done wirelessly via WLAN, for example.

The local network 18 is via an access device 21 , such as a DSL router with integrated DSL modem, connected to a public network, which is the Internet 19 can act. Furthermore, you can re-compute the computer 13 recognize that too with the local network 18 is connected and thus at least indirectly with the ER glasses 7 can communicate in the manner described above.

Unless the ER glasses 7 in the local network 18 is arranged, this can automatically and preferably cyclically data to a server 20 transmit, which may be formed for example as a hardware server. The transmission of the data may be from a processor device of the ER glasses 7 be initiated and / or executed. The data may be the detection data of the detection device 22 act and / or any user input from the carrier 2 that by the ER glasses 7 have been detected (for example, activation information described above for displaying information objects 26 on the second display device 24 ).

In the case shown is the server 20 realized as host computer or hardware server with access restriction, on which one or more software server (with respective access restriction) are arranged and executed according to the client-server model. The ER glasses 7 transmitted data contain clearly identifying characteristics (eg the Mac address of the ER glasses 7 ), so that a tunnel connection to the server 20 can be built. The tunnel connection is carried out by converting and transmitting a communication protocol that is embedded in a higher-level communication protocol, for example, to ensure a tapping and tamper-proof connection between the participants of the communication. The protocol can be a WebRTC protocol that allows real-time communication between participants.

In summary, therefore, an encrypted real-time communication link between the ER glasses 7 and the PC 6 with the second display device 24 being constructed.

On the server 20 User accounts are set up with any identification features of ER glasses 7 are associated. Is based on the transmitted data recognized that the ER glasses 7 with the server 20 is connected via the user account (for example, by logging in hereby or here) an external access to the ER glasses 7 be enabled. Consequently, the user account provides an access authorization to the ER glasses 7 while the server 20 is understood as restricted access IT system (ie only accessible to users who have a corresponding user account and log in hereby or hereby).

As a result, data can be transmitted via the provided tunnel connection from the server 20 to the ER glasses 7 be transferred and / or vice versa. In particular, it allows any information objects stored on the server 26 can be quickly loaded and displayed as soon as the QR code on the object 10 was decoded.

As an alternative or in addition to any MAC addresses, identifying identifiers may also be encoded by a QR code that uses the ER glasses 7 is detectable. After decoding the QR code, a tunnel connection between the server automatically assigned to the user account can be made 20 and the AR glasses 7 getting produced.

About the server 20 can not others in the local network 18 with the ER glasses 7 integrated computer-based devices (such as the PC 6 of the prospective customer 8th ) after access authorization, access to the ER glasses 7 be enabled. The computer 6 is in a conventional way to the Internet 19 connected and can do it with the server 20 and also the local network 18 the ER glasses 7 communicate.

Superior (and also independent of the further details of the described embodiments) may be provided in the context of the method and the arrangement that a direct authentication of the ER glasses 7 in the public network, for example by means of a SIM (subscriber identity module) card.

Alternatively to the described loading of presumably relevant information objects 26 in the computer device 13 or the PC 6 At the beginning and / or in preparation for a sales pitch, corresponding information objects can be created 26 also in the server 20 deposited and read out in the context of the provided real-time communication connection quickly and as needed to on the respective display devices 20 . 24 to be displayed. A common advantage of the described variants is that no information objects 26 from the ER glasses 7 itself must be generated and / or sent to the display device 24 to be displayed. As a result, a utilization of communication links is reduced and enables a particularly fast communication.

Claims (11)

Method for providing information to a head-worn augmented reality device ( 7 ), the augmented reality device ( 7 ) an optical detection device ( 22 ) and a first display device ( 20 ), which is set up in a field of vision ( 9 ) of a carrier ( 2 ) of the augmented reality device ( 7 ) Information objects ( 26 ) to display; and wherein in an environment of the augmented reality device ( 7 ) at least one predetermined optically recognizable feature ( 12 ) is arranged, with the at least one predetermined information object ( 11 ) is associated; the method comprising: - optical detection of the environment by means of the detection device ( 22 ) and detection data based thereon; Transfer of the acquisition data to at least one second display device ( 24 ) spatially separated from the augmented reality device ( 7 ) is arranged; Recognition of the optically recognizable feature ( 12 ) based on the data collected; - Display at least one information object ( 26 ), which with the optically recognizable feature ( 12 ) is associated, by means of the first display device ( 20 ). The method of claim 1, wherein the optically recognizable feature ( 12 ) on one of the carrier ( 2 ) to be explained object ( 10 ) is attached or attachable and the information objects ( 26 ) serve the carrier ( 2 ) over the object ( 10 ) to inform. The method of claim 1 or 2, further comprising: providing an activation facility by which the wearer 2 ) the augmented reality Contraption ( 7 ) displaying the information object ( 26 ) by means of the second display device ( 24 ) can activate. Method according to one of the preceding claims, wherein a plurality of information objects ( 26 ), and wherein the method further comprises: displaying the information objects ( 26 ) by means of the first display device ( 20 ), whereby an activation possibility only for a part of the information objects ( 26 ) provided. Method according to one of the preceding claims, wherein a plurality of information objects ( 26 ), at least one of the information objects ( 26 ) at least temporarily only on the second display device ( 24 ), and in particular wherein the display on the second display device ( 24 ) regardless of an activation input of the carrier ( 2 ) of the augmented reality device ( 7 ) he follows. Method according to one of the preceding claims, wherein the augmented reality device ( 7 ) and / or the second display device ( 24 ) is data transmitting connected to a memory device in which at least one information object ( 26 ), and wherein the method further comprises: - reading the information object ( 26 ) from the memory device in order to store it on the first and / or the second display device ( 24 display. Method according to one of the preceding claims, wherein the optically recognizable feature ( 12 ) at least one of the following comprises: - a machine-readable code ( 11 ), in particular a QR code; - a recognizable by text recognition string; A feature recognizable by image recognition; A feature recognizable by pattern recognition; A feature recognizable by template matching. Method according to one of the preceding claims, further comprising: recording the acquisition data and / or at least one information object ( 26 ) displayed on the second display device ( 24 ) is displayed, in particular, over a period of time in which detection data is sent to the second display device ( 24 ) be transmitted. Method according to one of the preceding claims, wherein the augmented reality device ( 7 ) and the second display device ( 24 ) are connected at least indirectly via a real-time capable communication system, in particular wherein the communication system is set up for communication on the basis of a WebRTC protocol. Method according to one of the preceding claims, wherein the augmented reality device ( 7 ) via a non-public communication network ( 18 ) with a server ( 20 ), the server ( 20 ) via a public network ( 19 ) and optionally also with the second display device ( 24 ) connected is. Arrangement ( 1 ), comprising: a head-worn augmented reality device ( 7 ) comprising an optical detection device ( 22 ) and a first display device ( 20 ), which is set up in a field of vision ( 9 ) of a carrier ( 2 ) of the augmented reality device ( 7 ) Information objects ( 26 ) to display; at least one predetermined optically recognizable feature ( 12 ) in an environment of the augmented reality device ( 7 ) and with the at least one predetermined information object ( 26 ) is associated; and at least one second display device ( 24 ) spatially separated from the augmented reality device ( 7 ) is positioned or positionable; the arrangement ( 1 ) is arranged to: - the environment by means of the detection device ( 22 ) and to generate detection data based thereon; The detection data to the second display device ( 24 ) transferred to; - the visually recognizable feature ( 12 ) using the data collected; and at least one information object ( 26 ), which with the optically recognizable feature ( 12 ) is associated, by means of the first display device ( 20 ).

Images