DE102018201711B4 - Arrangement and method for providing information in a head-portable augmented reality device - Google Patents

Abstract

A method for providing information in an augmented reality device (7) which can be worn on the head, the augmented reality device (7) comprising an optical detection device (22) and a first display device (20) which is set up for this purpose in to display a field of view (9) of a wearer (2) of the augmented reality device (7) information objects (26); and wherein optically recognizable features (12) are arranged in an environment of the augmented reality device (7), with which in each case at least one predetermined information object (11) is associated; characterized in that the method comprises: - optical detection of the environment by means of the Detection device (22) and generation of detection data based thereon - transmission of the detection data to at least one second display device (24) which is arranged spatially separate from the augmented reality device (7); - detection of optically recognizable features (12) on the basis of the acquisition data; - automatically displaying at least one information object (26) associated with one of the optically recognizable features (12) by means of the first display device (20), a plurality of information objects (26) being determined that are associated with one or more of the features (12) are associated, and the method further comprises: providing an activation Possibility of using which the carrier (2) of the augmented reality device (7) can activate the display of at least one of the information objects (26) by means of the second display device (24), an activation possibility only for a part of the determined information objects (26 ) provided.

Description

The invention relates to an arrangement and a method for providing information in a head-portable augmented reality device. In particular, the invention relates to a method and an arrangement with an augmented reality device (for example augmented reality glasses) with which optically recognizable features (for example a machine-readable code) arranged in an environment are recognized and information objects associated therewith are displayed can be.

Augmented reality devices are available in various forms on the market. An expanded reality (hereinafter also ER and in English: “augmented reality” (AR)) can generally be understood to mean that additional information objects can be faded in, for example, in the field of view of a wearer (ie a user) of an augmented reality device in the form of a text field. In particular, this can be done in such a way that the information object overlaps with the wearer's field of vision (or, in other words, overlaps with his view of reality or the environment). The information objects can be transparent or opaque from the point of view of the wearer.

Augmented reality devices are predominantly used in such a way that information objects are displayed predominantly or exclusively to the wearer of the device. Opportunities for the interactive exchange of information and / or information objects with another person who is not the bearer of the device are, however, limited.

There is a general need to enable people to exchange information over long distances. In particular, there is a need to be able to convey information over larger distances as comprehensively and as realistically as possible. For example, it can happen that a person (hereinafter also: an interested party) is interested in a certain topic or a certain object. However, the object itself or a person who can explain this object or the topic (hereinafter also referred to as a demonstrator or a presenter) can be at a distance that, for example, does not allow a direct verbal exchange of information between the persons. In other words, the interested party and the demonstrator can be spatially separated from one another. In such cases, customary means of communication have hitherto been used, for example in the form of a telephone connection, a video chat or the like. However, this is accompanied by individual disadvantages. In particular, it can prove difficult for the demonstrator to be able to react flexibly and comprehensively to inquiries from the interested party.

• - US 2006 / 0 284 791 A1 zeigt u.a. das Betrachten eines Objekts durch einen ersten Benutzer mit einer ER-Vorrichtung, wobei einem anderen Nutzer mit einer ER-Vorrichtung eine virtuelle Ansicht des Objekts eingeblendet wird.
• - WO 2017 / 149 120 A1 zeigt eine Möglichkeit zur Kommunikation zwischen einem Wartungspersonal in einem Wartungsbereich und einem entfernten Experten mittels einer ER-Vorrichtung.
• - US 2007 / 0 002 037 A1 zeigt eine ER-Kommunikationsvorrichtung, bei der eine entfernte Person Instruktionen an einen Nutzer vor Ort übermitteln kann und den Instruktionen entsprechende Inhalte einblendbar sind.
• - US 2016 / 0 049 004 A1 zeigt ein ER-Kommunikationssystem, bei dem Manipulationen virtueller Objekte durch einen der Kommunikationspartner einem anderen Kommunikationspartner angezeigt werden.
• - DE 101 53 511 A1 zeigt ein Service-System, bei dem ein mit einem Gerät befasster Techniker vor Ort eine Videobrille trägt und ein hiervon erfasstes Bild an ein entferntes Experten-Center übertragen wird.
• - DE 101 59 610 A1 zeigt eine Lösung zur Erstellung einer Dokumentation von Arbeitsvorgängen, wobei eine kopfgetragene Kamera Trackinginformationen erzeugt, denen Informationen zwecks Dokumentation von Arbeitsvorgängen zuordenbar sind.
• - DE 199 53 739 A1 zeigt eine Einrichtung und ein Verfahren zur objektorientieren Markierung und Zuordnung von Informationen zu selektierten technologischen Komponenten, wobei die Markierung und Zuordnung virtuell durch einen Experten erfolgt, um einen Anwender vor Ort zu unterstützen.
• - DE 10 2005 061 211 A1 zeigt das Erzeugen einer Mensch-Maschine-BenutzerOberfläche, bei der einem Benutzer virtuelle Steuermittel zum Überwachen und/oder Kontrollieren eines Geräts oder Prozesses bereitstellbar sind.

Technological background to augmented reality devices and in particular to information exchange between several people using such devices can be found in the following disclosures:

• - US 2006/0 284 791 A1 shows, inter alia, the viewing of an object by a first user with an ER device, a virtual view of the object being shown to another user with an ER device.
• - WO 2017/149 120 A1 shows a possibility for communication between maintenance personnel in a maintenance area and a remote expert by means of an ER device.
• - US 2007/0002 037 A1 shows an ER communication device in which a remote person can transmit instructions to a user on site and content corresponding to the instructions can be displayed.
• - US 2016/0 049 004 A1 shows an ER communication system in which manipulations of virtual objects by one of the communication partners are displayed to another communication partner.
• - DE 101 53 511 A1 shows a service system in which a technician who is equipped with a device wears video glasses on site and an image captured therefrom is transmitted to a remote expert center.
• - DE 101 59 610 A1 shows a solution for creating a documentation of work processes, wherein a head-mounted camera generates tracking information to which information can be assigned for the purpose of documenting work processes.
• - DE 199 53 739 A1 shows a device and a method for object-oriented marking and assignment of information to selected technological components, the marking and assignment being carried out virtually by an expert in order to support a user on site.
• - DE 10 2005 061 211 A1 shows the generation of a human-machine user interface in which a user can be provided with virtual control means for monitoring and / or controlling a device or process.

An object of the present invention is therefore to facilitate the exchange of information between spatially separated people.

This object is achieved by a method and an arrangement according to the attached independent claims. Advantageous further developments are specified in the dependent claims. Furthermore, it goes without saying that the features mentioned in the introductory description can also be provided individually or in any combination in the solution disclosed here, unless otherwise stated or evident.

The invention is generally based on the idea of providing a wearer of an augmented reality device with information associated with content in which another person is currently interested. For this purpose, optically recognizable features in an environment of the carrier can be recognized and / or analyzed or decoded, whereupon information objects can be displayed to the carrier that are associated with these characteristics. Thus, while the wearer is moving in the environment, he can receive specific information and convey it to the other person, for example verbally, in writing and / or by means of image or video transmission.

• - optisches Erfassen der Umgebung mittels der Erfassungseinrichtung und darauf basierendes Erzeugen von Erfassungsdaten;
• - Übertragen der Erfassungsdaten an wenigstens eine zweite Anzeigeeinrichtung, die optional räumlich getrennt von der erweiterte-Realität-Vorrichtung angeordnet ist (und/oder außerhalb eines Sichtfelds der Erfassungseinrichtung oder eines Blickfelds des Trägers);
• - Erkennen des optisch erkennbaren Merkmals anhand der Erfassungsdaten;
• - Anzeigen wenigstens eines Informationsobjekts, das mit dem optisch erkennbaren Merkmal assoziiert ist, mittels der ersten Anzeigeeinrichtung.

In detail, a method for providing information in an augmented reality device that can be worn on the head is proposed, wherein the augmented reality device comprises an optical detection device and a first display device that is set up to show the expanded device in a field of view of a wearer. Reality device displaying information objects; and wherein at least one predetermined optically recognizable feature is arranged in an environment of the augmented reality device, with which at least one predetermined information object is associated; the method comprising:

• - Optical detection of the environment by means of the detection device and generation of detection data based thereon;
• - transmitting the detection data to at least one second display device, which is optionally arranged spatially separate from the augmented reality device (and / or outside a field of view of the detection device or a field of view of the wearer);
• Recognizing the optically recognizable feature on the basis of the acquisition data;
• - Displaying at least one information object, which is associated with the optically recognizable feature, by means of the first display device.

The optically recognizable feature is preferably attached or attachable to an object to be explained by the carrier and the information objects serve to inform the carrier about the object.

The augmented reality device may be worn or portable by a demonstrator who wishes to explain a content (e.g., a course content) or an object (e.g., a purchase, such as a car or property) to a prospect. The interested party, however, can have or view the second display device and be generally spatially separate from the demonstrator (i.e. not be on site).

The augmented reality device can be designed in the form of augmented reality glasses. The augmented reality glasses can include a support frame to be positioned on a wearer's head and / or an area that can be positioned in front of at least one wearer's eye. This area can include the display device. The augmented reality glasses are preferably based on conventional variants, such as, for example, the BT-2000 augmented reality glasses available from Seiko Epson Corporation, Suwa (Japan), which represent an example of a so-called smart glass. However, the augmented reality glasses according to the invention can enable and provide the additional functions described below compared to such known variants.

The augmented reality device further comprises an optical detection device. This can generally be set up to optically record information, objects, content and / or in general the surroundings. In particular, the detection device can be set up to detect and preferably simultaneously detect an optically recognizable feature which is explained below and which is associated with information objects, and also an environment of the augmented reality device. According to one variant, the optical detection device comprises or is implemented as a camera device.

The optical detection device can be aligned or alignable in such a way that its field of vision coincides with a field of view of the wearer or at least partially overlaps it. In other words, the detection device can be set up to detect an area that is also in a field of view of the wearer and / or is viewed by the wearer.

The display device of the augmented reality device can comprise, for example, a (preferably partially or fully transparent) screen or a projection surface. In particular, the display device can be in an area of the device be integrated, which is arranged or can be arranged in the field of vision of the wearer, for example in a see-through area and / or spectacle lens. In turn, transparent and / or opaque information objects can be superimposed on the display device, which can be overlaid with a field of view of the wearer.

The environment of the augmented reality device can be, for example, a sales room, a classroom, a property, a private room or a workshop. The optically recognizable feature can in particular be designed such that it can be recognized by the detection device of the augmented reality device. Preferably, the optically recognizable feature can additionally or alternatively also be recognized by a person (in particular by any demonstrator or interested party). As described below, it can be a matter of general environmental or object features (in particular features already present there) that, for example, have been 4 selected and / or defined for optical recognition. It can also be features that have been specially made for optical recognition and / or positioned in the environment. The carrier of the augmented reality device (or the demonstrator) can select, define, manufacture and / or position such features.

According to the method, the surroundings are optically detected by means of the detection device. This can include, in particular, an image acquisition. According to one variant, the detection device detects surrounding images, for example in the form of individual images, a video and / or moving images. In particular, if the detection device comprises or is implemented as a camera device, it can capture camera images of the surroundings as acquisition data, which in turn can be present individually or in the form of a video or as moving images.

The detection data can be generated by the detection device itself or by another suitable unit of the augmented reality device. Alternatively, the detection data can be generated by a separate device (i.e. in particular separately from the augmented reality device) to which the detection device transmits detected information or signals. The acquisition data can contain image, video and / or moving image data or data from which images, videos and / or moving images can be generated. The data can be generally closed (i.e., as a record) or transferable in the form of a data stream (e.g., a video stream).

The second display device can be comprised of a conventional PC, a smartphone, a tablet or a comparable mobile device. The second display direction can generally be viewed by a person (for example a prospect) who is spatially separate from the wearer of the augmented reality device. In particular, this person or the second display device can be located in a different room, a different building or a different geographical area than the bearer of the augmented reality device. In this way it can be ensured that the second display device is outside the field of vision of the wearer.

The detection data obtained from the second display device can be displayed there, for example as an essentially continuous video and / or a video stream. In general, the transmission of the acquisition data can make it possible for a person viewing the second display device to understand, view and / or generally obtain information about the surroundings of the augmented reality device. In particular, if the detection area (or the field of view) of the detection device at least partially coincides with a field of view of the wearer or overlaps with it, the second display device can make it easier to understand a look of the wearer at the surroundings. In other words, the surroundings of the augmented reality device can be displayed or displayed to a person looking at the second display device from the perspective using the acquisition data, which corresponds to a perspective of the wearer of the augmented reality device or at least approximates it. In summary, the viewer of the second display device can be shown the surroundings of the augmented reality device in a particularly comprehensive and realistic manner, in particular in such a way that he can easily understand a position and / or perspective of the wearer of the augmented reality device.

According to the invention, a plurality of second display devices can also be provided, so that so-called 1: n communication is made possible between the demonstrator and possible interested parties (one demonstrator compared with n possible interested parties, where n is a number greater than one).

A communication exchange between an interested party looking at the second display device and a demonstrator who carries the augmented reality device is preferably not only made possible by transmitting the acquisition data. Instead, communication via text and / or audio input (ie verbally), made possible by at least the interested party to the demonstrator. This can be done, for example, by means of a corresponding text and / or audio transmission device, which can include a keyboard and / or a microphone. The latter can be integrated in a computer unit (for example a PC) which also includes the second display device. The audio signals can generally be in the form of voice inputs by a user.

The augmented reality device preferably has at least one loudspeaker in order to be able to output voice or general audio signals to the carrier, in particular based on voice inputs from the interested party looking at the second display device. Any text entries, however, can be displayed directly on the first display device. Text and / or audio input can also be converted to other output formats instead of outputting them directly to the demonstrator (for example from text to audio or vice versa).

The augmented reality device preferably comprises an audio transmission device, for example comprising a microphone, in order to be able to detect and transmit voice or general audio signals from the wearer. These can in turn be transmitted to a loudspeaker that is positioned in the vicinity of or on the second display device.

In summary, it can thus be provided within the scope of the invention that a transmission of audio signals and / or verbal communication from the interested party to the demonstrator (and / or vice versa) is also possible.

This means that the interested party can instruct the demonstrator, for example by text or speech, to move in the surroundings in a predetermined manner or in a predetermined direction. In particular, the interested party can instruct the demonstrator to move in an area that contains an object and / or content that interests him. The demonstrator can then use the generated and transmitted data to ensure that the interested party can view the object or the content in a realistic manner. For this purpose, the demonstrator can, for example, suitably align the augmented reality device or its own field of vision. Furthermore, the demonstrator can provide the prospect with speech information about the object or content, which in turn can be made possible via audio transmission.

In order to recognize the optically recognizable feature on the basis of the acquisition data, the augmented reality device can carry out image recognition and / or have suitable image recognition software, as is generally available on the market. In general, the optically recognizable feature can thus be recognized arithmetically on the basis of or in or from the acquisition data. This applies in particular if the latter is available as image or video data. In particular, the acquisition data can be evaluated essentially continuously and preferably in real time, so that the augmented reality device can recognize the optically recognizable feature essentially independently (i.e. without additional user input). In other words, the detection device can automatically detect the optically recognizable feature and / or automatically determine whether the optically recognizable feature is in the detected environment.

Alternatively, the wearer can suitably align the augmented reality device so that the optically recognizable feature lies in a detection area of the detection device. The wearer can then enter an activation command, for example by voice or by haptic actuation, so that a recognition or a recognition process is triggered. In other words, the recognition can also be initiated or triggered selectively and / or as required by the carrier.

The recognition can generally include that the augmented reality device recognizes the presence, the type and / or any information encoded by means of the optically recognizable feature. For example, as part of the recognition, the augmented reality device can recognize the presence and / or the type of a QR code and preferably also decode it directly. In other words, the augmented reality device can generally determine whether and preferably also which optically recognizable feature is in the environment. This can in turn be done by evaluating the acquisition data, for example using image evaluation software.

Using predetermined optically recognizable features can be understood to mean that the type and / or properties of these features are known and defined, so that they can be searched for the features as part of an image evaluation.

Furthermore, a separate step of determining an information object can be provided that is associated with the optically recognizable feature. This can be done by querying a database, which can be stored, for example, on a storage device and / or a server explained below. Consequently, the determination of the information object can be the sending of a corresponding request and / or Include information that or which optical feature was recognized to a further unit (for example to the server or the storage device). This step can be carried out by a control device and or a processor device of the augmented reality device.

The method further provides for at least one information object, which is associated with the optically recognizable feature, to be displayed by means of the first display device. This is preferably done as promptly as possible or immediately after the optically recognizable feature is recognized. The wearer of the augmented reality device can thus be quickly provided with relevant information, for example regarding an object or a content on which the optically recognizable feature is arranged.

The information object can be, for example, a text, a text field or a document, an audio playback or audio file, a video playback or video file, an image or an image file, a PDF document or a media attachment, a screenshot, a graphical user interface, a colored area or the colored design of a graphical user interface. In particular, the information objects can be provided as icons and / or as links which, when actuated, enable a link to other content, for example by loading or playing an audio, video or image file.

In general, the information objects can relate to objects and / or enable links to objects which enable information to be derived by a viewer.

The information objects that are displayed on the first display device to inform the wearer are preferably a text or text fields. Information objects that are displayed on the second display device for viewing by the interested party can be present in particular as image, video or audio files (or as links to such files), as a PDF document or as a specially designed (in particular colored) user interface.

One advantage that can be achieved with the described method is, for example, that an interested party can use the second display device to follow the surroundings of the augmented reality device in a realistic manner and in particular from the (human) perspective of the wearer. He can then express his interest in certain objects or contents in the environment, for example by means of text or voice input. Such objects or contents can have been marked beforehand with an optically recognizable feature, which can also form a separate step of the method according to the invention. The wearer of the augmented reality device can then approach the object or content and initiate a recognition process for the optically recognizable feature or, in particular if this is done automatically, wait for such a recognition process. Once the optically recognizable feature has been recognized, an information object associated therewith can be displayed as directly as possible in the wearer's field of vision. The carrier can pass on the information conveyed in this way, for example, verbally to the interested party.

An application scenario of the described method is, for example, using the augmented reality device in a car dealership, the carrier being a seller. Predetermined optically recognizable features were attached to the individual cars on a sales area of the dealership, for example in the form of QR codes. A spatially separated prospect contacts the seller via a communication network explained below, so that acquisition data (in particular video data) can be displayed on a display device of a prospect's PC. The seller can present the individual cars to the customer and, if necessary, by recognizing the respective optically recognizable features, additional information can be faded in to inform the prospective customer as comprehensively as possible. It is not necessary for the seller to interrupt the customer conversation or to make time-consuming manual entries. Instead, the preferably even automatically activated recognition of the optically recognizable feature on a car is sufficient, which, for example, can also go largely unnoticed from the prospect's point of view. It is understood that numerous other application scenarios are also envisaged, in which the method disclosed herein can bring further advantages. Corresponding examples are explained in more detail below.

It is important to point out that all of the above or following method steps can be carried out by at least one computer device, processor device or general computing unit, unless otherwise stated or apparent.

It is provided according to the invention that an activation possibility is provided or can be provided by means of which the wearer of the augmented reality device can activate a display of the information object by means of the second display device. This can be done, for example, via a computing or processor unit of augmented reality Device or a server connected to it. The activation option can be provided, for example, in the form of a virtual icon on a graphical user interface, the operator being able to initiate the display by means of the second display device by actuating the icon. Additionally or alternatively, the activation option can be provided by appropriately assigning an input element which can be actuated haptically or, in other words, by touch or contact, activation being possible by actuating (for example pressing, tapping, shaking or moving) the input element. Furthermore, the activation option can be provided by providing and / or activating a voice input command. The activation option can be provided promptly and preferably essentially immediately as soon as the information object is displayed.

This variant makes it possible for the wearer to decide whether information is displayed not only to him but also to any interested party that the second display device is observing. The carrier can consequently only have some information objects, which, for example, solely serve to promote sales success and / or bridge knowledge gaps, only be displayed on his own display device. Information that, on the other hand, promotes communication and the exchange of information with the interested party can be activated by the carrier, so to speak, so that it is also displayed on the second display device.

In principle, different classes of information objects can thus be provided, namely a first class that is only displayed on the first display device for viewing by the wearer, and a second class that is also displayed on the second display device when activated by the wearer can be.

It is further provided according to the invention that a plurality of information objects are determined which are associated, for example, with the optically recognizable feature or a plurality thereof, the information objects being displayed or being able to be displayed by means of the first display device, and with an activation option only for a part of the Information objects is provided. Consequently, there is a first part (or a first class) of information objects that can only be displayed on the first display device, and a second part (or a second class) of information objects that can also be displayed on the second display device.

In a further development of the method and the arrangement, there is also a further class of information objects which, at least temporarily, can only be displayed or are displayed on the second display device. In detail, a plurality of information objects can be determined that are associated, for example, with the optically recognizable feature or a plurality thereof, at least one of the information objects being displayed, at least temporarily, exclusively on the second display device (ie not additionally also on the first display device). . In particular, the display on the second display device can take place independently of an activation input by the wearer of the augmented reality device. As soon as possible and essentially immediately after the optically recognizable feature has been recognized, at least one information object associated therewith can be displayed on the second display device. This information object can be displayed on the second display device in addition to the acquisition data (in particular if these are available as a video stream and / or are displayed), for example below or above the acquisition data or can be shown directly in any video window.

This enables the prospective customer to be additionally shown suitable information objects for a currently relevant object or content which, for example, allow more extensive information and / or can generally adapt the display content of the second display device to the currently relevant object or content. The interested party can, so to speak, be attuned to the currently relevant object or content and, if necessary, view additional information (for example in the form of image files or PDF documents). Since the corresponding information objects are not necessarily required by the wearer and may be aimed specifically at the interested party, they are preferably only displayed on the second display device.

According to the invention, however, it can also be provided that a need-based display of these information objects can also be activated on the first display device. The carrier thus specifically requests the corresponding information objects, for example if the interested party asks about it. This can in turn be activated by the carrier by means of a corresponding command input.

In a development of the method and the arrangement, the augmented reality device and / or the second display device are connected in a data-transmitting manner to a storage device in which at least one information object is stored, the information object being readable from the storage device in order to display it on the first and / or the second display device. The storage device can be comprised by a server or can be implemented as such. The storage device can contain a database in which the information objects can be stored and in particular are linked to the optically recognizable feature. Depending on which optically recognizable feature was recognized, the information object associated therewith can then be read out from the storage device.

In the context of the present disclosure, a server can be understood to mean a computer program and / or a computer device (for example a hardware server), it being possible for other computer programs or computer devices (and / or processor units) to access the server, for example in order to access data therefrom relate or establish a communication connection.

In the case of a plurality of associated information objects, all information objects can be stored on the storage device or the server and can be read out therefrom. Optionally, you can also save them in temporary storage devices that are connected to the first or the second display device. This can relate, for example, to a memory device of a PC or a mobile device, which include the second display device. The actual display of the read or loaded information objects can, however, be selectively activated by the carrier in the above manner.

The previously described reading or loading of information objects can also take place independently of an actually recognized recognition of an optically recognizable feature. For example, at the start of the method, all of the information objects that are linked to the optically recognizable features in the environment can be loaded in the manner described.

In other words, all information objects that might possibly be to be displayed in the course of the method by recognizing an optically recognizable feature associated therewith can be loaded in advance. This increases the speed at which the corresponding information objects can subsequently be displayed on the first and / or the second display device. In particular, the information objects do not have to be read in individually and often over longer communication paths as soon as an optically recognizable feature has been recognized, let alone sent to the second display device by the augmented reality device or a memory unit linked to it. The information exchange can instead take place essentially in real time and the load on a communication network between the augmented reality device and the second display device can be kept low.

Preferably, however, the information objects are generally stored centrally on a storage device or a server, which the augmented reality device and / or the second display device can access via real-time communication. This increases data security since the information objects can be managed centrally on the storage device or the server.

• - einen maschinenlesbaren Code;
• - einen QR-Code (Quick-Response-Code);
• - eine per Texterkennung erkennbare Zeichenfolge, insbesondere per OCR (Optical Character-Recogn ition);
• - ein per Bilderkennung erkennbares Merkmal;
• - ein per Mustererkennung (Pattern Matching) erkennbares Merkmal;
• - ein per Template-Matching erkennbares Merkmal.

The optically recognizable feature can be provided in the form of one of the following examples or comprise at least one of the following:

• - a machine readable code;
• - a QR code (quick response code);
• a character string recognizable by text recognition, in particular by OCR (Optical Character Recogn ition);
• - A characteristic recognizable by image recognition;
• - a feature recognizable by pattern recognition (pattern matching);
• - A characteristic recognizable by template matching.

The machine-readable code (for example in the form of a QR code or a bar code) can be recognized and decoded by suitable recognition or evaluation software. In a known manner, such codes can receive identification information (for example in the form of a link), which enable a link to other content. In the present case, a link to the corresponding code, a link to the information object and / or the information which information object is linked to the code can thus be provided by recognizing and / or decoding a corresponding code. This information object can then be loaded or read (for example from the server described or the storage device) and displayed using one of the display devices. In principle, however, it is also possible for an information object (for example in the form of a text) to be encoded directly in such a machine-readable code.

One advantage of QR codes is that they can be created with little effort (for example using free software) and / or can be printed out using conventional printers. You can then click on the relevant content or Objects. A demonstrator can thus create and place optically recognizable features themselves without significant costs or technical outlay. The QR code can generally be defined according to any common standard, for example as a micro QR code or secure QR code.

For image recognition, pattern recognition and / or template matching, predefined contours, edge or color gradients, patterns, image templates or other optical properties can be predefined as optically recognizable features (for example by the demonstrator). These can then be determined using conventional image evaluation algorithms on the basis of the acquisition data or recognized therein.

Text recognition can be used, for example, to automatically recognize and read in vehicle identification numbers in the area of a vehicle's windshield.

According to a further embodiment of the method and the arrangement, the acquisition data and / or at least one information object that is displayed on the second display device can be recorded, for example over a period in which both the first and the second display device are activated and / or in which acquisition data are output to the second display device.

The simultaneous activation of the display devices and, in particular, any data transmission between them indicate that a communication process or a so-called communication session is taking place between the demonstrator and an interested party. Recording can generally be understood to mean storing the corresponding data. In particular, the acquisition data can be stored as a video file. The information objects can be stored separately from them or in such a way that they can be overlaid with the acquisition data (or the video displayed based on them). In particular, the display contents of the first and / or the second display device, including the information objects displayed thereon, can be recordable.

These variants enable the recorded data to be made available to the prospective customer in order to understand it again. In other words, the demonstration that has taken place can also be viewed again in an offline mode. This can also be done by interested parties who were not directly involved in the actual demonstration, but would like to access the recorded content, for example, in a video portal accessible via the Internet. This also enables 1: n communication, which has already been mentioned above.

In particular, it can be provided in this context to create a protocol that can be displayed and / or operated using a graphical user interface. Based on the recorded acquisition data and / or displayed information objects, the protocol can preferably summarize temporally ordered information about the course and / or the events of a communication that took place between the carrier and the interested party. For example, video sequences, audio recordings, displayed information objects, exchanged text messages or voice messages can be marked or displayed along a time axis and ordered according to their respective associated times.

In general, the augmented reality device and the second display device can be connected or connectable at least indirectly via a real-time communication system (or communication network), in particular wherein the communication system is set up for communication based on a WebRTC protocol. The communication system can be at least partially wired and / or at least partially wireless. Furthermore, the communication system can comprise a storage device and / or a server, wherein the storage device and / or the server can be connected both to the augmented reality device and thus to the second display device.

The WebRTC protocol (Web Real-Time Communication) relates to an open standard promoted by the World Wide Web Consoritum (W3C), which defines a collection of communication protocols and programming interfaces (API). This makes it possible to enable real-time communication via computer-computer connections using web servers. Computers to be connected accordingly can be provided both by the augmented reality device, for example in the form of a processor, and in connection with the second display device (for example in the form of a PC or mobile device, which comprise the second display device).

According to a further embodiment of the method and the arrangement, the augmented reality device is connected to a server via a (for example local) non-public communication network, the server being accessible via a (for example global) public communication network and in particular one Allows communication link with the second display device. The communication networks can in turn be at least partially wired and / or at least partially wireless.

The non-public communication network can be an IP-based network. In general, the non-public communication network can be implemented as a WLAN or Ethernet network and / or the public communication network as the Internet. The non-public communication network can be connectable to the public communication network via an access device (for example in the form of a (eg DSL) modem). Provided that it is connected to the non-public communication network, the augmented reality device can automatically and preferably cyclically transmit data to and / or receive data from a server. The server can be connected to both the non-public and the public communication network, but can in principle be outside the public communication network (i.e. spatially separated from it).

In particular, the augmented reality device can transmit uniquely identifying data and / or features (or, in other words, uniquely identifying identification features) to the server, so that a tunnel connection (or generally a secure connection) can be established. The data or features can be, for example, a MAC address of the augmented reality device. Furthermore, the data and / or features can be transmitted from the non-public communication network to the public communication network and / or the server by means of the access device.

The server can assign the transmitted characteristic data and / or features to a user account and thus enable external access to the augmented reality device by this user. In general, the server can also allow other computer-based devices (for example other PCs or mobile devices) that are not integrated in the non-public communication network to access the augmented reality device, in particular after access authorization has been made possible on the basis of the described user account.

Furthermore, it can be provided that uniquely identifying data and / or features are encoded using a machine-readable code (for example a QR code). This code can be recognized and decoded by means of the augmented reality device in the manner described above, whereupon a user account linked to the augmented reality device can be identified and / or a tunnel connection to the server can be established automatically.

Establishing a tunnel connection can generally be a prerequisite for enabling the described communication processes between a demonstrator and interested parties, in particular the recognition of further machine-readable codes and the loading and / or exchange of information objects based thereon.

As described, possible application scenarios of the method relate to a real estate sales call, a plant sales call, a machine sales call, a medical technology sales call or a sporting goods sales call, each of which can be carried out in real time or online. Likewise, a wearer of the augmented reality device can inform a prospect and preferably several prospects (see 1: n communication above) about a specific topic, object or content. In particular, training in real estate, systems, machines, medical devices, sporting goods or the like can be carried out in real time in this way.

• - die Umgebung mittels der Erfassungseinrichtung zu erfassen und darauf basierend Erfassungsdaten zu erzeugen;
• - die Erfassungsdaten an die zweite Anzeigeeinrichtung zu übertragen oder auszugeben;
• - das optisch erkennbare Merkmal anhand der Erfassungsdaten zu erkennen; und
• - wenigstens ein Informationsobjekt, das mit dem optisch erkennbaren Merkmal assoziiert ist, mittels der ersten Anzeigeeinrichtung anzuzeigen.

The invention further relates to an arrangement comprising: an augmented reality device which can be worn on the head and which comprises an optical detection device and a first display device which is set up to display information objects in a field of view of a wearer of the augmented reality device; at least one predetermined optically recognizable feature, which is arranged in an environment of the augmented reality device and with which at least one predetermined information object is associated; and
at least one second display device that is positioned or positionable spatially separate from the augmented reality device;
the arrangement being set up for this:

• - Detect the environment by means of the detection device and generate detection data based thereon;
• - to transmit or output the acquisition data to the second display device;
• - to recognize the optically recognizable feature on the basis of the acquisition data; and
• - to display at least one information object, which is associated with the optically recognizable feature, by means of the first display device.

The optically recognizable feature is preferably attached or attachable to an object to be explained by the carrier and the information objects serve to inform the carrier about the object.

The device can comprise any further development and any further feature in order to provide or carry out all of the above or following steps, operating states and functions. In particular, the device can be configured to carry out a method according to any of the above or the following aspects. Furthermore, the arrangement for executing the listed measures can include suitable computing, control and / or processor devices. These can be provided in the augmented reality device and / or in a device which comprises the second display device.

• 1 eine Prinzipskizze einer erfindungsgemäßen Anordnung gemäß einem ersten Ausführungsbeispiel, die ein erfindungsgemäßes Verfahren ausführt;
• 2A-2B schematische Ansichten von Informationsobjekten, die mittels der ersten und zweiten Anzeigeeinrichtung der Anordnung aus 1 angezeigt werden; und
• 3 eine Prinzipskizze einer erfindungsgemäßen Anordnung gemäß einem zweiten Ausführungsbeispiel, die ein erfindungsgemäßes Verfahren ausführt.

In the following, embodiments of the invention are explained on the basis of the attached schematic figures. Features that match their type and / or function can be provided with the same reference symbols across all embodiments. They represent:

• 1 a schematic diagram of an arrangement according to the invention according to a first embodiment, which carries out a method according to the invention;
• 2A-2B schematic views of information objects by means of the first and second display device of the arrangement 1 are displayed; and
• 3rd a schematic diagram of an arrangement according to the invention according to a second embodiment, which carries out a method according to the invention.

In 1 is an arrangement according to the invention 1 shown, which carries out a variant of a method according to the invention. The order 1 may generally provide or be implemented as a real-time audiovisual reality perception communication system.

The order 1 comprises an augmented reality device 7 in the form of augmented reality glasses (hereinafter referred to as ER glasses) 7 designated). This is from one person 2nd worn, which in the case shown is a demonstrator of a prospect explained below 8th an object 10th would like to explain and sell.

The person 2nd is in a sales room 4th that is spatially spaced from a location of the prospect 8th is, for example, several kilometers apart. The ER glasses 7 comprises a first display device 20 that are at eye level of the wearer 2nd is located and by means of the information objects in a field of view in a known manner 9 of the carrier 2nd can be shown. The field of vision 9 is otherwise not blocked or covered, so the wearer 2nd the real environment inside the sales room 4th can look at.

ER glasses are also included 7 a detection device 22 in the form of a camera. This captures the surroundings of the ER glasses 7 optically or visually and thereby generates acquisition data optionally in the form of image and / or video data. In the case shown, video data are generated which are shown to the interested party in the manner described below 8th be made available in real time. A field of view of the camera is essentially identical to the field of view 9 of the carrier 2nd what about proper positioning and / or alignment of the camera within the ER glasses 7 is made possible.

In the sales room there is also an optional computer device 13 arranged. This includes a router, not shown separately, in order to provide a non-public local communication network that is WLAN-based. Consequently, between the ER glasses 7 , which also comprises a processor unit in a known manner, and the computer device 13 a WLAN connection for data exchange can be established. It should be noted that all related to the computer setup 13 explained functions according to the invention also from the ER glasses 7 can be carried out by yourself. The computer setup 13 is therefore only optional.

The computer setup 13 is also with a public global communications network in the form of the Internet 15 via the schematically indicated connection 16 connected. Consequently, data from the ER glasses 7 about the computer setup 13 to the internet 15 be transmitted.

The prospective 8th is in a private room 5 and looks at a second display 24th that are part of a conventional PC 6 or alternatively a mobile device, not shown, such as a smartphone. The computer 6 is also via a schematically indicated connection 17th with the Internet 15 connected. The Internet 15 itself can be provided or have a server, not shown separately, via which the PC 6 and the computer device 13 (and thus indirectly also the ER glasses 7 ) can communicate with each other. All communication can take place in real time and preferably according to a WebRTC protocol.

Within the scope of the present disclosure and also detached from the further details of the The embodiments shown can be in the room 4th in which the carrier 2nd is a sales room 4th for example for vehicles. But it can also be a sales room 4th act for other items or generally a workshop, presentation room, classroom or training room. The space 5 in which the prospect 8th is, according to further variants according to the invention, can also be a classroom or a training room in which there are also several interested parties 8th can be located. The space 4th of the carrier 2nd can therefore be used generally as a demonstration room and the room 5 of the prospect 8th can be generally referred to as a prospect, audience or customer room.

In the example shown it is provided that detection data of the detection device 22 in the form of real-time video data from the ER glasses 7 to the second display device 24th be transmitted. The prospective 8th can thus be the environment within the sales room 4th from the (human) perspective of the wearer 2nd consider what enables a particularly realistic information transfer. In addition to the video connection, there is also an audio connection so that the wearer 2nd and the prospect 8th verbally communicate with each other. Alternatively or additionally, the interested party 8th Text messages via a keyboard of the PC 6 Enter that on the first display device 20 the ER glasses 7 are displayed.

Information objects are also stored on the server, not shown, with a machine-readable code explained below 11 are associated. At the beginning of a communication connection or, in other words, communication session, all information objects are created using machine-readable codes 11 inside the sales room 4th are associated in a storage device of the computer device 13 and the pc 6 loaded. These can therefore be called up and displayed particularly quickly as soon as a corresponding command is issued. Alternatively, the corresponding information objects 26 be kept centrally in the server and only if necessary to the ER glasses 7 or the second display device 24th be issued. However, both variants allow the information objects 26 not just from ER glasses 7 or the PC 6 must be generated and / or sent, which limits the communication effort and increases the achievable communication speed accordingly.

With the machine-readable code 11 on the object to be explained 10th is arranged (in the case shown on a painting), it is a QR code. This was in advance from the seller 2nd created and printed using conventional software and on the object 10th positioned. More specifically, the QR code is printed on a label attached to a frame of the object 10th is appropriate.

The QR code generally represents an optically recognizable feature 12th represents both the carrier 2nd as well as from the detection device 22 (or within the detection data generated thereby) can be recognized. In the example shown, the detection data of the detection device 22 continuously in the form of image and / or video data from the computer device 13 (or the ER glasses 7 itself) evaluated the presence of a corresponding optically recognizable feature 12th to investigate. Standard image analysis software can be used for this. In addition, information was stored in advance as to which optically recognizable features 12th is currently in the sales room 4th and / or according to which characteristics 12th is currently to be searched.

Alternatively or additionally, the outline of the painting and / or motifs within the painting could have been stored as an image template or as a pattern, the latter in turn optically recognizable features 12th form. Features of this type can also be ascertained and recognized in the context of image recognition, pattern recognition and / or so-called template matching.

The QR code contains a link, a reference, identifying information and / or a link to an information object associated with the QR code 26 . The information objects 26 were already in the above described manner in the computer device 13 and the PC 6 loaded or can be accessed centrally from the server, not shown. As soon as the QR code has been recognized and decoded, which in turn can be done using conventional decoding software, an information object associated therewith then becomes 26 on the display device 20 the ER glasses 7 displayed. This is, for example, text information that can relate to a historical background of the painting, a possible acceptable price range or other relevant information. This information can initially only the carrier 2nd are displayed.

The ER glasses are preferred 7 and / their detection device 22 Detect and / or decode the QR code yourself. To do this, use the ER glasses 7 (and in particular any processor device thereof) suitable programs (or apps) are executed.

A possible display content of the display device 20 is in 2A shown. Here recognizes one first the outline of the display device 20 with a glass of ER glasses 7 matches. If nothing is shown in it, the display device is 20 essentially transparent. It can be seen at the bottom of the display device 20 three information objects arranged side by side 26 in the form of individual text fields. More specifically, the decoded QR code is a multiplicity of information objects 26 linked in the form of individual text fields.

The carrier 2nd can read through these text fields and the relevant information to the prospect during the ongoing sales discussion 8th pass on. In addition, an activation option is provided, for example in the form of a key assignment of a handheld input device, not shown, by means of which the carrier 2nd displaying at least one of the information objects 26 also on the second display device 24th can activate. In other words, the wearer can 2nd thus selectively and user-controlled determine whether and which of the information objects 26 the interested party 8th are displayed.

It can also be provided that not all of the information objects 26 there is a corresponding activation option so that some of the information objects 26 in advance as only for the wearer 2nd can definitely be defined.

If the carrier 2nd the display of one of the information objects also on the second display device 24th activated, this information object 26 in addition or as an alternative to the display at the bottom, also enlarged within the first display device 20 are displayed.

In 2 B is the second display device 24th including the content and information objects displayed therein 26 shown. A video window can be seen first 28 , in which the detection data of the detection device 22 the ER glasses 7 displayed as a real-time video stream. You can also see one at the bottom of the video window 28 superimposed enlarged information object 26 recognize in the form of a text field.

This is an information object 26 that from the carrier 2nd for display on the second display device 24th was activated.

In addition, three lower information objects can be seen 26 , which at the bottom of the second display device 24th and outside the video window 28 are positioned. These are information objects 26 that are automatically displayed immediately after the QR code is recognized, and only on the second display device 24th and not on the first display 20 the ER glasses 7 . These information objects 26 can be in the form of image data and additional views of the object 10th present or convey other information content.

Optionally, the carrier 2nd the ER glasses 7 also cause (for example by means of a suitable key actuation, a voice command or the like) that all or at least selected ones initially only on the second display device 24th displayed information objects 26 also on the first display device 20 are displayed. This can be relevant, for example, if the prospect 8th to one of the information objects 26 asks a specific question.

Using the arrangement shown 1 and the method which can be carried out in this way can thus be carried out by the carrier 2nd the spatially separated interested parties 8th comprehensive about the object 10th Inform and also the flow of information through selective display and / or non-display of information lens 26 Taxes. Furthermore, by recognizing the QR code, he can quickly receive information for the prospect 8th the object 10th to be able to explain comprehensively. Because the communication takes place in real time, especially when using an optional WebRTC protocol, and the information objects 26 Thanks to pre-loading or being able to be called up and displayed quickly from a central server, the sales pitch can take place essentially or completely without interruption.

In this context, the use of a QR code also provides an additional advantage, since it can be decoded particularly quickly. The carrier 2nd does not have to interrupt the sales talk, but can promptly and essentially immediately after recognizing the QR code the desired information about the property 10th receive.

The acquisition data and / or all displayed information objects 26 can be recorded during the sales pitch. In particular, the content of the second display device 24th and / or at least the video window 28 with the ones under the control of the wearer 2nd displayed information objects 26 recorded, for example in the form of a video file. This can then be sent and / or called up as required so that people who are not involved in the conversation can also understand the information exchanged.

In 3rd is an order 1 which carries out a method according to the invention is shown according to a further exemplary embodiment. The The basic principle for information exchange explained above and the possible display contents in accordance with the 2A-2B are identical and are not explained again.

Instead, the embodiment of FIG 3rd essentially in terms of its network and / or communication architecture from the previous example. You can see the sales room again in detail 4th , the object placed in it 10th with the machine readable code 11 as well as the demonstrator 2nd who the ER glasses 7 wearing.

The ER glasses 7 is wireless (e.g. via WLAN) with an IP-based, non-public, local network 18th , for example connected to a local area network (arranged in a company). For the duration of the connection, the ER glasses 7 assigned an IP address, which can be done wirelessly using WLAN, for example.

The local network 18th is via an access facility 21 , such as a DSL router with an integrated DSL modem, connected to a public network that is the Internet 19th can act. Furthermore, you can again the computer device 13 recognize that also with the local network 18th is connected and thus at least indirectly with the ER glasses 7 can communicate in the manner described above.

Unless the ER glasses 7 in the local network 18th is arranged, this can automatically and preferably cyclically send data to a server 20 transmit, which can for example be designed as a hardware server. The data can be transmitted from a processor device of the ER glasses 7 initiated and / or executed. The data can be the acquisition data of the acquisition device 22 act and / or any user input by the carrier 2nd made by the ER glasses 7 were recognized (for example, activation information described above for displaying information objects 26 on the second display device 24th ).

In the case shown is the server 20 realized as a host computer or hardware server with restricted access, on which one or more software servers (with respective restricted access) are arranged and executed in accordance with the client-server model. The ER glasses 7 The data transmitted contain clearly identifying features (e.g. the Mac address of the ER glasses 7 ), so that a tunnel connection to the server 20 can be built. The tunnel connection is made by converting and transmitting a communication protocol that is embedded in a higher-level communication protocol, for example, to ensure a tap-proof and tamper-proof connection between the communication participants. The protocol can be a WebRTC protocol that enables real-time communication between the participants.

In summary, an encrypted real-time communication link between the ER glasses 7 and the PC 6 with the second display device 24th being constructed.

On the server 20 user accounts are set up with any identification features of the ER glasses 7 are associated. It is recognized from the transmitted data that the ER glasses 7 with the server 20 external access to the ER glasses can be made via the user account (for example by logging in here or via this) 7 be made possible. As a result, the user account provides access authorization for the ER glasses 7 represent while the server 20 is to be understood as an access-restricted IT system (ie it is only accessible to users who have a corresponding user account and log in here or here).

As a result, data about the tunnel connection provided can be obtained from the server 20 to the ER glasses 7 be transmitted and / or vice versa. In particular, it is possible to load and display as soon as the QR code on the object 10th was decoded.

As an alternative or in addition to any MAC addresses, identifying identification features can also be encoded by a QR code, which is by means of the ER glasses 7 is detectable. After decoding the QR code, a tunnel connection between the server and the user account can be created automatically 20 and the AR glasses 7 getting produced.

Via the server 20 can also not be in the local network 18th with the ER glasses 7 integrated computer-based facilities (such as the PC 6 of the prospect 8th ) access to the ER glasses after successful access authorization 7 be made possible. The computer 6 is doing it in a conventional way with the Internet 19th connected and can be connected to the server 20 and also the local network 18th the ER glasses 7 communicate.

Overriding (and also independent of the further details of the described exemplary embodiments) it can be provided in the context of the method and the arrangement that direct authentication of the ER glasses 7 in the public network, for example using a SIM (subscriber identity module) card.

As an alternative to the described store of probably relevant information objects 26 into the computer facility 13 or the PC 6 At the beginning and / or in preparation of a sales pitch, you can use appropriate information objects 26 even in the server 20 stored and read out quickly and as required within the framework of the real-time communication connection provided, in order to be displayed on the respective display devices 20 , 24th to be displayed. A common advantage of the described variants is that there are no information objects 26 from the ER glasses 7 self-generated and / or sent to be on the display device 24th to be displayed. This reduces the load on the communication connections and enables particularly fast communication.

Claims (9)

A method for providing information in an augmented reality device (7) which can be worn on the head, the augmented reality device (7) comprising an optical detection device (22) and a first display device (20) which is set up for this purpose in display information objects (26) to a field of view (9) of a wearer (2) of the augmented reality device (7); and wherein in an environment of the augmented reality device (7) optically recognizable features (12) are arranged, with which in each case at least one predetermined information object (11) is associated; characterized in that the method comprises: - optical detection of the surroundings by means of the detection device (22) and generation of detection data based thereon; - Transfer of the acquisition data to at least one second display device (24), which is arranged spatially separated from the augmented reality device (7); - recognizing optically recognizable features (12) on the basis of the acquisition data; - automatic display of at least one information object (26) associated with one of the optically recognizable features (12) by means of the first display device (20), a plurality of information objects (26) being determined which are associated with one or more of the features ( 12), and the method further comprises: - providing an activation option by means of which the carrier (2) of the augmented reality device (7) displays at least one of the information objects (26) by means of the second display device (24) can activate, an activation possibility being provided only for a part of the determined information objects (26). Procedure according to Claim 1 The optically recognizable feature (12) is attached or attachable to an object (10) to be explained by the carrier (2) and the information objects (26) serve to inform the carrier (2) about the object (10). Method according to one of the preceding claims, wherein a plurality of information objects (26) are determined, wherein at least one of the information objects (26) is displayed at least temporarily exclusively on the second display device (24), and in particular wherein the display on the second display device (24) takes place independently of an activation input of the carrier (2) of the augmented reality device (7). Method according to one of the preceding claims, wherein the augmented reality device (7) and / or the second display device (24) is connected for data transmission to a storage device in which at least one information object (26) is stored, and wherein the method further comprises : - Reading out the information object (26) from the storage device in order to display it on the first and / or the second display device (24). Method according to one of the preceding claims, wherein the optically recognizable feature (12) comprises at least one of the following: - a machine-readable code (11), in particular a QR code; - a character string recognizable by text recognition; - A characteristic recognizable by image recognition; - a characteristic recognizable by pattern recognition; - A characteristic recognizable by template matching. The method of any one of the preceding claims, further comprising: - Recording the acquisition data and / or at least one information object (26) which is displayed on the second display device (24), in particular over a period in which acquisition data are transmitted to the second display device (24). Method according to one of the preceding claims, wherein the augmented reality device (7) and the second display device (24) are at least indirectly connected via a real-time communication system, in particular the communication system being set up for communication based on a WebRTC protocol. Method according to one of the preceding claims, wherein the augmented reality device (7) is connected to a server (20) via a non-public communication network (18), the server (20) being accessible via a public network (19) and optionally also being connected to the second display device (24). Arrangement (1), comprising: an augmented reality device (7) which can be worn on the head and which comprises an optical detection device (22) and a first display device (20) which is set up in a field of view (9) of a wearer ( 2) to display information objects (26) to the augmented reality device (7); a plurality of predetermined optically recognizable features (12) which are arranged in an environment of the augmented reality device (7) and with which at least one predetermined information object (26) is associated; and at least one second display device (24) that is positioned or positionable spatially separate from the augmented reality device (7); characterized in that the arrangement (1) is set up to: - detect the surroundings by means of the detection device (22) and generate detection data based thereon; - to transmit the detection data to the second display device (24); - to recognize optically recognizable features (12) on the basis of the acquisition data; and - automatically display information objects (26) associated with the optically recognizable features (12) by means of the first display device (20); wherein if a plurality of information objects (26) are identified which are associated with one or more of the features (12), the arrangement (1) is further configured to: - provide an activation option by means of which the carrier (2) of the Augmented reality device (7) can activate the display of at least one of the information objects (26) by means of the second display device (24), an activation possibility being provided only for a part of the determined information objects (26).

Images