DE102022116004A1 - Techniques for generating conversation topics in a virtual environment - Google Patents

Abstract

The present invention relates to techniques for generating topics of a point of interest “POI” in a virtual environment, comprising the following steps
• Representation of a virtual POI on a playback unit, in particular a display unit and/or loudspeaker unit, of a user terminal;
• Capture audio signals from the POI and optionally additional capture of other data identifying the POI;
• Use of the recorded audio signals and optionally additionally the further identifying data as input of a classification algorithm, the classification algorithm being set up to recognize a keyword for the one POI based on the recorded audio signals and optionally additionally the further identifying data;
Playback of the recognized keyword of the POI.

Description

The present invention relates to a technical solution for generating topics of conversation in a virtual environment and displaying these topics of conversation on a user terminal. In particular, so-called VR glasses can be used as a user terminal for the method.

User terminals such as computers, smartphones, tablets and/or VR glasses are known from the prior art, with which users can communicate virtually with other users in a virtual environment, which is preferably implemented on a cloud server. The user terminals have input devices such as a keyboard, mouse and/or a microphone, with which a user can communicate or “move” in the virtual world. The user terminals also have display units, for example a screen, by means of which the user can see where they are in the virtual environment, and additional information from the virtual world can be displayed to the user via the display units. In addition to the display units, the user terminals can also include loudspeakers, which also provide the user with information about the virtual world.

The user terminals, particularly in the case of smartphones, can communicate with a server via a communication network using a radio module via radio technology, for example mobile communications and/or WLAN. However, the user terminals, particularly in the case of computers, can also communicate with the server via a wired connection via the communication network.

It is also known from the prior art that such VR glasses are used to show a user a virtual reality that comes as close as possible to the real world. For example, the user can use their VR glasses to visit a virtual museum. If the user turns around, the display of the VR glasses on the display unit follows this rotation.

Online conference systems are also known, which were increasingly used, especially during the Corona pandemic, whereby several groups of people can talk to each other via these online conference systems. In principle, it is possible to hold a virtual trade fair or conference using such online conference systems.

At virtual conferences there are often presentations, lectures but also market stalls (e.g. from outfitters) and/or gatherings of visitors exchanging ideas about topics. Below, all of these possible meetings in the virtual environment are referred to as virtual point-of-interest POIs. In general, one can say that interactions between people in the virtual environment take place at a POI, in particular so-called avatars, which represent real people in the virtual environment.

If a user takes part in such a virtual conference, it is usually difficult for him to quickly see what is being discussed in a lecture or what the current status of this lecture is. The user therefore does not know what he may have already missed, especially in the case of a lively discussion in a break-out session or at a virtual market stand, where an outfitter presents items to visitors and answers questions about the product portfolio.

Although it is possible to provide information about the possible content of a lecture or a discussion through short summaries if the respective provider of the POI has provided this information in advance, this information is time-consuming to create and cannot be taken into account if a lecture is a “completely new has taken a turn.” It is also often the case that the most interesting topics are discussed in individual Q&A sessions.

Visitors to a virtual conference are therefore faced with the problem of deciding which lecture or discussion they would like to attend virtually when they enter the virtual conference.

Of course, users have the opportunity to go into any room - the different POIs can be virtually separated from each other by different rooms - and listen to or watch the content presented there, but this means that a lot of time is lost until the POI is actually found corresponds to the interests of the user. Especially during this time, important information about the POI that is of interest to the user can be missed, so that as time passes it becomes increasingly difficult to participate in a conversation about the POI that is of interest to the user.

The present invention is therefore based on the object of specifying technical means for generating and/or displaying topics of virtual POIs.

The present invention solves this problem through the features of the independent claims.

The features of the various aspects of the invention or the various exemplary embodiments described below can be combined with one another, unless this is explicitly excluded or technically excluded. If the terms person or user who uses the method are used below, these are synonyms within the meaning of the invention.

• • Darstellung zumindest eines virtuellen POIs auf einer Wiedergabeeinheit, insbesondere einer Anzeigeeinheit und/oder Lautsprechereinheit, eines Nutzerendgerätes;
  • ◯ bei der Anzeigeeinheit handelt es sich insbesondere um ein Display eines Nutzerendgeräts, wie beispielsweise eines Computers, eines Smartphones und/oder einer VR Brille, wobei Steuerbefehle von einem Server generiert werden - auf dem die virtuelle Umgebung implementiert ist - sodass die virtuelle Umgebung auf der Anzeigeeinheit bildlich dargestellt werden kann. Im einfachsten Fall kann es sich bei der virtuellen Umgebung um eine zweidimensionale Darstellung handeln in welcher POIs, beispielsweise als abgetrennte Bereiche, angezeigt werden.
  • ◯ Die Darstellung eines POIs kann prinzipiell auch durch eine Lautsprechereinheit erfolgen, indem POIs mit Wörtern klanglich beschrieben werden. Dies ist insbesondere dann vorteilhaft, um blinden Menschen einen Zugang zu der virtuellen Umgebung und einer Interaktion mit den POIs zu ermöglichen.
• • Erfassen von Audiosignalen des POIs und optional zusätzliches Erfassen von weiteren den POI kennzeichnenden Daten;
  • ◯ Die Audiosignale sind hierbei den entsprechenden POIs zugeordnet. Bei den weiteren den POI kennzeichnenden Daten kann es sich um Positionsdaten innerhalb der virtuellen Umgebung und/oder Zeitdaten handeln. Die weiteren den POI kennzeichnenden Daten werden vor allem dazu verwendet, um POIs voneinander unterscheiden zu können, wobei die Zeitdaten bevorzugt dazu verwendet werden, um festzustellen wann innerhalb eines POIs bestimmte Themen diskutiert wurden. Bei lediglich einem einzigen POI oder wenn die POIs gezielt voneinander getrennt in der virtuellen Umgebung dargestellt sind, ist eine solche Unterscheidung allerdings nicht zwingend notwendig, um festzulegen, welches Audiosignal welchem POI zugeordnet werden soll, insbesondere, wenn sich der Nutzer durch einen sogenannten Avatar zwischen verschiedenen POIs hin und her bewegt.
  • ◯ Bevorzugt können verschiedene POIs mittels eines POI Unterscheidungs-Algorithmus erkannt werden, der bevorzugt auf dem Server implementiert ist, der die virtuelle Umgebung bereitstellt. Eine zweckmäßige Lösung ist es hierbei, wenn die virtuelle Umgebung verschiedene definierte Räume für jeden POI bereitstellt und alle Audiosignale von Nutzern, die sich, beispielsweise mit Ihrem Avatar, in diesem Raum befinden, dem entsprechenden POI zugeordnet werden. Räume können beispielsweise durch kartesische Koordinaten definiert werden, wobei die kartesischen Koordinaten der POIs sowie die Position der Nutzer innerhalb der virtuellen Umgebung dem Unterscheidungs-Algorithmus zur Auswertung übergeben werden können. Insbesondere ist der Unterscheidungs-Algorithmus eingerichtet eine Mehrzahl von POIs zu unterscheiden. Hierzu kann der Unterscheidungs-Algorithmus in einer alternativen Variante durch ein entsprechendes trainiertes neuronales Netzwerk ausgestaltet sein, das einerseits mit einem ersten Datensatz trainiert wird, der Nutzer aufweist, die als einem POI zugehörig markiert sind und andererseits mit einem zweiten Datensatz trainiert wird, der Nutzer aufweist, die als nicht zu dem POI zugehörig markiert sind. Eine andere Möglichkeit ist es, dass der Unterscheidungs-Algorithmus eine Routine aufweist, die beispielsweise die Positionsdaten der Nutzer - respektive deren Avatare - zueinander auswertet. Wie schon vorstehend angedeutet: eine einfache Form dieser Auswertungsroutine ist es festzustellen, dass ein POI vorliegt, wenn dieser durch einen Bereich definiert ist und/oder der Abstand von jeweils einem Avatar zu einem anderen Avatar kleiner als ein bestimmter Pixelabstand ist.
  • ◯ In der virtuellen Umgebung, die auch als ein Onlinekonferenz-System ausgestaltet sein kann, werden die Audiosignale in der Regel mittels Nutzerendgeräten an den die virtuelle Umgebung bereitstellen den Server übermittelt. In diesem Fall liegen die Audiosignale also bereits vor und müssen lediglich entsprechend an weitere Routinen, weitere Algorithmen usw. übergeben werden. Initial erfasst werden diese Audiosignale mit dem Nutzerendgerät, insbesondere dem Computer oder dem Smartphone, mit dem der Nutzer mit der virtuellen Umgebung oder dem Onlinekonferenz System kommuniziert.
• • Verwendung der erfassten Audiosignale und optional zusätzlich der weiteren kennzeichnenden Daten als Input eines Klassifizierungsalgorithmus, wobei der Klassifizierungsalgorithmus eingerichtet ist basierend auf den erfassten Audiosignalen und optional zusätzlich der weiteren kennzeichnenden Daten zumindest ein Schlagwort für den zumindest einen POI zu erkennen;
  • ◯ Der Klassifizierungsalgorithmus ist bevorzugt auf dem die virtuelle Umgebung bereitstellenden Server implementiert, da hierdurch ein zusätzliches Übertragen der notwendigen Daten an eine andere Rechnereinheit vermieden wird.
  • ◯ vorzugsweise ist der Klassifizierungsalgorithmus auf dem die virtuelle Umgebung bereitstellenden Server implementiert. In diesem Fall befinden sich das Onlinekonferenz System oder die virtuelle Welt ohnehin schon auf einem Server, sodass die entsprechenden Daten, insbesondere die Audiosignale, bereits vorliegen und nicht erneut übertragen werden müssen;
  • ◯ wertet der Klassifizierungsalgorithmus Gespräche von mehr als einem POI aus, werden Information hinterlegt, die es ermöglichen die Audiosignale den entsprechenden POIs zuzuordnen, damit im Nachhinein die erstellten Schlagwörter den richtigen POIs zugeordnet werden können. Informationen, die POIs kennzeichnen werden insbesondere aus den weiteren identifizierenden Daten gebildet. In der virtuellen Welt könnte dies bedeuten, dass ein POI durch die Position x1, y1 definiert ist und dass diesem POI das Audiosignal A1 zugeordnet wird.
  • ◯ In einem ersten Schritt zur Ermittlung der entsprechenden Schlagwörter können die Audiosignale mit schon im Stand der Technik bekannten Softwarelösungen (Speech-to-texttools) in Text umgewandelt werden; insbesondere können in diesem Zusammenhang bekannte Methoden der automatischen Spracherkennung (ASR) und der Natural Language Understanding (NLU) eingesetzt werden;
  • ◯ Zur Ermittlung der entsprechenden Schlagwörter, wobei die Schlagwörter die Gesprächsthemen des jeweiligen POI repräsentieren, kann der Klassifizierungsalgorithmus auf bereits vorhandene Algorithmen virtueller Sprachassistenten zurückgreifen. Virtuelle Sprachassistenten weisen Routinen auf, die bereits in der Lage sind „ThemenVSchlagwörter aus Audiosignalen entsprechend zu generieren, da dies für die nachfolgende Verarbeitung notwendig ist. Alternativ oder zusätzlich kann der Klassifizierungsalgorithmus zur Ermittlung der entsprechenden Schlagwörter ein trainiertes neuronales Netzwerk nutzen, wobei das neuronale Netzwerk mit Datensätzen trainiert wird, die jeweils mit einem Schlagwort markiert sind. Beispielsweise könnte ein solcher Datensatz wie folgt aussehen: das Audiosignal „Hast du gestern Abend den Europapokalsieg von Eintracht Frankfurt gesehen?“ kann mit dem Schlagwort „Fußball“ markiert sein.
• • Wiedergabe des erkannten Schlagwortes des POIs, insbesondere Erzeugung von Steuerbefehlen eingerichtet zur Wiedergabe des erkannten Schlagwortes des POIs auf einer Anzeigeeinheit des Nutzerendgerätes und Übertragung der Steuerbefehle an das Nutzerendgerät.
  • ◯ Zur Wiedergabe des erkannten Schlagworts werden geeignete Mittel zur Wiedergabe verwendet, wie beispielsweise eine Anzeigeneinheit, insbesondere ein Display, oder eine Lautsprechereinheit. Es kann aber zweckmäßig sein, dass der Nutzer einen POI oder mehrere POIs mit einem Mittel zur Markierung in einem vorgelagerten Prozess markiert, wobei lediglich die Schlagworte des oder der markierten POIs wiedergegeben werden. Mittel zur Markierung können beispielsweise ein Finger auf einem Touch Display, eine Computer Tastatur und/oder eine Maus sein. Bevorzugt wird die Auswertung des Klassifizierungsalgorithmus nur bei den zuvor markierten POIs durchgeführt, wodurch Ressourcen gespart werden können. Es kann insbesondere die vorstehend erwähnte Zuordnung der POIs zu ihren Audiosignalen verwendet werden, um die erkannten Schlagwörter auf der Anzeigeeinheit den richtigen POIs zuzuordnen.

According to the invention, a method for generating topics of a point-of-interest “POI” in a virtual environment is specified, comprising the following steps:

• • Representation of at least one virtual POI on a playback unit, in particular a display unit and/or loudspeaker unit, of a user terminal;
  • ◯ the display unit is in particular a display of a user terminal, such as a computer, a smartphone and/or VR glasses, with control commands being generated by a server - on which the virtual environment is implemented - so that the virtual environment on the Display unit can be represented graphically. In the simplest case, the virtual environment can be a two-dimensional representation in which POIs are displayed, for example as separate areas.
  • ◯ The representation of a POI can in principle also be done using a loudspeaker unit by describing POIs aurally with words. This is particularly advantageous in order to enable blind people to access the virtual environment and interact with the POIs.
• • Capture audio signals from the POI and optionally additional capture of other data identifying the POI;
  • ◯ The audio signals are assigned to the corresponding POIs. The further data characterizing the POI can be position data within the virtual environment and/or time data. The other data characterizing the POI is used primarily to distinguish POIs from one another, with the time data preferably being used to determine when certain topics were discussed within a POI. However, if there is only a single POI or if the POIs are specifically displayed separately from one another in the virtual environment, such a distinction is not absolutely necessary in order to determine which audio signal should be assigned to which POI, especially if the user uses a so-called avatar moves back and forth to different POIs.
  • ◯ Preferably, different POIs can be recognized using a POI distinction algorithm, which is preferably implemented on the server that provides the virtual environment. A practical solution here is if the virtual environment provides different defined rooms for each POI and all audio signals from users who are in this room, for example with their avatar, are assigned to the corresponding POI. For example, rooms can be defined by Cartesian coordinates, whereby the Cartesian coordinates of the POIs and the position of the users within the virtual environment can be passed on to the differentiation algorithm for evaluation. In particular, the differentiation algorithm is set up to distinguish a plurality of POIs. For this purpose, in an alternative variant, the differentiation algorithm can be designed by a corresponding trained neural network, which on the one hand is trained with a first data set that has users who are marked as belonging to a POI and on the other hand is trained with a second data set, the user that are marked as not belonging to the POI. Another possibility is that the differentiation algorithm has a routine that, for example, evaluates the position data of the users - or their avatars - in relation to one another. As already indicated above: a simple form of this evaluation routine is to determine that a POI is present if it is defined by an area and/or the distance from one avatar to another avatar is smaller than a certain pixel distance.
  • ◯ In the virtual environment, which can also be designed as an online conference system, the audio signals are usually transmitted via user devices to the server that provides the virtual environment. In this case, the audio signals are already available and simply need to be passed on to further routines, further algorithms, etc. These audio signals are initially recorded with the user device, in particular the computer or smartphone, with which the user communicates with the virtual environment or the online conference system.
• • Use of the captured audio signals and optionally the additional identifying data as input to a classification algorithm, the classification algorithm being set up based on the captured to recognize at least one keyword for the at least one POI from audio signals and optionally additionally the further identifying data;
  • ◯ The classification algorithm is preferably implemented on the server providing the virtual environment, as this avoids additional transfer of the necessary data to another computer unit.
  • ◯ preferably the classification algorithm is implemented on the server providing the virtual environment. In this case, the online conference system or the virtual world is already on a server, so that the relevant data, in particular the audio signals, are already available and do not need to be retransmitted;
  • ◯ If the classification algorithm evaluates conversations from more than one POI, information is stored that makes it possible to assign the audio signals to the corresponding POIs, so that the keywords created can subsequently be assigned to the correct POIs. Information that identifies POIs is formed in particular from the further identifying data. In the virtual world this could mean that a POI is defined by the position x1, y1 and that the audio signal A1 is assigned to this POI.
  • ◯ In a first step to determine the corresponding keywords, the audio signals can be converted into text using software solutions that are already known in the state of the art (speech-to-text tools); In particular, well-known methods of automatic speech recognition (ASR) and natural language understanding (NLU) can be used in this context;
  • ◯ To determine the corresponding keywords, whereby the keywords represent the topics of conversation of the respective POI, the classification algorithm can rely on existing algorithms of virtual voice assistants. Virtual voice assistants have routines that are already able to generate topic keywords from audio signals, as this is necessary for subsequent processing. Alternatively or additionally, the classification algorithm can use a trained neural network to determine the corresponding keywords, the neural network being trained with data sets that are each marked with a keyword. For example, such a data set could look like this: the audio signal “Did you see Eintracht Frankfurt’s European Cup win last night?” can be tagged with the keyword “football”.
• • Playback of the recognized keyword of the POI, in particular generation of control commands set up to reproduce the recognized keyword of the POI on a display unit of the user terminal and transmission of the control commands to the user terminal.
  • ◯ To reproduce the recognized keyword, suitable means of reproduction are used, such as a display unit, in particular a display, or a loudspeaker unit. However, it may be expedient for the user to mark a POI or several POIs with a means of marking in an upstream process, whereby only the keywords of the marked POI or POIs are reproduced. Means for marking can be, for example, a finger on a touch display, a computer keyboard and/or a mouse. The evaluation of the classification algorithm is preferably only carried out for the previously marked POIs, which can save resources. In particular, the above-mentioned assignment of the POIs to their audio signals can be used to assign the recognized keywords on the display unit to the correct POIs.

This has the advantage that the user is shown which topics of conversation are being “discussed” in the form of keywords in a specific POI, so that the user can specifically select the POI of their choice. In this way, the annoying, time-consuming “scanning” of which POI is the right one for the user can be eliminated. The result of the analysis can in particular represent the conversation content as keywords in the form of a tag cloud, which is displayed assigned to the POI. This enables a quick overview of the topics so that the conversation content of a POI can be determined within a few seconds. The results are preferably continuously expanded by constantly passing additional audio signals from the POI to the classification algorithm for evaluation. This also results in better information about the topics of conversation and their course over a longer period of time. It is also possible for the user to specify how long ago the POI's audio signals should be evaluated. A time period that is too long consumes a lot of computer resources and may not provide an up-to-date representation of the topics discussed in the form of the keywords. The period is preferred, for example selected within the period of 5-15 minutes.

The user expediently navigates to POIs within the virtual environment using navigation means. Using these navigation tools, the user can also mark POIs. Such navigation means are, for example, a finger with which a touchscreen of a smartphone, a computer or a tablet can be operated. But it can also be a keyboard and/or a computer mouse. The navigation means have the advantage that they allow the user to interact with the virtual environment. For blind users in particular, the navigation tool can also be voice-based.

In a preferred embodiment, the user marks the at least one POI using marking means, the classification algorithm being carried out for the marked POI and/or the keywords of the marked POI being displayed. The audio signals can be recorded and processed in parallel from a subset of the POIs, in particular from just one POI.

These features offer the advantage that, in a resource-saving manner, not all audio signals from all POIs are evaluated in parallel, but only those of the POIs that were marked in a previous step. Advantageously, at least one topic can be assigned and displayed to the POIs when the virtual environment is created, so to speak, as a heading, so that the user has at least an initial orientation as to which POIs he should mark. In another embodiment, the evaluation of the POIs, in particular all POIs, can be carried out in parallel, although only the keywords of those that are marked are displayed. This can advantageously provide better clarity for the user.

Preferably, the audio signals are recorded and/or evaluated for each POI for a predefined period of time or for a dynamic period of time. The dynamic period results from the fact that the POI is evaluated until a predefined probability threshold is certain that the classification algorithm has found the correct keyword or keywords.

The predefined fixed period of time has the advantage that an evaluation does not take too long, especially if there is uncertainty regarding the appropriate keyword, whereas the dynamic period has the advantage that the keywords displayed to the user were correctly recognized with a predefined probability.

In a preferred embodiment, the recognized keywords of the POIs are compared with the user's preferred keywords, with POIs that have preferred keywords being identified. For this comparison, synonyms of the keywords are used in particular, which can be stored in a database. This offers the advantage that the user is specifically made aware of POIs that correspond to their preferences. For example, it is possible to display such POIs with matching preferred keywords as a pop-up window, especially if the user is currently interacting with another POI.

These preferred keywords can be automatically extracted from a user's online profile and/or entered manually by the user.

Extraction from the online profile offers the advantage that the user does not have to enter the preferred keywords separately, but rather that these can be expanded through the user's normal activities with his online profile and/or are kept up to date. The manual entry allows the user to use the version with the preferred keywords without having to have an online profile or this enables the user to enter special preferred keywords.

The classification algorithm preferably uses a trained neural network to recognize topics of conversation and/or uses existing routines of virtual voice assistants.

Both alternatives are suitable for this task and can be constantly trained or improved, especially through feedback.

In one embodiment, a recording duration of the audio signals and optionally the further identifying data is increased in order to provide the classification algorithm with a larger data set.

The larger the data set available to the classification algorithm, the more reliably the real talking point of the POI can be identified. A typical conversation between a group at a POI also includes sentences that actually do not correspond to the topic of the conversation. For example, if the POI group is talking about cars, but one person in the group says "I'll be right back, I'm just going to the toilet" and if the audio signal only included this sentence, the classification algorithm would "mistake" the topic of conversation Detect “toilet use” and display accordingly. However, if the data record of the audio signal is recorded, for example, over a period of more than 15 seconds, in particular more than 1 minute, short-term topics of conversation and other people's statements are less important and can be compensated for.

The recorded audio signals and optionally the further identifying data can be composed of different, in particular unrelated, time periods.

This has the advantage of not having to record audio signals from POIs where activity pauses for a period of time. When activity resumes, audio signals are recorded, assembled and evaluated again.

The keywords preferably have a coding that reflects the probability with which the keyword was correctly recognized. The classification algorithm can determine the probability with which it has identified the correct keyword. In the simplest embodiment, the probability increases proportionally with the duration of the capture of the POI's audio signals. For example, the topic of conversation can be displayed in green if the algorithm has a certain probability, for example over 75%, that it has correctly identified the topic, otherwise the topic of conversation can be displayed in red.

According to a second aspect of the invention, a server is specified, wherein the server has means for carrying out the steps of the method described above. In particular, the server has a processor that can implement the algorithms described above. The virtual environment is preferably implemented on this processor or on another processor assigned to the server. In addition, the server can have interfaces for communication via a communication network with the user terminal, whereby the user records the audio signals with his user terminal, sends them to the server via the communication network and can interact with the virtual environment with the user terminal.

According to a third aspect of the invention, a system is specified comprising the server described above, a user terminal and a communication network, wherein the server and the user terminal are set up to communicate with one another via the communication network. The system is set up to carry out the method described above. In particular, the user terminal has at least one microphone unit for recording the user's audio signals and a playback unit for displaying the POIs.

According to a fourth aspect of the invention, a computer program is specified, comprising commands which, when the program is executed by the previously described server, cause the server to carry out the steps of the method described above.

Further advantageous design features of the present invention are defined in the patent claims.

• 1: zeigt schematisch das erfindungsgemäße System zum Generieren und Anzeigen von Schlagwörtern Themen, die Teilnehmer eines POIs diskutieren;
• 2: zeigt das erfindungsgemäße Verfahren illustrativ in einer virtuellen Umgebung;
• 3: zeigt die erfindungsgemäßen Verfahrensschritte, die von einem Computerprogramm auf einem Server ausgeführt werden.

Preferred exemplary embodiments of the present invention are explained below with reference to the accompanying figures:

• 1 : shows schematically the system according to the invention for generating and displaying keywords topics that participants in a POI discuss;
• 2 : shows the method according to the invention illustratively in a virtual environment;
• 3 : shows the method steps according to the invention, which are carried out by a computer program on a server.

Numerous features of the present invention are explained in detail below using preferred embodiments. The present disclosure is not limited to the specifically mentioned combinations of features. Rather, the features mentioned here can be combined in any way to form embodiments according to the invention, unless this is expressly excluded below.

1 schematically shows the system 10 according to the invention, in particular a communication system 10, for generating and displaying keywords topics that participants in a virtual POI discuss. The system 10 includes a user terminal 20, a server 40 and a communication network 60, whereby the user terminal 20 exchanges data with the server 40 by means of the communication network 60, or network 60 for short. A virtual environment is implemented on the server 40, which has at least one POI with which a user can interact using his user terminal 20. The POI corresponds to a “place of interest” where different users of the virtual environment can meet and discuss various topics. The POI can also correspond, for example, to a virtual trade fair stand where manufacturers produce products and other users can ask questions about the product if necessary.

The user terminal 20 is, for example, a smartphone 20, a VR glasses 20 or a computer 20, the user terminal 20 comprising a microphone unit 22 and a display unit 24, in particular a display 24. Means for interacting with the user terminal 20 can be connected to the user terminal 20 or the user terminal 20 can include the means for interaction. The means for interacting with the user terminal 20 are, for example, a computer mouse or a computer keyboard. The user terminal 20 can also be set up so that the user can interact with the user terminal 20 and in particular with the virtual environment displayed on the user terminal 20 using his finger or his voice. Furthermore, the user terminal 20 has a communication interface for communication via the communication network 60 with the server 40. The server 40 also has a corresponding communication interface. The user terminal 20 can in particular have a radio module as a communication interface.

2 shows the method according to the invention using an exemplary embodiment in a virtual environment 100, as can be displayed, for example, on a display unit of the user terminal 20.

The virtual environment shows three different POIs (POI 1, POI 2, POI 3) that are defined by a delimited area within the virtual environment 100. The user 110 of the virtual environment, who would like to find out more about the topics of one of the POIs, is represented as an avatar 110 by the circle with a cross in the middle.

The procedure can now proceed as follows. The user 110 visits a virtual conference that is represented in the virtual environment 100 and is essentially in a virtual lobby that represents the conference in its entirety. All existing marketplaces and/or lecture rooms are visually depicted as POIs and visible to the user 110 from this lobby.

The position of the user is known in the virtual environment 100 in a simple embodiment, for example as 2D or 3D coordinates, in particular relative to the POIs. In this way, the user 110 can go to specific POIs or several users can dynamically create new POIs. So that participants of a specific POI do not hear the conversations of another POI in a disruptive manner, an algorithm can be implemented in the virtual environment 100, which transmits the volume of the audio signals at a weaker level or even completely prevents it, the further two avatars 110 are from one another in terms of their 2D or 3D coordinates are removed. If a person then navigates their avatar 110 from a first POI to a second POI, the audio signals of the first POI become steadily quieter, whereas the audio signals of the second POI become steadily louder, thereby simulating the conditions of a real world.

With the finger or the mouse, the user 110 can now approach the various POIs and/or click on them to get information about which topics were presented or discussed in the POI at this moment or a few moments ago.

With a touch or click on the corresponding POI, a visual keyword cloud appears and/or the generated keywords are transmitted to the user as words via a loudspeaker unit of the user terminal 20. In 2 the user clicks on POI 2, whereupon the keyword cloud “E-cars; current developments; battery capacity; Range” appears. Preferably, the real voice of the speaker can be used or reproduced during the linguistic reproduction.

The lecture or discussion can be transmitted via microphone, in particular via directional microphone, as audio signals to speech processing on the server 40, with the speech processing applying ASR and NLU methods to the received audio signals and thus constantly providing the keyword cloud with current information about the topics currently being discussed filled, whereby the keyword cloud, as already indicated above, can be read to the user by an audio engine.

The keyword cloud can be generated by trained neural networks as described in the general section. However, there are also other embodiments that can sometimes be combined. In one embodiment, the keyword cloud is formed from the frequency of the spoken words. However, the keyword cloud is preferably formed based on the classifications of the NLU, with intents and entities being used in particular for the keyword cloud for categorization.

The keyword cloud, which includes at least one keyword, can be matched with the user's preferences or interests to recommend a "content recommendation factor" for each individual user in relation to the content of the POI's conversations. These preferences or interests can be derived in various ways both manually and automatically.

In the manual case, the user selects or enters 100 predefined keywords when registering or entering the virtual environment.

In the automated case: In today's business environment, many users have a social media profile on various social media sites. The user has the option of linking at least one of these profiles to the virtual environment when registering or entering the lobby, from which metadata can be automatically extracted from the social media pages to create the preferred keywords.

Based on the user's preferences, a corresponding recommendation can then be made by calculating an overlap between the user preferences and the current content of the keyword cloud formed. If the overlap exceeds a certain limit, the POI is “particularly recommended” - for example by being highlighted in color.

In addition, based on the user's preferences, it is possible to interact directly with other participants in the conversation and recommend that they participate because the interests and preferences show a high level of similarity.

If the microphone is used in combination with a loudspeaker unit to interact with the virtual environment on the user terminal, blind or visually impaired users in particular can use the method to participate in virtual environments, especially in the form of conferences.

Is the keyword cloud generated and is it as in the 2 displayed, the user can mark POI 2 and join it.

A modification of the method described above would be to capture an entire lecture or discussion in its entirety using an audio signal, convert it into text and make it available to a user.

3 shows the method steps 300 according to the invention, which are carried out by a computer program on a server.
Step 310: Display of a virtual POI on a playback unit, in particular a display unit and/or loudspeaker unit, of a user terminal;

Step 320: Acquiring audio signals of the POI and optionally additionally capturing further data identifying the POI;
Step 330: Use of the captured audio signals and optionally additionally the further characterizing data as input of a classification algorithm, the classification algorithm being set up to recognize a keyword for the one POI based on the captured audio signals and optionally additionally the further characterizing data.

Step 340: Generation of control commands set up to reproduce the recognized keyword of the POI on a display unit of the user terminal and transmission of the control commands to the user terminal.

Claims (11)

A method for generating topics of a point of interest “POI” in a virtual environment, comprising the following steps • Representation of a virtual POI within a virtual environment on a playback unit, in particular a display unit and/or a loudspeaker unit, of a user terminal; • Capture audio signals from the POI and optionally additional capture of other data identifying the POI; • Use of the recorded audio signals and optionally additionally the further identifying data as input of a classification algorithm, the classification algorithm being set up to recognize a keyword for the one POI based on the recorded audio signals and optionally additionally the further identifying data; • Playback of the recognized keyword of the POI. Procedure according to Claim 1 , characterized in that different POIs are recognized using a POI distinction algorithm. Method according to one of the preceding claims, characterized in that the user navigates to POIs using navigation means. Method according to one of the preceding claims, characterized in that the user marks the POI by means of marking means, the classification algorithm being carried out for the marked POI and/or the keywords of the marked POI being displayed. Method according to one of the preceding claims, characterized in that the audio signals come from a subset of POIs, in particular special from a POI, recorded and processed in parallel. Method according to one of the preceding claims, characterized in that the audio signals are recorded for each POI for a predefined period of time or for a dynamic period of time. Method according to one of the preceding claims, characterized in that recognized keywords of the POIs are compared with the user's preferred keywords, and that POIs which have preferred keywords are identified. Procedure according to Claim 7 , characterized in that preferred keywords are automatically extracted from an online profile of the user and/or that preferred keywords are entered manually by the user. Server for data processing, comprising means for carrying out the steps of the method Claim 1 . System comprising a server according to Claim 9 and a user terminal, wherein the user terminal is set up to communicate with the server via a communication network, the user terminal comprising at least one microphone unit for recording the audio signals of a user and a playback unit for displaying the POIs. Computer program comprising commands that are used when the program is executed by a server Claim 9 prompt them to follow the steps of the procedure Claim 1 to carry out.

Images