DE102023117678A1 - SYSTEMS AND METHODS FOR THE INTERACTIVE DISPLAY OF A VIRTUAL ENVIRONMENT ON A USER DEVICE WITH LIMITED COMPUTER CAPACITY - Google Patents

Abstract

A user device for rendering images of a virtual environment includes a display and a browser application for connecting to a server for receiving streaming image data for rendering views of a virtual environment as a first layer of display. Control commands initiated by a user on the user device are transmitted to the server. The user device then receives updated streaming image data to update the displayed first layer. Upon detecting a predetermined event related to the virtual world/control commands, the user device sends a request with a corresponding event identifier to a content management system (CMS), whereby the browser application and the CMS maintain corresponding lists of event identifiers. The CMS then transmits additional display information to the user device outside the virtual environment, which the user device displays as a second partially/regionally transparent layer above the first layer.

Description

CROSS REFERENCE TO RELATED PATENT APPLICATIONS

This application claims priority to Irish Patent Application No. S2022/0111 filed on July 6, 2022, which is hereby incorporated by reference in its entirety into the present application.

FIELD OF THE INVENTION

The present invention relates to a system, a computer-readable medium and a method for interactively displaying a three-dimensional world on a user device.

BACKGROUND OF THE INVENTION

It is hereinafter understood that the terms virtual world, virtual environment, virtual experience and virtual reality may be used interchangeably for the purposes of the present invention.

Virtual reality (VR) has experienced significant development over the last decade. Virtual reality allows the user to experience and interact with a computer-generated, virtual environment that may be similar to or completely different from the real world. A key feature of virtual reality is that it creates a feeling of immersion in the user, where immersion can be defined as the user's perception of being physically present in the virtual environment. Therefore, VR provides the user with a virtual environment.

The areas of application of virtual reality are wide-ranging and include, for example, entertainment, games, business and education. Due to the variety of contexts in which virtual environments are used today, there is an increasing need for the compatibility of these virtual environments with external applications.

In particular, there is an increasing need to combine the virtual environment with a means of communicating with other users both inside and outside the virtual environment or to enable users to carry out orders and/or payments.

There is also a need to seamlessly integrate other applications into the virtual environment. It is entirely conceivable that the user would still like to access or use external applications within the virtual world without having to leave the virtual world, as this detracts from the feeling of immersion in the virtual environment. In particular, it would be desirable to provide a way to embed external websites, communication channels, access to contacts, e-commerce platforms, payment networks and the like.

However, many of the above applications that are most interesting to combine with the virtual environment involve the transmission of highly sensitive personal data. Therefore, there is the additional problem of ensuring that third parties cannot easily gain access to such personal information as the user navigates the virtual environment. As the use of such virtual environments becomes more ubiquitous, there is increased demand for a variety of user devices that enable access to such virtual environments. A user device can be understood as any computing device that can be connected to the Internet via a browser application. Examples of such user devices include personal computers, laptops, tablets, cell phones and video game consoles.

In addition, in order to provide the user with an immersive experience, it is necessary that the rendering of the virtual world is done at high frame rates and high resolution so as not to negatively impact the user's experience. Currently, most user devices that are not specifically designed for high-end gaming applications are not capable of displaying such three-dimensional virtual environments in detail. Furthermore, user devices capable of such display are typically very expensive and immobile.

SUMMARY OF THE INVENTION

It is therefore an aim of the present invention to provide a solution that enables the smooth display of three-dimensional and high-resolution (e.g. HD or 4K quality) environments on a generic user device. In particular, it is desirable that the rendering of the virtual world can be seamlessly combined with other applications, enabling a more immersive experience for the user, as well as providing security against the loss of sensitive information to third parties, allowing the user to freely interact with other users or applications can interact within the virtual environment without having to fear the loss of sensitive information to third parties. Furthermore, the aim of the present invention is to provide a framework for representing virtual environments that is compatible with a wide range of possible user devices.

• - ein (tragbares) Benutzergerät, auf dem eine Browser-Anwendung läuft, und
• - mindestens einen Server, der konfiguriert ist, um Folgendes zu leisten:
  1. a. Steuerbefehle von dem Benutzergerät zu empfangen;
  2. b. Ansichten einer virtuellen Welt in Abhängigkeit von den Steuerbefehlen als Bilddaten zu rendern; und
  3. c. einen Strom der Bilddaten an das Benutzergerät zu übertragen,
• - sowie eine Software-Komponente, die so konfiguriert ist, dass sie zusätzliche Informationen bereitstellt.

This task is solved in particular by a system that has the following:

• - a (portable) user device running a browser application, and
• - at least one server configured to:
  1. a. receive control commands from the user device;
  2. b. render views of a virtual world as image data depending on the control commands; and
  3. c. transmit a stream of image data to the user device,
• - as well as a software component configured to provide additional information.

1. a) das Auftreten mindestens eines Ereignisses im Zusammenhang mit der virtuellen Welt und/oder den Steuerbefehlen zu erkennen;
2. b) beim Erkennen des mindestens einen Ereignisses eine Anfrage an die Software-Komponente zu senden;
3. c) zusätzliche Informationen von der Software-Komponente als Antwort auf die Anfrage zu empfangen;
4. d) den Bildstrom vom Server in einer ersten Schicht anzuzeigen und die empfangenen zusätzliche Informationen und/oder grafischen und/oder textuellen Elemente, die von den empfangenen zusätzlichen Informationen abgeleitet sind, in einer zweiten (teil-/bereichsweise transparenten) Schicht anzuzeigen, wobei die zweite Schicht über der ersten Schicht angezeigt wird.

According to this system, the browser application is configured to:

1. a) detect the occurrence of at least one event related to the virtual world and/or the control commands;
2. b) to send a request to the software component when the at least one event is detected;
3. c) receive additional information from the software component in response to the request;
4. d) display the image stream from the server in a first layer and display the received additional information and / or graphical and / or textual elements derived from the received additional information in a second (partially / partially transparent) layer, whereby the second layer is displayed above the first layer.

An important feature of the present invention is that the computational “heavy lifting,” i.e., rendering of the three-dimensional environment, is performed by the server and then streamed to the user device. The image stream, which forms the first layer, and the additional information received from the browser, which forms the second layer, are then only overlaid in the user device, namely in the browser. This not only enables the computational limits of the user device to be easily overcome, but also only requires the user device to be compatible with the browser and can therefore be used equally in a wide range of devices.

Outsourcing rendering to a server alleviates the long-standing problem that high-resolution rendering of a three-dimensional environment at high refresh rates, i.e. 60 frames per second or more, is computationally intensive and places high demands on the performance of the hardware on which the rendering is running. Process is running. The processing capacity of most user devices is not designed for such calculations, severely limiting either the frame rate, the resolution, or both when executing such a rendering process directly.

In addition, the combination of layers in the browser application, where a software component provides additional information to be displayed in the second layer, allows easy embedding of external content and applications into the virtual world. The combination of layers in the browser application also simplifies security issues because all components inherit credentials and authorization from a single instance.

It is important to note that overlaying in this context can correspond to any form of combining and/or merging the image data with the additional information in a manner that results in the additional information being perceived as preceding the image data, regardless depends on the actual order of the levels. HTML5 offers a simple way to implement a layer structure according to the invention.

The software component can be located either on a further, second server (in particular a web server) or on the server that provides the image data. Furthermore, it can be advantageous if the software component is located externally, for example in the cloud or on another server.

In one embodiment, an event may be defined by a list of conditions. The conditions preferably correspond to parameters or properties of the virtual world or the control commands. An event can be considered to have occurred when all defined conditions are fulfilled at the same time.

The additional information provided by the software component may contain textual, graphical or acoustic information. In some embodiments, it may provide links to access additional applications or direct the user to additional content.

According to an embodiment of the present invention, the control commands change a first position or a first orientation of a first avatar within the virtual world or cause the first avatar to perform an action in the virtual world, and the rendered views of the virtual world correspond to the first position first orientation and/or the action of the first avatar.

According to this embodiment, the user can navigate in the virtual world and interact with the objects therein. An advantage of this embodiment is that the browser application only needs to process the input of the control commands and transmit them to the server, while the server performs the rendering of the virtual world according to the received commands. This allows the user to freely interact with the virtual world without the user device having to bear large computing loads. This increases the immersiveness of the virtual environment for the user.

According to a (further) embodiment, the software component stores customization options for the appearance of the virtual world and/or the avatar and is configured to receive a selection of a customization option by the browser and, upon receipt of the selection, to provide information to the server that cause the server to render the avatar and/or virtual world according to the selection.

An advantageous effect of this embodiment is that the user device and/or the browser does not need to store information about the appearance of the virtual world. This further reduces the computing power and memory requirements of the user device.

An adaptation option refers to all options that relate to the visual representation of the three-dimensional world, the avatar or the second layer above, such as color, appearance, different skins, etc.

In a further embodiment of the invention, the server is connected to a multiplayer component, the multiplayer component providing the server with a second position and a second orientation of at least a second avatar controlled by a second user device, and the server providing a Renders representation of the second avatar in the second position with the second orientation within the virtual world. Furthermore, the multiplayer component is configured so that the representation of the virtual world on the first and second user devices is synchronized.

This embodiment allows multiple users to interact with the same virtual world and with each other within that world. In particular, the multiplayer component can be set up to connect to a second server that is controlled by the second user device. This has the advantageous effect that multiple systems according to the present invention can be easily connected to one another without burdening the user device with additional workloads.

However, the multiplayer component is not limited to connecting different servers. A connection can also be established between the server controlled by the first user device and conventional personal hardware suitable for displaying three-dimensional environments, such as VR glasses, game consoles, personal computers, etc.

In a (further) embodiment of the present invention, the server itself is adapted to monitor the movement of the first avatar to detect conditions for an event and to transmit at least one identifier to the user device when these conditions are met. The software component includes a database, the database storing the additional information associated with each identifier.

According to one aspect of the invention, the receipt of such an identifier by the browser constitutes an event. This can be the case, for example, when the avatar enters a certain area in the virtual environment. This indirect capture of events by the browser has the advantage of redirecting further workload away from the user device. In addition, the provision of the additional information by the software component in this case is reduced to a simple lookup process in which the received identifier is compared with the stored identifiers and the information belonging to the received identifier in the database is transmitted to the browser device.

According to a further aspect of the invention, the conditions for transmitting the at least one identifier include a proximity of the avatar's position to a target position in the virtual world and/or a proximity of the avatar's orientation to a third orientation in the virtual world.

This makes it possible to easily formulate conditions for events in the virtual world based on the avatar's location. In particular, this allows users who are not trained in the design of three-dimensional environments to specify conditions for events in the three-dimensional world, e.g. B. via an application programming interface (API).

It should be noted that the term position is not limited to the exact point with the shape (x,y,z) in the three-dimensional world, but can also refer to a specific area of the world, such as: B. a room or a specific floor or floor or any type of area or volume that e.g. B. are defined by several coordinates. In particular, the third position does not necessarily have to be fixed, but can, for example, correspond to the second position of a second avatar or a moving vehicle or another movable or deformable object.

The conditions for transmitting the identifier may also be based on a timer and/or on the fulfillment of all of a first list of predetermined conditions based on the first location and/or the first orientation and/or the input of control commands and/or one of the first or second avatar.

In this way, different conditions can be combined when defining an event. For example, proximity to a specific third position in the three-dimensional world can only lead to the detection of an event in combination with a specific timer value or in combination with a specific orientation of the first avatar.

A further embodiment of the system according to the present invention can be achieved if the additional information received from the software component enables external applications to be embedded in the virtual world.

The term embedding in this context refers to the display of textual, visual or acoustic information that is received from another application running on the user device and integrated into the second layer to be displayed before the image stream.

This allows data provided by external applications to be seamlessly incorporated into the second layer to be displayed before the image stream. In particular, the user can access additional applications without having to leave the virtual world. In addition, the information provided by the additional software when an event is detected can be used to trigger the integration of such external applications.

In particular, the additional information provided by the software component and displayed in the second layer may contain links that enable the browser application to open further applications. In this case, embedding external functions into the virtual environment inherits the security functions already provided by the browser application.

In a particular embodiment, the additional information received by the software component establishes a communication channel between the user device and the second user device, in particular a chat function and/or an audio channel and/or a video connection.

This enables direct communication between the different users participating in the three-dimensional world, especially in multiplayer settings. This makes the multiplayer experience more attractive for the user as the user can easily communicate or cooperate with other users.

In a further embodiment, the browser application is provided with authorization to initiate and control the execution of the external applications on the user device.

Since the additional information is displayed within the second layer of the browser application, it is necessary that the browser application can act with the necessary authentication and authorization to initiate and allow the execution of additional applications on the user device in accordance with the data control received either from the server rendering the virtual world or the software component.

This also helps to make the experience even more immersive for the user, as there is no need to manually allow the use of certain applications. This is particularly advantageous for e-commerce applications. In a particular embodiment, the browser application can rely on established online payment systems as plug-ins to enable easy payment in the virtual environment. Since the browser application does not require the user's separate permission to trigger corresponding transactions, the corresponding security features are also passed on to the virtual environment. This significantly simplifies and accelerates the use of external applications in the virtual world.

According to a further aspect of the invention, the software component consists of a content management system (CMS), whereby the customization options can be set via an application programming interface (API).

This further increases the user-friendliness when customizing the CMS, as even non-expert users can use the customization options more easily. In a particular example, the CMS is a web application that consists of a web server and a database backend.

In this case, the CMS creates and maintains the virtual environment by storing a generated configuration in the database backend for each user device. According to this embodiment, the CMS application controls the virtual environment.

• ein (tragbares) Benutzergerät, auf dem eine Browser-Anwendung läuft, eine erste Verbindung zu mindestens einem Server herstellt, der Server Steuerbefehle von dem Benutzergerät empfängt und Ansichten einer virtuellen Welt in Abhängigkeit von den genannten Steuerbefehlen als Bilddaten rendert, wobei ein Strom der Bild- und/oder Audiodaten an das Benutzergerät übertragen wird;
• der Browser das Auftreten mindestens eines Ereignisses in Bezug auf die virtuelle Welt und/oder die Steuerbefehle erkennt und beim Erkennen des mindestens einen Ereignisses eine Anfrage an eine Software-Komponente sendet, auf die Anfrage hin der Browser zusätzliche Informationen von der Software-Komponente erhält;
• der Browser den Bildstrom vom Server in einer ersten Schicht und die empfangenen Zusatzinformationen und/oder von den empfangenen Zusatzinformationen abgeleitete grafische und/oder textuelle Elemente in einer zweiten (teil-/bereichsweise transparenten) Schicht anzeigt, wobei die zweite Schicht die erste Schicht überlagernd angezeigt wird.

Furthermore, the aforementioned problems are solved by a method in which:

• a (portable) user device running a browser application establishes a first connection to at least one server, the server receives control commands from the user device and renders views of a virtual world as image data depending on said control commands, a stream of the image - and/or audio data is transmitted to the user device;
• the browser detects the occurrence of at least one event relating to the virtual world and/or the control commands and, upon detecting the at least one event, sends a request to a software component, upon which request the browser receives additional information from the software component;
• the browser displays the image stream from the server in a first layer and the additional information received and/or graphical and/or textual elements derived from the additional information received in a second (partially/partially transparent) layer, the second layer being displayed overlying the first layer becomes.

According to a further aspect, the layer structure according to the invention can be changed in a third-party application, preferably a third-party website, which is preferably hosted on a third-party server. This can be achieved by integrating the first and second layers into a higher-level framework, which can be part of the third-party application or website. In this case, the web page rendered by the browser application may be split into two or more frames one of which displays the virtual world with the first and second layers, respectively, which can be called a VR frame. At least one of the other frames may display content originating from the third-party server and/or another server and therefore may be referred to as a third-party frame. The content reproduced in the at least one other frame can be that of a web shop or, more generally, a retailer platform.

The above-mentioned frameworks can be implemented through any available technology, e.g. B. iFrames and/or templates and/or sections of HTML5.

In one aspect of the invention, there is communication established between the at least one of the layers (first and second layers of the virtual word) and the third-party application, e.g., the third-party website. In particular, events associated with the first and/or second layer may be forwarded to the third-party application and/or the third-party server. The communication is preferably carried out on the client side, e.g. B. built between the frame and optionally to a server, e.g. B. the third-party server. This enables an immediate reaction to the event, e.g. B. generated by the third-party website.

Once the browser application detects an event, a request is sent to the software component, which returns additional information, which is then displayed in the second layer. The additional information can in particular contain a link or a button by which the user can interact with the control commands.

Once the browser application registers the ID associated with an event in the first and/or second layer, the parent framework may further be caused to pass information associated with the event ID to the at least one other third-party framework, connected to the third-party application and/or server. In particular, the event ID can be associated, preferably using a transfer table, with an action on the third-party site, such as placing an item in a checkout cart. This information can then be passed from the parent framework to a component of the third-party application. Alternatively, the information may only cause a local change in the third-party framework of the browser/client, which can be transmitted to the third-party application or server at a later time.

The term "overarching framework" is chosen to emphasize the conceptual structure of the present aspect of the invention and need not refer to an actual framework, but more generally to the component of a third-party application or website into which the VR framework and a or multiple third-party frames are embedded.

In an alternative aspect, the browser may pass the additional information to the third-party application and/or server immediately after detecting the event, without first passing it to the third-party framework.

In all of the above cases, the propagation of the information associated with the event ID may or may not be dependent on the user entering appropriate control commands once the presentation graphic is displayed in the second layer. For example, the user may need to click a button in the second layer before the associated information is triggered to be passed through the higher-level framework. Alternatively, for certain event IDs, it may be preferable to cause the associated information to be passed through the parent framework immediately, without requiring further input from the user.

This aspect is particularly suitable for e-commerce applications, for example. Here the event can be triggered by an action of the avatar in the virtual world, which results in additional information being displayed on the second layer. Such additional information can be, for example, product information, but also a purchase button that triggers the purchase of the displayed item. When the user enters an appropriate control command (e.g. by clicking the “Buy” button), the associated information may be transmitted in the manner described above.

For example, items for sale on a third-party website can be added to a shopping cart through actions in the virtual world. By separating the first and second layers or the third-party website into different frames, parts of the third-party website remain directly accessible and visible to the user when navigating in the virtual world.

With the method according to the invention, the same advantages can be achieved as have already been described in connection with the system according to the invention. It should be noted that the features described in connection with the system according to the invention also apply to the method according to the invention. Likewise, features of the method according to the invention can be transferred to the system according to the invention by configuring the browser application, the server or the additional software so that they are suitable for carrying out the corresponding method features.

BRIEF DESCRIPTION OF DRAWINGS

The invention is also described below with regard to further details, features and advantages, which are explained in more detail with reference to the figures. The described features and combinations of features, as shown below in the figures described with reference to the figures, can be used not only in the combination specified in each case, but also in other combinations or on their own, without thereby departing from the scope of the invention becomes.

• 1 zeigt ein System bestehend aus einer Browser-Anwendung, die auf einem Benutzergerät (nicht dargestellt) läuft, einer Software-Komponente und einem Server;
• 2 zeigt das erfindungsgemäße System in einer Multiplayer-Einstellung, verbunden mit einem weiteren Benutzergerät;
• 3a zeigt die Erkennung eines Ereignisses durch Übermittlung einer ID vom Server an die Browser-Anwendung;
• 3b zeigt die Erkennung eines Ereignisses durch die Browser-Anwendung;
• 4 zeigt eine Ausführungsform, bei der das CMS einen Webserver und eine Backend-Datenbank beinhaltet;
• 5a zeigt einen beispielhaften Datenbankeintrag für die Definition eines Ereignisses;
• 5b zeigt einen beispielhaften Datenbankeintrag, der die ID von Ereignissen mit den Zusatzinformationen verknüpft;
• 6 zeigt ein Beispiel für ein Benutzergerät nach dem System der Erfindung;
• 7 zeigt ein Beispiel für die Einbettung der virtuellen Realität in ein Client-Fenster, das die Kommunikation mit einem Drittparteien-Server herstellt;
• 8A bis 8E zeigen repräsentative Konfigurator- und gerenderte Anzeigebildschirme für die Zuordnung und Speicherung von Beispielbildern in der Software-Komponente gemäß dem erfindungsgemäßen System; und
• 9A bis 9D zeigen repräsentative Konfigurator- und gerenderte Anzeigebildschirme für die Zuordnung und Speicherung von beispielhaften Videos gemäß der Erfindung in der Software-Komponente.

Embodiments of the invention are described below with reference to the figures.

• 1 shows a system consisting of a browser application running on a user device (not shown), a software component and a server;
• 2 shows the system according to the invention in a multiplayer setting, connected to another user device;
• 3a shows the detection of an event by transmitting an ID from the server to the browser application;
• 3b shows the recognition of an event by the browser application;
• 4 shows an embodiment in which the CMS includes a web server and a backend database;
• 5a shows an example database entry for defining an event;
• 5b shows an example database entry that links the ID of events with the additional information;
• 6 shows an example of a user device according to the system of the invention;
• 7 shows an example of embedding virtual reality into a client window that establishes communication with a third-party server;
• 8A until 8E show representative configurator and rendered display screens for the assignment and storage of example images in the software component according to the system according to the invention; and
• 9A until 9D show representative configurator and rendered display screens for assigning and storing exemplary videos according to the invention in the software component.

DETAILED DESCRIPTION

1 shows the basic structure of a system according to the present invention. When entering the virtual world by accessing an environmental URL, a representative display provided by the browser application 2 on a display of a user device with limited computing capacity (not shown) communicates to software via a first connection 8 -Component 4, which runs on a separate device to carry out initial authentication checks and is placed in a queue by the software component 4 for access to a server 3.

Representative authentication checks can include, for example, a login process in which the browser application 2 sends a request for a digital token, for example an HTTP POST request, to the software component 4 via the first connection 8. The corresponding digital token received by the browser application 2 enables access to this virtual world, for example through a queuing service for allocating a server, such as server 3, from a group of servers that enable access to this virtual world. If access to the virtual World is password protected, the submitted request may also contain a valid password and/or Single Sign-On (SSO) identifiers if the browser application 2 or user device includes appropriate SSO functionality. Further, the user device can log in with the user's email and password if an administrator has previously created an account for that email.

The digital token may be a cryptographically signed JSON Web (“JWT”) token in accordance with Internet Engineering Task Force (IETF) RFC 5741, which may be stored as a cookie and/or RAM of the user device, or any other suitable type of digital token . Once the user device has the digital token, subsequent queries from the browser application 2 may result in an authentication check by the software component 4 to determine whether the user device has a valid token, i.e. one that has not expired, in order to access access the virtual world.

It should be understood that typical cloud streaming servers may not be immediately available for deployment by the user device, especially when demand for such servers is high, so there will be a delay in allocating and granting access to a cloud Streaming servers, like Server 3, can exist. If the cloud streaming capacity running the desired virtual world (binary executable) is already provisioned and available, the user device can be immediately redirected to the server 3. If this is not the case, the user device may receive a “ticket” along with the ticket's position in the queue and periodically poll the queue with that ticket. Advantageously, the queue management software often prefers servers that are located close to the user device in order to reduce any latency of the user device.

The software component 4 sets up a second connection 9 with the server 3. Alternatively, the server 3 connects to the software component 4 to signal its presence and status. In particular, the software component 4 can be configured so that it selects one from a large number of accessible servers according to predetermined criteria while the browser application 2 is in the queue. These criteria may depend, for example, on the user's geolocation or IP address, connection, number of available servers or authentication credentials. The server 3 loads a list of event conditions via the second connection 9 (see 5a) from the software component 4 and stores it in an in-memory address map, ie a hash map or database on the server. The browser application 2 also receives a copy of a partial/range or complete list of event conditions.

The server is reserved for the user device for a preset time, e.g. 10 seconds, and the server address (a URL) is sent to that user device for use by the browser application 2. The browser application 2 then establishes a connection to this server 3 and authenticates itself, for example via a WebSocket protocol connection. A negotiation process begins immediately via this WebSocket and standard WebRTC ICE (Interactive Connectivity Establishment) candidates are exchanged. ICE candidates are used to establish a third connection 10, for example a WebRTC connection, between the server 3 and the browser application 2. According to a further aspect of the invention, TURN (Traversal Using Relay around NAT protocol) standard relay servers can also be used. After a handshake between the browser application 2 and the server 3, the flow of image data and audio begins.

The image (video) and/or audio data corresponding to the three-dimensional environment are transmitted from the server 3 to the browser application 2 via the third connection 10, where they are displayed in the first layer 5.

The control commands are sent from the browser application 2 to the server 3 via the same third connection 10. Depending on the type of user device used to access the three-dimensional environment, the control commands can be: click events, mouse movements, keyboard inputs, joystick movements, VR haptics, touchscreen reactions, accelerometer inputs, gyroscope measurements, webcam Data and spoken commands, i.e. microphone inputs.

In addition, the third connection 10 includes a bidirectional data channel for exchanging further information between the server 3 and the browser application 2.

If the browser application breaks the connection to the server 3, the server 3 updates its status via the second connection 9 to the software component 4 to declare itself available again.

When an event is detected by the browser application 2, the browser application (depending on the availability of the required data in the browser application) sends a request to the software component 4 via the first connection 8 and receives additional information via the same connection to the event from the software component 4. The browser application 2 then displays the received additional information and/or graphical and/or textual elements derived from the received additional information 6 in the second (partially/regionally transparent) layer 7, where the second layer 7 is displayed lying above the first layer 5. As used herein, "partially/regionally transparent layer" means, without limitation, semi-transparent or transparent areas over at least a portion of such layer, with at least other areas of the layer having images similar to the images in the superimposed, ie, lower layer, such as the first Layer 5, hide or merge with them. Alternatively, the browser application may have previously downloaded any additional information required to execute the event from the software component 4. Those described below 8A until 8E show representative configurator and rendered display screens for assigning and storing example images in the software component 4 for displaying pop-up images when the occurrence of an associated triggering event is detected. They also show 9A until 9D representative configurator and display screens for associating and storing sample videos in the software component for displaying videos when the occurrence of an associated triggering event is detected.

When the server 3 receives control commands from the browser application 2 via the third connection 10, the position and orientation of the first avatar 11 in the virtual world are updated and the image data transmitted to the browser application 2 is updated accordingly. In addition, the server 3 compares the new position and orientation of the avatar with the list of positions and orientations corresponding to the possible event conditions stored in the second database on the server 3.

Other possible event conditions include the assignment of an in-game actor (element) ID to a video, the scheduled absolute time (i.e., real-world time), the elapsed time since the start of the stream, and others.

2 shows the system in a multiplayer environment. The server 3 receives the position and orientation of a second avatar 14 from a multiplayer component 13. The multiplayer component 13 is connected to at least one second user device 12 that controls the movement of a second avatar 14. It is understood that this connection to a second user device can also be an indirect connection via another server within the meaning of the present invention. In this case, the second user device is to be understood as meaning the entire system according to the present invention. After receiving the second position and the second orientation, the server 3 renders a second avatar 14 as part of the image data displayed in the first layer 5.

3a shows the indirect detection of event conditions in the system by the server 3 of 1 and 2 . As the first avatar moves in the virtual environment, the server monitors the properties of the input commands and the virtual world and checks whether all conditions for a predefined event are met. If this is the case, the server sends an ID corresponding to the detected event to the browser application. The browser application then detects the presence of the ID in the third connection's data stream as an event. The browser application then sends a request to the software component that contains the received ID, or has already fetched data with such an ID. The software component then compares the received ID with a list of IDs stored in the database. As soon as the appropriate ID has been found in the database, the software component transmits the additional information associated with the received ID to the browser application, unless the additional information has already been fetched in advance. Finally, the browser processes the additional information received and displays it in the second layer, overlaying the image data stream.

3b shows the direct recognition of an event state by the browser application 2 in a further embodiment of the invention. In this case, the browser application 2 monitors the control commands and the virtual environment directly. If all conditions that define an event are met, sen The browser application sends the ID request corresponding to the respective event to the software component 4. The software component 4 then compares the received ID with a list of IDs that are stored in the database. As soon as the matching ID has been found in the database, the software component sends the additional information associated with the received ID to the browser application. Finally, the browser processes the additional information received and displays it in the second layer, overlaying the image data stream.

4 shows an embodiment in which the software component 4 consists of a CMS. The CMS is divided into a web server 15 and a backend database 16. The CMS receives customization options via the first connection 8 and saves the generated configuration in the backend database. In particular, it monitors the status of the virtual environment via the second connection 9.

5a shows the data structure of an event, consisting of the event ID and a variety of conditions. The conditions are based in part on the possible locations, orientations and actions in the virtual world as well as on the command inputs. A first condition can e.g. B. be that the first location of the first avatar 11 is within a certain region around a third location in the virtual world. A region can be defined as the set of all points within a predefined distance from the third location or as a predefined set of points that contain the third location (such as a room in which the third location is located). A second condition can e.g. B. be the value of a timer that corresponds to a predefined value. In this example, an event is detected if the location of the first avatar 11 is within the range of locations defined by the first condition AND the value of the timer is equal to the predefined value.

5b shows the data structure according to which the additional information is stored within the software component. According to this embodiment, the database includes an ID list containing the IDs of all configured events along with the corresponding additional information.

In 6 1, a user device 17 is shown according to an embodiment of the present invention. The browser application 2 is executed by the user device CPU 19, with the first layer 5 (not shown) and the second layer 6 (not shown) being displayed on the user device display 20. The control commands are entered via a keyboard 18 of the user device.

In 7 a client, for example a browser application 2, is shown which displays a third party website. The website is divided into a third-party frame 72 and a VR frame 74, for example using iFrames. For convenience, the window/frame in which the third-party frame 72 and the VR frame 74 reside is referred to as the parent frame 75.

In the embodiment, the third-party frame 72 is arranged next to the VR frame 74, which contains at least the first layer 5 and the second layer 7 of the virtual world, as already described, for example with reference to 2 . In the embodiment of 7 The display of the display graphic of parts of the additional information 6, which were triggered by the occurrence of an event, can include the display of a button in the second layer 7, which enables, for example, the addition of a product to a shopping list. The user can use the display graphic 6, e.g. B. the button, interacting, causing the browser 2 to communicate with the third-party frame 72 via the higher-level frame 75. In one embodiment, such communication could cause the third-party website to add a particular item to a shopping cart, the particular item being related to what is shown in the VR frame 74, e.g. in the first layer 5 and/or the second layer 7.

Alternatively, the browser 2 and/or the parent frame 75 may transmit the received information from the VR frame 74 directly to an external third-party server 71 via a fourth connection 73.

In one embodiment, the communication between the VR frame 74 and the parent frame 75 uses a proprietary messaging protocol. This means that when the higher-level frame 75 interacts - clicking on the button - with the display graphic 6, it receives a message that identifies an event or preferably an item that is to be placed in the shopping cart. The type of display graphic 6, the time of appearance, the behavior and any ID that identifies an event or an article can be configured in the software component 4. The identifier may be sent to the parent frame 75 as a payment debit of the custom message so that the third-party website can respond to receipt of the identifier by any means available, e.g. B. by triggering further actions within the third-party website.

    let postMessage = JSON.stringify({
    "_ABC_action":"addtobag",
    "skuBaseld": "11111",
    "INCREMENT":1,
    "QTY":1
    });
    if ('parentIFrame' in window) {
    window. parent1Frame.sendMessage(postMessage);
    }});

The custom message can be a JSON string that can be generated and passed as follows:

 let postMessage = JSON.stringify({
    "_ABC_action": "addtobag",
    "skuBaseld": "11111",
    "INCREMENT":1,
    "QTY":1
    });
    if ('parentIFrame' in window) {
    window.window. parent1Frame.sendMessage(postMessage);
    }});

This approach allows the user to easily perform actions in virtual reality that affect the behavior of the third-party website. By distributing the video stream in the first layer 5, the display graphic 6 in the second layer 7 and through the possibility of configuring the display graphic 6, in particular the ID to be transferred, the invention enables a very efficient configuration of the system. For example, it is possible to add the option to purchase specific items within virtual reality without having to change virtual reality. In one embodiment, the administrator of the system only needs to set up a new presentation graphic 6, the corresponding ID and an event that triggers the presentation graphic. All of these steps can be carried out with software component 4.

Of course, this part of the invention is not limited to purchasing items and/or services. The approach can also be used for training and other educational services.

Suitable CMS for the software component 4 of 4 In certain embodiments, for example, the computer graphics game development software “Unreal Engine” offered by Epic Games, Inc. can be used. In the 8A until 8E Representative configuration and display screens are shown for the assignment and storage of an exemplary pop-up image in such a CMS, which should appear in level 7 when a corresponding event occurs. 8A shows an example pop-up configuration screen where a developer can enter a name and title, respectively, in the Pop-up Slug and Title fields for the image to be attached via the Image field. In this example, the name of the pop-up slug is “ArcticPopUp” for the added image of the arctic landscape. An associated description of the image can also be entered in the “Description” field.

8B shows an example event condition configuration screen for use by the developer to create appropriate event conditions based on associated “triggers” and “actions”. 8C and 8D show sub-configuration screens for triggers and actions available via the in 8B event conditions configuration screen shown. As in 8C shown, "Region" was selected as the "Trigger Type", and in 8D The selected “Action Type” for this “Open Pop-up” trigger is shown with the associated image name “ArcticPopUp” for the image to be displayed. 8E shows the resulting ArcticPopUp pop-up image and description displayed when a user's avatar enters the region for the region-triggered event condition.

Accordingly, in the 9A until 9D representative configurator and display screens for the assignment and storage of a URL of a video are shown in an exemplary CMS 4, which is to be displayed in level 7 when a corresponding event occurs. In 9A An example configurator screen is shown for entering the video name “Unreal Slug” and a corresponding “title” of the video, both of which are “SampleVideo” in this example. In addition, the configurator screen contains 9A an add video button labeled “+Add”. After selecting this button, the in appears 9B sub-configuration screen shown where the developer can enter the “title” of the respective video and the associated “video URL” from which the video should be streamed. In addition, the developer can use this sub-configuration screen also enter certain attributes of the video to be displayed, e.g. B. whether the video runs in a loop, i.e. is repeated, or whether it should be streamed live.

Another configurator screen (not shown) similar to the configurator screen of 8B is used to enter the triggers associated with the playback of videos associated with the configurator screens of the 9A and 9B were specified. See e.g. b. 9C and 9D , in which playback of a video is triggered when a user's avatar enters a defined area near a rendered screen in the virtual environment that displays the static image "OPERATION #OCEAN DETOX" in a first display layer 9C displays. As soon as the avatar enters the defined region, a video begins to play in the second layer 7, which contains this static image, as in 9D shown, with the title screen “THE OCEANS ARE FULL OF TRASH” overlaid.

The ones in the 8A-8E and 9A-9D The triggers and actions described are for illustrative purposes only, and other alternative triggering events and actions are conceivable within the scope of the invention, including, but not limited to, actions performed by the avatar, regional triggers, temporal triggers, specific positions and/or viewpoints of the Avatars and others as described above. Although these figures illustrate triggers and actions based on the appearance of a user's avatar in the virtual environment, it should be readily understood that actions to display content in Layer 7 may alternatively be triggered by user device control commands.

It is to be understood that although various aspects of the present disclosure have been shown and described by way of example, the invention described herein is not limited thereto but may be embodied in other ways as suggested by the disclosure and as set forth in the following exemplary claims. For example, despite the disclosure relating to a system and method for representing a virtual reality, such a system or method may additionally be implemented in an augmented reality or holographic system or the like in accordance with the invention described above.

Reference symbol list

1

system

2

Browser application

3

server

4

Software component

5

first layer

6

Display graphic of additional information

7

second layer

8th

first connection

9

second connection

10

third connection

11

first avatar

12

second user device

13

Multiplayer component

14

second avatar

15

Web server

16

Backend database

17

User device

18

Keyboard

19

CPU of the user device

20

User device display

71

Third Party Servers

72

Third Party Framework

73

fourth connection

74

VR framework

75

parent frame/window

Claims (22)

Method for displaying images of a virtual environment on a first user device, comprising the following steps: executing a browser application on the first user device to establish a first connection, preferably over a network, between the first user device and at least a first server and to receive streaming image data from the first server; rendering views of a virtual environment as a first layer on the first user device based on the received streaming image data; transmitting control commands from the first user device to the first server and receiving correspondingly updated streaming image data of the virtual environment based on the control commands; upon detecting the occurrence of at least one predetermined event relating to the virtual environment and/or the control commands, transmitting a request with an event identifier to a second server that executes a software component, wherein the browser application and the software component maintain corresponding lists of predetermined event identifiers; receiving additional display information from the software component in response to the request, said additional display information representing content generated outside of the rendered virtual environment; and Rendering images derived from the received additional display information as a second partially transparent layer on the first user device, the second layer being displayed over the first layer to enable the display of the external content within the rendered virtual environment make possible. Procedure according to Claim 1 , wherein the control commands change at least one of the following - a first position - or a first orientation of a first avatar within the virtual environment; or wherein the control commands cause the first avatar to perform an action in the virtual environment, and wherein the rendered views of the virtual environment correspond to at least one of: - the first position, - the first orientation, or - the action of the first avatar. Method according to one of the preceding claims, wherein the software component stores customization options for the appearance of the virtual environment and/or the first avatar and is configured to receive a selection of a customization option by the browser and, upon receipt of the selection, to send information to the first Server delivers, causing the first server to display the avatar and/or virtual environment according to the selection. Method according to one of the preceding claims, wherein the at least one server is connected to a multiplayer component, the multiplayer component providing the at least one server with a second position and a second orientation of at least a second avatar controlled by a second user device and the first server renders a representation of the second avatar at the second position with the second orientation within the virtual environment. A method according to any one of the preceding claims, wherein the first server is configured to monitor the movement of the first avatar to detect conditions for an event and to transmit at least one event identifier to the first user device when those conditions are met, and wherein the software component includes a database, the database storing the additional information associated with each event identifier. Method according to one of the preceding claims, in particular according to Claim 5 , wherein the conditions for the transmission of the at least one identifier are based on the proximity of the position of the first avatar to a third position in the virtual environment and / or the proximity of the orientation of the first avatar to a third orientation in the virtual environment. Method according to one of the preceding claims, in particular according to Claim 5 , wherein the conditions for transmitting the event identifier are based on a timer and/or the satisfaction of all conditions of a first list of predetermined conditions based on at least one of the following: the first location, the first orientation, the entry of control commands, or one of the first or second avatar. Method according to one of the preceding claims, wherein the additional display information received from the software component enables embedding of external applications into the virtual environment. Method according to one of the preceding claims, in particular according to Claim 8 , wherein the additional information received from the software component establishes a communication channel between the first user device and the second user device to enable at least one of the following, a chat function, an audio channel or a video connection. Method according to one of the preceding claims, in particular according to Claim 9 , wherein the browser application is provided with permission to initiate and control the execution of the external applications on the first user device. Method according to one of the preceding claims, in particular according to Claim 3 , where the software component includes a content management system and the customization options can also be set via an application programming interface. A system for rendering images of a virtual environment on a first user device, comprising: a first user device configured to: run a browser application; to receive control commands entered by a user, preferably over a network, and to transmit such control commands to at least one first server, the at least one first server being configured to (i) generate image data indicative of rendered views of a virtual environment , which are based in part on the received control commands, and (ii) transmits a stream of image data to the first user device, Receiving the stream of image data transmitted to the first user device, and where the browser application is configured to: as a first layer to render views of the virtual environment based on the received streaming image data, detect the occurrence of at least one event related to the virtual environment and/or the control commands, upon detecting the at least one event, transmitting a request to the second server executing the software component to generate the additional image information, and Rendering images derived from the received additional display information as a second partially/regionally transparent layer, displaying the second layer overlying at least a portion of the first layer to facilitate the display of the external content in the rendered virtual environment to enable. System after Claim 12 , wherein the control commands change at least one of the following: a first position or a first orientation of a first avatar within the virtual environment; or cause the first avatar to perform an action in the virtual environment, and wherein the rendered views of the virtual environment correspond to at least one of the first position, the first orientation, and/or the action of the first avatar. System according to one of the Claims 12 or 13 , wherein the software component stores customization options for the appearance of the virtual environment and/or the first avatar and is configured to receive a selection of a customization option by the browser and upon receipt of the off selection provides information to the first server, causing the first server to render the first avatar and/or virtual environment according to the selection. System according to one of the Claims 12 until 14 , wherein the at least one server is connected to a multiplayer component, the multiplayer component providing the at least one server with a second position and a second orientation of at least a second avatar controlled by a second user device, and the server a Renders a representation of the second avatar at the second position with the second orientation within the virtual environment. System according to one of the Claims 12 until 15 , especially after Claim 13 , wherein the server is configured to monitor the movement of the first avatar to detect conditions for an event and to transmit at least one event identifier to the first user device when those conditions are met, and wherein the software component includes a database , where the database stores the additional information associated with each event identifier. System according to one of the Claims 12 until 16 , especially after Claim 16 , wherein the conditions for the transmission of the at least one identifier are based on a proximity of the position of the first avatar to at least a third position in the virtual environment or on a proximity of the orientation of the first avatar to a third orientation in the virtual environment. System according to one of the Claims 12 until 17 , especially after Claim 15 , wherein the conditions for transmitting the event identifier are based on at least a timer or the satisfaction of all conditions of a first list of predetermined conditions based on at least one of the following: the first location, the first orientation, the entry of control commands or one of the first or action carried out by the second avatar. System according to one of the Claims 12 until 18 , whereby the additional display information received from the software component allows external applications to be embedded in the virtual environment. System according to one of the Claims 12 until 19 , especially after Claim 19 , wherein the additional display information received from the software component establishes a communication channel between the first user device and the second user device to enable a chat function, an audio channel or a video connection. System according to one of the Claims 12 until 20 , especially after Claim 20 , wherein the browser application is provided with permission to initiate and control the execution of the external applications on the first user device. System according to one of the Claims 12 until 21 , especially after Claim 14 , where the software component includes a content management system and the customization options can also be set via an application programming interface.

Images