FR3137770A1 - SYSTEMS AND METHODS FOR INTERACTIVE RENDERING OF A VIRTUAL ENVIRONMENT ON A USER DEVICE HAVING LIMITED COMPUTATIONAL CAPACITY - Google Patents

Abstract

The present invention relates to a user device for rendering images of a virtual environment including a display. A browser application is intended to establish a connection to a server to receive streaming image data to render views of a virtual environment as the first layer of a display. Control instructions initiated by a user on the user device are transmitted to the server. In response, the user device receives updated streaming image data to update the first displayed layer. Upon detecting an occurrence of a predetermined event related to the virtual world/control instructions, the user device transmits a request with a corresponding event identifier to a content management system (CMS), the application of browser and CMS maintaining corresponding lists of event IDs. The CMS then transmits additional display information external to the virtual environment to the user device, which the user device renders as a partially transparent second layer overlying the first layer.

Description

SYSTEMS AND METHODS FOR INTERACTIVE RENDERING OF A VIRTUAL ENVIRONMENT ON A USER DEVICE HAVING LIMITED COMPUTATIONAL CAPACITY REFERENCE TO RELATED PATENT APPLICATIONS

This application claims priority over Irish Patent Application No. S2022/0111, filed July 6, 2022, which is incorporated herein by reference in its entirety.

FIELD OF INVENTION

A system, computer-readable medium, and method for interactively rendering a three-dimensional world on a user device is provided.

TECHNOLOGICAL BACKGROUND OF THE INVENTION

In the following, it should be understood that the terms virtual world, virtual environment, virtual experience and virtual reality can be used interchangeably for the purposes of the present invention.

Virtual reality (VR) has seen significant development over the past decade. Virtual reality allows a user to experience and interact with a virtual, computer-generated environment that may be similar to or completely different from the real world. A central characteristic of virtual reality is that a feeling of immersion is elicited in the user, where immersion can be defined as the user's perception of being physically present in the virtual environment. Therefore, VR provides the user with a virtual environment.

The application areas of virtual reality are widespread and include, for example, entertainment, gaming, business and education. Due to the wide variety of contexts in which virtual environments are used today, there is an ever-increasing demand for compatibility of these virtual environments with external applications.

In particular, there is an increased need to combine the virtual environment with possibilities for communication with other users, both inside and outside the virtual environment or to allow users to perform orders and/or payments.

Additionally, there is a need to incorporate other applications into the virtual environment in an easy manner. The user may really want to access or use external applications while remaining in the virtual world, instead of having to leave the virtual world to do so, as this detracts from the feeling of immersion in the virtual environment. In particular, it would be desirable to offer a possibility of integration of external websites, communication channels, contact access, e-commerce platforms, payment networks and the like.

However, many applications like those cited above that are most interesting for combining with the virtual environment involve the transfer of highly sensitive personal data. Thus, there is the added problem of ensuring that third parties cannot easily access this personal information, as the user navigates through the virtual environment. As the use of such virtual environments becomes ubiquitous, there is an increased demand for a wide variety of user devices that are capable of enabling access to such virtual environments. By user device, we mean any computing means that can be connected to the Internet via a Browser application. Examples of such user devices include personal computers, laptops, tablets, cellular phones and video game consoles.

Furthermore, in order to provide an immersive experience to the user, it is necessary that the rendering of the virtual world occurs at high frame rates and with high resolution, so as not to negatively impact the user experience . Currently, most user devices that are not specifically designed for high-end gaming applications are not able to render such three-dimensional virtual environments in detail. Furthermore, user devices that are capable of such rendering tend to be very expensive and immobile.

It is therefore an objective of the present invention to provide a solution that enables the seamless presentation of three-dimensional and high-resolution (e.g., HD or 4K quality) environments on a generic user device. In particular, it is desirable that the rendering of the virtual world can be combined with other applications in an easy way, allowing a more immersive experience for the user, while providing security against the loss of sensitive information to third parties, allowing the user to interact freely with other users or applications from the virtual environment, without fear of loss of sensitive information to third parties. Furthermore, the objective of the present invention is to provide a framework for rendering virtual environments that is compatible with a wide range of possible user devices.

This problem is solved in particular by a system comprising:

1. recevoir des instructions de commande en provenance du dispositif utilisateur ;
2. restituer des vues d’un monde virtuel en fonction des instructions de commande en tant que données d’image ; et
3. transmettre un flux des données d’image au dispositif utilisateur,

ainsi qu’un composant logiciel configuré pour fournir des informations supplémentaires.
a user device (portable) running a browser application and at least one server configured to

1. receive control instructions from the user device;
2. rendering views of a virtual world based on control instructions as image data; And
3. transmit a stream of image data to the user device,

as well as a software component configured to provide additional information.

According to this system, the browser application is configured to:

a) detect the occurrence of at least one event linked to the virtual world and/or to the control instructions;

b) upon detection of at least one event, send a request to the software component;

c) receive additional information from the software component in response to the request;

d) displaying the image stream from the server in a first layer and the additional information received and/or the graphic and/or textual elements derived from the additional information received in a second (partially transparent) layer, the second layer being displayed in superimposing the first layer.

An important feature of the present invention is that the “heavy lifting” of calculation, i.e. rendering the three-dimensional environment, is done by the server and then transmitted continuously to the user device. The image stream, forming the first layer, and the additional information received by the browser, forming the second layer, are then superimposed only on the user device, specifically in the browser. This not only allows the user to easily overcome the computational limitations of the user device, but simply requires compatibility of the user device with the browser, thus also being applicable to a wide range of devices.

Outsourcing rendering to a server alleviates the long-standing problem that high-resolution rendering of a three-dimensional environment at high frame rates, i.e. 60 frames per second or more, is computationally expensive, which places high demands on the capabilities of the hardware on which the rendering process is executed. The processing capacity of most user devices is not suitable to handle such calculations, thus severely limiting the frame rate, resolution, or both, when directly performing such a rendering process.

Furthermore, the combination of layers in the browser application, in which a software component provides additional information to be displayed in the second layer, allows direct integration of external content and applications into the virtual world. Combining layers in the browser application further simplifies security concerns because all components inherit credentials and authorization from a single instance.

It is important to note that an overlay in this context can correspond to any form of combination and/or fusion of the image data with the additional information in such a way that the additional information is perceived as being in front of the image data. image, regardless of the actual layer order. HTML5 offers a direct possibility for the implementation of a layered structure according to the invention.

The software component can be located either on an additional second server (especially a web server) or on the server providing the image data. Additionally, it may be advantageous if the software component is located externally, for example in the cloud or a different server.

In one embodiment, an event may be defined by a list of conditions. The conditions preferably correspond to parameters or properties of the virtual world or control instructions. An event can be considered to have occurred when all defined conditions are met at the same time.

Additional information provided by the software component may contain textual, graphical or audio information. In certain embodiments, links may be provided to call other applications, or redirect the user to other content.

According to one embodiment of the present invention, the control instructions change a first position or a first orientation of a first avatar in the virtual world, or cause the first avatar to perform an action in the virtual world, and the rendered views of the virtual world correspond to the first position, the first orientation and/or the action of the first avatar.

According to this embodiment, the user can navigate in the virtual world and interact with objects therein. An advantage of this embodiment is that the browser application only needs to process the input of control instructions and transmit them to the server, while the server renders the virtual world according to the instructions received. Thus, the user can interact freely with the virtual world, without the user device having to support heavy computational loads. This increases the feeling of immersion in the virtual environment for the user.

According to one (another) embodiment, the software component stores customization options regarding the appearance of the virtual world and/or the avatar and is configured to receive a selection of a customization option by the browser, and upon receipt, the selection provides information to the server, causing the server to render the avatar and/or virtual world corresponding to the selection.

An advantageous effect of this embodiment is that the user device and/or the browser do not need to store information regarding the appearance of the virtual world. This further reduces the requirements for computing power and memory on the user device.

It should be understood that a personalization option includes all options related to the visual representation of the three-dimensional world, the avatar or the second superimposed layer, such as color, appearance, different skins, etc.

In another embodiment of the invention, the server is connected to a multiplayer component, the multiplayer component providing a second position and a second orientation of at least one second avatar, controlled by a second user device, to the server, and the server renders a representation of the second avatar at the second position with the second orientation in the virtual world. Additionally, the multiplayer component is configured to ensure that rendering of the virtual world on the first and second user devices is synchronized.

This embodiment allows several users to interact with the same virtual world and with each other in said world. In particular, the multiplayer component may be adapted to connect to a second server, which is controlled by the second user device. This has the advantageous effect that multiple systems according to the present invention can be interconnected directly, without burdening the user device with additional workloads.

The multiplayer component is not limited to connecting different servers, however. A connection can also be established between the server controlled by the first user device and a piece of conventional personal hardware, suitable for rendering three-dimensional environments, such as VR glasses, game consoles, personal computers, etc.

In a(nother) embodiment of the present invention, the server itself is adapted to monitor the movement of the first avatar to detect conditions for an event, and to transmit at least one identifier to the user device each time these conditions are met, and the software component includes a database, the database storing additional information associated with each identifier.

Thus, according to one aspect of the invention, the reception of such an identifier by the browser constitutes an event. This may be the case, for example, when the avatar enters a certain region of the virtual environment. This indirect detection of events by the browser has the advantage that the additional workload is removed from the user device. Furthermore, the provision of the additional information by the software component is in this case reduced to a simple query procedure, in which the received identifier is compared with the stored identifiers and the information in the database regarding the received identifier is then transmitted to the browser device.

According to an additional aspect of the invention, the transmission conditions of the at least one identifier include a proximity of the position of the avatar relative to a target position in the virtual world and/or a proximity of the orientation of the avatar in relation to a third orientation in the virtual world.

This makes it possible to directly formulate conditions for events in the virtual world, based on the location of the avatar. In particular, this allows users not trained in the design of three-dimensional environments to predefine conditions for events in the three-dimensional world, for example using an application programming interface (API).

It should be noted that the term position is not limited to the strict point of the shape (x,y,z) in the three-dimensional world, but can also refer to a certain region of the world, such as a room or a level or a particular floor or any type of zones or volume, which is defined for example by several coordinates. In particular, the third position does not necessarily have to be fixed, but can for example coincide with the second position of a second avatar or a moving vehicle or any other mobile or deformable object.

The conditions for transmitting the identifier may also be based on a timer and/or satisfy all of a first list of predetermined conditions, based on the first location and/or the first orientation and/or the input of control instructions and/or an action performed by the first or second avatar.

In this way, various conditions can be combined in the definition of an event. For example, proximity to a certain third position in the three-dimensional world can only lead to the detection of an event when it is combined with a certain timer value, or is in combination with a certain orientation of the first avatar.

An additional embodiment of the system according to the present invention can be realized, when the additional information received from the software component allows integration of external applications into the virtual world.

In this context, the term embedding refers to the display of textual, visual or audio information received from another application running on the user device, and incorporated into the second layer, to be displayed in front of the stream of images.

This allows the easy incorporation of data provided by external applications into the second layer, to be displayed in front of the image stream. Notably, this allows the user to access other applications without needing to leave the virtual world. Additionally, the information provided by the additional software upon detection of an event can be used to prompt the incorporation of such external applications.

In particular, additional information provided by the software component and displayed in the second layer may include links that allow the browser application to open other applications. In this case, the integration of external features into the virtual environment inherits the security features already provided by the browser application.

In a particular embodiment, the additional information received from the software component establishes a communication channel between the user device and the second user device, in particular a chat function and/or an audio channel and/or a video connection.

Particularly in multiplayer setups, this allows direct communication between different users, participating in the three-dimensional world. This makes the multiplayer experience more engaging for the user, as the user can easily communicate or cooperate with other users.

In a further embodiment, the browser application is provided with permission to prompt and control the execution of external applications on the user device.

As the additional information is displayed in the second layer of the browser application, it is therefore necessary to allow the browser application to act with the necessary authentication and authorization to initiate and manage the execution of other applications on the user device, in accordance with the data received from the server rendering the virtual world or from the software component.

This also makes it easier for the user to feel immersed, as there is no need to manually authorize the use of certain applications. This is particularly useful for e-commerce applications. In a particular embodiment, the browser application may utilize established online payment systems as plug-ins, to facilitate simplified payment in the virtual environment. Since the browser application does not require separate authorization from the user to initiate relevant transactions, again, relevant security features are delegated to the virtual environment. This significantly simplifies and speeds up the use of external applications in the virtual world.

According to a further aspect of the invention, the software component consists of a content management system (CMS), and the customization options can be set via an application programming interface (API).

This further increases user-friendliness, when it comes to CMS customization, as it allows non-expert users to manipulate the customization options with greater ease. In a particular example, the CMS is a web application consisting of a web server and a database backend.

In this case, the CMS creates and maintains the virtual environment, saving a generated configuration for each user device in the database backend. According to this embodiment, the CMS application controls the virtual environment.

Furthermore, the above-mentioned problems are solved by a method in which

a user device (portable) running a browser application establishes a first connection to at least one server, the server receives control instructions from the user device and renders views of a virtual world based on said control instructions as image data, transmitting a stream of image and/or audio data to the user device;

the browser detects the occurrence of at least one event linked to the virtual world and/or the control instructions and, upon detection of the at least one event, sends a request to a software component, in response to the request, the browser receives additional information from the software component;

the browser displays the image stream from the server in a first layer and the additional information received and/or the graphic and/or textual elements derived from the additional information received in a second (partially transparent) layer, the second layer being displayed in superimposing the first layer.

In yet another aspect, the layered structure of the invention can be modified in a third-party application, preferably a third-party web page, preferably hosted on a third-party server. This can be achieved by integrating the first and second layers into a parent frame, which may be part of the third-party application or website. In this case, the web page rendered by the browser application can be divided into two or more frames, one of which displays the virtual world with the first and second layers respectively, which can be called the VR frame. At least one of the other frames can display content from the third-party server and/or any other server, and can therefore be called a third-party frame. The content rendered in the at least one other frame may be that of an online store or, more generically, of a merchant platform.

The frames referred to above may be implemented by any available technology, for example frames and/or templates and/or sections of HTML5.

According to one aspect of the invention, there is communication between the at least one of the layers (first and second layers of the virtual world) and the third-party application, for example, the third-party web page, which is established. More specifically, events that are associated with the first and/or the second layer can be transmitted to the third-party application and/or the third-party server. Preferably, the communication is established on the client side, for example between the frame and optionally transmitted to a server, for example, the third party server. This allows an immediate response to the event to be generated, for example from the third party web page/third party side.

Once the browser application detects an event, a request is sent to the software component, which returns additional information, which is then displayed in the second layer. In particular, the additional information may include a link or a button, with which the user can interact with the order instructions.

Once the browser application registers the ID associated with an event in the first and/or the second layer, the parent frame may further be prompted to transmit information associated with the event ID to the at least one another third-party frame associated with the third-party application and/or the third-party server. In particular, the event ID can be associated, preferably by means of a staging table, with an action on the third party side, such as placing an item in a checkout cart. This information can then be further relayed by the parent frame to a component of the third-party application. Alternatively, the information may only cause a local change in the browser/client's third-party frame, which may be presented to the third-party application or third-party server at a later time.

The term "parent frame" is chosen to emphasize the conceptual structure of the present aspect of the invention and may not refer to an actual frame, but rather more generally to the component of a third-party application or web page, in which the RV frame and one or more third-party frames are integrated.

In an alternative aspect, the browser may transmit the additional information to the third-party application and/or third-party server immediately after detecting the event, without first relaying it to the third-party frame.

In all of the above cases, the relay of information associated with the event ID may or may not be conditional on the user entering respective control instructions, once the graphical representation is presented in the second layer. So, for example, the user may have to click on a button in the second layer, before the transmission of associated information is initiated by the parent frame. Alternatively, for some event IDs, it may be preferable to immediately initiate transmission of the associated information through the parent frame, without requiring additional input from the user.

This aspect is particularly suitable, for example, for e-commerce applications. Here, the event can be triggered by a certain action of the avatar in the virtual world, causing additional information to be displayed on the second layer. This additional information may, for example, include product information, as well as a buy button, which may encourage the purchase of the displayed item. When the user enters a corresponding order instruction (such as clicking the buy button), associated information may be relayed in the manner described above.

This makes it possible, for example, to place items, which are for sale on the third-party website, in a checkout basket, through actions in the virtual world. The separation of the first and second layers and the third-party web page respectively into different frames allows parts of the third-party web page to remain accessible and visible directly to the user while navigating in the virtual world.

With the method according to the invention, the same advantages can be obtained as those which have already been described in association with the system according to the invention. It should be noted that the characteristics described in the context of the system according to the invention apply to the method according to the invention. Likewise, the characteristics of the method according to the invention are transferable to the system according to the invention by configuring the browser application, the server or the additional software in such a way that it is adapted to the implementation of the characteristics of corresponding process.

In the following, the invention is also described with regard to other details, features and advantages, which are explained in more detail with reference to the figures. The characteristics and combinations of characteristics described, as shown below in the figures described with reference to the figures, are applicable not only in the combination indicated in each case, but also in other combinations or alone, without departing from the scope of the invention.

In the following, embodiments of the invention are described with reference to the figures, in which

There represents a system consisting of a browser application running on a user device (not shown), a software component and a server;

There represents the system according to the invention in a multiplayer configuration, connected to another user device;

There represents the detection of an event by passing an ID from the server to the browser application;

There represents the detection of an event via the browser application;

There represents an embodiment in which the CMS includes a web server and a backend database;

There represents an example database entry for defining an event;

There represents an example database entry relating the event ID to additional information;

There represents an example of a user device according to the system of the invention;

There represents an example of integrating virtual reality into a client window, establishing communication with a third-party server;

Figures 8A to 8E represent a representative configurator and display screens rendered for associating and storing in the software component examples of images according to the invention; And

Figures 9A to 9D represent a representative configurator and display screens rendered for associating and storing in the software component examples of videos according to the invention.

DETAILED DESCRIPTION

There illustrates the basic configuration of a system according to the present invention. Upon entering the virtual world by accessing an environment URL, a representative display provided by a browser application 2 on a display of a user device with limited computing capacity (not shown) establishes communication via a first connection 8 with a software component 4, running on a separate server, to perform initial authentication checks and to be queued by the software component 4 to access a server 3.

Representative authentication checks may include, for example, a logon process whereby browser application 2 transmits a digital token request, such as an HTTP POST request, to software component 4 on the first connection 8. The corresponding digital token received by the browser application 2 allows access to such a virtual world, for example, by a queuing service for the allocation of a server, such as server 3, from a group of servers that allow access to such a virtual world. Additionally, if access to the virtual world is password protected, the transmitted request may include a valid password, and/or single sign-on (SSO) credentials, if the browser application 2 or the user device includes corresponding SSO functionality. It is further up to the user device to log in with the user's email address and password, if an administrator has already created an account for that email address.

The digital token may be a JSON Web Token ("JWT") cryptographically signed in accordance with Internet Engineering Task Force (IETF) RFC 5741, which may be stored as a cookie and/or user device RAM, or any other type of appropriate digital token. Once the user device has the digital token, subsequent requests from the browser application 2 may result in an authentication check by the software component 4 to determine whether the user device has a valid token, i.e. -say not having expired to access the virtual world.

It will be readily understood that typical cloud streaming servers may not be immediately available for provisioning by the user device, particularly if demand is high for such servers, so there may be a delay in assigning and granting access to a cloud streaming server, such as server 3. If the cloud streaming capability running the desired virtual world (/binary executable) is already provisioned and available, the user device can be quickly redirected to the server 3. Otherwise, the user device can receive a "ticket" together with the position of the ticket in the queue, and periodically poll the queue waiting with this ticket. Advantageously, queue management software often favors servers located close to the user device to reduce any latency that the user device may experience.

The software component 4 establishes a second connection 9 with the server 3. Alternatively, the server 3 connects to the software component 4 to signal its presence and its status. In particular, the software component 4 may further be configured to select one of a plurality of accessible servers according to predetermined criteria, while the browser application 2 is queued. These criteria may, for example, depend on the user's geolocation or IP address, connection, number of available servers or authentication details. Via the second connection 9, the server 3 downloads a list of event conditions (see ) from software component 4 and stores them in an in-memory address map, i.e. hash map or database on the server. a copy of a partial or complete list of event conditions information.

The server is reserved for a predefined duration, for example 10 seconds, for the user device and the server address (a URL) is sent to that user device for use by the browser application 2. The browser application 2 then connects and authenticates with this server 3, for example using a WebSocket protocol connection. Over this WebSocket connection, a negotiation process immediately begins and WebRTC ICE (Interactive Connectivity Establishment) standard candidates are exchanged. ICE candidates are used to establish a third connection 10, for example a WebRTC connection, between the server 3 and the browser application 2. According to another aspect of the present invention, relay servers of the TURN standard (Traversal Using Relay protocol around NAT) can be used. After a connection between browser application 2 and server 3, the image and audio data flow begins.

On the third connection 10, the image (video) and/or audio data corresponding to the three-dimensional environment are transferred from the server 3 to the browser application 2, where they are displayed in the first layer 5.

On the same third connection 10, control instructions from browser application 2 are sent to server 3. Depending on the type of user device used to access the three-dimensional environment, the control instructions may include: events clicks, mouse movements, keyboard inputs, joystick movements, VR haptics, touchscreen responses, accelerometer inputs, gyroscope measurements, webcam data and voice instructions, c 'i.e. microphone input.

Furthermore, the third connection 10 includes a bidirectional data channel for the exchange of additional information between the server 3 and the browser application 2.

When the browser application disconnects from server 3, server 3 updates its state for software component 4 via the second connection 9 to declare itself available again.

When an event is detected by the browser application 2, the browser application sends a request (based on the availability of the required data in the browser application) to the software component 4 via the first connection 8 and receives additional information about the event coming from software component 4 via the same connection. The browser application 2 then displays the additional information received and/or the graphic and/or textual elements derived from the additional information received 6 in the second (partially transparent) layer 7, the second layer 7 being displayed by superimposing the first layer 5 As used herein, the term "partially transparent layer" means, without limitation, semi-transparent or transparent regions on at least a portion of such layer, at least other portions of the layer having images which block or merge images present in the overlaid layer, i.e. the sublayer, such as the first layer 5. Alternatively, the browser application may have downloaded all the information additional necessary to execute the event in advance from the software component 4. Figures 8A to 8E, described below, show a representative configurator and display screens rendered for associating and storing in the software component 4 images in example for displaying pop-up images when the occurrence of an associated trigger event is detected. Likewise, Figures 9A to 9D show a representative configurator and display screens for associating and storing in the software component example videos for displaying videos when the occurrence of an associated trigger event is detected.

When the server 3 receives control instructions from the browser application 2 via the third connection 10, the position and orientation of the first avatar 11 in the virtual world are updated, and the image data transferred to browser app 2 are updated accordingly. In addition, server 3 compares the new position and orientation of the avatar with the list of positions and orientations corresponding to possible event conditions, stored in the second database on server 3.

Other possible event conditions include associating an in-game actor (item) ID with a video, predicted absolute time (i.e. actual time), elapsed time since the start of broadcast and others.

There represents the system in a multiplayer configuration. The server 3 receives the position and orientation of a second avatar 14 from a multiplayer component 13. The multiplayer component 13 is connected to at least a second user device 12, which controls the movement of a second avatar 14 It is understood that this connection to a second user device can also be an indirect connection via another server according to the present invention. In this case, the second user device should be understood as the entire system according to the present invention. Upon receiving the second position and the second orientation, the server 3 renders a second avatar 14 as part of the image data displayed in the first layer 5.

There shows the indirect detection of event conditions in the system, via the server 3 of Figures 1 and 2. While the first avatar moves in the virtual environment, the server monitors the properties of the input instructions and the virtual world and checks if all conditions of a predefined event are met. If so, the server passes an ID corresponding to the event that was detected to the browser application. The browser application then detects the presence of the ID in the data stream of the third connection as an event. The browser application then sends a request to the software component, containing the received ID, or has already previously retrieved data containing such an ID. The software component then compares the received ID to a list of IDs stored in the database. Once the matching ID in the database is found, the software component passes the additional information associated with the received ID to the browser application in case the additional information has not already been pre-fetched . Finally, the browser processes the additional information received and displays it in the second layer, overlaying the image data stream.

There shows direct recognition of an event condition through browser application 2 in a further embodiment of the invention. In this case, the browser application 2 directly monitors the control instructions and the virtual environment. When all conditions defining an event are met, the browser application sends the request for an ID corresponding to the relevant event to software component 4. Software component 4 then compares the received ID to a list of IDs stored in the database. Once the matching ID in the database is found, the software component passes the additional information associated with the received ID to the browser application. Finally, the browser processes the additional information received and displays it in the second layer, overlaying the image data stream.

There represents an embodiment in which the software component 4 consists of a CMS. The CMS is divided into a web server 15 and a backend database 16. The CMS receives customization options via the first connection 8 and saves the generated configuration in the backend database. In particular, it monitors the state of the virtual environment via the second connection 9.

There represents the data structure of an event, consisting of the event ID and a plurality of conditions. Conditions are based in part on locations, orientations, and possible actions in the virtual world, as well as instructional inputs. For example, a first condition may be whether or not the first location of the first avatar 11 is in a certain region around a third location in the virtual world. A region can be defined as the set of all points located within a predefined distance from the third location, or as a predefined set of points containing the third location (such as a room in which the third location is located). A second condition can for example be the value of a timer equal to a predefined value. In this example, an event will be detected, if the location of the first avatar 11 is within the range of locations defined by the first condition AND if the timer value is equal to the predefined value.

There represents the data structure according to which the additional information is stored in the software component. According to this embodiment, the database includes an ID list which contains the IDs of all configured events together with corresponding additional information.

There represents a user device 17 according to one embodiment of the present invention. The browser application 2 is executed by the CPU 19 of the user device, displaying the first layer 5 (not shown) and the second layer 6 (not shown) on the display of the user device 20. Control instructions are entered via a keyboard 18 of the user device.

There represents a client, such as a browser application 2 displaying a third-party web page. The web page, for example using itrames, is separated into a third party frame 72 and a VR frame 74. For ease of reference only, the window/frame containing the third party frame 72 and VR frame 74 is called parent frame 75. .

In the embodiment, the third-party frame 72 is arranged next to the RV frame 74 which contains at least the first 5 and second 7 layers of the virtual world as already described, for example with reference to the . In the embodiment of the , the display of a graphical representation of parts of the additional information 6, which has been triggered by the occurrence of an event, is represented in the second layer 7, may include the display of a button, for example which allows you to add a product to a purchasing card. The user can interact with the graphical representation 6, for example the button, by causing the browser 2 to communicate via the parent frame 75 with the third party frame 72. In one embodiment, such communication can cause the third party web page to add a particular item to a shopping cart, the particular item being linked to what is represented in the RV frame 74, for example in the first and/or second layer 5,6.

Alternatively, the browser 2 and/or the parent frame 75 can communicate the information received from the RV frame 74 directly on a fourth connection 73 to an external third-party server 71.

In one embodiment, communication between RV frame 74 and parent frame 75 uses a custom message protocol. This means that the parent frame 75 when interacting – by clicking on the button – with the graphic representation 6 receives a message which identifies an event or preferably an item which must be added to the shopping cart. The type of graphical representation 6, the appearance time, the behavior and any ID identifying an event or item can be configured in the software component 4. The ID can be transmitted as a payload of the personalized message to the parent frame 75 allowing the third party web page to react upon receipt of the ID by any available means, for example by triggering other actions in the third party web page.

The custom message can be a JSON string which can be generated and transmitted as follows:

let postMessage = JSON.stringify({

"_ABC_action":"addtobag",

"skuBaseId": "11111",

“INCREMENT”:1,

"QTY":1

});

if ('parentIFrame' in window) {

window.parentIFrame.sendMessage(postMessage);

}});

The approach allows the user to easily perform actions within VR that will affect the behavior of the third-party web page. By distributing the video stream in the first layer 5, the graphic representation 6 in the second layer 7 and by allowing the configuration of the graphic representation 6, in particular the ID which will be transmitted, the invention allows a very efficient configuration of the system. For example, it is possible to add the possibility of purchasing certain items in virtual reality without having to modify the virtual reality. In one embodiment, the system administrator just needs to configure a new graph 6, the respective ID and an event that triggers the graph. All these steps can be performed on software component 4.

Of course, this part of the invention is not limited to the purchase of articles and/or services. The approach can also be used for training and any other educational service.

In certain embodiments, appropriate CMS usable for the software component 4 of the include, for example, the Unreal Engine computer graphics game development software offered by Epic Games, Inc. Figures 8A to 8E are representative configurator and display screens for associating and storing in such a CMS an example contextual image to be presented in layer 7 upon the occurrence of a corresponding event. There represents an example of a contextual configurator screen allowing a developer to enter in the “Pop-up Slug” and “Title” fields, a name and a title, respectively, of the image to be attached via the “image” field. In this example, the Pop-up Slug name is "ArcticPopUp" for the added image of the Arctic landscape. An associated description of the image can also be entered in the “Description” field.

There represents an example event condition configuration screen to be used by the developer to create respective event conditions based on the associated “Triggers” and “Actions” fields. Figures 8C and 8D represent Trigger and Action sub-configuration screens accessible from the event condition configuration screen shown on the . As shown on the , “Trigger Type” of “Region” has been selected, and on the The “Action Type” selected for such a trigger is “Open Pop-up” with the associated image name of “ArcticPopUp” for the image to be displayed. There illustrates the resulting displayed pop-up image of "ArcticPopUp" and associated description when a user's avatar enters the region for a region-triggered event condition.

Correspondingly, Figures 9A to 9D are representative configurator and display screens for associating and storing in an example CMS 4 a URL of a video to be displayed in layer 7 upon the occurrence of an event corresponding. In reference to the , an example configurator display screen is shown for entering the name of the video as "Unreal Slug", and a corresponding "Title" of the video, both of which are "SampleVideo" in this example. In addition, the configurator display screen of the includes an action button to add the "videos", which is labeled "+add." » By selecting this button, a sub-configurator display screen shown on the would appear to allow the developer to enter “Title” of the respective video and associated “Video URL” from which the video would be streamed. Additionally, this sub-configurator display screen also allows the developer to associate certain attributes of the video to be displayed, including whether the video is looped, i.e. repeated, or whether it should be played in direct.

Another configurator screen (not shown) similar to the configurator screen of the can be used to enter triggers associated with playing videos specified using the configurator screens in Figures 9A and 9B. See, for example, Figures 9C and 9D, which include triggers for playing a video when a user's avatar enters a defined region near a rendered display screen in the virtual environment displaying in a first display layer the static image “OPERATION #OCEAN DETOX” on the . When the avatar enters the defined region, a video begins to be played in the second layer 7 which superimposes this static image, as shown in the figure. , with the title screen “THE OCEANS ARE FULL OF TRASH. »

The triggers and actions described with reference to Figures 8A-8E and 9A-9D are for illustrative purposes only, and other alternative trigger events and actions are contemplated by the invention, including but not limited to limited to, actions taken by the avatar, regional triggers, temporal triggers, specific positions and/or viewpoints of the avatar, and others as described above. Additionally, although these figures were illustrative of triggers and actions based on occurrences in the virtual environment by a user's avatar, it should be easily understood that actions to display content in layer 7 can alternatively be triggered by control instructions from the user device.

It will be understood that, although various aspects of the present disclosure have been illustrated and described by way of example, the invention described herein is not limited thereto, but may otherwise be variously implemented as suggested by the disclosure and as set forth in the scope of the following exemplary claims. For example, despite the disclosure concerning a system and a method for rendering a virtual reality, such a system or method can further be implemented in an augmented or holographic reality system, or the like, in accordance with the invention described previously.

List of References:

1 System

2 Browser application

3 Server

4 Software component

5 First layer

6 Graphical representation of additional information

7 Second layer

8 First connection

9 Second connection

10 Third connection

11 First Avatar

12 Second user device

13 Multiplayer component

14 Second avatar

15 Web server

16 Backend database

17 User device

18 Keyboard

19 user device CPU

20 User device display

71 Third-party server

72 Third-party frame

73 Fourth connection

74 VR frame

75 Parent frame/window

Claims (22)

Method for rendering images of a virtual environment on a first user device, comprising the steps consisting of:
executing a browser application on the first user device to establish a first connection, preferably over a network, between the first user device and at least one first server and to receive streaming image data from the first server;
rendering as a first layer on the first user device views of a virtual environment based on the received streaming image data;
transmitting control instructions from the first user device to the first server and receiving corresponding updated streaming image data of the virtual environment based on said control instructions;
upon detecting the occurrence of at least one predetermined event linked to the virtual environment and/or the control instructions, transmitting a request with an event identifier to a second server running a software component, in which the browser application and the software component maintain corresponding lists of predetermined event identifiers;
receiving additional display information from the software component in response to the request, wherein the additional display information represents content generated outside of the rendered virtual environment; And
rendering as a second partially transparent layer on the first user device images derived from the received additional display information, wherein the second layer is displayed overlying the first layer to enable display of external content in the rendered virtual environment . The method of claim 1, wherein the control instructions change at least one of a first position or orientation of a first avatar in the virtual environment, or cause the first avatar to perform an action in the virtual environment, and wherein the rendered views of the virtual environment correspond to at least one of the first position, first orientation or action of the first avatar. Method according to one of the preceding claims, in which the software component stores personalization options as to the appearance of at least one of the virtual environment or of the first avatar and is configured to receive a selection of a personalization option by the browser, and upon receipt, the selection provides information to the first server, causing the first server to render at least one of the avatar or virtual environment corresponding to the selection. Method according to one of the preceding claims, in which the at least one server is connected to a multiplayer component, the multiplayer component providing a second position and a second orientation of at least one second avatar, controlled by a second user device, at at least one server, and the first server renders a representation of the second avatar at the second position with the second orientation in the virtual environment. Method according to one of the preceding claims, wherein the first server is adapted to monitor the movement of the first avatar to detect conditions for an event, and to transmit at least one event identifier to the first user device each time these conditions are satisfied, and the software component includes a database, the database storing additional information associated with each event identifier. Method according to one of the preceding claims, in particular claim 5, in which the transmission conditions of the at least one identifier are based on a proximity of the position of the first avatar relative to a third position in the virtual environment and/ or close to the orientation of the first avatar relative to a third orientation in the virtual environment. Method according to one of the preceding claims, in particular claim 5, wherein the conditions for transmitting the event identifier are based on a timer and/or satisfying all of a first list of predetermined conditions, on based on at least one of the first location, the first orientation, input of control instructions, or an action performed by the first or second avatar. Method according to one of the preceding claims, in which the additional display information received from the software component allows integration of external applications into the virtual environment. Method according to one of the preceding claims, in particular claim 8, wherein the additional information received from the software component establishes a communication channel between the first user device and the second user device for activating at least one of a chat function, an audio channel, or a video connection. Method according to one of the preceding claims, in particular claim 9, wherein the browser application is provided with authorization to initiate and control the execution of external applications on the first user device. Method according to one of the preceding claims, in particular claim 3, in which the software component comprises a content management system, and the personalization options can additionally be defined via a programming interface application. System for rendering images of a virtual environment on a first user device comprising:
a first user device configured to:
run a browser application,
receive control instructions entered by a user and transmit these control instructions, preferably over a network, to at least one first server, wherein the at least one first server is configured to (i) create indicative image data rendered views of a virtual environment based, in part, on the received control instructions, and (ii) transmitting a stream of the image data to the first user device,
receive the image data stream transmitted to the first user device, and
receiving from a second server additional image information produced by a software component executing on said second server, said additional image information representing content generated outside of the rendered virtual environment; And
in which the browser application is configured to:
rendering as a first layer views of the virtual environment based on the received streaming image data,
detect the occurrence of at least one event linked to the virtual environment and/or to the control instructions,
upon detecting the at least one event, transmitting a request to the second server running the software component to produce the additional image information, and
rendering as a partially transparent second layer images derived from the received additional display information, wherein the second layer is displayed overlapping at least a portion of the first layer to enable display of external content in the rendered virtual environment . The system of claim 12, wherein the control instructions change at least one of a first position or orientation of a first avatar in the virtual environment, or cause the first avatar to perform an action in the virtual environment, and in which the rendered views of the virtual environment correspond to at least one of the first position, the first orientation and/or the action of the first avatar. System according to one of claims 12 and 13, in which the software component stores personalization options as to the appearance of the virtual environment and/or the first avatar and is configured to receive a selection of a personalization option by the browser, and upon reception, the selection provides information to the first server, causing the first server to render the first avatar and/or the virtual environment corresponding to the selection. System according to one of claims 12 to 14, in which the at least one server is connected to a multiplayer component, the multiplayer component providing a second position and a second orientation of at least one second avatar, controlled by a second user device , at least one server, and the server renders a representation of the second avatar at the second position with the second orientation in the virtual environment. System according to one of claims 12 to 15, in particular claim 13, wherein the server is adapted to monitor the movement of the first avatar to detect conditions for an event, and to transmit at least one event identifier to the first user device each time these conditions are satisfied, and the software component comprising a database, the database storing additional information associated with each event identifier. System according to one of claims 12 to 16, in particular claim 16, in which the conditions of transmission of the at least one identifier are based on a proximity of the position of the first avatar in relation to a third position in the virtual environment or on a proximity of the orientation of the first avatar relative to a third orientation in the virtual environment. System according to one of claims 12 to 17, in particular claim 15, in which the conditions for transmitting the event identifier are based on at least one timer or satisfying all of a first list of predetermined conditions , based on at least one of the first location, the first orientation, input of control instructions, or an action performed by the first or second avatar. System according to one of claims 12 to 18, in which the additional display information received from the software component allows integration of external applications into the virtual environment. System according to one of claims 12 to 19, in particular claim 19, wherein the additional display information received from the software component establishes a communication channel between the first user device and the second user device to activate a function chat, audio channel, or video connection. System according to one of claims 12 to 20, in particular claim 20, wherein the browser application is provided with authorization to initiate and control the execution of external applications on the first user device. System according to one of claims 12 to 21, in particular claim 14, in which the software component comprises a content management system, and the customization options can additionally be defined via a programming interface of application.