JP7411585B2 - Centralized rendering - Google Patents

Description

(Cross reference to related applications)
This application has the benefit of U.S. patent application Ser. No. 16/011,413, filed June 18, 2018, the contents of which are incorporated herein by reference in their entirety for all purposes. claim.

TECHNICAL FIELD This disclosure relates generally to systems and methods for visually rendering graphical data on a display, and more particularly, to systems and methods for visually rendering data from multiple computer applications on a single display. .

Various techniques exist for rendering computer application graphical data to a display. It may be desirable for these techniques to render graphical data realistically, ie, consistent with the viewer's expectations based on the physical world, and so efficiently. It may also be desirable for rendering techniques to be adaptable to computer systems of various topologies, including, for example, computer systems in which multiple applications contribute graphical data to be displayed on a single display.

Conventional systems are often unable to realistically and efficiently render content in such multi-application systems. For example, in some such systems, rendering graphical data from multiple applications on a single display can result in data being incorrectly sorted on the display, detracting from the realism of the display. , producing unexpected visual results. Furthermore, graphical data from one application may not be able to realistically interact with graphical data from another application, such as through lighting and shadow effects, or through shaders. In addition, some such systems are limited in their ability to use rendering optimizations, such as culling of invisible surfaces, to increase computational efficiency.

Systems involving augmented reality (AR) or "mixed reality" particularly require better solutions to the problem of rendering graphical data from multiple applications onto a single display. For example, AR systems have the potential for multiple users to interact within a shared virtual space with virtual content from all users rendered on a single display. Such interactions are believable and meaningful to the user, and may require that the graphical output of the AR system be convincing and consistent with the user's visual expectations, and may require different types and numbers of users, User hardware, and user software, are flexible enough to adapt to the different ways users may wish to engage with the system, to sustain continuous operation at high frame rates, and on mobile devices. It is desirable to be efficient enough to maximize battery life. Additionally, applications and application data associated with individual users in an AR system provide security (which can be compromised by data access among untrusted users), especially as the number of users of the system grows. It may be desirable to remain independent from other users, both to maintain scalability. Furthermore, such systems may benefit from minimizing technical constraints on users and user applications; for example, limiting the hardware requirements for users to participate in an AR system may be more Encourage more users to participate. This is done by limiting the extent to which an individual user or an application launching on the user's hardware needs to perform complex rendering operations, e.g. by limiting the extent to which an individual user or an application launching on the user's hardware has to perform complex rendering operations, e.g. This can be accomplished by offloading applications to a shared system.

Embodiments of the present disclosure describe a computer system in which multiple applications contribute graphical data to be displayed on a single display. Embodiments of the present disclosure can be used to render graphical data realistically, ie, consistent with a viewer's expectations based on the physical world, and efficiently. According to embodiments of the present disclosure, first graphical data may be received from a first client application, and second graphical data may be received from a second independent client application. The first and second graphical data may be combined into a "unified" data structure, such as a scene graph, that may be used to describe the relationships between the nodes represented by the first and second graphical data. good. The unified data structure can therefore be used to render a scene reflecting the first and second graphical data on a display in a practical and efficient manner.
The present invention provides, for example, the following.
(Item 1)
A method,
receiving first graphical data comprising a plurality of first nodes from a first client application of the computer system;
receiving second graphical data comprising a plurality of second nodes from a second client application of the computer system;
Generating a scene graph and
including;
The scene graph describes an occlusion relationship between at least one first node of the plurality of first nodes and at least one second node of the plurality of second nodes,
The method wherein the scene graph is configured to create a rendered scene based on the occlusion relationship, wherein at least one second node occlusions at least one first node.
(Item 2)
The method of item 1, further comprising traversing the scene graph by a processor of the computer system.
(Item 3)
3. The method of item 2, wherein the computer system is configured to communicate with a display, and the method further includes displaying output on the display.
(Item 4)
Displaying the output includes displaying at least one first node of the plurality of first nodes and at least one second node of the plurality of second nodes. The method described in item 3.
(Item 5)
5. The method of item 4, wherein displaying the output comprises displaying at least one first node.
(Item 6)
3. The method of item 2, further comprising applying an optimization to the output at the computer system.
(Item 7)
7. The method of item 6, wherein applying the optimization includes culling a surface.
(Item 8)
3. The method of item 2, further comprising applying a visual effect to the output at the computer system.
(Item 9)
9. The method of item 8, wherein applying the visual effect includes calculating a light intensity value.
(Item 10)
9. The method of item 8, wherein applying the visual effect includes running a shader.
(Item 11)
3. The method of item 2, further comprising, in the computer system, applying a physical effect to the output.
(Item 12)
12. The method of item 11, wherein applying the physical effect includes detecting a collision.
(Item 13)
The first client application is a first application running on the computer system, and the second client application is a second application running on the computer system, and the first client application is a second application running on the computer system. The method of item 1, wherein a client application is sandboxed on the computer system to the second client application.
(Item 14)
the first graphical data corresponds to a first client scene graph associated with the first client application;
the second graphical data corresponds to a second client scene graph associated with the second client application;
the first client scene graph is sandboxed on the computer system with respect to the second client scene graph;
the first client scene graph is sandboxed on the computer system with respect to the scene graph;
2. The method of item 1, wherein the second client scene graph is sandboxed on the computer system with respect to the scene graph.
(Item 15)
The method of item 1, wherein the scene graph corresponds to a version of a versioned scene graph.
(Item 16)
The first graphical data is communicated to the scene graph using a first processing thread of the computer system, and the second graphical data is communicated to the scene graph using a first processing thread of the computer system that is independent of the first processing thread. The method of item 1, wherein the scene graph is communicated using a second processing thread.
(Item 17)
A system comprising one or more processors, the one or more processors configured to perform a method, the method comprising:
receiving first graphical data comprising a plurality of first nodes from a first client application of the computer system;
receiving second graphical data comprising a plurality of second nodes from a second client application of the computer system;
Generating a scene graph and
including;
The scene graph describes an occlusion relationship between at least one first node of the plurality of first nodes and at least one second node of the plurality of second nodes,
The system wherein the scene graph is configured to create a rendered scene based on the occlusion relationship, wherein at least one second node occlusions at least one first node.
(Item 18)
18. The system of item 17, the method further comprising causing a processor of the computer system to traverse the scene graph.
(Item 19)
19. The system of item 18, wherein the computer system is configured to communicate with a display, and the method further includes causing the computer system to present output on the display.
(Item 20)
Displaying the output includes displaying at least one first node of the plurality of first nodes and at least one second node of the plurality of second nodes. The system described in item 19.
(Item 21)
21. The system of item 20, wherein displaying the output comprises displaying at least one first node.
(Item 22)
19. The system of item 18, the method further comprising causing the computer system to apply an optimization to the output.
(Item 23)
23. The system of item 22, wherein applying the optimization includes culling a surface.
(Item 24)
19. The system of item 18, the method further comprising causing the computer system to apply visual effects to the output.
(Item 25)
25. The system of item 24, wherein applying the visual effect includes calculating a light intensity value.
(Item 26)
25. The system of item 24, wherein applying the visual effect includes executing a shader.
(Item 27)
19. The system of item 18, wherein the method further includes causing the computer system to apply a physical effect to the output.
(Item 28)
28. The system of item 27, wherein applying the physical effect includes detecting a collision.
(Item 29)
The first client application is a first application running on the computer system, and the second client application is a second application running on the computer system, and the first client application is a second application running on the computer system. 18. The system of item 17, wherein a client application is sandboxed on the computer system to the second client application.
(Item 30)
the first graphical data corresponds to a first client scene graph associated with the first client application;
the second graphical data corresponds to a second client scene graph associated with the second client application;
the first client scene graph is sandboxed on the computer system with respect to the second client scene graph;
the first client scene graph is sandboxed on the computer system with respect to the scene graph;
18. The system of item 17, wherein the second client scene graph is sandboxed on the computer system with respect to the scene graph.
(Item 31)
18. The system of item 17, wherein the scene graph corresponds to a version of a versioned scene graph.
(Item 32)
The first graphical data is communicated to the scene graph using a first processing thread of the computer system, and the second graphical data is communicated to the scene graph using a first processing thread of the computer system that is independent of the first processing thread. 18. The system of item 17, wherein the scene graph is communicated using a second processing thread.
(Item 33)
18. The system of item 17, wherein the system comprises the computer system.
(Item 34)
The first client application is a first application executed via the one or more processors, and the second client application is a second application executed via the one or more processors. 18. The system of item 17, wherein the first client application is sandboxed on the system with respect to the second client application.

1A-1E illustrate an example computer system including a graphical display according to an embodiment of the present disclosure.

FIG. 2A illustrates an example flow of data in an example computer system, according to an embodiment of the present disclosure.

FIG. 2B illustrates an example renderer output corresponding to an example computer system, according to an embodiment of the present disclosure.

FIG. 2C illustrates an example flow of data in an example computer system including multiple independent applications, according to an embodiment of the present disclosure.

FIG. 2D illustrates an example renderer output corresponding to an example computer system that includes multiple independent applications, according to an embodiment of the present disclosure.

FIG. 3A illustrates components of an example computer system that may render 3D data from multiple independent applications to a display using a unified scene graph, according to embodiments of the present disclosure.

FIG. 3B illustrates aspects of an example client application for an example computer system, including multiple independent client applications, in accordance with an embodiment of the present disclosure.

FIG. 3C illustrates aspects of an example client-server interface for an example computer system that includes multiple independent client applications, according to an embodiment of the present disclosure.

FIG. 3D illustrates aspects of an example host application 340 for an example computer system that includes multiple independent client applications, according to an embodiment of the present disclosure.

FIG. 3E illustrates aspects of an example renderer 360 for an example computer system that includes multiple independent client applications, according to an embodiment of the present disclosure.

FIG. 4 illustrates an example system architecture that may be implemented within any portable or non-portable device, according to an example of the present disclosure.

In the following description of embodiments, reference is made to the accompanying drawings, which form a part hereof, and in which is shown, by way of illustration, specific embodiments that may be practiced. It is to be understood that other embodiments may be used and structural changes may be made without departing from the scope of the disclosed embodiments.

1A-1E illustrate various example computer systems with displays. FIG. 1A shows an exemplary desktop computer connected to an external monitor. FIG. 1B shows an exemplary laptop that includes a display. FIG. 1C shows an example mobile device that includes an integrated display. FIG. ID shows an exemplary television including a display. FIG. 1E shows an example computer system that includes a head-mounted display. This disclosure is not limited to any particular type of computer system, to any particular type of display, or to any particular means of connecting a computer system to a display. The present disclosure is further not limited to two-dimensional displays, but specifically contemplates three-dimensional displays, such as stereoscopic displays.

In some example computer systems, the data to be presented graphically (as a "rendered scene") on a display represents objects (such as 2D or 3D geometric primitives, including polygons, etc.) in three-dimensional space. The presentation of 3D Data on a display, including data ("3D Data"), refers to an image ("Display This includes presenting "scenes in which the For example, in a software application running on a computer system (such as a video game that uses a 3D engine), 3D data may contain information about the spatial coordinates, orientation, and/or visual properties of objects within a three-dimensional game world, as well as the game world's spatial coordinates, orientation, and/or visual properties. It may also include data describing the viewing origin and viewing axis within the image. 3D data may also include data related to textures associated with the object to be rendered, shader parameters associated with the object, and other information that affects how the object may be displayed. For example, during a "rendering" or "drawing" phase, a game instructs a software and/or hardware "pipeline" to create a rendered scene for presentation on a display as a displayed scene. You may. In such embodiments, it is generally desirable for the resulting images to reflect the user's expectations about the visual world. In particular, it is generally desirable for a first opaque object that is closer to the viewing origin to occlude a second object behind the first object. Objects that are not occluded correctly can be confusing to the user and may not clearly indicate where the object is located in three-dimensional space. In some example computer systems, occlusion is achieved through sorting, where objects that are closer to the viewing origin are sorted or drawn over objects that are farther from the viewing origin.

Sorting multiple objects for presentation on a display such that one object realistically occludes another is a method of determining relationships between objects, e.g., spatial relationships between objects in three-dimensional space. Request information about. Some example computer systems utilize scene graphs to represent relationships (eg, hierarchical relationships) between one or more objects, such as objects to be rendered as a scene. As used herein, a scene graph is any data structure that represents such relationships. For example, in a scene graph, rendered objects to be presented may be represented as nodes in the graph, and relationships between nodes represent logical or spatial relationships between the objects. The renderer may then traverse the scene graph and render or prepare at least one of the objects for display in a manner that will achieve appropriate occlusion according to techniques known in the art. can. In other words, a renderer may create a scene of objects that have nodes, but the presentation on the display is such that an object that is occluded by another object in the renderer is only partially within the resulting displayed scene. As will be presented, only a subset of the object may be rendered (the output in such an embodiment is the non-occluded portion of the object). Such selective display may occur when the user desires only the content embodied by the second object launched from the second application to be visible within a given period of time. It may be useful to obscure the content embodied by the first object launched from the application. In some embodiments, a scene graph is an intermediate data structure that sits between an application containing 3D data and a renderer for rendering that 3D data for presentation to a screen, and includes several In embodiments, an application writes scene information to a scene graph, which may later be used by a renderer to render a scene or output a scene to be displayed.

FIG. 2A shows an example flow of data in an example computer system 200. In system 200, a single application 210 can write data to scene graph 240 that renderer 250 can use to render objects 220 for presentation on display 290. For example, object 220 may include a number of objects (e.g., polygons) that together comprise 3D representations of two human hands, namely hand 232 and hand 236, and the application may, for example, During the display, the object 220 may be directed to be presented on the display from the perspective of a viewing origin that is oriented along the viewing axis. In the example, hand 232 and hand 236 are interlocked in a handshake, and because of the relative positioning of the hands, the viewer may notice that some parts of hand 232 are part of hand 236 relative to the viewing origin and visual axis. , and expect some polygons that make up hand 236 to occlude parts of hand 232. The application 210 identifies polygons between the polygons that make up the object 220 that can be used to identify polygons that should occlude other polygons, i.e., should be sorted to display on top of others. Information describing the relationships between objects 220, such as the spatial relationships of objects 220, can be written to scene graph 240. For example, scene graph 240 shows that polygon 234 (belonging to hand 232) is located between the viewing origin and the polygons that make up hand 236, and should therefore occlude those polygons in hand 236, and polygon 238 (belonging to hand 236) is located between the viewing origin and the polygon comprising hand 232, and may therefore reflect that it should occlude those polygons in hand 232. Renderer 250 may then output a subset, such as only object 220 or hand 232 or an unoccluded portion of hand 232 or only hand 236, as output to display 290 consistent with the desired occlusion for presentation.

FIG. 2B shows an example output of the renderer 250 of the example computer system 200 shown in FIG. 2A. In the example described above with respect to FIG. 2A, based on the relative positions of objects 220, the viewer may notice that some objects (such as polygons 234) that make up hand 232 occlude hand 236 and that We expect some objects (such as polygon 238) to occlude hand 232. The example output shown in FIG. 2B is consistent with expected occlusion. That is, object 220 of FIG. 2A is correctly displayed to present a handshake on display 290 that is consistent with the viewer's expectations.

In the exemplary computer system 200 shown in FIGS. 2A and 2B, scene graph 240 is written directly to application 210 by only a single application. Renderer 250 then traverses scene graph 240 and renders hand 232 and hand 236 with appropriate occlusion. Conventional systems that use a scene graph as part of a rendering process, such as example computer system 200, correctly render objects when the scene graph (e.g., scene graph 240) receives direct input from multiple independent applications. It may not be occluded. In these situations, unlike example computer system 200, there is no single application that can provide a scene graph with all of the object-related data that may be needed to properly sort objects on the display.

FIG. 2C shows an example flow of data in example computer system 201 using two independent applications and illustrates the occlusion problem described above. Unlike the example computer system 200 described above, the example computer system 201 includes two independent applications: application 212 and application 214. In example computer system 201, application 212 and application 214 both write data to scene graph 240 and render their separate 3D data to a single display 290. In FIG. 2C, application 212 attempts to render and present objects 222 (including the objects that make up hand 232), and application 214 attempts to render and present objects 224 (including the objects that make up hand 236). try. In this example, when hand 232 and hand 236 are displayed side by side within the same 3D environment, as in the example illustrated in FIGS. 2A and 2B, the viewer may notice that part of hand 232 is 236 and would be interlocked in a handshake, as one would expect parts of hand 236 to occlude hand 232.

The embodiment shown in FIG. 2C may have difficulty achieving realistic occlusion of objects to be rendered. In an example, application 212 may write data corresponding to object 222 (including hand 232) to scene graph 240, and application 214 may write data corresponding to object 224 (including hand 236) to scene graph 240 in the same scene. Graph 240 can be filled in. However, in example computer system 201, if application 212 and application 214 are independent ("sandboxed") applications, then application 212 uses application 214's objects 224 (including hand 236 and its constituent objects) to Similarly, application 214 cannot access data associated with application 212's objects 222 (including hand 232 and its constituent objects). That is, in some embodiments, neither application 212 nor application 214 can completely identify the relationship between object 222 and object 224. Accordingly, neither application 212 nor application 214 writes information to scene graph 240 that may be necessary to identify objects that occlude other objects, or the order in which objects should be sorted on the display. I can't.

FIG. 2D shows an example output of renderer 250 of example computer system 201 shown in FIG. 2C. In the example described above with respect to FIG. 2C, based on the relative positioning of objects 222 and 224, the viewer may notice that some portions of hand 232 occlude hand 236 and that some portions of hand 236 occlude hand 236. Expect to occlude hand 232. However, unlike FIGS. 2A and 2B, scene graph 240 in FIG. C is unable to correctly sort objects 222 and 224 and produce the desired occlusion, as explained above. Instead, in the example shown, all of objects 222 are sorted onto all of objects 224. The example output shown in FIG. 2D is therefore inconsistent with the expected occlusion. The resulting handshake image therefore does not accurately reflect the objects in application 212 and application 214, and the image may additionally appear unnatural and confusing to the viewer.

Other disadvantages of conventional scene graphs, such as scene graph 240 of FIGS. 2A-2D, are evident when used in conjunction with multiple independent applications, such as in example computer system 201. For example, rendering efficiency can be achieved using data corresponding to the entire scene to be rendered, such as in application 210 of FIG. 2A. For example, by knowing which surfaces will be occluded, the example computer system 200 of FIG. 2A instructs the system 200 to cull those surfaces, thereby avoiding unnecessary expenditure of computational resources. can be avoided. This culling may not be possible in a multi-application system, such as the example computer system 201 of FIG. 2C, because each application may not possess scene knowledge to determine which surfaces should be culled. . Additionally, in some implementations involving only a single application, such as example computer system 200, beneficial effects may be applied to objects based on the presence of other objects in the scene. For example, applying realistic lighting and shading effects to an object may require data corresponding to nearby objects. Additionally, some shader effects benefit from such data. Similarly, effects generated by particle systems or collision detection systems may benefit from such data. Such effects may occur if the 3D data is provided by multiple independent applications, as no single application may be able to provide all of the node relationship information necessary to apply such effects. may be limited or impossible in certain systems.

This disclosure presents systems and methods that use a unified scene graph and address the above disadvantages of systems that render 3D data from multiple independent applications. A unified scene graph may be used in place of a traditional scene graph, such as scene graph 240 of FIG. 2C, in a system (such as example computer system 201 of FIG. 2C) where multiple independent applications provide 3D data to be rendered. can be done. As described herein, in some embodiments, a unified scene graph receives 3D data from a plurality of individual input sources, writes information corresponding to the 3D data to a central location, and writes information corresponding to the 3D data to a central location. A system may be included for maintaining that information for access by a renderer that creates a rendered scene comprising objects based on the 3D data. The rendered scene has realistic object occlusion, computational efficiency, visual effects (such as lighting and shading), or physics that would otherwise be difficult or impossible to achieve in systems that do not utilize a unified scene graph. It may be used to generate visual effects (such as collision detection) or output (such as graphical output) that reflects a partial representation of the occluded object.

In some examples, an example computer system includes multiple applications, each application including 3D data representing one or more objects within a common 3D environment. Multiple applications may each exist within a "sandboxed" environment such that this remains independent of other applications; for example, each individual application's data is shared with each other application. While each application's 3D data may correspond to the same 3D environment, each application may not have access to the data of each other application. , maintains its own instance of the 3D environment. For example, each application may represent players in an online multiplayer video game, where each player exists within the same game world instance or 3D environment, but lacks direct access to other players' data. . In such embodiments, it may be desirable for all players to be rendered simultaneously in a single instance of the game world, but for each player to render the 3D data of each other client participant. Maintaining the necessary information may be undesirable (or computationally prohibitive). Additionally, for security purposes, it may be desirable to limit the information of a player that is available to other players.

In some embodiments, multiple sandboxed applications can each independently write information corresponding to their 3D data to a local scene graph, which information is later written to a common centralized scene graph. written. The unified scene graph can then be traversed by a renderer to render the scene for presentation on a display as an image based on the collective 3D data provided by each application. By communicating 3D data from each of multiple sandboxed applications into a single, unified scene graph, the renderer can eliminate occlusion, requiring or benefiting from simultaneous knowledge of all applications' 3D data. , lighting effects, and rendering optimization (such as surface culling) can be applied. These benefits are achieved while limiting the required computational overhead for each sandboxed application; from a single application perspective, all an application needs to do is update a single scene graph. , only reflecting that 3D data; other operations are performed by other components of the system. Additionally, security benefits can be obtained by maintaining separation between sandboxed applications.

FIG. 3A illustrates components of an example computer system 300 that may render 3D data from multiple independent applications to a display using a unified scene graph. The illustrated embodiment utilizes a client-server topology, however, the present disclosure is not limited to client-server implementations. In example computer system 300, first client application 310 and second client application 320 each communicate 3D data (in some embodiments, via a network) to client-server interface 330. In some embodiments, client applications 310 and 320 are "sandboxed" applications that operate independently of each other and independently communicate their 3D data to client-server interface 330. Client-server interface 330 may receive updated 3D data from client applications 310 and 320 and communicate the 3D data (in some embodiments, via a network) to server-side host application 340. can. In some embodiments, client-server interface 330 uses multi-threading techniques and uses multiple processor threads to receive, process, and/or communicate 3D data to host application 340. In some embodiments, the client-server interface includes logic to control (such as by throttling) the rate at which 3D data is communicated to host application 340. Host application 340 uses the 3D data received from the client-server interface to update unified scene graph 350 so that unified scene graph 350 reflects the 3D data received from client applications 310 and 320. Can be done. In some embodiments, unified scene graph 350 comprises multiple versions of the scene graph, and known versioning techniques are used to allow updates to unified scene graph 350 to occur in parallel. . Renderer 360 then traverses unified scene graph 350, applies optimizations and effects as appropriate, and outputs the output for display on display 370, such as a computer monitor (e.g., at least one of client applications 310 and 320). (a graphical output comprising only the occluded portions of one client application without any occluded application data).

FIG. 3B illustrates aspects of an example client application 310 for the example computer system 300 shown in FIG. 3A. In the example shown, 3D data 312 represents graphical objects (geometric primitives, eg, polygons, etc.) within a 3D environment that are to be presented on display 370. 3D data 312 may be updated (314) by client application 310. For example, if the client application 310 is an application with a rendering loop that iterates 60 times per second, the client application 310 updates the 3D data 312 60 times per second, which should be reflected in the rendered output. may reflect changes in its data during the course of its operation. In some examples, 3D data 312 is represented as a local scene graph 316, which may be local to each client application 310. In some examples, local scene graph 316 may include data (such as nodes) that corresponds to data in unified scene graph 350. When 3D data 312 is updated 314, client application 310 can update local scene graph 316 to reflect the most recent version of 3D data 312. When local scene graph 316 is updated, it can be used by client application 310 to generate 317 client data 318. In some examples, client data 318 may represent local scene graph 316 in its entirety. In some embodiments, client data 318 may represent changes made to local scene graph 316 since the previous client data 318 was sent to client-server interface 330. For example, client data 318 may include nodes added to or deleted from local scene graph 316, changes in relationships between nodes within local scene graph 316, or changes in the nature of nodes within local scene graph 316. may be included. In some embodiments, client data 318 uses identifiers, such as identification numbers, that correspond to scene graph nodes to determine the relationship between data from local scene graph 316 and corresponding data on centralized scene graph 350. May be identified. Client data 318 can then be communicated to client-server interface 330 for eventual communication to host application 340. In some embodiments, communication of client data 318 to client-server interface 330 may occur via a network. In some embodiments, a client helper application may be used in conjunction with client application 310 to generate client data 318 from local scene graph 316 or from 3D data 312.

Aspects described with respect to client application 310 may similarly describe client application 320 or other client applications (along with client application 310) comprising example computer system 300. The systems and methods described herein may be extended to include any number of client applications and client data, and this disclosure is not limited to any such number, and further provides several benefits. It will be appreciated by those skilled in the art that (eg, improved computational efficiency) may become more apparent with an increasing number of client applications. As explained above, client applications 310 and 320 may be sandboxed applications that do not share data or functionality. For example, in example computer system 300, client application 320 may have its own 3D data and local scene graph independent of 3D data 312 and local scene graph 316 belonging to client application 310. However, in some implementations, including example computer system 300, a single client-server interface 300 is shared by multiple client applications, such as client applications 310 and 320.

FIG. 3C illustrates aspects of an example client-server interface 330 for the example computer system 300 shown in FIGS. 3A and 3B. In an example, client data 318 and client data 328 are client data that is communicated to or updated by individual client applications 310 and 320, as described above with respect to FIG. 3B. In some embodiments, client data 318 and 328 may be updated on client-server interface 330 at different rates. This means that, for example, if one client application runs on less powerful computing hardware than another client application (causing the client application to update its client data less frequently), one When a client application communicates with the client-server interface 330 over a network with lower bandwidth than another client application, or when client data associated with one client application is associated with another client application. (and requires more processing time to occur). Different rates of updating client data on the client-server interface 330 may also be the result of temporary sudden changes in operating conditions, such as when a network failure temporarily takes the client application offline. For example, it may be desirable for computer system 300 to tolerate different rates of updating client data, such that, for example, a network failure affecting one client application can result in a centralized scene graph using client data from other client applications. 350 or the rate at which the scene is rendered. Also, when updating the unified scene graph 350, client data from one client application is updated to the other client's It may be desirable to ensure that client data from an application is not delayed too far or too far ahead of it.

In some embodiments, the role of the client-server interface 330 is to handle differences or increases or decreases in the rate of updating client data. Referring to FIG. 3C, example client-server interface 330 may receive client data 318 and 328 via independent processing threads 332 and 334, respectively, and may include a thread manager 336 for handling the threads. . Utilizing multiple threads to update client data from different sources, such as client application 310 and client application 320, ensures that problems with one source do not block or otherwise adversely affect data from other sources. It is possible to prevent this from happening. In the example shown, thread manager 336 inputs client data 318 and 328 from client applications 310 and 320 using threads 332 and 334, respectively, and host data 319 and 329 (respectively, client data 318 and 328). , corresponding to client applications 310 and 320 and threads 332 and 334) may be output to host application 340. Thread manager 336 is configured to process threads 332 and 334, identify and address throughput problems or other problems associated with threads 332 and 334, and/or control output of host data 319 and 329. May contain logic. For example, if client data 318 and client data 328 are updating at approximately the same rate (via threads 332 and 334, respectively), thread manager 336 simply updates host data 319 and 329 (corresponding to client data 318 and 319, respectively) may be updated. However, if client data 318 is updated at a much faster rate than client data 328, thread manager 336 may throttle client data 318 (e.g., by communicating this to host application 340 at a lower frequency). However, this may be prevented from exceeding client data 328 by much. Thread manager 336 may also control the overall rate at which host data is updated. For example, thread manager 336 may throttle the rate at which host data 319 and/or 329 is updated so that the host data is updated faster than host application 340 can process it (client application 310 and/or 329 , which may result in undesirable desynchronization of the output to the unified scene graph 350 and/or the display 370).

FIG. 3D illustrates aspects of an example host application 340 for the example computer system 300 shown in FIGS. 3A-3C. Described herein are operations performed within thread 341 within host application 340, such that thread 341 may be executed in parallel with additional threads within host application 340. In some embodiments, multi-threaded processing within host application 340 allows multiple client applications or multiple sets of host data to (in some embodiments, by updating different versions of a centralized scene graph) It may have the advantage of allowing the same unified scene graph 350 to be updated simultaneously. This, in turn, may increase the overall throughput of client data into rendered scenes for presentation on a display. In some embodiments, multi-threading may require that locks be placed on the centralized scene graph data, for example, to prevent threads from inadvertently writing to the same data. However, in some embodiments, one or more of the described operations may not be performed within a thread.

In the example shown in FIG. 3D, host data 319 (corresponding to client application 310 and client data 318) is updated (342) as described above with respect to FIG. 3C. Host application 340 may then identify changes that host data 319 may make to the previous version of unified scene graph 350. For example, host application 340 may indicate that host data 319 would add a node, delete a node, change the relationship between two nodes, or change the nature of a node with respect to unified scene graph 350. May be identified. (In some embodiments, such as the embodiment shown in FIG. 3D, host application 340 may use host data handler 344 to perform these operations or others). Host application 340 may identify a version of unified scene graph 350 to be created or updated according to host data 319 (352). In some embodiments, prior to writing to version 352 of centralized scene graph 350, host application 340 may lock the version to prevent other processes from modifying it in parallel. . Host application 340 may make changes to version 352 to reflect host data 319 (eg, by adding or deleting scene graph nodes in version 352 to correspond to host data 319). In some examples, host application 340 may then unlock version 352 and update the value of the version number corresponding to version 352 (356). Host application 340 may then update 342 the host data and repeat the process shown in FIG. 3D. When the unified scene graph 350 is updated to reflect individual host data derived from individual client applications, the unified scene graph 350 indicates that the individual client applications are "sandboxed" and independent of each other. Even if obtained, it will reflect collective host data from multiple client applications.

FIG. 3E illustrates aspects of an example renderer 360 for the example computer system 300 shown in FIGS. 3A-3D. In some embodiments, renderer 360 constitutes part of host application 340. In some examples, renderer 360 may be part of another component of example computer system 300 or may be a separate component or application. In some embodiments, renderer 360 may be implemented in different physical hardware than one or more components of example computer system 300, and may be implemented in physical hardware that is different from one or more components of example computer system 300, and may communicate with one or more of those components via a network. May communicate with things.

In the example shown in FIG. 3E, renderer 360 operates on version 352 of unified scene graph 350. In the example, the role of the renderer is to create a rendered scene comprising data, such as output or graphical output, for presentation on display 370 based on version 352 of unified scene graph 350. As part of this process, renderer 360 may traverse version 352 (362) using known scene graph traversal techniques. During or after traversal 362, renderer 360 may optionally update 364 unified scene graph 350 to reflect the results of the traversal. For example, as part of traverse 362, renderer 360 may identify abandoned nodes that should be removed from unified scene graph 350. Following traversal 362 and/or update 364, renderer 360 may apply various optimizations 366 to the scene. For example, renderer 360 may cull obscure or invisible surfaces to avoid consuming unnecessary computational resources. Following traversal 362 and/or update 364, renderer 360 may apply one or more visual effects 367 to the scene. For example, in some examples, renderer 360 may apply lighting or shadow effects, apply one or more shaders, apply particle effects, and/or apply physical effects. Finally, renderer 360 can output data to a graphical output pipeline, and the results can display output on display 370.

The above exemplary processes of a computer system may be provided by any suitable logic circuitry. Suitable logic circuits may include one or more computer processors (eg, CPU, GPU, etc.) that, when executing instructions implemented in a software program, implement the process. In addition, such processes may also be implemented in hardware logic circuits, such as programmable logic (e.g., PLDs, FPGAs, etc.) or customized logic (e.g., ASICs, etc.) that implement the logic design providing the process. can be provided through a corresponding logic design. Furthermore, such processes can be provided through implementations that combine one or more processors of both software and hardware logic circuits.

FIG. 4 illustrates an example system 400 that may be used to implement any or all of the above embodiments. The above embodiments (in whole or in part) may be used in any portable (including wearable) or non-portable device, such as a communication device (e.g., mobile phone, smart phone), multimedia device (e.g., MP3 player). , TV, radio), portable or handheld computers (e.g., tablets, netbooks, laptops), desktop computers, integrated desktops, peripheral devices, head-mounted devices (e.g., which may include integrated displays); may be embodied in any other system or device compatible with inclusion in example system architecture 400, including combinations of two or more of the types of devices. The embodiments described above may be implemented in two or more physically separate devices, such as two or more computers, communicating via a wireless network. The above embodiments may be embodied in two or more physically distinct devices, such as belt packs, that communicate data to and/or from a head-mounted display. FIG. 4 generally depicts a system that includes one or more computer readable media 401, a processing system 404, an I/O subsystem 406, radio frequency (RF) circuitry 408, audio circuitry 410, and sensor circuitry 411. 400 is a block diagram of one embodiment of the 400. FIG. These components may be coupled by one or more communication buses or signal lines 403.

It should be clear that the architecture shown in FIG. 4 is only one example architecture of system 400, and that system 400 may have more or fewer components, or a different configuration of components, than those shown. . The various components shown in FIG. 4 may be implemented in hardware, software, firmware, or any combination thereof, including one or more signal processing and/or application specific integrated circuits.

Referring to example system architecture 400 of FIG. 4, RF circuitry 408 can be used to transmit and receive information via a wireless link or network to one or more other devices; Contains well-known circuitry for performing its functions. RF circuitry 408 and audio circuitry 410 may be coupled to processing system 404 via peripheral interface 416. Interface 416 may include various known components for establishing and maintaining communications between peripherals and processing system 404. Audio circuitry 410 can be coupled to an audio speaker 450 and a microphone 452 to process audio signals received from interface 416 and to perform any known method for processing audio signals received from interface 416 to enable users to communicate with other users in real time. Can include circuitry. In some examples, audio circuit 410 can include a headphone jack (not shown).

Sensor circuit 411 may include, but is not limited to, one or more light emitting diodes (LEDs) or other light emitters, one or more photodiodes or other light sensors, one or more photothermal sensors, magnetometers, accelerometers, gyroscope, barometer, compass, proximity sensor, camera, ambient light sensor, thermometer, GPS sensor, electro-oculography (EOG) sensor, and remaining battery life, power consumption, processor speed, CPU load, and equivalent It can be coupled to a variety of sensors, including various system sensors that can sense objects. In such embodiments involving head-mounted devices, one or more sensors may be associated with the user's eyes, such as tracking the user's eye movements or identifying the user based on images of the user's eyes. It may also be employed in conjunction with functionality.

Peripheral interface 416 may couple input and output peripherals of the system to processor 418 and computer readable media 401. One or more processors 418 may communicate with one or more computer-readable media 401 via controller 44 . Computer-readable medium 401 may be any device or medium (excluding signals) that can store code and/or data for use by one or more processors 418. In some examples, media 401 can be a non-transitory computer readable storage medium. Media 401 can include a memory hierarchy, including, but not limited to, cache, primary memory, and secondary memory. The memory hierarchy includes magnetic and/or optical storage such as RAM (e.g. SRAM, DRAM, DDRAM), ROM, FLASH, disk drives, magnetic tape, CD (compact disc), and DVD (digital video disc). Can be implemented using any combination of devices. Media 401 may also include transmission media for carrying information-bearing signals indicative of computer instructions or data (excluding the signal and excluding the carrier wave on which the signal is modulated). For example, transmission media may include, but are not limited to, the Internet (also referred to as the World Wide Web), an intranet, a local area network (LAN), a wide local area network (WLAN), a storage area network (SAN), a metropolitan area network. (MAN), and the like.

One or more processors 418 may launch various software components stored within media 401 to perform various functions for system 400. In some embodiments, the software components include an operating system 422, a communications module (or set of instructions) 424, an I/O processing module (or set of instructions) 426, and a graphics module (or set of instructions). 428 and one or more applications (or sets of instructions) 430. Each of these modules and the applications described above performs one or more of the functions described above and the methods described herein (e.g., computer-implemented methods and other information processing methods described herein). can correspond to a set of instructions for These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules; therefore, various subsets of these modules may be combined or combined in various embodiments. They may be rearranged differently. In some embodiments, medium 401 may store a subset of the modules and data structures identified above. Additionally, media 401 may store additional modules and data structures not described above.

Operating system 422 may include various procedures, sets of instructions, software components, and/or drivers for controlling and managing common system tasks (e.g., memory management, storage device control, power management, etc.). , facilitate communication between various hardware and software components.

Communication module 424 can facilitate communication with other devices via one or more external ports 436 or via RF circuitry 408 and receive signals from RF circuitry 408 and/or external port 436. It can include various software components for handling data.

Graphics module 428 may include various known software components for rendering, animating, and displaying graphical objects on one or more display surfaces. The display surface may include a 2D or 3D display. A display surface may be coupled directly or indirectly to one or more components of example system 400. In embodiments involving touch-sensitive displays (eg, touch screens), graphics module 428 may include components for rendering, displaying, and animating objects on the touch-sensitive display. In some examples, graphics module 428 may include components for rendering to a remote display. In some embodiments, such as those that incorporate a camera, the graphics module 428 includes rendered graphical objects and camera data (such as those captured from a head-mounted camera) or photographic data (such as images captured by a satellite). etc.) for creating and/or displaying an image formed by compositing images. In some examples, the graphics module may include components for rendering images to a head-mounted display device. In some examples, the image includes a view of an element of virtual content (e.g., an object in a three-dimensional virtual environment) and/or a view of the physical world (e.g., camera input showing a user's physical surroundings). But that's fine. In some examples, a display may present a composite image of virtual content and a view of the physical world. In some examples, the view of the physical world may be a rendered image, and in some examples, the view of the physical world may be an image from a camera.

The one or more applications 430 may include, but are not limited to, browsers, address books, contact lists, email, instant messaging, word processing, keyboard emulation, widgets, JAVA-enabled applications, encryption, digital rights. Any application installed on the system 400 may include management, voice recognition, voice replication, location capabilities (such as those provided by the Global Positioning System (GPS)), music players, and the like.

I/O subsystem 406 may be coupled to an ocular I/O device 412 and one or more other I/O devices 414 for controlling or performing various functions. For example, ocular I/O device 412 may include various components for processing ocular input (e.g., sensors for eye tracking) or user gesture input (e.g., optical sensors). 432 can communicate with processing system 404 . One or more other input controllers 434 can send electrical signals to and receive electrical signals from other I/O devices 414. Other I/O devices 414 may include physical buttons, dials, slider switches, sticks, keyboards, touch pads, additional display screens, or any combination thereof.

I/O processing module 426 receives information from, but is not limited to, ocular I/O device 412 via ocular I/O device controller 432 or from other I/O devices 414 via I/O controller 434. Various software components may be included to perform various tasks associated with ocular I/O device 412 and/or other I/O devices 414, including receiving and processing input, including receiving and processing input. In some examples, I/O device 414 and/or I/O processing module 426 may perform various tasks associated with gestural input, which may be provided by tactile or non-tactile means. In some examples, gesture input may be provided by a camera or another sensor to detect movement of the user's eyes, arms, hands, and/or fingers, for example. In some embodiments, I/O device 414 and/or I/O processing module 426 may be configured to identify an object on a display with which a user desires to interact, e.g., a GUI element at which a user is pointing. may be configured. In some embodiments, eye I/O device 412 and/or I/O processing module 426 may perform eye tracking tasks, such as identifying an object or area on a display that a user is looking at ( (e.g., with the aid of optical or EOG sensors). In some embodiments, a device (such as a hardware "beacon") performs operations on the touch I/O device 412 and/or I/O processing module 426, such as identifying the location of a user's hand relative to a 2D or 3D environment. It may be worn or held by the user to assist with gesture-related tasks. In some examples, eye I/O device 412 and/or I/O processing module 426 are configured to identify a user based on sensor input, such as data from a camera sensor associated with the user's eye. may be done.

In some examples, graphics module 428 can display visual output to a user within a GUI. Visual output may include text, graphics, video, and any combination thereof. Some or all of the visual output may correspond to user interface objects. In some examples, I/O devices 412 and/or 414 and/or controllers 432 and/or 434 (along with any associated modules and/or sets of instructions in medium 401) can perform gesture and/or eye movements. Movement can be detected and tracked, and detected gestures and/or eye movements can be translated into interactions with graphical objects, such as one or more user interface objects. In embodiments where ocular I/O device 412 and/or ocular I/O device controller 432 are configured to track the user's eye movements, the user interacts directly with graphical objects by viewing them. be able to.

Feedback may be provided, such as by ocular I/O device 412 or another I/O device 414, based on the content being displayed and/or the state or states of the computing system. Feedback can be optical (e.g., light signals or displayed images), mechanical (e.g., tactile feedback, touch feedback, force feedback, or the like), electrical (e.g., electrical stimulation), olfactory, It may be transmitted acoustically (eg, beeps or the like), or the like, or any combination thereof, and in a variable or non-variable manner.

System 400 may also include a power system 444 for powering various hardware components, including a power management system, one or more power sources, a recharging system, a power outage detection circuit, a power converter or inverter, and a power status indicator. , and any other components typically associated with power generation, management, and distribution in portable devices.

In some examples, peripheral interface 416, one or more processors 418, and memory controller 420 may be implemented on a single chip, such as processing system 404. In some other examples, they may be implemented on separate chips.

In some embodiments, methods are disclosed. The method includes receiving first graphical data comprising a plurality of first nodes from a first client application of the computer system; and receiving first graphical data comprising a plurality of second nodes from a second client application of the computer system. and generating a scene graph, the scene graph describing a relationship, such as an occlusion relationship, between the first node and the second node. The scene graph, when traversed by a processor of the computer system, is configured to render a scene comprising nodes. In addition to, or in the alternative to, one or more of the above embodiments, the method may further include traversing the scene graph by a processor of the computer system. In addition to or in the alternative to one or more of the above embodiments, the computer system may be configured to communicate with a display, and the method further includes It may include presenting an output comprising at least one node that is not occluded by another node. In some embodiments, occlusion is the visual occlusion of one node by another when viewing a rendered scene of an object from a given perspective. In addition to, or in the alternative to, one or more of the above embodiments, the computer system may be configured to communicate with a display, and the method further comprises: displaying the output on the display, such as by displaying the rendered scene, or by displaying only those nodes of the first or second plurality of nodes that are not occluded, while not displaying other nodes; But that's fine. For example, if a second plurality of nodes occludes a portion of the first plurality of nodes, the displayed output will be the first plurality of occluded nodes that do not display any of the second plurality of nodes. may be the only node that is not In addition to or in the alternative to one or more of the above embodiments, the method may further include applying the optimization to the output at the computer system. In addition to or as an alternative to one or more of the above examples, applying the optimization may include culling the surface. In addition to, or in the alternative to, one or more of the above embodiments, the method may further include applying a visual effect to the output at the computer system. In addition to, or in the alternative to, one or more of the above examples, applying a visual effect may include calculating a light intensity value. In addition to, or in the alternative to, one or more of the above examples, applying a visual effect may include running a shader. In addition to, or in the alternative to, one or more of the above embodiments, the method may further include applying a physical effect to the output at the computer system. In addition to, or as an alternative to, one or more of the above examples, applying a physical effect may include detecting a collision. In addition to, or in the alternative to, one or more of the above embodiments, the first client application may be a first application running on a computer system, and the first client application may be a first application running on a computer system; may be a second application running on the computer system, and the first client application may be sandboxed on the computer system to the second client application. In addition to or in the alternative to one or more of the above embodiments, the first graphical data may correspond to a first client scene graph associated with the first client application; The graphical data of 2 may correspond to a second client scene graph associated with a second client application, the first client scene graph being sandboxed on the computer system with respect to the second client scene graph. The first client scene graph may be sandboxed on the computer system relative to the scene graph, and the second client scene graph may be sandboxed on the computer system relative to the scene graph. may be converted into In addition to or in the alternative to one or more of the above examples, a scene graph may correspond to versions of a versioned scene graph. In addition to, or in the alternative to, one or more of the above embodiments, the first graphical data may be communicated to the scene graph using a first processing thread of the computer system; The second graphical data may be communicated to the scene graph using a second processing thread of the computer system that is independent from the first processing thread.

In some embodiments, methods are disclosed. The method includes traversing a scene graph of a computer system with a display, the scene graph comprising first 3D data associated with a first application, the first 3D data including one or more nodes, the scene graph comprises second 3D data associated with the second application, the second 3D data comprises one or more nodes, and the first application is associated with the second application. a scene graph comprising a relationship between nodes of the first 3D data and nodes of the second 3D data; and an image corresponding to the scene graph on the display. Displaying, the image corresponding to an output of traversing the scene graph, the image reflecting either a partial representation or a complete representation of the relationship, may include. In addition to, or in the alternative to, one or more of the above examples, the relationship may be a spatial relationship. In addition to, or in the alternative to, one or more of the above embodiments, the method may further include, at the computer system, applying an optimization to the output of traversing the scene graph. . In addition to or as an alternative to one or more of the above examples, applying the optimization may include culling the surface. In addition to, or in the alternative to, one or more of the above embodiments, the method may further include applying, at the computer system, a visual effect to the output of traversing the scene graph. . In addition to, or in the alternative to, one or more of the above examples, applying a visual effect may include calculating a light intensity value. In addition to, or in the alternative to, one or more of the above examples, applying a visual effect may include running a shader. In addition to or in the alternative to one or more of the above embodiments, the method may further include applying, at the computer system, a physical effect to the output of traversing the scene graph. good. In addition to, or as an alternative to, one or more of the above examples, applying a physical effect may include detecting a collision. In addition to or as an alternative to one or more of the above examples, a scene graph may correspond to versions of a versioned scene graph. In addition to or in the alternative to one or more of the above embodiments, graphical data corresponding to the first 3D data may be communicated to the scene graph by a host application running on the computer system. good. In addition to or in the alternative to one or more of the above embodiments, graphical data corresponding to the first 3D data may be communicated to the host application by a client of the host application. In addition to or in the alternative to one or more of the above embodiments, first graphical data corresponding to the first 3D data is generated by a host application using a scene graph using a first processing thread. and second graphical data corresponding to the second 3D data may be communicated to the scene graph by a host application using a second processing thread that is independent from the first processing thread. .

In some embodiments, a computer system is disclosed. The system includes one or more processors and stores instructions that, when executed by the one or more processors, cause the one or more processors to perform one or more of the methods described above. and a memory.

In some embodiments, a non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium, when executed by the one or more processors, transmits data from a first client application of the computer system to the one or more processors, comprising a plurality of first nodes. receiving graphical data; and generating a scene graph, the second graphical data comprising a plurality of second nodes from a second client application of the computer system; describes a relationship between a first node and a second node, the scene graph is configured to render a scene based on the occlusion relationship when traversed by a processor of the computer system; One or more of the first or second nodes within the first or second plurality of nodes may store instructions that occlude the other or cause the method to be performed. In some embodiments, occlusion is the visual occlusion of one node by another when viewing a rendered scene of an object from a given perspective. In addition to, or in the alternative to, one or more of the above embodiments, the method may further include traversing the scene graph by a processor of the computer system. In addition to, or in the alternative to, one or more of the above embodiments, the computer system may be configured to communicate with the display, and the method further provides for the display to be present on the scene graph. or by displaying only those nodes of the first or second plurality of nodes that are not occluded while not displaying other nodes, etc. May include. For example, if a second plurality of nodes occludes a portion of the first plurality of nodes, the displayed output will be the first plurality of occluded nodes that do not display any of the second plurality of nodes. may be the only node that is not In addition to or in the alternative to one or more of the above embodiments, the method may further include applying the optimization to the output at the computer system. In addition to or as an alternative to one or more of the above examples, applying the optimization may include culling the surface. In addition to, or in the alternative to, one or more of the above embodiments, the method may further include applying a visual effect to the output at the computer system. In addition to, or in the alternative to, one or more of the above examples, applying a visual effect may include calculating a light intensity value. In addition to, or in the alternative to, one or more of the above examples, applying a visual effect may include running a shader. In addition to, or in the alternative to, one or more of the above embodiments, the method may further include applying a physical effect to the output at the computer system. In addition to, or as an alternative to, one or more of the above examples, applying a physical effect may include detecting a collision. In addition to, or in the alternative to, one or more of the above embodiments, the first client application may be a first application running on a computer system, and the first client application may be a first application running on a computer system; may be a second application running on the computer system, and the first client application may be sandboxed on the computer system to the second client application. In addition to or in the alternative to one or more of the above examples, the first graphical data may correspond to a first client scene graph associated with the first client application; The graphical data of 2 may correspond to a second client scene graph associated with a second client application, the first client scene graph being sandboxed on the computer system with respect to the second client scene graph. The first client scene graph may be sandboxed on the computer system relative to the scene graph, and the second client scene graph may be sandboxed on the computer system relative to the scene graph. may be converted into In addition to or in the alternative to one or more of the above examples, a scene graph may correspond to versions of a versioned scene graph. In addition to, or in the alternative to, one or more of the above embodiments, the first graphical data may be communicated to the scene graph using a first processing thread of the computer system; The second graphical data may be communicated to the scene graph using a second processing thread of the computer system that is independent from the first processing thread.

In some embodiments, a non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium, when executed by one or more processors, causes the one or more processors to traverse a scene graph of a computer system with a display, the scene graph comprising a first the scene graph comprises first 3D data associated with the second application, the first 3D data comprising one or more nodes, the scene graph comprising second 3D data associated with the second application, the first 3D data comprising one or more nodes; The 3D data comprises one or more nodes, the first application is sandboxed on the computer system to the second application, and the scene graph comprises the nodes of the first 3D data and the second 3D data. and displaying an image corresponding to the scene graph on the display, the image corresponding to the output of traversing the scene graph, the image corresponding to the relationship between the nodes of the data, Instructions may be stored for performing the method, including reflecting the method. In addition to, or in the alternative to, one or more of the above examples, the relationship may be a spatial relationship. In addition to, or in the alternative to, one or more of the above embodiments, the method may further include, at the computer system, applying an optimization to the output of traversing the scene graph. . In addition to or as an alternative to one or more of the above examples, applying the optimization may include culling the surface. In addition to, or in the alternative to, one or more of the above embodiments, the method may further include applying, at the computer system, a visual effect to the output of traversing the scene graph. . In addition to, or in the alternative to, one or more of the above examples, applying a visual effect may include calculating a light intensity value. In addition to, or in the alternative to, one or more of the above examples, applying a visual effect may include running a shader. In addition to or in the alternative to one or more of the above embodiments, the method may further include applying, at the computer system, a physical effect to the output of traversing the scene graph. good. In addition to, or as an alternative to, one or more of the above examples, applying a physical effect may include detecting a collision. In addition to or as an alternative to one or more of the above examples, a scene graph may correspond to versions of a versioned scene graph. In addition to or in the alternative to one or more of the above embodiments, graphical data corresponding to the first 3D data may be communicated to the scene graph by a host application running on the computer system. good. In addition to or in the alternative to one or more of the above embodiments, graphical data corresponding to the first 3D data may be communicated to the host application by a client of the host application. In addition to or in the alternative to one or more of the above embodiments, first graphical data corresponding to the first 3D data is generated by a host application using a scene graph using a first processing thread. and second graphical data corresponding to the second 3D data may be communicated to the scene graph by a host application using a second processing thread that is independent from the first processing thread. .

In some embodiments, a computer system is disclosed. The system includes one or more processors, the computer system configured to receive first scene data from a first client application, and the computer system configured to receive second scene data from a second client application. configured with a storage device and, when executed by the one or more processors, causing the one or more processors to generate a graphical data structure based on the first scene data and the second scene data. the graphical data structure, when provided as input to a rendering operation performed by one or more processors, is configured to provide an output corresponding to an image on a display. It may also include a memory for storing instructions for execution. In addition to, or in the alternative to, one or more of the above examples, the graphical data structure may be at least one of a display list and a display tree. In addition to, or in the alternative to, one or more of the above embodiments, the method may further include performing a rendering operation using the graphical data structure as input. In addition to, or in the alternative to, one or more of the above examples, the computer system may further include a display, and the method may further include displaying an image on the display. . In addition to or in the alternative to one or more of the above embodiments, the first client application may be a first application executed by one or more processors of the first device. Often, the second client application may be a second application executed by one or more processors of the first device. In addition to or in the alternative to one or more of the above embodiments, the first client application may be a first application executed by one or more processors of the first device. Often, the second client application may be a second application executed by one or more processors of the second device. In addition to or in the alternative to one or more of the above embodiments, the storage device may be further configured to receive third scene data from a third client application. In addition to or in the alternative to one or more of the above embodiments, the method may further include deleting the first scene data from the storage device. In addition to or in the alternative to one or more of the above embodiments, the graphical data structure may comprise first data and second data, and the method further comprises: based on the first data, in response to determining whether the data corresponds to an occluded view or a non-occluded view; The method may include rendering an image with a non-occluded view and, in response to determining that the first data corresponds to an occluded view, rendering the image without an occluded view. In addition to or in the alternative to one or more of the above embodiments, the storage device is further configured to convert the first scene data into a first version in response to receiving the first scene data. may be configured to be stored within the version control system as a version control system. In addition to or in the alternative to one or more of the above embodiments, the storage device further receives third scene data from the first client application and transfers the third scene data to the second client application. It may be configured to be stored as a version within a version control system. In addition to or in the alternative to one or more of the above embodiments, the method further includes, in response to generating the graphical data structure, deleting the first version from the storage device. May include. In addition to, or in the alternative to, one or more of the above embodiments, the method may be performed in parallel with the storage device receiving the third scene data. In addition to or in the alternative to one or more of the above embodiments, the storage device is configured to receive the first scene data in parallel with receiving the second scene data. You can. In addition to or in the alternative to one or more of the above embodiments, the storage device is configured to receive third scene data at a first interval corresponding to a first data rate. The method may further include adjusting the length of the first interval to correspond to the second data rate. In addition to or in the alternative to one or more of the above embodiments, the first scene data includes at least one of new data, deleted data, and a change in the relationship between the data. may be provided.

In some embodiments, a computer system is disclosed. The computer system may include a server, server data, a first client application, and a second client application, the computer system receiving first raw scene data from the first client application at the server; at the server, receiving second raw scene data from the second client application; and at the server, receiving the first raw scene data from the first client application, the second raw scene data from the second client application; incorporating scene data and server data into a centralized scene data structure, executing at least a portion of the data contained within the centralized scene data structure at the server, and based on the data executed within the centralized scene data structure; It may be configured to create a graphical data structure. In addition to, or as an alternative to, one or more of the above examples, the graphical data structure may be a display list or a display tree. In addition to, or in the alternative to, one or more of the above embodiments, the computer system may further include a rendering engine configured to render the graphical data structure into a processed image. In addition to or in the alternative to one or more of the above embodiments, the computer system may further include a display configured to display the processed image. In addition to, or in the alternative to, one or more of the above examples, the display may be capable of displaying virtual content while maintaining at least a partial view of the physical world. In addition to or in the alternative to one or more of the above embodiments, the first client application and the second client application are two different applications running on a single physical device. There may be. In addition to or in the alternative to one or more of the above embodiments, the first client application and the second client application are two different applications, each running on a separate physical device. There may be. In addition to or in the alternative to one or more of the above embodiments, the server may be configured to receive third raw scene data from a third client application. In addition to or in the alternative to one or more of the above embodiments, the server receives raw scene data from the first client application after executing the raw scene data from the first client application. It may be configured to delete. In addition to, or as an alternative to, one or more of the above embodiments, the rendering engine further includes an occlusion module that divides data within the graphical data structure into a first occluded category and a second occluded category. An occlusion module may be provided that is configured to separate into non-occluded categories and display a second non-occluded category. In addition to or in the alternative to one or more of the above embodiments, the server is configured to store first raw scene data from a first client application as a first version. You can. In addition to or in the alternative to one or more of the above embodiments, the server is configured to store third raw scene data from the first client application as a second version. You can. In addition to or in the alternative to one or more of the above embodiments, a computer system is configured to receive a first version of first raw scene data from a first client application by a server. storing the first version of the first raw scene data from the first client application from time to time until the first raw scene data from the first client application is read and executed; may be configured. In addition to or in the alternative to one or more of the above embodiments, the server receives second raw scene data from the first client at the same time as the server receives the second raw scene data from the second client. The first raw scene data may be configured to receive the first raw scene data. In addition to, or in the alternative to, one or more of the above embodiments, the server may be configured to slow down the rate at which the first client application sends raw scene data to the server. . In addition to or in the alternative to one or more of the above embodiments, the data received from the first and second client applications may include new data, deleted data, previously transferred data. and modified data.

Although the disclosed embodiments have been fully described with reference to the accompanying drawings, it is noted that various changes and modifications will become apparent to those skilled in the art. For example, elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. It is intended that such changes and modifications be included within the scope of the disclosed embodiments as defined by the appended claims.

Claims (17)

A method,
receiving first graphical data comprising a plurality of first nodes from a first client application of the computer system;
receiving second graphical data comprising a plurality of second nodes from a second client application of the computer system;
generating a scene graph;
The scene graph defines a first occlusion relationship between at least one first node of the plurality of first nodes and at least one second node of the plurality of second nodes. describe,
The scene graph defines a second occlusion relationship between at least one third node of the plurality of first nodes and at least one fourth node of the plurality of second nodes. further describe,
The scene graph is configured to create a rendered scene based on the first occlusion relationship, and at least one second node is configured to create a rendered scene based on the first occlusion relationship; occluding at least one first node to occlude a first portion of graphical data of 1;
In said second occlusion relationship, at least one third node occupies at least one third node such that a second portion of said first graphical data occupies a second portion of said second graphical data. How to occlude node 4. The method of claim 1, further comprising traversing the scene graph by a processor of the computer system. 3. The method of claim 2, wherein the computer system is configured to communicate with a display, and the method further includes displaying output on the display. Displaying the output includes displaying at least one first node of the plurality of first nodes and at least one second node of the plurality of second nodes. The method according to claim 3. 3. The method of claim 2, further comprising applying an optimization to the rendered scene at the computer system. 6. The method of claim 5, wherein applying the optimization includes culling a surface. 3. The method of claim 2, further comprising applying visual effects to the rendered scene at the computer system. 8. The method of claim 7, wherein applying the visual effect includes calculating a light intensity value. 8. The method of claim 7, wherein applying the visual effect includes running a shader. 3. The method of claim 2, further comprising applying physical effects to the rendered scene at the computer system. 11. The method of claim 10, wherein applying the physical effect includes detecting a collision. The first client application is a first application running on the computer system, and the second client application is a second application running on the computer system, and the first client application is a second application running on the computer system. 2. The method of claim 1, wherein a client application is sandboxed on the computer system to the second client application. the first graphical data corresponds to a first client scene graph associated with the first client application;
the second graphical data corresponds to a second client scene graph associated with the second client application;
the first client scene graph is sandboxed on the computer system with respect to the second client scene graph;
the first client scene graph is sandboxed on the computer system with respect to the scene graph;
2. The method of claim 1, wherein the second client scene graph is sandboxed on the computer system with respect to the scene graph. 2. The method of claim 1, wherein the scene graph corresponds to a version of a versioned scene graph. The first graphical data is communicated to the scene graph using a first processing thread of the computer system, and the second graphical data is communicated to the scene graph using a first processing thread of the computer system that is independent of the first processing thread. 2. The method of claim 1, wherein the scene graph is communicated using a second processing thread. A scene graph to be updated based on first updated graphical data from a first client application and further updated based on second updated graphical data from a second client application. identifying the
locking the scene graph to the second updated graphical data;
modifying the scene graph using the first updated graphical data;
unlocking the scene graph to the second updated graphical data;
2. The method of claim 1, further comprising: modifying the scene graph using the second updated graphical data. The modified scene graph includes a second node between at least one third node of the plurality of first nodes and at least one fourth node of the plurality of second nodes. Describe the occlusion relationship,
The modified scene graph is configured to create a rendered scene based on the second occlusion relationship, and the at least one third node is configured to create a rendered scene based on the second occlusion relationship; 17. The method of claim 16, wherein four nodes are occluded.

Images