CN114341808A

CN114341808A - Low latency, distributed applications for interactive worlds

Info

Publication number: CN114341808A
Application number: CN202080038381.6A
Authority: CN
Inventors: E·J·埃德尔曼; C·D·斯卡拉布里尼; A·J·比利亚洛沃斯; H·D·希克斯
Original assignee: Amazon Technologies Inc
Current assignee: Amazon Technologies Inc
Priority date: 2019-03-28
Filing date: 2020-03-28
Publication date: 2022-04-12
Also published as: EP3948539A1; WO2020198724A1

Abstract

The technology provides computationally intensive distributed computing applications and systems in a service provider environment that can be used to provide virtual worlds, simulations, virtual environments, games, and other multi-dimensional (e.g., 2D and 3D) virtual environments that can be viewed, interacted with, and accessed by users and other computing services. The distributed computing application may be a large-scale, low-latency distributed application used with a persistent interactive virtual world that simulates or is capable of hosting millions of concurrent users. By partitioning, the techniques may manage computing resources so that applications or application instances may efficiently utilize the CPU core (central processing unit core) of a large hardware instance, and so that developers may efficiently use fewer hardware instances.

Description

Low latency, distributed applications for interactive worlds

Background

Virtualization technologies for computing resources have provided benefits for many customers with different needs with respect to managing large-scale computing resources, and have allowed various different computing resources or computing services to be efficiently and securely shared by multiple customers. For example, virtualization techniques may allow a single physical computing machine to be shared among multiple customers by using a hypervisor to provide each customer with one or more virtualized computing resources (e.g., compute instances and software containers) hosted by the single physical computing machine. In addition, users or clients may access large amounts of dynamic and virtualized computing resources without having to manage the computer hardware on which those resources execute. Virtualized computing resources may be configured to obtain a variety of different additional resources or services via an API that provides access to the resources and services.

Centralized computing resources may be used to create electronic 2D (two-dimensional) virtual environments, 3D (three-dimensional) virtual environments, or multi-dimensional virtual environments such as electronic simulations, electronic worlds, and electronic games, and a large amount of centralized computing resources may be used to host such virtual environments. These virtual environments may be accessed by multiple users over a computer network or the internet. For example, a virtual world, a simulation, or an electronic game may be accessed by multiple users over the internet. Examples of such virtual worlds may be physical simulators, medical simulations, driving simulators, open adventure worlds, first-person shooter games, sports games, strategy games, or Massively Multiplayer Online (MMO) games, among others. Such virtualized worlds can be hosted using a set of virtualized resources in a service provider environment, including distributed applications, virtualized containers, compute instances (i.e., virtual machines), virtualized data stores, virtualized networks, virtualization services, and other virtualized computing resources executing on underlying hardware devices and underlying computer networks.

Virtualization technologies for computing resources have provided benefits for many customers with different needs with respect to managing large-scale computing resources, and have allowed different computing resources or computing services to be efficiently and securely shared by multiple customers. For example, virtualization techniques may allow a single physical computing machine to be shared among multiple customers by using a hypervisor to provide each customer with one or more virtualized computing resources (e.g., compute instances and software containers) hosted by the single physical computing machine. In addition, users or clients may access large amounts of dynamic and virtualized computing resources without having to manage the computer hardware on which those resources execute. Virtualized computing resources may be configured to obtain a variety of different additional resources or services via an API that provides access to the resources and services.

Drawings

FIG. 1 illustrates a system that may include a service provider environment hosting a compute-intensive distributed computing system for providing a virtual environment accessible to users, in accordance with examples of the present technology.

FIG. 2 illustrates different example components included in a service provider environment hosting a compute-intensive distributed computing system for providing a virtual environment accessible to users, in accordance with one example of the present technology.

FIG. 3 is a block diagram illustrating an example computing service for providing a compute-intensive distributed computing system using a distributed computing service manager in accordance with one example of the present technology.

FIG. 4A illustrates a system and related operations for assigning computing units in a distributed computing system using processing partitions organized by object domain, in accordance with examples of the present technology.

FIG. 4B illustrates a graphical example of creating a processing partition using object domains for assignment to hardware hosts in a distributed computing system, in accordance with examples of the present technology.

FIG. 4C illustrates a graphical example of creating a processing partition for assignment to a hardware host in a distributed computing system using object domains and spatial locality, in accordance with examples of the present technology.

FIG. 5 illustrates various example components included in a service provider environment for managing hardware hosts in a distributed computing system using processing partitions organized by object domain, according to one example of the present technology.

6A-6B are flow diagrams illustrating an example method for managing hardware hosts in a distributed computing system using processing partitions organized by object domain in accordance with examples of the present technology.

FIG. 7 is a flow diagram illustrating an example method for processing data objects assigned to hardware hosts using processing partitions organized by an object domain in accordance with examples of the present technology.

FIG. 8 illustrates hardware hosts and related operations for storing and replicating state of data objects in a distributed computing system, in accordance with examples of the present technology.

FIG. 9 illustrates a graphical schematic example of a serialization format for data objects in a distributed computing system for providing low replication cost and thread safe reading in accordance with examples of the present technology.

FIG. 10 is a flow diagram illustrating an example method for managing storage of data objects in a distributed computing system using a serialization format that provides low replication cost and thread-safe reading in accordance with examples of the present technology.

FIG. 11 is a flow diagram illustrating an example method for capturing changes to data objects in a distributed computing system using an ordered, append only log-based format to provide versioned snapshots of states, in accordance with examples of the present technology.

FIG. 12 is a flow diagram illustrating an example method for exchanging data objects in a distributed computing system between hardware hosts using an ordered, append only log-based format to provide versioned snapshots of states, in accordance with examples of the present technology.

FIG. 13 illustrates a system and related operations for assigning computing units in a distributed computing system using processing partitions with data objects organized by spatial locality, in accordance with examples of the present technology.

FIG. 14 illustrates a graphical example of creating a processing partition using spatial location information associated with a data object and assigning the processing partition to a hardware host in a distributed computing system, in accordance with examples of the present technology.

15A-15B are flowcharts illustrating an example method for managing hardware hosts in a distributed computing system using processing partitions organized by spatial location information associated with data objects in accordance with examples of the present technology.

FIG. 16 is a flow diagram illustrating an example method for processing data objects assigned to hardware hosts using processing partitions organized by spatial location information associated with the data objects in accordance with examples of the present technology.

FIG. 17 is a flow diagram illustrating an example method for splitting a processing partition organized by spatial location information in accordance with an example of the present technology.

FIG. 18 is a flow diagram illustrating an example method for merging processing partitions organized by spatial location information in accordance with an example of the present technology.

FIG. 19 shows an example of physical computer hardware upon which the present techniques may be executed.

Detailed Description

The technology provides computationally intensive distributed computing applications and systems in a service provider environment that can be used to provide virtual worlds, simulations, virtual environments, games, and other multi-dimensional (e.g., 2D and 3D) virtual environments that can be viewed, interacted with, and accessed by users and other computing services. Examples of such distributed virtual environments may be physical simulations, first person games, medical simulations, television or movie sets, driving simulators, airplane simulators, CAD (computer aided design) or CAM (computer aided manufacturing) applications, space simulators, and other virtual worlds. The distributed computing application may be a large-scale, low-latency distributed application used with a persistent interactive virtual world that simulates or is capable of hosting millions of concurrent users.

The techniques may manage computing resources in a service provider environment and may scale elastically to support thousands of applications (i.e., for handling a variety of different object types). The techniques may load balance application instances across hardware in response to virtual world load, so developers may amortize large world and simulation costs on a user basis. By partitioning, the techniques may manage computing resources so that applications or application instances may efficiently utilize the CPU core (central processing unit core) of a large hardware instance, and so that developers may efficiently use fewer hardware instances. In addition, this technology also enables developers to run complex worlds using a large number of hardware hosts, even worlds with millions of users connected to a single world or simulation. The technique may be based on using computing resources that are automatically scaled (e.g., scaled down when not in use or scaled up when load increases) and only charge for computing time when in world, simulation, or gaming activities.

The techniques may include a state structure that is a high-throughput computing and data service that may support gigabytes per second of data writes of data from a highly partitioned application or application instance co-located with the data. Examples of applications executed using this technique may include destruction applications, which may simulate a large number of destroys in the game world while manipulating tens of thousands of physical entities; an AI (Artificial Intelligence) application that enables thousands of competing AIs in a game; or "navigation web" generates an application that enables developers to recalculate the navigation web of AI vehicles or characters in the world that change at 30hz by many kilometers. The custom application may also access simulation data, data objects, entities, and actors from the state structure by subscribing to filters for receiving changed data objects and world assets from hardware hosts (e.g., for regions of the world or for object types).

A distributed computing system in a service provider environment may provide high throughput, low latency computing and data services (or state structures) to support the placement and acquisition of highly partitioned data millions of times per second using one or more hardware hosts. The hardware host may include a computer, server, physical host, or virtual machine instance that executes the containerized application or application instance and manages the memory shared across the containerized application or application instance. A Distributed Shared Memory (DSM) management system may facilitate client and application interaction with highly partitioned data, as well as enable data distribution across many hardware hosts. DSM enables highly partitioned data to be read with low latency and shared through subscription-based streams. An application or client creates a data object event stream that requests updates to the data object sent to the DSM, and these changes are replicated to the peer hardware hosts, allowing other applications and clients to see the updates. The DSM provides fault tolerance by separating code from data state. Thus, starting and stopping applications does not affect the data stored in the DSM.

The DSM may also enable highly partitioned data to be backed up to long term storage in the service provider environment. The DSM can also replicate highly partitioned data to a large number (thousands, millions, or billions) of clients. Region-of-interest management (e.g., spatial fields or object types) and bandwidth prioritization systems can be used to prioritize data updates to clients to maintain large-scale object world or simulated client view consistency over consumer-level internet connections.

The DSM may also enable client code (e.g., containerized applications or application instances) to modify highly partitioned data as close as possible to the data on the serverless fabric for minimum latency and maximum throughput. Applications and clients read data from the DSM by executing a one-time query that causes them to subscribe to updates to data objects that match the query filter. The filter may be a spatial structure (e.g., an axis-aligned bounding box), a tag or list of tags (e.g., "bot"), or any other query that may be evaluated on a field of a data object (e.g., "enemy player with low hit point"). A manager service on a hardware host may manage the lifecycle of applications and DSMs on the hardware host and may expose endpoints that allow DSMs on other hardware hosts to be linked together.

In an example use of this technique for gaming, a developer building a gaming experience in a service provider environment may have been limited by the computation and storage of a single server process on a single hardware instance. For example, a game console paired with a single server process may manipulate thousands of physical objects per frame, or a few dozen AI (artificial intelligence) that is intelligent enough to challenge several players. Developers have been able to deploy a single server process to computing instances on hardware hosts or droplets (draglets) using existing tools, but it can be difficult to leverage additional computing resources to support more users, physical objects, and AIs, while building complex clustering techniques that enable developers to distribute their virtual worlds, games, or simulations across servers. Building techniques that distribute computing workload across servers and are highly stateful, low-latency, and fault-tolerant are difficult, risky, and take years of effort for large and experienced teams.

With the present technology, developers can immediately begin building virtual worlds, simulations, and games that utilize increased simulation density (e.g., 10X-10000X) compared to a single server game. This may result in handling tens of thousands of physical objects or characterized by thousands of competing AIs per frame. Providing high performance runtime, distributed game applications and tools can help game developers build games in a service provider environment that exceed the limitations of a single server model.

In one configuration of the present technology, a distributed computing system may use object domains to organize computing units across hardware hosts. The object domain may define the processing partition (i.e., the subdivision of the data object) using the object type of the data object in the object-based model of the virtual environment. Distributing the load for processing data objects in a distributed computing system using object types enables the processing and rendering of large virtual worlds. The data objects may be geometric objects (e.g., polygons, spheres, cylinders, etc.), player models, opponent models, buildings, vehicles, monsters, alien people, plants, animals, or any other object that may be 2D or 3D modeled or animated. The object field may describe the object type of a data object that may be assigned to a processing group or a processing partition. An object type may be data associated with a data object that conceptually describes the data object. For example, the data object may be one of the following object types: a human, an animal, a plant, a building, a vehicle, a particle, a weapon, or another object type. The object domain may subdivide the data object using one object type or may use multiple object types. An object domain may be associated with one processing partition or multiple processing partitions. A mapping may be created between an object domain and a processing application that processes data objects subdivided within the object domain. The object types may be used to group data objects of the object types together, and the grouping may be considered a process partition computing unit.

The processing application may be used to process data objects assigned to the processing partitions by object type. Thus, the object domain may be used to define which applications and/or application instances may be used to perform the computations of the object domain and which applications and/or application instances may have exclusive change rights (e.g., write rights) to data objects of the associated object type. For example, a data object having a physical data type may be processed by a physical application. The physical object domain may define which object types belong to the physical object domain, define which physical processing applications have rights to the data object, and define data dependencies with other object types for processing the object types belonging to the physical object domain.

As described, an object type may be owned by an object domain having a processing application representing at least one computing process to be applied to the object type owned by the object domain. Thus, the object domain may provide an indexing process to organize the object data into discrete groups of computing units. The object domain may define which applications have permission to write to a particular object type (e.g., exclusive change permission). The object domain may define which applications have rights (e.g., read-only rights) to use additional object types during processing. This definition of ownership of data objects enables a single writer-multiple reader model to be implemented for applications and/or application instances. This definition of ownership of data objects also enables replication signaling of additional object types to be read by the object domain, allowing the distributed computing system to compute what data needs to be made available at which hardware hosts and at what priority.

Thus, data objects in a virtual environment may be organized into processing partitions by the distributed computing system according to their object types. The distributed computing system may use the processing partitions to allocate data objects to application instances of processing applications on the hardware hosts. For example, multiple "particle objects" may be assigned to multiple processing partitions. One or more of the processing partitions may then be assigned to a particle simulation application instance on the first hardware host for processing. Other processing partitions may be assigned to the particle simulation application instance on the second hardware host for processing.

In high performance computing systems (e.g., Hadoop clusters), the distribution of data to be processed in parallel has previously been accomplished by dividing unified data into portions and sending the data to separate nodes in the computing cluster. Since the data in these cases is uniform, a uniform type of processing is applied to the data. In contrast, partitioning data for parallel processing may be problematic where it may be desirable to apply different types of processing to separate portions of the data. Instead, the present technology may partition large data sets of data objects having various data types for a virtual world (e.g., a game world) into processing partitions for distributed or parallel processing. The technique is capable of grouping different types of data objects into groups using object types. The object type can be assigned to a hardware host having an application instance specifically programmed to handle the object type. The use of object types enables processing to be partitioned into processing partitions distributed among hardware hosts in a distributed computing system based on the type of processing to be applied to the object type. In addition, the techniques may also detect and manage dependencies between data object types, and managing dependencies may overcome bottlenecks in the processing of data objects that are partitioned into processing partitions. The use of data dependencies defined by object domains also enables processes to be organized into process partitions and distributed among hardware hosts in a distributed computing system to optimize where data used by application instances is located in the distributed computing system.

This technology further provides high throughput, low latency computing and data services that support millions of placements and acquisitions per second of highly partitioned data that is partitioned into processing partitions and distributed among hardware hosts in a distributed computing system. A distributed computing system may use a Distributed Shared Memory (DSM) service on a device to manage storage of data objects mapped to processing partitions at hardware hosts. The distributed shared memory device may write changes to the data object using a storage format shared across distributed shared memory services on the device to provide a versioned snapshot of the state. The distributed shared memory service may also enable the state of data objects to be read with low latency through subscription-based streams. Developers using this technology can achieve a higher level of performance, scalability, persistence, and reliability on a single server and on multiple servers, enabling a stable, spatially unlimited, or spatially intensive interactive experience for players.

The technique can write simulation state (e.g., 300X +) orders of magnitude more than past single server database solutions and replicate state across hardware hosts in a distributed computing system and to users across PCs, game consoles, and mobile devices. In one configuration of the present technology, a distributed computing system may use a serialization format to represent data objects to manage the storage of data objects for an application. As previously described, the data objects to be stored in the shared memory may be geometric objects (e.g., polygons, spheres, cylinders, etc.), player models, opponent models, buildings, vehicles, monsters, alien persons, plants, animals, images, or any other object capable of being modeled in 2D or 3D. The distributed computing system may use the serialized format to manage state changes to data objects with an ordered append-only based storage process. The ordered append-only log-based storage process provides low replication cost and thread-safe reads in a distributed computing system. Thus, data objects may be represented using a format shared across multiple distributed shared memory services on separate devices in a distributed computing system to provide versioned snapshots of state and low replication costs.

The data object may be represented in the memory of the distributed shared memory in the hardware host using a set of semantics established at the top of the byte array. A representation of a data object may be written to a memory device using a byte array that is divided into a plurality of sections. The segment may describe the in-memory content of the data object and include information on how to read the in-memory content to obtain the current state of the data object. One of the sections may include a log section containing a series of one or more log records. Changes to the data object caused by the processing application or application instance may be written to the log section using an ordered, log-based format to provide a versioned snapshot of the state. The log record may be used to represent a version ordered collection of changes to the data object.

The distributed shared memory of the hardware host may access the log section from the tail and read the log record backwards to get the latest state of the data object. The distributed shared memory may read log records backwards to quickly identify object fields of data objects that may have changed during processing and collect the most recent version-ordered changes to the object fields of the data objects. The distributed shared memory may also read a representation of the data object originally written to the memory device when a portion of the data object in a log record in the log section has not changed. The distributed shared memory may determine the latest state of the data object by collecting the latest version ordering changes to the object fields and collecting the original in-memory contents of the data object when needed.

In high performance computing systems (e.g., Hadoop clustering), the format of the data to be processed is typically a human-readable format, such as JSON or XML, or as a CSV file. These formats generally do not provide an efficient format for the distributed computing system to actually store data, whether in memory or on disk. Storing data in advanced or annotated formats may also be extremely inefficient for storage efficiency through wired communication and/or parallel processing.

The techniques enable data objects to be represented in memory, on a mass storage device, or transferred over a network connection using stored procedures and serialized formats that provide low replication cost and thread safe reading. The storage process according to the present technology enables the in-memory representation of a data object to be copied to another memory location, another hardware host, or a storage device without causing additional serialization processing because the in-memory representation has already been serialized. The storage process further enables the processing of data objects to be quickly and easily distributed among hardware hosts in a distributed computing system. The stored procedures may ensure that the processing application performs operations with the correct version of the data object and that updates to the data object are properly ordered. In addition, this technique may also manage in-memory changes to data objects to overcome bottlenecks in the processing of data objects when processing application modifications or reads. The storage process may further enable state exchange between applications and hardware hosts and improve the speed, efficiency, and operation of data storage and replication in a distributed computing system.

In accordance with the present technique, spatial analysis of data objects associated with a virtual environment may be used to organize computing units in a distributed computing system. Spatial analysis may determine spatial information about the data object, such as absolute position, relative position, proximity position, spatial dependencies, and the like. The data objects may be grouped together using spatial location information, and the set of data objects may be considered a processing partition (i.e., a subdivision of the data objects) for processing an application or application instance assigned to the data objects of the processing partition. The processing partitions may be defined, in part, on how the data objects are spatially grouped together. The processing partitions may be load balanced across hardware hosts in the distributed computing system. Allocating the load for processing data objects in a distributed computing system through the use of processing partitions and spatial organization enables large virtual worlds to be processed and presented to clients or users.

In one configuration, the techniques spatially subdivide a plurality of data objects into a plurality of processing partitions using one or more object fields that provide spatial location information for the data objects. Spatial location information (e.g., x, y, and z coordinates) associated with the plurality of data objects may be determined from object fields in the data objects and may be mapped to the processing partition. The processing partition may identify a set of data objects in the defined place. The processing partitions may be allocated and sent to hardware hosts in the distributed computing system for execution by an application of code having data objects. The processing partition may be sent to the hardware host to configure the hardware host to allow the application to process the data object for the identified location.

Thus, data objects that are spatially close together may be grouped for one or more processing applications (e.g., physical applications). In one configuration, this technique may use a tree structure to subdivide data objects spatially. The tree structure may be an octree or a quadtree covering three dimensions of a virtual world or two dimensions of a planar environment. Each node in the tree structure may identify a set of data objects having spatial proximity or spatial grouping to be processed by an application instance configured to process one object type. The nodes in the tree may be used to assign data objects to application instances of a processing application that processes the data objects. For example, multiple "particle objects" may be assigned to multiple processing partitions by spatially subdividing the particle objects by location. One or more of the processing partitions may then be assigned to one or more particle simulation application instances on the hardware host or across multiple hardware hosts for processing.

Partitioning data for parallel processing can be problematic where different types of processing can be applied to portions of the data. The present technology overcomes this difficulty by partitioning large data sets of data objects having different data types for a virtual world (e.g., a game world) into processing partitions for parallel processing. The technique takes into account the spatial relationship of the data objects in order to group the data objects and process them more quickly. The use of spatial analysis enables a process to be partitioned into process partitions distributed among hardware hosts in a distributed computing system in order to group related data as closely as possible. Furthermore, the techniques may also detect and manage spatial dependencies between data object types, and managing the spatial dependencies may overcome bottlenecks in the processing of data objects that are partitioned into processing partitions. Using the data dependencies defined by the object domains also enables processes to be organized into process partitions among hardware hosts in the distributed computing system to optimize where data used by application instances is located in the distributed computing system.

In accordance with the present technology, a distributed computing system may use object domains to identify and control processing applications executing on hardware hosts. The hardware host may be preloaded with a processing application using an application library. The application library may be stored by the hardware host and used by an application manager on the hardware host to launch an instance of the processing application according to the associated object type. As described above, the object domain may be used to define which applications and/or application instances may be used to perform computations of the object domain and which applications and/or application instances may have exclusive change rights (e.g., write rights) to data objects of the associated object type. Launching instances of processing applications in a distributed computing system using object types enables processing and rendering of large virtual worlds because processing can be allocated to any number of hardware hosts with copies of application libraries. According to one example, a hardware host may receive an assigned processing partition and identify, from an application library, a processing application for an object type associated with (e.g., matching or capable of performing a function on) the processing partition. The hardware host may launch an instance of the processing application to enable the hardware host to process a plurality of data objects grouped by object type, the data objects mapped to the processing partition.

In another configuration, the distributed computing system may enable hardware hosts to communicate through subscription-based streams. A processing application in the application library may define a subscription policy for obtaining additional data for use during processing of the data object. The subscription policy may identify what additional data object types are used during processing and at what source the additional object types may be located. The hardware host may send subscription requests using data dependencies, spatial relationships, object tracking, and queries defined in the subscription policy to enable the hardware host to transfer and replicate data objects used by instances of the processing application during processing.

Accordingly, the distributed computing system may enable hardware hosts to share data objects used by processing applications distributed across the hardware hosts. In one configuration, this technique may enable a first hardware host to receive multiple processing partition assignments and identify a first processing partition allocated to the first hardware host. The first hardware host may then determine a first object type associated with the first processing partition and use the first object type to launch an instance of a corresponding processing application from the application library. The first hardware host may also determine a second object type on which processing of the data object of the first object type by the processing application depends. The first hardware host may use the plurality of processing partition assignments to determine a second hardware host that is assigned a second processing partition, the second hardware host grouping data objects into the second processing partition in accordance with the second object type. The first hardware host may send a subscription request to the second hardware host to instruct the second hardware host to replicate (e.g., make a copy of or make changes to) the second plurality of data objects to the first hardware host and the instance of the processing application.

In another configuration, the techniques may enable the first hardware host to determine a subscription policy that identifies neighboring relationships between spatial subdivisions associated with a plurality of spatial subdivisions of the multidimensional virtual environment to filter data objects of the second object type that satisfy the filter. The first hardware host may identify the second processing partition using a neighboring relationship between a first spatial subdivision associated with the first processing partition and a second spatial subdivision associated with the second processing partition that satisfies a subscription policy.

In yet another example, the first hardware host may determine a subscription policy that identifies query criteria (e.g., a vehicle or a particular moving object with a low hit point) associated with a query for filtering data objects of the second object type. The first hardware host may identify the second processing partition by matching the second plurality of data objects to the query criteria using a subscription policy. The first hardware host may also determine a list of subscribers to the data object of the first object type. The first hardware host may send the data object to a list of subscribers. The first hardware host may further receive updates to the plurality of processing partition assignments. The first hardware host may determine migration of the second processing partition between the second hardware host and a third hardware host, and send a second subscription request from the first hardware host to the third hardware host.

FIG. 1 illustrates a system 100 that may include a service provider environment 102 hosting a compute-intensive distributed computing system for providing a user-accessible and/or viewable virtual environment, in accordance with examples of the present technology. The system 100 may include a service provider environment 102 and one or more clients 104 in communication with the service provider environment 102 using a network 106. The network 106 may include any useful computing network, including an intranet, the Internet, a local area network, a wide area network, a wireless data network, or any other such network or combination thereof. The components for such a system may depend at least in part on the type of network and/or environment selected. Communication over the network may be accomplished through wired or wireless connections, and combinations thereof. The service provider environment 102 is able to use the client 104 to manage and deliver computing, storage, and networking capabilities as software services for a community of end recipients. In this example, the service provider environment 102 can include an infrastructure manager 110, one or more infrastructure services 112, and one or more distributed computing systems 120.

The infrastructure manager 110 may manage and control physical machines and other physical devices for providing a compute-intensive distributed computing system. Infrastructure manager 110 may be a service comprising one or more computing systems, such as server computers, configured to control the physical and virtualized infrastructure resources of infrastructure service(s) 112 and distributed computing system(s) 120. The infrastructure services 112 and virtualized infrastructure resources of the distributed computing system 120 may include virtualized executable resources, virtualized storage resources, virtual network interfaces, and other virtualized networking components. Some examples of virtualized executable resources may include compute instances, containers (e.g., open source container orchestration system (kubernets)), compute functions, managed applications, and so forth. Some examples of virtualized storage services may include database services, block storage services, content delivery services, and the like. Some examples of virtualized networking components may include virtualized networking devices and physical network devices (i.e., routers, firewalls, load balancers, etc.) configured with the infrastructure services 112 and logical roles in the distributed computing system 120.

The infrastructure manager 110 may instantiate all or part of the infrastructure services 112 and the distributed computing system 120. The infrastructure manager 110 can identify the physical hosts and physical networking devices to be managed by the infrastructure manager 110. Some examples of physical hosts managed by the infrastructure manager 110 are server computers, embedded computers, computing hosts at the edge of the service provider environment 102, and other physical devices. Some examples of physical networking devices used by the infrastructure manager 110 are routers, switches, firewalls, load balancers, cache services, server computers, embedded devices, and other physical networking devices.

The infrastructure services 112 may be considered on-demand computing services hosted in servers, virtualized machines, grids, clusters, parallel, or distributed computing systems. Some examples of infrastructure services 112 that may be provided by the service provider environment 102 may include one or more computing services 114, one or more storage services 116, networking services, web services, streaming services, network accessible services, software as a service, storage as a service, on-demand applications, services for performing code functions, and so forth. In this example, the infrastructure service(s) 112 can include a distributed computing service 118, the distributed computing service 118 to establish and host the distributed computing system(s) 120 as a large-scale, low-latency, distributed computing system.

Distributed computing system(s) 120 may be considered an on-demand compute-intensive distributed computing system hosted in a server, virtualized machine, grid, cluster, parallel, or distributed computing system. Distributed computing system(s) 120 may be used to provide virtual environments (e.g., 2D (two-dimensional) or 3D (three-dimensional virtual environments)), simulations, and "computationally cumbersome games" that include persistent, interactive virtual worlds that may host millions of concurrent users or game players. Distributed computing system 120 may provide high throughput, low latency computing and data services that support millions of placements and acquisitions per second for highly partitioned data and data objects.

Distributed computing system(s) 120 may include a control plane 122, a data plane 124, and an entry plane 126. The control plane 122 may be responsible for coordinating the creation of distributed computing systems for simulations, virtual environments, games, etc., and the control plane 122 may be responsible for load balancing across hardware hosts in the distributed computing system. Hardware hosts, such as hardware hosts 142 and 144, may be physical hosts, hardware computing devices, or servers capable of hosting one or more virtualized computing resources or services. The control plane 122 may manage aspects of the distributed computing system, such as running simulation and virtual environments, including deploying code for execution, collecting metrics, uploading logs, and monitoring clusters of hardware hosts.

The control plane 122 may also provide a user accessible dashboard for performing auditable maintenance actions across a cluster of hardware hosts, and a development team can use the dashboard to access operating characteristics of the running simulation and virtual environment. Control plane 122 may include a system manager 130, which system manager 130 manages resources used in the distributed computing system through the simulation and virtual environment, such as a hardware host warming pool 132 and one or more cluster services 134. The hardware host warming pool 132 may include one or more hardware hosts with preloaded operating systems, applications, assets, and/or data waiting for assignments from the system manager 130. The warm hardware host may represent a physical host or virtualized infrastructure resource allocated to the simulation for later use. The clustering services 134 may include services that support the operation of the distributed computing system, such as health monitoring, ticketing, recording, metrics, deployment, workflow, and the like.

The data plane 124 may be responsible for the processing of simulations, virtual environments, games, etc. in a distributed computing system, and the data plane 124 may be responsible for communications across hardware hosts in the distributed computing system. The data plane 124 may include a hardware host activity pool 140 over which computing units of a simulated or virtual world are distributed using one or more hardware hosts, such as hardware hosts 142 and 144, on the hardware host activity pool 140. Hardware hosts 142 and 144 include one or more files 150a-b, one or more applications 152a-b, and one or more distributed shared memories 154 a-b. The file 150 may be data, resources, data objects, 3D models, 2D models, textures, images, animations, web pages, and the like. The application(s) 152 may include a class of user computing or execution processes that may be executed on data (e.g., data objects) associated with the file(s) 150, and the application describes application instances in a collective manner. For example, the application may be a physical application that modifies the data object using a physics-based digital method. Distributed shared memory 154 may manage data in memory used and shared across applications 152. The distributed shared memory 154 may also replicate changes to data in memory across hardware hosts in the hardware host active pool 140. For example, application(s) 152 may subscribe to a stream of data object events by requesting that updates to data in memory be delivered to application(s) 152 from distributed shared memory 154 a-b. In another example, application(s) 152 may create a data object event stream that includes modifications to data in memory that are sent to distributed shared memory 154 a-b. The distributed shared memories 154 a-154 b may replicate changes in the state of data in memory between Distributed Shared Memories (DSMs) of the hardware hosts 142 and 144.

The portal plane 126 can be responsible for facilitating communication with distributed computing systems of simulations, virtual environments, games, and the like. For example, the ingress plane 126 may manage connections from outside the distributed computing system(s) 120. The ingress plane 126 may include a front-end service resource manager 160, one or more remote ingress points 162, and one or more client gateways 164. The client(s) 104 may utilize an Application Programming Interface (API) to connect to the client gateway(s) 164, the client gateway(s) 164 authenticating the credentials and authorizing the connection to the remote entry point(s) 162. The remote entry point(s) 162 may include a multi-tenant queue that copies data from a virtual world or simulation to a physical host of the client(s) 104. The front-end service resource manager 160 may provide a graphical user interface for accessing, managing, monitoring, updating, or developing distributed applications, data objects, and/or assets executing in the service provider environment 102.

The techniques may provide a serverless virtual world or simulation service for building and operating computationally large worlds, such as massive multiplayer games with millions of connected players and city-scale simulations with millions of persistent objects. A developer may create millions of data objects and launch an application (e.g., a computing process) that modifies the data objects at a variable rate (e.g., 10-60 Hz). The technique can automatically manage where and when millions of data objects and applications are allocated to hardware hosts. The techniques may load balance application instances across hardware hosts in response to various conditions (such as load on a virtual world), or concatenate data processed by an application to reduce replication costs).

FIG. 2 illustrates various example components included in a service provider environment 200 of a compute-intensive distributed computing system for providing a virtual environment visible to a user, in accordance with one example of the present technology. In this example, service provider environment 200 may be capable of delivering computing, storage, and networking capabilities as software services to a community of end recipients. In one example, service provider environment 200 may be established for an organization by or on behalf of the organization. That is, the service provider environment 200 may provide a "private cloud environment. "in another example, service provider environment 200 may support a multi-tenant environment in which multiple customers may operate independently (i.e., a public cloud environment). In general, service provider environment 200 may provide the following model: infrastructure as a service ("IaaS") and/or software as a service ("SaaS"). Other models may be provided. For the IaaS model, the service provider environment 200 may provide computers as physical or virtual machines and other physical devices to serve as virtualized infrastructure resources in a virtual infrastructure.

Application developers can develop and run their applications on the service provider environment 200 without incurring the cost of purchasing and managing the underlying hardware and software. The SaaS model allows for the installation and operation of applications in the service provider environment 200. For example, an end client may access service provider environment 200 using a networked client device (such as a desktop computer, laptop computer, tablet computer, smartphone, game console, etc.) running a web browser or other standalone client application. Service provider environment 200 may include a plurality of computing devices arranged, for example, in one or more server banks or computer banks or other arrangements. Computing devices may use hypervisors, Virtual Machine Managers (VMMs), and other virtualization software to support computing environments. In this example, service provider environment 200 may include one or more server computers 202. The server computer(s) 202 may include a system manager module 210, a cluster manager module 210a, a world manager module 210b, a cluster service module(s) 212, a data store 214, a front-end service resource manager module 216, a remote entry point module 218, a client gateway module 220, one or more warm hardware hosts 224, one or more active hardware hosts 226, one or more processors 230, and one or more memory modules 232.

System manager module 210 may include hardware and software configured to create, deploy, and manage a high-throughput low-latency distributed computing system. The system manager module 210 can use the cluster manager module 210a and the world manager module 210b to coordinate the clustering using the hardware hosts to create a distributed computing system for simulation, virtual environments, and the like. Cluster manager module 210a can manage running aspects of the infrastructure associated with the distributed computing system, including deploying code for execution, collecting metrics, uploading logs, and monitoring cluster resources. The world manager module 210b can manage aspects of running multidimensional virtual environments, worlds, or simulations hosted by the distributed computing system, including distributing files and applications to hardware hosts and monitoring changes to the processing of data on the hardware hosts by the applications. The system manager module 210 can also use the cluster manager module 210a and the world manager module 210b to load balance computing operations across hardware hosts.

The cluster service module 212 may include hardware and software elements configured to provide services that support the operation of the distributed computing system, such as a monitoring service 240, a logging service 242, a ticketing service 244, a workflow service 246, a deployment service 248, and an operations console service 250. The monitoring service 240 may be used to monitor resources used by the distributed computing system. For example, the monitoring service 240 may collect metrics associated with the active hardware host(s) 226 to determine utilization of one or more hardware or software resources. The logging service 242 may be used to manage and analyze logs and operational data points from the distributed computing system. Ticketing services 244 may be used for ticketing of questions or ticketing of source code, executable code, or data that is checked in or out. The workflow services 246 may be used to manage and execute workflows in a distributed computing system. For example, the workflow services 246 may initialize, construct, or load virtual computing resources, data objects, hardware hosts, and the like. Deployment service 248 may be used to configure and deploy hardware hosts in a distributed computing system. The operations console service 250 may be used to interact with the control plane, data plane, and portal plane associated with the distributed computing system. The operations console service 250 may provide one or more user interfaces (textures or graphics), Application Programming Interfaces (APIs), and the like through which a user may enter commands or retrieve data associated with the distributed computing system.

The data repository 214 may include hardware and software elements configured to provide data services to distributed computing systems managed by the cluster manager module 210a and multidimensional virtual environments managed by the world manager module 210 b. The data store 214 may include one or more world/cluster configurations 260, one or more files 262, and one or more applications 264. The world/cluster configuration 260 may define a distributed computing system for the world or simulation. For example, the world may be defined using an object-based model of the virtual environment. The virtual environment may be a 2-dimensional, 3-dimensional, or multi-dimensional virtual world. The object-based model may use data objects to represent entities within the virtual environment. Files 262 may include data, resources, data objects, 3D models, 2D models, textures, images, animations, web pages, etc. used by the distributed computing system. In one example, the data objects in file(s) 262 may represent entities within the virtual environment. These entities may be characters, animations, geometric objects, actors, non-player characters, vehicles, buildings, vegetation, rocks, animals, monsters, and the like. Data objects in the file 262 may be represented in the data store 214 using attributes having key-value or name-value pairs and object fields having field identifiers and field values. Application(s) 264 may include executable code that processes file(s) 262. For example, application(s) 264 may process data objects that represent entities within the virtual environment. Some examples of application(s) 264 may include simulation, collision detection, physics engines, rendering, and the like.

Front-end services resource manager module 216 may include hardware and software elements configured to provide access, management, monitoring, updating, or development of world/cluster configuration(s) 260, file(s) 262, and application(s) 264. The front-end services resource manager module 216 may include one or more graphical user interfaces to build virtual worlds, define data objects, write applications, and the like. The remote entry point module 218 may include a multi-tenant queue that copies data from the data store 214 and/or active hardware host 226 to the physical host of the remote client. Further, the client may utilize an API to connect to the client gateway module 220, the client gateway module 220 authenticating credentials and authorizing a connection to the remote entry point module 218.

Warming pool hardware host(s) 224 may include hardware and software elements configured in a standby mode to execute application(s) 264 using file(s) 262. Cluster manager module 210a may assign multiple hardware hosts to warm pool hardware host(s) 224 in a standby or suspended mode of operation in anticipation of future needs. Cluster manager module 210a can migrate hardware host(s) between warming pool hardware host(s) 224 and active pool hardware host(s) 226 to scale computing resources when need arises. Active pool hardware host(s) 226 may include hardware and software elements configured in an active mode to execute application(s) 264 using file(s) 262. Active pool hardware host(s) 226 may include a hardware host manager module 270, an application manager module 272, one or more application instances 284, a distributed shared memory module 274, a file system data store 276, and an in-memory data store 278.

Hardware host manager module 270 may include hardware elements and software elements configured to manage data processing by one or more hardware hosts. The hardware host manager module 270 may include a runtime module that receives instructions from the cluster manager module 210a, and the world manager module 210b executes the instructions on the hardware host. The instructions may configure the hardware host in an active mode to execute application 264 using file 262. The instructions may identify the files 262 assigned to the hardware host and which of the applications 264 to use to process the assigned files 262. Hardware host manager module 270 may receive one or more of file(s) 262 for processing and store the files as local file(s) 280 in file system data store 276. In accordance with one example of the present technology, the hardware host manager module 270 may receive one or more index structures assigned to hardware hosts from the world manager module 210 b. The index structure may be used by the world manager module 210b to organize the files 262 into computing units, referred to herein as processing partitions, discussed in more detail below. Hardware host manager module 270 may use an index structure to manage the storage of local files 280 in file system data store 276.

The instructions may further identify application instance(s) 264 for processing one or more of local file(s) 280. Hardware host manager module 270 may receive one or more of applications 264 and store the applications as native application(s) 282 in file system data store 276. In accordance with one example of the present technology, hardware host manager module 270 may receive an application library that includes application(s) 264. The application library may be stored by the hardware host manager module 270 and used by the application manager module 272 to launch an instance of a native application 282 in the file system data store 276.

Application manager module 272 may include hardware and software elements configured to manage one or more application instances 284 on a hardware host. Examples of application instances 284 may be instances of physical applications, rendering applications, collision applications, transformation applications, occlusion applications, sound applications, or other types of applications. The application manager module 272 may identify the local files 280 assigned to the hardware host and determine which of the local applications 282 to instantiate. Application manager module 272 may determine the number of instances of local application(s) 282 to be instantiated as application instance(s) 284. Application manager module 272 may scale the number of application instances 284 as desired.

Distributed shared memory module 274 may include hardware and software elements configured to manage in-memory data store 278. In-memory data store 278 for hardware hosts may be used by application instances 284 for storage, and changes to in-memory data store 278 may be shared across active pool hardware hosts 226. Distributed shared memory module 274 may load one or more of local files 280 into in-memory data store 278, which is shared memory 286. The distributed shared memory module 274 may receive requests from the application instances 284 to access the shared memory 286. The distributed shared memory module 274 may send data from the shared memory 286, for example, by processing requests from the application instance 284 to read the data. The distributed shared memory module 274 may also receive requests from the application instance 284 to modify the shared memory 286. The distributed shared memory module 274 may store data to the shared memory 286, for example, by processing requests from the application instances 284 to write data. The distributed shared memory module 274 may further replicate changes to the shared memory 286 across the active hardware hosts 226.

The various processes and/or other functions contained within service provider environment 200 may be executed on one or more processors 230 in communication with one or more memory modules 232. Service provider environment 200 may include a plurality of computing devices arranged, for example, in one or more server banks or computer banks or other arrangements. Computing devices may use hypervisors, Virtual Machine Monitors (VMMs), and other virtualization software to support computing environments.

The term "data store" can refer to any device or combination of devices capable of storing, accessing, organizing, and/or retrieving data, which can include any combination and number of data servers, relational databases, object-oriented databases, clustered storage systems, data storage devices, data warehouses, flat files, and data storage configurations in any centralized, distributed, or clustered environment. The storage system components of data stores 214, 276, and 278 may include storage systems such as SANs (storage area networks), cloud storage networks, volatile or non-volatile RAM, optical media, or hard drive type media. As can be appreciated, the data stores 214, 276, and 278 may represent multiple data stores.

FIG. 2 illustrates that certain processing modules may be discussed in connection with the technology and may be implemented as computing services. In one example configuration, a module may be considered a service having one or more processes executing on a server or other computer hardware. Such services may be centrally hosted functions or service applications that may receive requests and provide output to other services or consumer devices. For example, a module providing a service may be considered an on-demand computing hosted in a server, virtualized service environment, grid, or cluster computing system. An API may be provided for each module to enable the second module to send requests to the first module and receive output from the first module. Such APIs may also allow third parties to interface with the module and make requests and receive output from the module. Although FIG. 2 illustrates an example of a system that may implement the techniques described above, many other similar or different environments are possible. The example environments discussed and illustrated above are merely representative and are not limiting.

FIG. 3 is a block diagram illustrating an example computing service 300 for providing a compute-intensive distributed computing system using a distributed computing service manager in accordance with one example of the present technology. The computing service 300 may be used to execute and manage multiple computing instances 304a-d on which the present techniques may execute. In particular, the depicted computing service 300 illustrates one environment in which the techniques described herein may be used. Computing service 300 may be one type of environment that includes different virtualized service resources that may be used, for example, to host computing instances 304 a-d.

The computing service 300 may be capable of delivering computing, storage, and networking capabilities as software services to a community of end recipients. In one example, the computing service 300 may be established by or on behalf of an organization for an organization. That is, the computing service 300 may provide a "private cloud environment. In another example, the computing service 300 may support a multi-tenant environment, where multiple customers may operate independently (i.e., a public cloud environment). In general, the computing service 300 may provide the following model: infrastructure as a service ("IaaS"), and/or software as a service ("SaaS"). Other models may be provided. For the IaaS model, the computing service 300 may provide computers as physical or virtual machines and other resources. The virtual machine may be run by the hypervisor as a guest, as described further below. In another configuration, service model delivery may include computing of an operating system, a programming language execution environment, a database, and a web server.

Application developers can develop and run their software solutions on computing services without incurring the cost of purchasing and managing the underlying hardware and software. The SaaS model allows for the installation and operation of application software in the computing service 300. For example, an end client may access the computing service 300 using a networked client device (such as a desktop computer running a web browser, a laptop computer, a tablet computer, a smartphone, etc.) or other lightweight client application. Those skilled in the art will recognize that the computing service 300 may be described as a "cloud" environment.

The computing service 300 specifically illustrated may include a plurality of server computers 302 a-d. The server computers 302a-d may also be referred to as physical hosts. Although four server computers are shown, any number may be used, and a large data center may include thousands of server computers. The compute service 300 may provide computing resources for executing compute instances 304 a-d. Compute instances 304a-d may be, for example, virtual machines. A virtual machine may be an example of a software implementation of a machine (i.e., a computer) that executes applications like a physical machine. In the example of a virtual machine, each of the server computers 302a-d may be configured to execute an instance manager 308a-d capable of executing an instance. Instance managers 308a-d may be a hypervisor, a Virtual Machine Manager (VMM), or another type of program configured to enable execution of multiple compute instances 304a-d on a single server. In addition, each of compute instances 304a-d may be configured to execute one or more applications.

Server computer 314 may be reserved to execute software components for implementing the present techniques or for managing the operation of computing service 300 and compute instances 304 a-d. For example, the server computer 314 may execute the distributed computing service manager 315 to provide high throughput, low latency computing and data services that support highly partitioned data placement and retrieval millions of times per second.

The server computer 316 may execute a management component 318. A user may access the management component 318 to configure various aspects of the operation of the compute instances 304a-d purchased by the customer. For example, a user may set up compute instances 304a-d and make changes to the configuration of compute instances 304 a-d.

A deployment component 322 can be employed to assist a customer in deploying computing instances 304 a-d. Deployment component 322 can access account information associated with compute instances 304a-d, such as the name of the owner of the account, credit card information, the country of the owner, and so forth. Deployment component 322 can receive a configuration from a user that includes data describing how compute instances 304a-d can be configured. For example, the configuration may include an operating system, providing one or more applications to be installed in compute instances 304a-d, providing scripts and/or other types of code to be executed for configuring compute instances 304a-d, providing caching logic that specifies how an application cache is to be prepared, and providing other types of information. Deployment component 322 can utilize user-provided configuration and caching logic to configure, fill in, and launch compute instances 304 a-d. The configuration, caching logic, and other information can be specified by a user of the access management component 318 or by providing the information directly to the deployment component 322.

Customer account information 324 may include any desired information associated with a customer of the multi-tenant environment. For example, the customer account information 324 may include a unique identifier of the customer, a customer address, billing information, licensing information, customization parameters for starting the instance, scheduling information, and the like. As described above, the customer account information 324 may also include security information used in the encryption of asynchronous responses to API requests. By "asynchronous," it is meant that the API response can be made at any time after the initial request and using a different network connection.

A network 310 may be used to interconnect the computing service 300 and the server computers 302a-d, 316. Network 310 may be a Local Area Network (LAN) and may be connected to a Wide Area Network (WAN)312 or the internet so that end customers may access computing service 300. Further, the network 310 may include a virtual network overlaid on a physical network to provide communication between the server computers 302 a-d. The network topology shown in fig. 3 has been simplified because more networks and networking devices may be utilized to interconnect the different computing systems disclosed herein.

FIG. 4A illustrates a system 400 and related operations for assigning computing units in a distributed computing system using processing partitions organized by object domain, in accordance with an example of the present technology. The system 400 may include a world manager 402. Some examples of the world manager 402 may include the system manager 130 described with reference to fig. 1 and the world manager module 210b described with reference to fig. 2.

The world manager 402 may identify one or more data objects 404 associated with an object-based model of a multi-dimensional virtual environment (e.g., a 3D simulation, a 3D game, etc.). World manager 402 may retrieve data objects 404 from an object data store. Data object 404 may include one or more object fields 420. Object field 420 may include a set of fields, attributes, or properties represented by a field name 422 and a field value 424, the field name 422 may serve as an identification tag or key. The field values 424 may include data values, strings, X, Y, Z coordinates, etc., which may be referenced by the processing application using the object identifier 430 to process the data object.

In this example, the data object 404 may include metadata that provides an object Identifier (ID)430, an object ID value 431, an object version 432, an object version value 433, an object type 434, and an object type value 425 using one or more of the object fields 420. The example object ID value 431 of the object identifier 430 field may be a "forest _ tree _ object" and the example object ID value 431 of the object version 432 field may be a "version 4.214". Object type 434 may map to an object domain and have an object type value 435 of "particle", "tree", "water", etc. The object domain describes which object types form part of the domain, thereby enforcing rights with respect to what processing applications have exclusive change rights, and data dependencies (e.g., read-only rights) for the object types used in the processing of the object types forming part of the domain. An object type may be owned by a domain that has a processing application that represents its computation. The type of processing may include any type of processing of data objects that may occur for a virtual world, simulation, or game.

The world manager 402 may then determine one or more processing partitions 406 using the object type 434 of the data object 404. In addition to the data objects 404 themselves, the world manager 402 may use the world configuration 408 to determine the processing partition 406, as discussed in more detail later. The processing partition 406 may define computing units in a distributed computing system hosting a multidimensional environment, which may be allocated to hardware hosts in the distributed computing system. Data objects 404 may be organized or grouped into processing partitions 406 by object type values 435 of object types 434 to create computing units of a hardware host. Thus, the processing partition 406 may group the data objects 404 into computing units according to the object domain represented by the object type. The metadata in a processing partition may reference a set of data objects or data objects mapped to the processing partition as a computational unit for processing.

In accordance with one example of the present technology, the world manager 402 may use the object field 420 of the data object 404 to determine the processing partition 406. Because the processing partitions 406 represent groups of data objects organized into computing units to be processed by the processing applications, the world manager 402 may organize the data objects 404 according to one or more of the object fields 420. In accordance with one example of the present technology, this data object management by world manager 402 may include the creation of one or more index structures, referred to as domain indexes, that map a set of data objects 404 to processing partitions 406 using object types 434 through object domains, as will be discussed in later sections. The index structure may index data objects 404 into processing partition 406. In accordance with another example of the present technology, this data object management by the world manager 402 may include the creation of one or more indexing structures, known as spatial indexes, which may be used to divide the processing of the data object 404 by spatial location information (e.g., spatial location, bounding box, etc.) in the object field 420, as will be discussed in later sections. World manager 402 may use object field(s) 420 to create multiple layers in grouping data objects 404 into computing units, thereby further organizing data object(s) 404 by the multiple fields, attributes, and properties represented by object field(s) 420.

World manager 402 may then determine one or more processing partition assignments 410 to assign processing partition 406 to one or more hardware hosts 412. Processing partition assignments 410 may be made based on object types and application instances that may process the object types, based on data dependencies and based on metrics of processing load on hardware or software (e.g., to avoid overload, etc.). The world manager 402 may use the processing partition assignments 410 to send the processing partitions 410 to the hardware hosts 412 to allocate processing for the data objects 404 in the distributed computing system. Processing partition assignment 410 may be used to configure hardware host 412 for processing data object 404.

The hardware host 412 may process the data object 404 using the process partition assignment 410. Examples of processing may include data object movement, data object collisions, rendering, explosion, physical simulation, world credit transactions, or other types of processing. The data object 404 for a particular one of the processing partitions 406 may be loaded at one of the hardware hosts 412 and then processed by a processing application. For example, an instance of a processing application may be built on a hardware host to handle the explosion of data object 404. An instance of a processing application may iterate across one or more data objects mapped to processing partitions assigned to hardware hosts to simulate what data objects 404 occurred during an explosion. Changes to data object 404 may then be written to memory in the hardware host (using DSM). Changes written to memory may also be shared with other hardware hosts.

The technique may use a multi-level index to organize data objects into processing partitions 406 that represent computing units in a distributed computing system. For example, the world manager 402 may create multiple layers as a hierarchical structure. A multi-tier indexing scheme may be used where object domain grouping is the most important tier. FIG. 4B illustrates the separation of a global view of a multi-dimensional virtual environment into more than one group or process partition according to object domain. The multidimensional environment may comprise a data object having two object domains, a first object domain "domain a" being represented by a cross and a second object domain "domain B" being represented by a solid dot in a global view. World manager 402 can decompose the global view into "domain A decomposition" and "domain B decomposition," which become the most important layers for grouping data objects by object type associated with "domain A" and "domain B". The decomposition process also allows layers of lesser importance in the multi-layer indexing scheme to use similar semantics while differing in terms of instances. For example, in FIG. 4C, if each of "Domain A" and "Domain B" includes a large number of data objects densely packed into different spaces, this decomposition process may add another layer (such as a spatial index) to further index the data objects into the processing partition using the spatial location information. As depicted, because the spatial index components of the two domains can be in different states, the world manager 402 can organize data objects into the processing partitions 406 and determine whether to allocate data objects in different states to different hardware hosts.

FIG. 4B also shows a rights model that can be used with the object domain. More specifically, the domain allows the developer to declare which application owns each of the object types and which additional object types depend on any particular domain. While other applications may read data objects whose domain is not owned, they may not mutate them (e.g., may not write to the data object). As shown, the "Domain A" permissions for the first application may include read-only permissions for data objects associated with "Domain B" and exclusive write permissions for data objects associated with "Domain A". The "domain B" rights for the second application may include read-only rights for data objects associated with "domain a" and exclusive write rights for data objects associated with "domain B".

Fig. 4B further illustrates how the rights model can be used with an object domain for replication signaling. More specifically, since the domain allows the developer to declare which application owns each of the object types and which additional object types depend on any particular domain, the privilege model can signal whether the data object is to be shared by the application or replicated across the distributed computing system. For example, a first application associated with "domain A" may declare that the first application owns a data object associated with the object type of "domain A". The first application may further subscribe to the data object event stream by requesting delivery of updates to the data object associated with the object type of "domain B". The replication signaling mechanism may act as an object filtering mechanism to reduce the number of objects that need to be replicated across hardware hosts. For example, a domain of an object type that requires another domain does not imply an inverse relationship.

FIG. 5 illustrates various example components included in a service provider environment 500 for managing hardware hosts in a distributed computing system using processing partitions organized by object domain, according to one example of the present technology. Service provider environment 500 may include one or more server computers 502 in communication with one or more hardware hosts 504 using a network 506. The network 506 may include any useful computing network, including an intranet, the Internet, a local area network, a wide area network, a wireless data network, or any other such network or combination thereof.

The server computer 502 may include a world manager module 510, a first data store 512, a second data store 514, a third data store 516, one or more processors 518, and one or more memory modules 520. The world manager module 510 may be configured to create, deploy, and manage a high-throughput low-latency distributed computing system. The world manager module 510 may use the hardware host 504 to coordinate the creation of distributed computing systems for virtual worlds, games, simulations, virtual environments, and the like. The world manager module 510 may manage aspects of running a distributed computing system, including deploying code for execution (e.g., application instances), collecting metrics, uploading logs, and monitoring resources. The world manager module 510 may access the first data store 512 to create and manage a world configuration 522 of the virtual world, one or more data objects 524 stored for the virtual world, and one or more applications 526. The world configuration 522 may identify the virtual world and define the assets and applications used by the virtual world. World configuration 522 may specify one or more indices for partitioning assets into processing partitions. One example of a world configuration 522 may use the following world model:

The data object(s) 524 may identify objects and entities in the virtual world and define properties and characteristics of the assets. A data object may define a set of object fields and a set of indexable fields that map to its existing fields. The data objects 425 may use object definitions and object schema definitions, for example, as follows:

with respect to the above, some examples of data object(s) 524 may include:

application(s) 526 can identify and store copies of processing applications and application instances that process computations and processing in the virtual world. An application may specify object domains to be processed during computation and processing in a virtual world. The application may identify or be linked to an object type to indicate a given object domain. The application may further specify an index that may be used by the world manager module 510 to partition the data object 524 into processing partitions or blocks. For example, an application may specify the maximum number of data objects that an application instance may write during a particular application period or processing cycle.

The application may also define a subscription policy for obtaining additional data for use during processing of the data object. The subscription policy may identify what additional data object types are used during processing and at what source the additional object types may be located. In addition, the subscription policy may specify a maximum delay that the application instance can tolerate to obtain additional object types from a specified source. Application(s) 526 may be defined using, for example, the following applications:

With respect to the above, some examples of applications 526 may include:

the world manager module 510 may also access the second data store 514 to create and manage one or more hardware host configurations 528 and store one or more hardware host metrics 530. For example, the world manager module 510 may store a configuration of the type of compute instance to be used as a hardware host, a configuration of a container on the hardware host, a capacity of one or more hardware or software resources of the compute instance, and one or more types of processing partitions that the compute instance may process. Additionally, the world manager module 510 can obtain and track one or more hardware host metrics 530. These hardware host metrics 530 may identify and track the online/offline status of the hardware host, utilization of hardware or software resources of the hardware host, and other performance metrics associated with the hardware host.

The world manager module 510 may access the third data store 516 to create and manage one or more processing partitions 532 and one or more processing partition assignments 534. The processing partition 532 may include a data structure that serves as an index to map the data objects 524 to computing units that may be assigned to hardware hosts. Processing partition 532 may include metadata identifying the processing partition and one or more of data objects 524 that map to the processing partition.

As previously described, the object domain may be used in the process of distributing data objects in a distributed computing system hosting a virtual environment. In this example, world manager module 510 may access first data store 512 to identify the object type of data object 524. The world manager module 510 may create a list of object types used in the virtual world. The world manager module 510 may then use the list of object types to create a processing partition 532 to provide a mapping between the data objects 524 and the processing partition 532. For example, a first object type (e.g., a person) may be mapped to a first processing partition, and a second object type (e.g., a vehicle) may be mapped to a second processing partition.

The world manager module 510 may also use the world configuration 522 and the application 526 to create a processing partition 532. For example, the world manager module 510 may use the world configuration 522 to determine the total number of processing partitions that may be supported in the virtual environment. In another example, the world manager module 510 can use the configuration of the application 526 to determine the maximum or minimum number of data objects that can be allocated to a processing partition. The world manager module 510 may also use the hardware host configuration 528 and the hardware host metrics 530 to create a processing partition 532. For example, the world manager module 510 may use the hardware host configuration 528 to determine the number of hardware hosts available, the type of hardware hosts, the hardware and software resources allocated to the hardware hosts, etc. in order to determine the total number of processing partitions. In another example, the world manager module 510 may use the hardware host configuration 528 to determine the capacity of resources defined in the hardware host configuration 528 to determine the total number of processing partitions in the processing partitions 532 or the total number of data objects 524 mapped to one of the processing partitions 532. In another example, the world manager module 510 may use the hardware host metrics 530 to determine whether utilization of hardware or software resources affects the number of processing partitions in the processing partitions 532 or the supported number of data objects 524 mapped to one of the processing partitions 532.

After determining the processing partition 532 using data object 524, world manager module 510 may determine one or more processing partition assignments 534. Processing partition assignment(s) 534 may include a data structure that serves as an index to assign processing partition(s) 532 to hardware host(s) 504. The processing partition assignments 534 may include metadata identifying at least one of the one or more processing partitions and the hardware hosts 504 assigned to the processing partitions. The world manager module 510 can use the world configuration 522 and the application 526 to determine the processing partition assignment 534. For example, the world manager module 510 may determine the processing partition assignment 534 using the assigned number of hardware hosts defined in the world configuration 522. In another example, the world manager module 510 can use the number of objects supported or a subscription policy defined for the application 526 to determine the process partition assignment 534.

The world manager module 510 can also use the hardware host configuration 528 and the hardware host metrics 530 to determine a processing partition assignment 534. For example, the world manager module 510 may use the capacity of the resources defined in the hardware host configuration 528 to determine the processing partition assignment 534. In another example, the world manager module 510 can use the hardware host metrics 530 to determine whether utilization of hardware or software resources affects the processing partition assignments 534.

After determining the process partition 532 using the data object 524, the world manager module 510 may send the process partition 532 to the hardware host 504 in preparation for the hardware host 504 to process the data object 524. For example, the world manager module 510 may send the processing partition assignment 534 to the hardware host 504. A first one of the hardware hosts 504 may determine which of the processing partitions 532 is assigned to the first hardware host. The first hardware host may process one or more of the data objects 524 mapped to those processing partitions 532 assigned to the first hardware host.

The hardware host 504 may include a hardware host manager module 540, an application manager module 542, one or more application containers 544a-n, a Distributed Shared Memory (DSM) module 546, a data store 548, one or more processors 550, and one or more memory modules 552. Data store 548 can include one or more native data objects 560, an application library 562, one or more native hardware host metrics 564, one or more native processing partitions 566, and one or more native processing partition assignments 568.

The hardware host manager module 540 may be configured to manage the operation of the hardware host 504 to process the data objects 524. Hardware host manager module 540 may store one or more data objects 524 as local data objects 560 in file system data store 548. The hardware host manager module 540 may also receive one or more applications 526 using the application library 562 for storage in the data store 548. The hardware host manager module 540 may configure the application library 562 with all or a portion of the application(s) 526. Thus, the application library 562 may be used to provide on-demand access to one or more applications 526 on the hardware host 504 without additional configuration.

The hardware host manager module 540 may also monitor utilization of hardware and software resources and store one or more local hardware host metrics 564 in the data store 548. The hardware host manager module 540 may monitor the operation of the application manager module 542, the application container(s) 544a-n, and the distributed shared memory module 546 or the hardware host 504 as a whole. The hardware host manager module 540 may report the local hardware host metrics 564 back to the world manager module 510 for storage as the hardware host metrics 530.

Hardware host manager module 540 may manage storage for local processing partition(s) 566. Hardware host manager module 540 can receive processing partition 532 from world manager module 510 and hardware host manager module 540 can store a copy of processing partition 532 in data store 548 using local processing partition 566. In addition, the hardware host manager module 540 may report back to the world manager module 510 changes to the local processing partition 566, which the world manager module 510 may use to update the processing partition 532 and the processing partition assignments 534.

The hardware host manager module 540 may manage the storage of the local processing partition assignment(s) 568. The hardware host manager module 540 may receive one or more of the processing partition assignments 534 from the world manager module 510. The hardware host manager module 540 can use the local processing partition assignment 568 to store a copy of the processing partition assignment 534 in the data store 548, and the hardware host manager module 540 can use the copy to determine the assigned processing partition. Further, the hardware host manager module 540 may also maintain copies of the processing partition assignments 534 for other hardware hosts 504 to facilitate communication and data routing with other hardware hosts 504.

In accordance with one example of the present technology, hardware host manager module 540 may analyze local processing partition assignment(s) 568 to determine an assigned processing partition of local processing partitions 566. The hardware host manager module 540 may request one or more of the processing partitions 532 from the world manager module 510, or the hardware host manager module 540 may receive the processing partitions 532 along with the processing partition assignments 534 from the world manager module 510.

The application manager module 542 may be configured to manage a lifecycle of an application on the hardware host 504 using the application library 562. For example, application manager module 542 can read local processing partition(s) 566 to determine which of application(s) 526 was used during processing of local data object(s) 560 mapped to local processing partition(s) 566. The application manager module 542 may retrieve one or more of the applications 526 from the application library 562 and instantiate the applications to process the local data objects 560 according to the object type. The application manager module 542 may load application instances into the application containers 544a-n and track the processing of the local data objects 560 by the application instances.

Distributed shared memory module 546 may manage data objects 524 using shared memory in a memory device. Distributed shared memory module 546 may read local processing partition(s) 566 to identify local data object(s) 560. For example, metadata referencing local data object(s) 560 can be in local processing partition(s) 566 and can be used to obtain and store data object(s) 524 locally at hardware host 504. Distributed shared memory module 546 may then load local data object 560 into shared memory as shared memory data object 570 for use as needed by application instances executing in application containers 544 a-n.

Distributed shared memory module 546 may also process requests from application instances executing in application containers 544a-n to access shared memory data objects 570. The requests to access the shared memory data object 570 may include create, read, update, and delete operations by application instances executing in the application containers 544 a-n. Since the shared memory data object 570 is processed by an application instance executing in the application containers 544a-n, the distributed shared memory module 546 may record state changes to the shared memory data object 570 using a thread-safe and low-copy cost storage process and a serialization format understood by the distributed shared memory module 546, as will be discussed in later sections.

Distributed shared memory module 546 may further copy state changes made to shared memory data object 570 to other application instances and across other ones of hardware hosts 504. Distributed shared memory module 546 may determine a subscriber list that identifies application instances on hardware hosts or other hardware hosts registered to receive a data object event stream associated with shared memory data object 570. When the shared memory data object 570 is processed by an application instance executing in an application container 544a-n, the distributed shared memory module 546 may send a state change to the shared memory data object 570 to the subscriber list. Distributed shared memory module 546 may also subscribe to data object event streams associated with distributed shared memory modules on other hardware hosts. Distributed shared memory module 546 may receive state changes from other hardware hosts to reflect in shared memory data object 570.

In accordance with one example operation of the present technique, distributed shared memory module 546 may determine changes to local processing partition 566 as shared memory data object 570 is processed by an application instance executing in application containers 544 a-n. The distributed shared memory module 546 may change the mapping between a local data object 560 and one of the local processing partitions 566, for example, when a data object is created (i.e., causing the addition of a mapping between the data object and one of the local processing partitions 566) and when a data object is deleted or removed from the virtual environment (i.e., causing the removal of a mapping between the data object and one of the local processing partitions 566). Distributed shared memory module 546 may change the mapping between local data objects 560 and local processing partitions 566, e.g., due to shared memory data objects 570 being processed by application instances executing in application containers 544a-n, resulting in changes to the object types, spatial locations, states, and other object properties used to group data objects 524 into processing partitions 532. Distributed shared memory module 546 may update local processing partition 566 with changes to the mapping between local data object 560 and local processing partition 566. The distributed shared memory module 546 may send any updates to the local processing partition 566 to the world manager module 510.

The world manager module 510 may receive updates from the hardware hosts 504 that reflect changes to the mapping between one or more of the data objects 524 and the processing partitions 532. For example, one of the hardware host(s) 504 may create a data object and add the data object to one of the local processing partition(s) 566. The world manager module 510 may obtain the local processing partition(s) 566 from the hardware host(s) for comparison with the processing partition(s) 532 to detect the change. In another example, one of the hardware host(s) 504 may perform operations to update or delete data objects, which results in a change in the mapping between the data object(s) 524 and the processing partition(s) 532. In yet another example, one of the hardware hosts 504 may split or merge one or more of the local processing partitions 566. For example, a processing application may specify the capability of the processing application to process multiple data objects of a given object type. A threshold (e.g., 60%) may be established for the processing application that, once approached, causes data objects handled by the first instance of the processing application to be split and redistributed between the first instance and the second instance of the processing application. Processing partitions may be merged when there is underutilization of hardware or software resources. The world manager module 510 may update the processing partition 532 with updates from the hardware host.

In response to an update to a processing partition 532, the world manager module 510 may also determine whether to update the processing partition assignment 534. World manager module 510 can update the processing partition assignment(s) 534 to load balance the processing of data object(s) 524 among hardware host(s) 504. The world manager module 510 may also update the process partition assignments 534 to ensure that frequently used data may be aggregated or collocated to the hardware host. The world manager module 510 may then send updates to the processing partition assignments 534 to the hardware host 504. Thus, the world manager module 510 can control and optimize the performance of the hardware host 504 during processing of the data objects 524.

Operating in accordance with another example of the present technology, the world manager module 510 may also monitor metrics associated with the hardware host 504 to manage the performance of the hardware host 504. The world manager module 510 can receive data associated with the metrics from the hardware host(s) 504 and use the data to determine an initial configuration for processing the partition assignments 534 or to determine whether to update the processing partition assignment(s) 534. The world manager module 510 can use data associated with the metrics to determine whether to migrate a processing partition from a first one of the hardware hosts 504 to a second one of the hardware hosts 504. For example, during processing of the data object 524, the utilization of hardware or software resources of the first hardware host may exceed a threshold and affect the performance of the first hardware host. The world manager module 510 may determine to change the assignment of some of the processing partitions 534 to balance processing load, control hardware or software utilization, reduce network bandwidth and latency, and so forth. The world manager module 510 may send the updated processing partition assignment 534 to the hardware host 504. Again, the world manager module 510 may manage the performance of the hardware hosts 504 during the processing of the data objects 524 by modifying the allocation of the processing partitions 532.

Various processes and/or other functions contained within service provider environment 500 may be executed on processors 518 and 550 in communication with memory modules 520 and 552, respectively. Service provider environment 500 may include a plurality of computing devices arranged, for example, in one or more server banks or computer banks or other arrangements. Computing devices may use hypervisors, Virtual Machine Monitors (VMMs), and other virtualization software to support computing environments.

The term "data store" can refer to any device or combination of devices capable of storing, accessing, organizing, and/or retrieving data, which can include any combination and number of data servers, relational databases, object-oriented databases, clustered storage systems, data storage devices, data warehouses, flat files, and data storage configurations in any centralized, distributed, or clustered environment. The storage system components of data stores 512, 514, 516, and 548 may include storage systems such as SANs (storage area networks), cloud storage networks, volatile or non-volatile RAM, optical media, or hard drive type media. Data stores 512, 514, 516, and 548 may represent multiple data stores, as may be appreciated.

FIG. 5 illustrates that certain processing modules may be discussed in connection with the technology and may be implemented as computing services. In one example configuration, a module may be considered a service having one or more processes executing on a server or other computer hardware. Such services may be centrally hosted functions or service applications that may receive requests and provide output to other services or consumer devices. For example, a module providing a service may be considered an on-demand computing hosted in a server, virtualized service environment, grid, or cluster computing system. An API may be provided for each module to enable the second module to send requests to the first module and receive output from the first module. Such APIs may also allow third parties to interface with the modules and make requests and receive output from the modules. Although FIG. 5 illustrates an example of a system in which the techniques described above may be implemented, many other similar or different environments are possible. The example environments discussed and illustrated above are merely representative and are not limiting.

6A-6B are flowcharts illustrating an example method for processing partitions using object domain allocation in accordance with one example of the present technology. The method 600 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine (e.g., a computer system or information processing device), by hardware components or application specific integrated circuits of an electronic device, or by a combination of software and hardware elements.

In operation 602, a world manager can receive a plurality of data objects associated with an object-based model representing a multidimensional virtual environment hosted by a distributed computing system. In operation 604, the world manager may determine a plurality of object types associated with the plurality of data objects. An example of the object type may be that a type "vehicle" is set for the object _1(object _1) and the object _2(object _2) may be a type "building". Thus, the two object groups may be separated by their type. In operation 606, the world manager may generate a mapping between the plurality of data objects and the plurality of processing partitions using the plurality of object types. As in the example, objects of type "vehicle" and objects of type "building" may be mapped to processing partition 1 and processing partition 2, respectively. The world manager may generate a data structure that indexes a plurality of data objects into a plurality of processing partitions using a plurality of object types. The data structure may include metadata about the processing partitions and data objects grouped by object type, where the data objects are mapped to the processing partitions.

In operation 608, the world manager may identify a plurality of hardware hosts in the distributed computing system to process the object-based model using the plurality of data objects. The world manager may identify hardware hosts from a pool of hardware hosts waiting for assignment, active hardware hosts with spare computing power, and the like. The active hardware host may execute an application instance in a container or in a compute instance that may operate on a processing partition. This means that the processing partition may be allocated for processing to a hardware host having an application instance that may process applications of the type mapped to the data object of the processing partition. In one example, a first hardware host may have an application instance that may handle a "vehicle" type and a second application instance that may handle a "building" when a physical event (e.g., a collision or explosion occurs).

In operation 610, the world manager may generate a plurality of processing partition assignments between a plurality of processing partitions and a plurality of hardware hosts. The world manager may generate a data structure that allocates the plurality of processing partition assignments to the plurality of hardware hosts. The data structure may include metadata about the hardware host and the processing partitions allocated for processing to the hardware host. In operation 612, the world manager may send a plurality of processing partition assignments to the plurality of hardware hosts to organize the plurality of hardware hosts to process the plurality of data objects using the plurality of object domains. Thus, data objects may be processed based on the type of object.

Method 600 continues with reference number "A" from FIG. 6A to FIG. 6B. In operation 614, the world manager may monitor the processing of the plurality of data objects mapped to the plurality of processing partitions by the plurality of hardware hosts. The world manager may receive data associated with utilization of hardware and software resources, changes to data objects, and changes to processing partitions. In operation 616, the world manager may receive a change to a processing of a plurality of data objects mapped to a plurality of processing partitions. The change may include an update to a processing partition, which may include mapping a newly created data object to the processing partition, modifying a data object mapped to the processing partition, or removing a mapping between a data object and the processing partition. For example, the number of data objects mapped to a processing partition may increase for other reasons (e.g., players entering the area), and load balancing may need to occur by correcting assignments. In another example, the object type may change from "building (building)" to "exploded building (expanded building)", which may affect the processing partition assignment of the data object. The changes to the processes of the plurality of processing partitions may also include changes due to over-utilization of hardware and software resources on a hardware host.

In operation 618, the world manager can use the change to determine whether to update the multiple processing partition assignments. The world manager may determine to update multiple processing partitions to manage or optimize performance of the hardware host. The world manager may load balance the processing of data objects across multiple hardware hosts, for example, in response to underutilization or over-utilization of hardware or software resources. The world manager may also redistribute the processing partitions due to changes in the data objects or processing partitions. If the world manager uses the change to determine not to update the plurality of processing partition assignments in step 620, the method 600 continues in operation 614, where the world manager returns to monitoring the plurality of hardware hosts in operation 614.

If the world manager determines at step 620 to update the plurality of processing partition assignments using the change, the method 600 continues at operation 622 where the world manager updates the plurality of processing partition assignments using the change at operation 622. The world manager may update the plurality of processing partition assignments, for example, to migrate a processing partition assignment from a first hardware host to a second hardware host due to exceeding a threshold of CPU usage associated with the first hardware host. The world manager may migrate the processing partition from the first hardware host to the second hardware host to concatenate data more frequently used by the second hardware host. The world manager may also create a new processing partition assignment for the hardware host that moved from the warming pool to the active pool. The method 600 continues using reference "B" from FIG. 6B and returns to FIG. 6A, where in operation 614 the world manager can send multiple processing partition assignments (including any updates) to multiple hardware hosts.

FIG. 7 is a flow diagram illustrating an example method 700 for processing data objects grouped by object domain to a processing partition in accordance with one example of the present technology. The method 700 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine (e.g., a computer system or information processing device), by hardware components or application specific integrated circuits of an electronic device, or by a combination of software and hardware elements.

In operation 702, a hardware host may receive a plurality of processing partition assignments, a processing partition assignment being an assignment between a plurality of processing partitions and a plurality of hardware hosts. The hardware host may receive a plurality of processing partition assignments from the world manager. In operation 704, the hardware host may identify a processing partition assigned to the hardware host using the processing partition assignment. In one configuration, a single processing partition may be assigned to a hardware host. In another configuration, a number of processing partitions may be assigned to a hardware host (e.g., dependent, independent, dependent, or independent processing partitions).

In operation 706, the hardware host may load the plurality of data objects mapped by the first object type into a processing partition in a memory device associated with the hardware host using the distributed shared memory. In operation 708, the hardware host may determine data dependencies used by the processing applications associated with the processing partitions. The data dependency may indicate additional object types that are used by the processing application to which the data objects mapped to the processing partition depend. The data dependency may indicate that the processing application may subscribe to receive data object event streams associated with additional object types. The data correlation may further indicate a subscription policy for filtering sources from which additional object types are obtained. An application instance on a hardware host may communicate with the distributed shared memory using a privilege model of the object type of the data object currently in use in processing. Read-only rights in the rights model may be used to determine data dependencies.

In operation 710, a hardware host using distributed shared memory may subscribe to additional data objects of a second object type using data dependencies. The subscription may be used to identify and filter sources of data object event streams associated with additional data objects of the second object type. The distributed shared memory of the plurality of hardware hosts may be a source for a stream of data object events associated with additional data objects of the second object type. The plurality of distributed shared memories may communicate with each other to exchange data objects or to exchange updates to data objects as part of a subscription policy identified for data dependencies.

In operation 712, the hardware host may receive a list of subscribers having data dependencies on a plurality of data objects grouped into the processing partition by the first object type. The subscriber list may be used to identify and filter destinations for a stream of data object events associated with a plurality of data objects of the first object type. A processing application on the distributed shared memory of the same hardware host and multiple hardware hosts may be a destination for a stream of data object events associated with multiple data objects of the first object type. As discussed above, an application instance on a hardware host may communicate with a distributed shared memory of a data object currently being used for processing. Additionally, multiple distributed shared memories may communicate with each other to exchange data objects or updates to data objects between subscriber lists.

In operation 714, the hardware host may process, using the processing application, a plurality of data objects grouped by a first object type to the processing application using the added data object. The application instance may read data associated with the data object from the distributed shared memory. An application instance may perform operations or computations on data objects, some of which may result in changes to the data objects. In operation 716, the hardware host may manage storage of the plurality of data objects in the memory device with the plurality of requests from the processing application using the distributed shared memory. The plurality of requests may include requests to modify data associated with the plurality of data objects mapped to the processing partition. The application instance may submit a request to the distributed shared memory to modify data associated with the data object. The distributed shared memory may process the request to record changes to the data object caused or implemented by the request.

In operation 718, the hardware host may send a plurality of changes to the plurality of data objects caused by the plurality of requests to the subscriber list using the distributed shared memory. As discussed above, an application instance on a hardware host may communicate with a distributed shared memory of a data object currently being used for processing. Additionally, multiple distributed shared memories may communicate with each other to exchange data objects or updates to data objects between subscriber lists.

FIG. 8 illustrates a system 800 and related operations for storing and replicating state of data objects in a distributed computing system, in accordance with examples of the present technology. The system 800 may include a distributed shared memory 802. Some examples of distributed shared memory 802 may include distributed shared memory module 274 described with respect to fig. 2 and distributed shared memory module 546 described with respect to fig. 5. Distributed shared memory 802 may manage the storage of data objects 404 for system 800. The distributed shared memory 802 may manage the storage of the data objects 404 using a storage process to create or load the data objects 404 into one or more memory devices using an in-memory representation of the data objects 404. Distributed shared memory 802 may also manage the storage of data objects 404 using a storage process to handle requests for reads and writes to the in-memory representation of data objects 404.

In accordance with the present techniques, distributed shared memory 802 may use a stored procedure that includes a serialized format for data objects 404. The distributed shared memory 802 may use a storage process to represent data objects as hierarchical data within memory and on disk using multiple segments. The distributed shared memory 802 may further utilize log records in one of the sections to represent modifications to the data object 404 to minimize the time cost of reflecting state changes in the highly distributed real-time computation.

For example, the distributed shared memory 802 may use a stored procedure and implement a serialized format of the data objects 404 to minimize the time cost of reflecting state changes based on round (epoch) simulations, where each round provides an execution interval. Round based simulation may use a scenario with 1 writer/N reader processes for data object 404, with each process potentially containing multiple threads, where the processes may have a round or execution interval lasting about 30 ms. A typical RPC (remote procedure call) framework (such as grpc/protobuf) starts to lag behind as the round frequency increases, and the size of the message used to reflect the state change increases. This may be due to the CPU time taken to deserialize the data object and the cost of synchronizing the entire data object when it is updated. The distributed shared memory 802 may use stored procedures in accordance with the present technology to meet performance requirements because the stored procedures provide platform-independent and language-independent data format specifications for use with the data objects 404.

The distributed shared memory 802 may provide a small, predictable amount of overhead because code written to use stored procedures in a serialized format should perform CRUD (create, read, update, delete) operations as close as possible to the native stored procedures. The distributed shared memory 802 may include overhead for using storage processes with a serialized format, which may maintain amortized O (1) costs. By keeping the memory layout the same as the store/line format, the distributed shared memory 802 also does not incur the serialization cost of using a stored procedure. Additionally, distributed shared memory 802 may use stored procedures to implement updates to data object 404 in the form of log records. Distributed shared memory 802 may group together as a log one or more modifications to data object 404 within a round or execution interval.

The distributed shared memory 802 may provide a mechanism for processing log records of applications and other distributed shared memories to retrieve and consume data objects 404. Thus, the distributed shared memory 802 may be highly concurrent because there is a separation between how the storage process enables the storage of data objects and the storage of metadata for tracking changes to the state of the data objects 404. Finally, the distributed shared memory 802 is efficient because the storage process can use binary encoding to represent the data of the data object, and therefore, reading the data field from the encoding simply uses a binary encoding scheme.

In this example, the distributed shared memory 802 may retrieve the data objects 404 from one or more sources (not shown), such as a file, an object data store, another distributed shared memory, and so forth. As discussed above, the data object 404 may be associated with an object-based model of a virtual environment (e.g., a 3D simulation, a 3D game, etc.). At the beginning of the lifecycle of the data objects 404 in the virtual environment, the distributed shared memory 802 may manage the storage of the data objects 404 in the memory devices during object creation and loading using a storage process in accordance with the present techniques. For example, the distributed shared memory 802 may generate one or more object representations 804 for the data objects 404 in the memory devices. While keeping the memory layout the same as the store/line format, distributed shared memory 802 may copy data objects 404 directly to the memory devices. Alternatively, the distributed shared memory 802 may parse the data object 404 to write the data object to the memory device using a stored procedure. The object representation 804 may include one or more sections in a memory device. Some examples of sections may include metadata sections, virtual table sections, object data sections, journal sections, and so forth. For example, at creation time, object representation(s) 804 may include object metadata section 806, virtual table 808, and object data section 810. At other times during the lifecycle of the data object 404, the distributed shared memory 802 may add one or more segments and portions of segments to the object representation 804 using a storage process.

The distributed shared memory 802 may use the object metadata section 806 to describe the in-memory/on-disk/on-wire content of the data object 404, as well as how to read the object representation 804. The object metadata section 806 may include one or more metadata fields. The object metadata section 806 may include a length field indicating the length (e.g., in bytes) of the object metadata section 806 and an object length field indicating the total combined length of the object metadata section 806, the virtual table 808, and the object data section 810. The object metadata section 806 may further include: an object Identifier (ID)812 field that provides a unique identifier for identifying the data object; and a base version field indicating the version of the data object represented by the object data section 810. The object metadata section 806 may include: a virtual table offset field that provides a memory offset to the beginning of the virtual table 808 (e.g., from the beginning of the object metadata section 806), and an object data offset field that provides a memory offset to the beginning of the object data section 810 (from the beginning of the object metadata section 806).

The distributed shared memory 802 may use the virtual table 808 to describe one or more portions of the object data segment 810. Virtual table 808 may include one or more table fields and table entries. For example, the virtual table 808 may include an offset array from the beginning of the object data segment 810 that points to the portion of the object data segment 810. In one example, the desired memory location in the object data segment 810 can be found by using the offset found in the corresponding table entry of the virtual table 808. In another example, the nth data field of the data object may be found by using an offset found in the nth index of the virtual table 808. The virtual table 808 may include a length field that indicates the length (e.g., in bytes) of the virtual table 808 and one or more table entries or field offsets that provide an array of table entries or field offsets that represent each of the data entries or data fields in the object data section 810. The distributed shared memory 802 may use the object data section 810 to contain data entries, data fields, data values, etc. of the data object 404. Object data section 810 may include a length field that provides the length (e.g., in bytes) of object data section 810 and a byte array that represents the data values in the data object.

In accordance with another example of the present technology, the distributed shared memory 802 may manage the storage of state changes for the data objects 404 during the lifecycle of the data objects 404 in the virtual environment. For example, distributed shared memory 802 may provide a mechanism for processing applications and other distributed shared memory to retrieve and process data objects 404. The distributed shared memory 802 may receive one or more requests 820. The request 820 may be issued by one or more processing applications. The request 820 may instruct the distributed shared memory 802 to create or delete an object representation 804 in a memory device, read the object representation 804 from the memory device, and write changes to the data object 404 represented by the object representation 804.

In one example, the distributed shared memory 802 may analyze the request 820 to identify a read request associated with the data object 404 to be processed using the object representation 804 in the memory device. The distributed shared memory 802 may read the object metadata section 806 of the object representation(s) 804 to determine whether the request(s) 820 match, for example, a particular object ID812, object version 814, and object type 816. Upon satisfaction of the initial validation check, the distributed shared memory 802 may obtain the memory location of the virtual table 808 from the object metadata section 806. The distributed shared memory 802 may read the virtual table 808 to determine the field offset in the object data section 810 to obtain the in-memory content of the data object 404. The distributed shared memory 802 may read the object data section 810 using the field offset to obtain the in-memory content of the data object 404 from the object representation 804. Distributed shared memory 802 may include in-memory contents of data object(s) 404 obtained from object representation(s) 804 and may send a response to one or more requesters associated with request(s) 820.

In another example, distributed shared memory 802 may analyze request 820 to identify write requests associated with data object 404 and data to be written to data object 404 using object representation 804 in the memory device. The distributed shared memory 802 may again read the object metadata section 806 of the object representation(s) 804 to determine whether the request(s) 820 match, for example, a particular object ID and object version. Upon satisfying the initial validation check, the distributed shared memory 802 may append the version-ordered changeset to an object representation 804 in the memory device.

The distributed shared memory 802 may use a series of log records in a log section to reflect state changes of the object representation 804 in the memory device. As shown in fig. 8, object representation(s) 804 may also include a log section 830. The distributed shared memory 802 may use the log section 830 to represent a change in state of the data object 404 represented in the object representation 804. The distributed shared memory 802 may access the current state of the data object 404 by reading the log section 830 from the tail and reading back to obtain the latest information for the data object 404 contained in the object representation 804 in the memory device. Distributed shared memory 802 may manage the current state of data object(s) 404 by appending one or more log records 832 (e.g., log records 832a-n) to object representation(s) 804 to reflect version-ordered changes or write sets. The distributed shared memory 802 may use the log record 832 to describe the changeset to the data object 404 and to which version of the data object the change applies. The distributed shared memory 802 may append the log records 832 to the object representation 804 in reverse order to facilitate end-to-head read mechanisms as well as single-pass writes and thread-safe reads.

FIG. 9 illustrates a graphical example of an object representation 900 used by distributed shared memory to manage storage of data objects in a distributed computing system that provides low replication cost and thread safe reading in accordance with examples of the present technology. The distributed shared memory may implement the object representation 900 using a set of semantics built on top of the byte array. Object representation 900 may provide low replication costs because the byte array may be replicated "as is" to another memory location, to a file on disk, across a network to another hardware host, and so on, because the serialized format of object representation 900 reduces the additional processing of formatting data of different storage or transmission media.

The distributed shared memory may partition the object representation 900 into one or more sections in memory, on wires, and on disks as files. Object representation 900 may include a metadata section 902, a virtual table section 904, an object data section 906, and a journal section 908. Object representation 900 may provide thread safe reads because the organization of the extents in object representation 900 prevents reads from extents that may be written to at the same time. Object representation 900 may include a resizable byte array (RBArray) that provides an abstraction around the resizable byte array that allows for atomic "reservation" of bytes. The object representation 900 may enable a request to resize an array to allocate additional bytes to complete as an atomic process before servicing another pending request to resize the array. Thus, object representation 900 makes it thread safe for the atom to reserve a queue that can allow additional writes to object representation 900.

The distributed shared memory may use the metadata section 902 of the object representation 900 to describe the byte array and provide information on how to read the byte array. Metadata section 902 may include one or more metadata fields, such as metadata length 910, object length 912, object identifier 914, base version 916, virtual table offset 918, and object data offset 920. The metadata length 910 may specify the length of the metadata section 902 in bytes. Object length 912 may specify the total length of the byte array without log section 908. The object identifier 914 may specify a unique identifier to identify this object. Some examples of object identifiers 914 may include UUIDs or GUIDs.

Base version 916 may specify the version of the data object represented by object data section 906. The virtual table offset 918 may specify an offset (from the beginning of the byte array) to the beginning of the virtual table section 904. Object data offset 920 may specify an offset (from the beginning of the byte array) to the beginning of object data section 906.

The distributed shared memory may use virtual table segment 904 of object representation 900 to describe the offset into a particular portion of object data segment 906. The virtual table section 904 may include a virtual table length 930 that specifies the length of the virtual table 932 in bytes. Virtual table 932 may include an array of offsets starting from the beginning of object data section 906. The nth field of the data object represented by the byte array may be found by using the offset found in the nth index of the array in virtual table 932.

The distributed shared memory may use the object data section 906 of the object representation 900 to store data objects. Object data section 906 may include an object data length 940 that specifies the length of object data 942 in bytes. Data associated with the data object may be retrieved from a file, object store, or the like, and copied into object data 942. Object data 942 may include an array of bytes that represent a data object. Object data 942 may include: integers, floating points, characters, strings, binary large objects (blobs), and the like, as well as combinations thereof. As described above, the object identifier 914 identifies the data object represented in the object data section 906. Base version 916 specifies the version of the data object represented in object data section 906.

The distributed shared memory may use the log section 908 of the object representation 900 to represent a state change of the data object. The distributed shared memory may use the log section 908 to write a version-ordered changeset to the data object whose contents are represented in the object data section 906. The log section 908 may include a series of one or more log records 950 (e.g., log records 950 a-d). The distributed shared memory may read the log section 908 from the end and back to obtain the current state of the data object. The distributed shared memory may use log records 950 to describe a set of changes to a data object. The distributed shared memory may write log records 950 to log section 908 by appending log records 950 in reverse order to facilitate a tail-to-head read mechanism, a single-pass write, and a thread-safe read.

In this example, log record 950d may include one or more log fields, such as length 960, which specifies the length of log record 950d in bytes. The log record 950d may include a version field 962. Distributed shared memory may use version field 962 to specify the version of the data object to which log record 950d applies. The distributed shared memory may also use the version field 962 to specify a portion of the data record or version of a particular data field to which the log record 950d applies. Log record 950d may also include an object identifier 964 and a modified field table 966, object identifier 964 providing an object identifier for the data object to which log record 950d applies.

In accordance with one example of the present technology, the distributed shared memory may use the modified field table 966 to indicate where the state of the data object has changed. Modified field table 966 may include a reference indicating that a portion of log record 950d includes a change to data associated with a data object, which may be different from data currently associated with the data object and stored in a portion of object data section 906. For example, the distributed shared memory may use modified field table 966 to include references indicating which object fields of the data object have been changed. Modified field table 966 may include a length 972 (in bytes) to describe the size of modified field table 966, as well as bit field contents 974 to represent the actual modified field using an index. In this example, distributed shared memory may set one or more bits of bit field contents 974 to indicate that the object field has changed. Each bit in a byte may correspond to an ordered set of object fields. Setting a bit of a corresponding object field in bit field contents 974 may provide a reference indicating a change to the object field.

In accordance with one example of the present technology, the distributed shared memory may use translation tables 968 to indicate where the state of the data object has changed. The log record 950d may further include a translation table 968 and a table content 978, the translation table 968 providing an internal virtual table that includes a length 976 (in bytes) to describe the size. The distributed shared memory may use table content 978 as a mapping between an offset pointing to the modified data section 970 and the changed object fields that may also be identified in the modified field table 966. The distributed shared memory may use the modified data section 970 to store changes to the data object. Modified data section 970 may include an array of bytes representing changes to object fields in log record 950 d. In accordance with one example of the present technique, the distributed shared memory may append the log records 950a-d in memory one after another or consecutively. In another example, the distributed shared memory may include a previous record pointer or another type of linked list scheme in the log records 950 a-d. The distributed shared memory may use the previous record pointer to provide gap detection to detect incomplete logs.

In accordance with another example of the present technology, a distributed shared memory may aggregate multiple requests to modify data associated with a data object during a round or execution interval in a distributed computing system. The distributed shared memory may use translation tables 968 and/or modified field tables 966 to include references to portions of the data object affected by multiple requests. The distributed shared memory may append a single log record for the round to the log section 908 of the object representation 900. A single log record may include multiple changes to the data object implemented by multiple requests that are referenced as changes in the translation table 968 and/or the modified field table 966 that affect multiple portions of the data object.

In accordance with yet another example of the present technology, the distributed shared memory may aggregate multiple requests to modify data associated with a data object during a round or execution interval in the distributed computing system when the requests apply to the same portion of the data object. The distributed shared memory may append a single log record for the round to the log section 908 of the object representation 900. A single log record may include the latest series of changes to the data object implemented by multiple requests that are referenced as a single change in the translation table 968 and/or the modified field table 966 that affect portions of the data object.

In accordance with another example of the present technology, the distributed shared memory may append a plurality of log records to the log section 908 of the object representation 900 during a round or execution interval in the distributed computing system. The distributed shared memory may determine to merge multiple log records into a single log record (e.g., at the end of a round, when a read is made, or at another point in time) in order to save storage space or improve later read times to determine the current state of the data object. The distributed shared memory may write a single log record to the log section 908 to replace multiple log records. The single log record may include a plurality of references indicating that a plurality of changes to a data object aggregated from the plurality of log records reflect changes to a plurality of portions of the data object.

In accordance with additional examples of the present technology, the distributed shared memory may use the object representation 900 to communicate state changes associated with data objects. The distributed shared memory may receive instructions to send the current state of the data object to a recipient, such as a processing application, another distributed shared memory, an object data store, and so on. The distributed shared memory may determine whether the recipient includes a previous state of the data object. For example, the distributed shared memory may track whether all or part of object representation 900 has been previously sent to a recipient. When the distributed shared memory determines that the recipient includes a previous state of the data object, the distributed shared memory may send one or more log records 950 (e.g., all log records) in the log section 950 to the recipient. The distributed shared memory may send one or more incremental log records to the recipient that represent differences between the current state of the data object and previous states maintained at the recipient. The distributed shared memory may determine that the recipient does not include the data object and the previous state of the second object representation 900 or the entire log section 908. The distributed shared memory may consolidate the log sections 908 into a single log record as described above to represent the current state based on storage or network conditions. For example, the distributed shared memory may consolidate the log sections 908 into a single log record to reduce the storage used by the object representation 900. In another example, the distributed shared memory may consolidate the log sections 908 into a single log record to reduce network bandwidth for data transfer.

FIG. 10 is a flow diagram illustrating an example method 1000 performed by distributed shared memory for managing storage of data objects in a distributed computing system that provides low replication cost and thread safe reading in accordance with examples of the present technology. The method 1000 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine (e.g., a computer system or information processing device), by hardware components or application specific integrated circuits of an electronic device, or by a combination of software and hardware elements.

In operation 1002, a distributed shared memory of a hardware host may receive data objects associated with an object-based model of a virtual environment hosted by a distributed computing system. The distributed shared memory of the hardware host may retrieve data associated with the data object from the object data store. In another example, the distributed shared memory may receive data associated with the data object from another hardware host, a world manager, a data service, or a client. In operation 1004, the distributed shared memory may determine a memory location of the data object in a memory device associated with the hardware host. The memory devices may include volatile and non-volatile memory devices. The distributed shared memory may set aside a portion of memory in the memory device to manage the shared memory space for the representation of the data object. The distributed shared memory may allocate memory at memory locations of the data objects and share memory with processing applications on the hardware hosts and with other distributed shared memory on other hardware hosts in the distributed computing system. In this example, the distributed shared memory may share a portion of the random access memory as a shared memory space.

In operation 1006, the distributed shared memory may determine a format of the data object. The format of the data object may include an on-disk format and an in-memory format. The on-disk format may be the same or different than the in-memory format. The distributed shared memory may use the same format for file storage and memory storage to reduce the need to perform serialization prior to storage or transfer over a network. If the distributed shared memory determines in operation 1008 that the format of the data object is used by the distributed shared memory, the distributed shared memory may write the data object to a memory location in the memory device in operation 1010. For example, the distributed shared memory may copy data associated with a data object directly to a memory device, regardless of whether the data is obtained from an object data store, a file, a network packet, or the like. The distributed shared memory may perform memory copy (memcopy) to allocate data objects to the shared memory space.

If the distributed shared memory determines in operation 1008 that the format of the data object is not used by the distributed shared memory, the distributed shared memory may identify metadata associated with the data object, as in operation 1012. For example, the distributed shared memory may parse data associated with the data object to determine an object identifier that uniquely identifies the data object and an object version of the data associated with the data object. In operation 1014, the hardware host may write metadata to the in-memory representation of the data object at the memory location using the metadata section.

In operation 1016, the distributed shared memory may identify object data associated with the data object. The distributed shared memory may parse data associated with the data object to determine digital data, strings, binary blobs, etc. The distributed shared memory may apply a plurality of data transformation rules to the data. In one example, the distributed shared memory may determine a schema for the data object that identifies one or more object fields and the format of the data contained in the object fields. The distributed shared memory may apply data transformation rules to data contained in the object fields to transform and prepare the data for storage in the memory devices. In operation 1018, the distributed shared memory may write the object data to the representation of the data object at the memory location using the object data segment. Continuing with the previous example, the distributed shared memory may use the object data section to store data contained in an object field associated with the data object in a byte array.

In operation 1020, the distributed shared memory may identify one or more object fields associated with the data object. In one example, the distributed shared memory may identify one or more portions of data associated with a data object to determine one or more object fields and resolve the object data to values for the object fields. In another example, the distributed shared memory may detect a pattern of data objects and use the pattern to identify object fields and structures of the data objects. In operation 1022, the distributed shared memory may write a mapping between the object field and the portion of the data object segment containing the value of the object field using the virtual table. The distributed shared memory may use a virtual table to describe an offset array from the beginning of the object data segment. The distributed shared memory may access a memory location in an object data section of an nth field of the data object by using an offset found in an nth index of the virtual table.

FIG. 11 is a flow diagram illustrating an example method 1100 performed by distributed shared memory to manage storage of state of data objects in a distributed computing system using an ordered, append only log based format to provide versioned state snapshots in accordance with examples of the present technology. The method 1100 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine (e.g., a computer system or information processing device), by hardware components or application specific integrated circuits of an electronic device, or by a combination of software and hardware elements.

In operation 1102, a distributed shared memory of a hardware host may receive a request to modify data associated with a data object. The distributed shared memory may receive the request from a processing application configured to process a data object on the hardware host according to an object type of the data object. The distributed shared memory may receive a request in response to a processing application issuing a request via an API for accessing data objects, wherein storage of the data objects is managed by the distributed shared memory.

In operation 1104, the distributed shared memory may determine whether the request to modify data associated with the data object matches an object identifier and version associated with its representation storing the data object managed by the distributed shared memory. The distributed shared memory may analyze the request to determine whether the request specifically identifies the data object. The distributed shared memory may further analyze the request to determine whether the request is directed to storing a version of a data object being managed by the distributed shared memory.

If the distributed shared memory determines in operation 1106 that the request does not match the object identifier and version associated with the representation of the data object, the distributed shared memory may use the request to generate an error message in operation 1108. The distributed shared memory may include details associated with the version and object identifier referenced by the request in an error message. The error message may also include data, memory contents, stack, heap, and other debug information associated with the processing application.

If the distributed shared memory determines in operation 1106 that the request matches the object identifier and version associated with the representation of the data object, the distributed shared memory may identify a change to the data object in the request, as in operation 1110. The change may include any change to the data object, such as a change in a field identifier, a change in a field value, or any other write to the data object. The distributed shared memory may analyze the request to identify a change. The distributed shared memory may interpret the request according to an API for modifying data objects, wherein storage of the data objects is managed by the distributed shared memory. In one example, the request may indicate that the object field is modified using the provisioning value. The distributed shared memory may determine the object fields and the provisioning values from the request.

In operation 1112, the distributed shared memory may write the changes to the memory locations associated with the representation of the data object using the log record. For example, the distributed shared memory may write log records to log sections of representations of data objects whose storage is managed by the distributed shared memory. The distributed shared memory may append log records to a series of log records in a log section.

In operation 1114, the distributed shared memory may identify an object field associated with the change. The distributed shared memory may analyze the request to identify an object field in the request. Using the previous example, the request may indicate one or more object fields to be modified and new corresponding values for the object fields, and the distributed shared memory may track which object fields are being modified. In operation 1116, the distributed shared memory may write a translation table to the log record that maps the object field to the portion of the log record that includes the change.

In operation 1118, the distributed shared memory may generate a reference indicating that the change reflects a change to an object field. The reference may include: a bit, byte, flag, resource identifier, or other data that identifies an object field. In operation 1120, the distributed shared memory may log the modified field table including the reference. As discussed above, the distributed shared memory may use the modified field table to indicate which portions of the data object have state changes. For example, a hardware host may set one or more bits of a byte or group of bytes to indicate that the object field corresponding to the bit has changed. Each bit in the byte may be mapped to or associated with an object field in the ordered set of object fields. Setting the bit of the corresponding object field may provide a reference indicating that the change reflects a change to the object field. The distributed shared memory may write the byte or group of bytes with one or more bits set to the log record as a modified field table.

In accordance with one example of the present technology, the distributed shared memory may perform the method 1100 with multiple requests to modify data associated with a data object during a round or execution interval in a distributed computing system to aggregate or merge the requests into a single log record. The distributed shared memory may use the modified field table to include references to portions of the data object affected by the multiple requests. The distributed shared memory may append a single log record for the round to the log segment in which it stores the representation of the data object managed by the distributed shared memory. In accordance with yet another example of the present technology, the distributed shared memory may aggregate multiple requests to modify data associated with a data object during a round or execution interval in the distributed computing system when the requests apply to the same portion of the data object. The distributed shared memory may append a single log record for the round to the log segment in which it stores the representation of the data object managed by the distributed shared memory. A single log record may include the latest series of changes to a data object implemented by multiple requests that are referenced as a single change in the modified field table.

FIG. 12 is a flow diagram illustrating an example method 1200 performed by a distributed shared memory for exchanging states of data objects in a distributed computing system between hardware hosts in accordance with examples of the present technology. The method 1200 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine (e.g., a computer system or information processing device), by hardware components or application specific integrated circuits of an electronic device, or by a combination of software and hardware elements.

In operation 1202, a first hardware host that uses a distributed shared memory to store a representation of a data object may receive an instruction to send a current state of the data object to a second hardware host. The first hardware host may receive the instruction as a data request from the second hardware host. In another example, the first hardware host may receive instructions in the form of a subscriber list. The first hardware host may publish changes made to the data object using the subscriber list. The subscriber list may include processing applications on the first hardware host, the second hardware host, and other hardware hosts and services in the distributed computing system. The first hardware host may publish the changes directly to the subscriber, or the first hardware host may notify the subscriber that the changes are available. The subscriber may then retrieve or request the change from the first hardware host.

In operation 1204, the first hardware host may determine whether the second hardware host includes a previous state of the data object. For example, the first hardware host may use the distributed shared memory to determine whether data associated with the representation of the data object has been previously sent to or obtained by the second hardware host. The first hardware host may communicate with the second hardware host to determine whether the second hardware host includes a previous state of the data object. The first hardware host may analyze the instruction for an indication of whether the second hardware host includes a previous state of the data object. In another example, the first hardware host may use the distributed shared memory to track communications sent to the second hardware host indicating a previous state of whether the second hardware host contains the data object.

If the first hardware host determines in operation 1206 that the second hardware host does not include a previous state of the data object, the first hardware host may send a representation of the data object to the second hardware host using distributed shared memory in operation 1208. The distributed shared memory of the first hardware host may initiate a memory copy that sends the in-memory representation of the data object to the distributed shared memory of the second hardware host. The distributed shared memory of the first hardware host does not incur serialization costs when sending the in-memory representation of the data object to the distributed shared memory of the second hardware host because the in-memory layout is the same as the storage/line format in accordance with the present techniques. The distributed shared memory of the second hardware host may then load the memory copy directly into the memory device.

If the first hardware host determines in operation 1206 that the second hardware host includes a previous state of the data object, the first hardware host may identify, in operation 1210, a portion of the log segment associated with the representation of the data object that represents a difference between the current state and the previous state of the data object using the distributed shared memory. The distributed shared memory of the first hardware host may determine that a single log record represents a difference between a current state and a previous state of the data object. In another example, the distributed shared memory of the first hardware host may determine that the plurality of log records represent a difference between a current state and a previous state of the data object.

In operation 1212, the first hardware host may send the portion of the log segment to the second hardware host using the distributed shared memory. The distributed shared memory of the first hardware host may send a single log record to the second hardware host, the single log record identified as representing a difference between a current state and a previous state of the data object. In another example, the distributed shared memory of the first hardware host may send a plurality of log records to the second hardware host, the log records identified as representing differences between the current state and the previous state of the data object.

In yet another example, the distributed shared memory of the first hardware host may consolidate one or more log records identified as representing differences between the current state and the previous state of the data object into a single log record to be sent to the second hardware host. The first hardware host may use the distributed shared memory to send a single log record to the second hardware host to reduce network bandwidth for data transfers.

In accordance with one example of the present technology, the distributed shared memory may use method 1200 to save or archive the current state of data objects in an object data store or file. The distributed shared memory may determine to merge multiple log records representing the current state into a single log record in order to save storage space or improve later read time to determine the current state of the data object. For example, the distributed shared memory may write a single log record to a log segment to replace multiple log records before storing the single log record as a file on disk. The single log record may include a plurality of references indicating that a plurality of changes to the data object aggregated from the plurality of log records reflect changes to a plurality of portions of the data object representing a current state of the data object.

FIG. 13 illustrates a system 1300 and related operations for assigning computing units in a distributed computing system using processing partitions with data objects organized by spatial locality, in accordance with examples of the present technology. The system 1300 can include a world manager 1302 and one or more hardware hosts 1304. According to examples of the present technology, the world manager 1302 may assign computing units to hardware hosts 1304 in a distributed computing system using processing partitions organized by spatial locality.

World manager 1302 may include one or more server computers configured to manage the processing of data objects in a distributed computing system. Some examples of world managers 1302 may include cluster manager 130 described with reference to FIG. 1 and cluster manager module 210 described with reference to FIG. 2. To manage processing, in accordance with one example of the present technology, the world manager 1302 may determine one or more spatial subdivisions 1306 associated with an object-based model of a virtual environment hosted by the distributed computing system. The spatial subdivision 1306 may include a spatial region that divides world space into two or more subsets (e.g., non-overlapping disjoint subsets). The spatial region may be an arbitrary and/or uniform division of space. The spatial region may define a spatial cell as a square or rectangle, or may define a spatial cell as: arbitrary points, lines, areas, polyhedrons (e.g., polygons and polyhedrons), and conic sections such as ellipses, circles, and parabolas. The space may be a 2-dimensional space, a 3-dimensional space, or any multi-dimensional space that may be subdivided. The spatial sub-division 1306 may be hierarchical, meaning that a space (or region of space) may be divided into several spatial regions, and then the spatial regions may be recursively subdivided again to create additional spatial regions. The spatial subdivision(s) 1306 may also include multiple layers or levels of subdivisions, which may overlap one another and may include different layers or levels that provide different metrics, criteria, and granularity for subdividing the virtual world.

The spatial subdivision 1306 may be represented using one or more tree structures, such as discussed below with respect to fig. 14. In one example, world space associated with a virtual environment may be subdivided using octants, and the spatial subdivisions 1306 may be represented using an octree, which is a tree structure in which each internal node has eight children. Octrees may be used to subdivide the three-dimensional space associated with a virtual world by recursively subdividing a region into eight octants. In another example, quadrants may be used to subdivide the world space associated with the virtual environment, and the space subdivision 1306 may be represented using a quadtree, which is a tree structure in which each internal node has four children. A quadtree may be used to partition a two-dimensional space associated with a virtual world by recursively subdividing a region into four quadrants. Other methods may be used to subdivide the world space associated with the virtual environment and represent the spatial subdivisions 1306 using data structures to provide nodes to represent particular units of spatial information and nodes where data (e.g., data objects) may be associated with particular units of spatial information. Some additional examples may include: grids, Thiessen polygon (Voronoi) diagrams, space-filling curves, KD-trees, bounding volume hierarchies, and the like.

The world manager 1302 may also determine one or more processing partitions 1308 associated with the spatial subdivision 1306. For example, world manager 1302 may use spatial subdivision 1306 to create processing partition 1308. In one example, a processing partition of processing partitions 1308 may be associated with a single spatial subdivision of spatial subdivisions 1306. In another example, a single processing partition may be associated with multiple of spatial sub-partitions 1306. In another example, multiple processing partitions may be associated with one or more of spatial sub-partitions 1306. Additionally, multiple processing partitions may be associated with a single spatial subdivision.

The world manager 1302 may determine a processing partition 1308 by mapping the data objects 404 to the processing partition 1308 associated with the spatial subdivision 1306. World manager 1302 may use the spatial location information to index data objects 404 into processing partition 1308. In another example, world manager 1302 may instruct hardware host 1304 to create processing partition 1308 using spatial subdivision 1306. The world manager 1302 may define a world space associated with the virtual environment and instruct the hardware host 1304 to create a processing partition 1308 during processing of the object-based model of the virtual environment. In another example, hardware host 1304 may be responsible for indexing data object 404 into processing partition 1308 using spatial location information. Hardware host 1304 may index data object 404 when data object 404 is created, deleted, and modified by an instance of a processing application on hardware host 1304. Such operations of an instance of a processing application may result in the addition, deletion, splitting, merging, and other modifications of processing partition 1308 by hardware host 1304. The world manager 1302 may monitor the hardware hosts 1304 to receive additions, deletions, splits, merges, and other modifications to the processing partitions 1308 associated with the spatial subdivision 1306. In accordance with the present technology, the processing partition(s) 1308 may define computing units in a distributed computing system that are organized by spatial location information associated with the data object(s) 404, which may be distributed to the hardware host(s) 1304. As previously discussed, the data object(s) 404 may be associated with an object-based model of a virtual environment (e.g., a 3D simulation, a 3D game, etc.). Data object 404 may include one or more object fields 420. In this example, data object 404 may include one or more object fields 420 to provide an object identifier 430, an object version 432, and an object type 434 as metadata for data object 404. The object fields 420 may also include one or more object properties or attributes defined using field values or key-value pairs, such as a location field 1316 that may specify a first spatial attribute and a location field 1318 that may specify a second spatial attribute.

Location fields

1316 and 1318 may contain: spatial location information, such as data values, strings, X, Y, Z coordinates, vectors, boundaries, grids, ranges, etc., that may be referenced by a processing application in order to process the data object 404. Other examples of spatial location information contained in data object 404 may include: location, place, and orientation information associated with points, lines, polyhedrons (e.g., polygons and polyhedrons), conic sections (e.g., ellipses, circles, parabolas), or other types of data object models.

In accordance with one example of the present technology, the object field 420 of the data object 404 may be used to determine the processing partition 1308. Since the processing partition 1308 represents a group of data objects organized into computing units for processing by a processing application, the data objects 404 may be organized according to spatial location information in one or more object fields 420. In accordance with one example of the present technology, this data object management by the world manager 1302 and hardware hosts 1304 may include the creation of one or more indexing structures, referred to as spatial indexes, that map the data objects 404 to the processing partitions 1308 using spatial location information from the object fields 420. World manager 402 may use object field 420 to create multiple layers in a packet, thereby further organizing data objects 404 by multiple attributes, such as the object domains discussed previously.

Processing partition 1308 may be represented by one or more spatial index structures that use spatial location information associated with data objects 404 to group data objects 404 at or near spatial cells defined by spatial subdivision 1306. The spatial index elements or nodes may represent processing partitions and include data or metadata identifying the processing partitions and sets or groups of data objects organized into processing partitions by spatial unit (e.g., location or position) associated with the spatial subdivision. The spatial index representing the processing partition 1308 may contain a tree structure representing the spatial subdivision 1306. The nodes of the tree structure representing the spatial subdivision 1306 may map or index the data objects 404 to the processing partitions 1308 using the spatial location information. The spatial index representing processing partition 1308 may also contain metadata information about the relevant regions of spatial subdivision 1306, such as parents, neighbors (special siblings), children, levels, and the like.

The world manager 402 may use the processing partition 1308 to determine one or more processing partition assignments 1312. Processing partition assignment 1312 allocates processing partition 1308 to hardware host 1304. Processing partition assignment 1312 may be represented by a data structure (e.g., a list, index, tree, etc.) that identifies the hardware hosts in hardware host 1304 and the processing partitions 1308 assigned to the hardware hosts. Processing partition assignment 1312 may be made based on the number of spatial sub-partitions in spatial sub-partitions 1306, the object type of data objects 404 at a given location, the application instances that may process the object type, the spatial relationship (e.g., between spatial sub-partitions 1306 or between data objects 404), the data dependencies between data objects 404, and metrics associated with hardware host 1304, such as the processing load on hardware or software resources (e.g., to avoid overload, etc.).

World manager 1302 may send a processing partition assignment 1312 to hardware host 1304. In one example, world manager 1302 may send processing partition assignment 1312 directly to hardware host 1304. In another example, world manager 1302 may notify hardware host 1304 of a processing partition assignment 1312, and hardware host 1304 may retrieve the processing partition assignment 1312 from another source. Hardware host 1304 may then process data object 404 using processing partition assignment 1312. Examples of processing may include: data object movement, data object deletion, data object insertion, data object collision, rendering, explosion, physical simulation, worldwide credit transaction, or other types of processing. Hardware host 1304 may use processing partition assignment 1312 to identify the allocation of processing partition 1308. World manager 1302 may send processing partition 1308 to hardware host 1304 using processing partition assignment 1312. In another example, hardware host 1304 may retrieve the allocation of processing partition 1308 from another source.

FIG. 14 illustrates a graphical example of organizing computing units to assign to hardware hosts in a distributed computing system 1400 using spatial location information associated with data objects, in accordance with an example of the present technology. FIG. 14 illustrates first using object domains (e.g., "Domain A" and "Domain B") as discussed above and then using a spatial index to divide world space into more than one group or process partition. If the world space of a virtual environment hosted by a distributed computing system includes two domains, where each domain may have a large number of data objects densely packed into different spaces, the computing units of the two domains may be in different states, as depicted. Thus, FIG. 14 illustrates the use of multiple spatial subdivisions (e.g., "subdivision A" and "subdivision B") to further divide world space into more than one group or process partition.

The distributed computing system 1400 may include a spatial index manager 1402, which spatial index manager 1402 may include one or more server computers configured to index data objects in a virtual world hosted by the distributed computing system 1400 into computing units or processing partitions using spatial location information. Some examples of the spatial index manager 1402 may include a world manager 1302 and a hardware host 1304 described with reference to FIG. 13. Spatial index manager 1402 may determine a plurality of processing partitions associated with a plurality of spatial subdivisions of the domain of subdivision a and subdivision B. For example, spatial index manager 1402 may initially create and manage one or more spatial indexes 1408 using one or more object fields 1404 and index configuration 1406 (as described later).

The spatial index manager 1402 may analyze spatial location information associated with the object field(s) 1404 to determine a set of data objects that may be organized into processing partitions by location, proximity, size, shape, spatial relationship, spatial dependency, spatial causal relationship, and the like. For example, spatial index manager 1402 may analyze spatial location information to determine whether data objects can be grouped into processing partitions by location or place. In another example, data objects that are near or approximately at the same location or place as each other may be grouped into the same processing partition. Spatial index manager 1402 may further analyze the spatial location information to determine whether a spatial relationship exists for the data object. For example, a first data object may have a spatial relationship with a second data object by being positioned in proximity to the second data object. In another example, the first data object and the second data object may have a spatial relationship with the bounding box by being located within a plane or volume associated with the bounding box. The spatial index manager 1402 may determine whether some of the data objects satisfy the spatial causal relationship by being within a predefined distance of each other. Other examples may include the data object satisfying a spatial relationship with gravity or an electromagnetic field, being within a defined geometric shape, being positioned on a defined topology, being proximate to or intersecting a set of vertices and edges describing a polygon, mesh, crash box or other shape, range, camera view, etc.

Spatial index manager 1402 may use index configuration 1406 to determine how to represent the processing partition associated with spatial index 1408. Spatial index manager 1402 may determine the capacity of a processing partition using a plurality of data objects that may be assigned to the processing partition. Index configuration 1406 may specify the number of data objects that may be indexed by spatial index 1408. The number of data objects that may be indexed by spatial index 1408 may be determined based on the configuration of the processing application and the capacity of the processing application to process the plurality of data objects. The number of data objects that may be indexed by the spatial index 1408 may also be determined based on a configuration of the virtual world, which may define whether there is a one-to-one, one-to-many, and many-to-one correspondence between processing partitions and the spatial index 1408, a structure of the spatial index 1408, one or more functions for indexing the data objects 404, a configuration of hardware and software resources, a number of hardware hosts in the distributed computing system, and so forth. The index configuration 1406 can also include index specific configurations set as part of the world definition (e.g., to use octrees with maximum levels and predefined ranges).

In this example, spatial index manager 1402 uses object field(s) 1404 and index configuration 1406 to create, update, and manage spatial index(s) 1408. Since the processing partitions represent groups of data objects organized into compute units to be processed by the processing applications, spatial index manager 1402 may organize the data objects according to spatial location information in object fields 1404. Spatial index manager 1402 may use object field(s) 1404 to create spatial index(s) 1408 with spatial location information to map a set of data objects to a processing partition. As shown, the spatial index(s) 1408 can use the object field(s) 1404 to create multiple layers in the grouping, thereby further organizing the data objects by multiple attributes, such as object domain and spatial locality.

Spatial index 1408 may be represented by a tree

structure having nodes

1410, 1412, 1414, 1416, 1418, 1420, and 1422. In this example, node 1410 may represent a root node, and

nodes

1412, 1414, 1416, 1418, 1420, and 1422 may represent child nodes. The root node 1410 may include metadata 1430, the metadata 1430 providing information associated with the processing partition represented by the root node 1430 and the set of data objects 404 organized into the processing partition. Metadata 1430 may also include information about relevant areas of spatial index 1408, such as parent nodes, neighboring nodes (special siblings), child nodes, and the like. Metadata 1430 may identify the index level and first level used in subdividing the space. For example, metadata 1430 may specify that a first level for partitioning a world space is an object type and a second level is a spatial location. The

child nodes

1412, 1414, 1416, 1418, 1420, and 1422 may also include similar metadata (not shown).

As data objects are generated in world space, spatial index manager 1402 may be responsible for indexing the data objects into processing partitions using spatial index 1408. Upon initialization of world space, spatial index manager 1402 may assign a single node in spatial index of data objects 1408 based on object type or spatial subdivision. Spatial index manager 1402 may receive instructions to index data objects and, once located within world space, spatial index manager 1402 may map the data objects to processing partitions by reading object fields 1404 and by adding metadata associated with the data objects to nodes associated with the processing partitions in spatial index 1408. Spatial index manager 1402 may maintain an atomic reference to a working copy of spatial index 1408, and only spatial index manager 1402 may access the working copy. Spatial index manager 1402 may index the data object using the work copy and then flip the atomic reference to spatial index 1408 so that other threads (e.g., world manager or application manager) can access the current state of the processing partition.

To determine where to add the data object to the spatial index 1408, the spatial index manager 1402 may use a hash code octree implementation to calculate the location of the data object using the normalized range. The spatial index manager 1402 may compute a hash of the data object by normalizing spatial location information associated with the data object to a normalized range. The computed hash may identify nodes in the spatial index(s) 1408 that correspond to spatial subdivisions of world space that include the location of the data object. For example, spatial index manager 1402 may normalize a position dimension associated with a data object to a floating point number from 0 to 1. The spatial index manager 1402 may then multiply the normalized position by a power of 2 that is associated with the number of levels specified for the octree. Spatial index manager 1402 may effectively hash data objects into the deepest possible level of spatial index 1408. Spatial index manager 1402 may then use the computed hash to traverse spatial index 1408 from root node 1410 in order to find internal or child nodes that surround the data object. During traversal, the spatial index manager 1402 can compare one or more hash bits that match the "next" level to children of the current node. The spatial index manager 1402 may use the hash to determine the index of the next child in the child array.

During the lifecycle of the data object in world space, a request to modify data associated with the data object may be issued from an instance of a processing application associated with the object type of the data object. Spatial index manager 1402 may use changes to data objects caused by the request to re-index the data objects into spatial index 1408. For example, during re-indexing, spatial index manager 1402 may determine to update the mapping between data objects and processing partitions in response to a request to modify data. The spatial index manager 1402 may also determine an update mapping in response to an instruction from a world manager or another process to re-index a data object. Spatial index manager 1402 may re-index the data objects using changes to the data objects caused by the request and update metadata associated with the data objects in spatial index 1408. The re-indexing may occur when ownership of a data object is transferred from a first hardware host to a second hardware host. In this case, spatial index manager 1402 may take ownership of the data object and update the metadata associated with the data object in spatial index 1408.

Spatial index manager 1402 may also use spatial location information associated with data objects to determine whether to change or maintain a mapping between data objects and processing partitions. The spatial index manager 1402 may re-index the data objects by adding metadata associated with the data objects at different nodes in the spatial index 1408, e.g., because movement of the data objects reflected in the spatial location information causes the data objects to be at locations associated with different spatial subdivisions of the world space. The spatial index manager 1402 may also re-index the data object to remove metadata associated with the data object from the nodes in the spatial index 1408 because the data object is no longer located in world space (e.g., de-spawned or corrupted).

After re-indexing, spatial index manager 1402 may also determine whether to split or merge nodes in spatial index 1408. When the number of data objects mapped to the first processing partition represented by the first node approaches, meets, or exceeds a first defined threshold, spatial index manager 1402 may determine to split the first node into a second node and a third node. The first defined threshold may be determined as a first percentage (e.g., 60%) of a capacity of the processing application to process the plurality of data objects. When the number of data objects mapped to the first and second processing partitions represented by the first and second nodes approaches, meets, or is below a second defined threshold, spatial index manager 1402 may determine to merge the first and second nodes to create a third node. The second defined threshold may be determined as a second percentage (e.g., 10%) of the capacity of the processing application to process the plurality of data objects.

Spatial index manager 1402 may read spatial index 1408 and perform any splitting or merging as needed. In one example, when a first defined threshold determined to be a first percentage (e.g., 60%) of the capacity of the processing application to process the plurality of data objects is met or exceeded, spatial index manager 1402 may split the node and add the node to spatial index 1408 (or update metadata of existing nodes to reflect the split). For example, the world space region associated with child node 1412 may be subdivided into four subdivided regions that may be associated with child node 1414 and three siblings. When the first threshold is met or exceeded, the spatial index manager 1402 may allocate some of the data objects indexed by the child node 1412 across the child node 1414 and one or more of the three siblings or a combination thereof. Spatial index manager 1402 may use spatial location information associated with the data object to index the data object into child node 1414 and three siblings, the spatial location information locating the data object into a corresponding subdivision region.

In some cases, data objects that hash out of the normalized range into the child nodes of spatial index(s) 1408 may be mapped to root node 1410. If there are too many data objects at root node 1410, an "up" split may be performed. Additional nodes may be added to the spatial index 1408 as siblings to the root node 1408 representing additional quadrants or octants of the extended world space. Each additional quadrant or octant added at the side of root node 1410 may be considered for splitting up to become a new root node. Alternatively, a completely new root node may be created. To determine the position of the upward expansion, the data object at root 1410 may be classified as a super region that encompasses the existing world space. Using, for example, four super regions, the best super region for upward expansion can be determined by which split removes most objects from the root node 1410 and keeps the size of the node below the limit. Due to the nature of this change, the object hash may be recomputed by adding a bit representing the expansion space to the current object hash. For range changes that are powers of 2 (double, quadruple), the existing positional hashes can be converted by bit shifting each dimension. The actual range change may also be performed by re-hashing all the data objects and recreating the spatial index 1408. A "shadow" octree can be created by the spatial index manager 1402 and then published when completed.

If there are too many objects in the nodes near the bottom of the tree, a hierarchy addition may be performed. The hierarchy addition may include adding a bit to the end of the data object hash. Hierarchy addition can occur when nodes are created within the bottom two hierarchies.

In another example, spatial index manager 1402 may merge nodes and remove nodes from spatial index 1408 (or update metadata of existing nodes to reflect the merge) when a second defined threshold determined to be a second percentage (e.g., 10%) of the capacity of the processing application to process the plurality of data objects is met or exceeded. For example, four subdivided regions including world space regions associated with the child node 1422 and three siblings may be merged into the world space region associated with the child node 1420. Any child nodes associated with child node 1422 may also be merged into the world space region associated with child node 1420. The spatial index manager 1402 may combine data objects indexed by one or more of the child node 1422, the three siblings, and any descendant or combination of the child node 1422 and the three siblings with the child node 1420. To merge the nodes together, the spatial index manager 1402 may take ownership of three siblings of the child node 1420 in level N, the child node 1422 in level N +1, and the child node 1420 in level N + 1. If the total number of data objects indexed by the child node 1422 and the three siblings is close to the lower threshold or minimum number of data objects, the spatial index manager 1402 may merge the nth level child node 1422 and the three siblings of the N +1 level into the node 1420. The spatial index manager 1402 may then delete the child node 1422 and the three siblings.

15A-15B are flow diagrams illustrating an example method 1500 for managing hardware hosts in a distributed computing system using process partitions organized by spatial location information associated with data objects in accordance with examples of the present technology. The method 1500 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine (e.g., a computer system or information processing device), by hardware components or application specific integrated circuits of an electronic device, or by a combination of software and hardware elements.

In operation 1502, a world manager can determine a plurality of spatial subdivisions associated with an object-based model representing a virtual environment hosted by a distributed computing system. The world manager may determine the plurality of spatial subdivisions using a configuration associated with an object-based model representing the virtual environment. The world manager may parse the configuration to determine a size of world space associated with the virtual environment, a data structure (e.g., an octree or a quadtree) to represent the plurality of spatial subdivisions, a configuration of the data structure (e.g., a number of levels, a number of nodes within a level, and a size of each node in terms of a minimum or maximum number of data objects that may be indexed by the node).

The world manager may also determine a plurality of spatial subdivisions using a configuration associated with a processing application that processes data objects in the object-based model. The world manager may read the configuration to determine a plurality of object types associated with data objects processed by a processing application, application resources used by the processing application, types of the data structures used by the processing application to access the data objects, minimum and maximum numbers of data objects that may be processed by a processing application, subscription types that specify which additional object types to subscribe to in order to process the data objects, and subscription policies for obtaining the additional data. The subscription policy may identify what additional data object types are used during processing and what source locations the additional object types may be located at.

The subscription policy of the processing application may be used to determine which additional data objects to synchronize to the hardware host where the processing instance of the processing application is located. In one example, the subscription policy may specify that the processing application subscribes to neighbor nodes in a data structure (e.g., index) that references the data object. Using the data structure, the hardware host can discover the neighbors of the node to which the data object being processed by the processing application is indexed (e.g., using spatial location information), and the hardware host can cause the processing application to subscribe to data from those neighboring nodes. In another example, a subscription policy may specify object tracking, which identifies the data object, and the subscription policy may subscribe to data from the node to which the data object indexes, the neighbor, parent, child, etc. of the node.

The subscription policy may be used to initiate a subscription to a data object from different hardware hosts in the distributed computing system. For example, a processing application may be instantiated on a first hardware host and a neighbor node may be assigned to a second hardware host. The first and second hardware hosts may communicate to share state changes to data objects indexed into neighboring nodes. For example, a first hardware host may identify a neighbor node as being allocated to a second hardware host using processing partition assignment. The first hardware host may send instructions, messages, subscription requests, etc. to the second hardware host to subscribe to and obtain data associated with the data object mapped to the neighbor node (e.g., for dependencies between the first hardware host and the second hardware host). The second hardware host may add the first hardware host to a subscriber list to receive data associated with the data object mapped to the neighbor node. The second hardware host may determine a change to the data object mapped to the neighbor node and send all or a portion of the data associated with the data object mapped to the neighbor node to the first hardware host.

In another example, the processing application may issue a subscription to data associated with all players near an enemy for use by the AI to control an enemy unit. This is also useful for processing applications or clients that perform rendering. In addition, the subscription policy may specify a maximum delay that an application instance can tolerate getting additional object types from a specified source.

Referring again to FIG. 15A, the world manager may establish the plurality of spatial subdivisions as spatial regions that divide the space into two or more subsets. The multiple spatial subdivisions may be hierarchical, meaning that a space (or region of a space) may be divided into several spatial regions, and then the spatial regions may be recursively subdivided again to create additional spatial regions. The plurality of spatial subdivisions may also include multiple layers or hierarchies of subdivisions, which may overlap one another and may include different layers or hierarchies that provide different metrics, criteria, and granularities for subdividing the virtual world.

As discussed above, the plurality of spatial subdivisions may be represented using one or more tree structures, e.g., as discussed above with respect to fig. 14. World space associated with a virtual environment hosted by a distributed computing system may be subdivided using octants, and the plurality of spatial subdivisions may be represented using octrees to subdivide three-dimensional space associated with the virtual world. In another example, the virtual world may be subdivided using quadrants and the plurality of spatial subdivisions may be represented using quadtrees to divide a two-dimensional space associated with the virtual world. Other methods may be used to subdivide the virtual world and represent the multiple spatial subdivisions using data structures to provide nodes to represent particular units of spatial information and which data may be associated with particular units of spatial information.

In operation 1504, the world manager may determine a mapping between the plurality of data objects and a plurality of processing partitions associated with the plurality of spatial subdivisions. For example, the world manager may initialize the spatial index using an octree associated with a plurality of spatial subdivisions. The world manager may represent the processing partition using nodes in an octree to associate the processing partition with a single spatial subdivision of the plurality of spatial subdivisions. The world manager may also represent multiple processing partitions with a single node in the octree and associate the multiple processing partitions with a single spatial subdivision. Further, the world manager can represent a processing partition with a plurality of nodes in an octree and associate a single processing partition with a plurality of spatial subdivisions.

In operation 1506, the world manager may identify a plurality of hardware hosts in the distributed computing system to process the object-based model using the plurality of data objects. The world manager may identify hardware hosts from a pool of hardware hosts waiting to be allocated, or the world manager may identify active hardware hosts with spare computing capacity, etc. The active hardware host may execute an application instance in a container or in a compute instance that may operate on a processing partition. In operation 1508, the world manager may assign the plurality of processing partitions to the plurality of hardware hosts using the plurality of spatial indices. More specifically, the world manager may assign one or more spatial indices to hardware hosts. The world manager may also assign one or more hardware hosts to the spatial index. The world manager may assign the processing partitions to the hardware hosts based on: spare computing capacity, hardware hosts with dedicated hardware (e.g., GPUs), load balancing, collocated data, and the like.

In operation 1510, the world manager may generate a plurality of processing partition assignments between the plurality of processing partitions and the plurality of hardware hosts. This means that the processing partitions can be allocated to hardware hosts that can process application instances of applications of the type that map to the processing partitions and organize using the spatial locality. For example, the world manager may assign a spatial index to a hardware host having a physical application that modifies data objects using physics simulations, a conflict detection application that determines conflicts between data objects, a rendering application that renders the data objects into one or more images or videos, and so on. In operation 1512, the world manager may send the plurality of processing partition assignments to the plurality of hardware hosts to organize the plurality of hardware hosts to process the plurality of data objects using the spatial locality.

The method 1500 continues with reference "a" to fig. 15B of fig. 15A. In operation 1514, the world manager can use the mapping to monitor spatial indexing of the plurality of data objects by the plurality of hardware hosts to process the plurality of data objects. The world manager may monitor the plurality of hardware hosts to track object changes, to determine whether changes have occurred to the plurality of processing partitions, and to determine metrics associated with the hardware hosts. The data object may be modified as a result of processing by a processing application that may change spatial location information associated with the data object. The processing partition may also be modified due to changes in spatial location information due to processing of the data object by the processing application. Changes to spatial location information may introduce re-indexing of data objects by the hardware host. Changes in spatial location information may also result in splitting or merging of processing partitions.

At operation 1516, the world manager can determine changes to the plurality of processing partitions associated with the plurality of spatial sub-partitions. Updating a processing partition with the change may include mapping a newly created data object to the processing partition, modifying a data object mapped to the processing partition, or removing a mapping between a data object and the processing partition. Updating a processing partition with the change may include splitting a first processing partition into a second processing partition and a third processing partition, such that the data objects indexed into the first processing partition are allocated to the second processing partition and the third processing partition. In another example, updating the processing partition with the change may include merging the first processing partition and the second processing partition into a third processing partition, such that the data objects indexed into the first processing partition and the second processing partition are assigned to the third processing partition.

At operation 1518, the world manager can use the change to determine whether to update the multiple processing partition assignments. The world manager may determine to update multiple processing partitions to manage or optimize performance of the hardware host. The world manager may load balance the processing of data objects across multiple hardware hosts. Additionally, the world manager may redistribute the processing partitions due to changes to the data objects or hardware hosts. If the world manager determines not to update the plurality of processing partition assignments using the change in step 1520, the method 1500 continues in operation 1514, where in operation 1514 the world manager returns to monitoring the plurality of hardware hosts.

If the world manager uses the changes to determine to update multiple processing partition assignments in step 1520, the method 1500 continues in operation 1522, where the world manager uses the updates from the hardware hosts to update multiple processing partitions in operation 1522. The world manager may update the plurality of processing partitions, for example, changing a processing partition assignment for a first hardware host and allocating a spatial index owned by the first hardware host to a second hardware host. Thus, the world manager can migrate a processing partition from a first hardware host to a second hardware host. The method 1500 continues using reference "B" from FIG. 15B and returns to FIG. 15A, where in operation 1512 the world manager can send the plurality of processing partition assignments to the plurality of hardware hosts.

FIG. 16 is a flow diagram illustrating an example method 1600 for processing data objects assigned to hardware hosts using processing partitions organized by spatial location information associated with the data objects, in accordance with examples of the present technology. The method 1600 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine (e.g., a computer system or information processing device), by hardware components or application specific integrated circuits of an electronic device, or by a combination of software and hardware elements.

In operation 1602, a distributed shared memory associated with a hardware host in a distributed computing system may receive a plurality of processing partition assignments representing assignments between a plurality of processing partitions and a plurality of hardware hosts. The distributed shared memory may receive a plurality of processing partition assignments from a cluster or world manager associated with the distributed computing system. The plurality of processing partition assignments may identify a plurality of hardware hosts in the distributed computing system and nodes of a tree structure allocated to the plurality of hardware hosts. A node may include a spatial index structure that represents a set of data objects grouped or indexed by spatial location information. The set of data objects may also be organized by object type. Thus, a node may represent a collection of data objects of the same object type and satisfying spatial dependencies.

In operation 1604, the distributed shared memory may identify a processing partition assigned to a hardware host associated with the spatial subdivision using the plurality of processing partition assignments. The distributed shared memory may analyze a list that provides a plurality of processing partition assignments between the plurality of hardware hosts and the plurality of processing partitions to find an allocation to the hardware hosts. The hardware host may filter the list to identify the processing partitions assigned to the hardware host.

In operation 1606, the distributed shared memory may load the plurality of data objects mapped to the processing partition grouped by location associated with the spatial subdivision to a memory device associated with the hardware host. For example, the distributed shared memory may read nodes of the tree structure corresponding to the process partitions to obtain metadata about data objects mapped to the process partitions. The distributed shared memory may retrieve data objects from a file, an object data store, across a network, from another distributed shared memory, and so forth.

In operation 1608, the hardware host may process the plurality of data objects using the instance of the processing application on the hardware host. The hardware host may determine an object type of the data object to identify a processing application configured to execute code on the data object of the object type. The hardware host may also determine a location of the data object to identify a processing application configured to execute code on the data object according to the location. The hardware host may create an instance of the processing application as a containerized application to process the data object.

In operation 1610, the hardware host may determine whether to update the processing partition with changes associated with the spatial subdivision. The hardware host may determine in operation 1608 that the process partition is updated in response to processing of the data object that caused the change in spatial location information associated with the data object. For example, the processing application may create new data objects located within the virtual environment, move data objects, change data associated with existing data objects, and remove data objects from the virtual environment. The create, delete, and update operations may add or remove spatial dependencies between data objects used to organize the data objects into processing partitions.

In a particular example, a new data object may be created at a given location, and the new data object triggers a spatial dependency with one or more of the data objects. The hardware host may determine to update the process partition to add the new data object to the process partition because the data object satisfies the spatial dependency. In another example, a new data object introduced to a given location that triggers a spatial dependency may cause the number of data objects to be processed by a processing application to increase and approach a set upper threshold. When the number of data objects to be processed by a processing application approaches a threshold, the hardware host may determine to split a first processing partition into two processing partitions, allocate some of the data objects to remain mapped to the first processing partition, and allocate some of the data objects to map to a new second processing partition. The hardware host may also create a second instance of the processing application to process the data object mapped to the second processing partition.

In another example, some of the data objects may be moved to a different location during processing than the remaining data objects, and spatial dependencies with the remaining ones of the data objects are removed. In yet another example, some of the data objects may be destroyed in the virtual during processing. The hardware host may determine to update the processing partition to remove from the processing partition data objects that no longer satisfy the spatial dependencies or no longer exist in the virtual environment. Removing a data object from a given location may result in a reduction in the number of data objects processed by the processing application. As described above, the number of data objects processed by a processing application may approach or exceed a defined threshold. As the number of data objects handled by the processing application approaches the defined threshold, the hardware host may determine to merge the first processing partition with the second processing partition, thereby combining data objects mapped to the second processing location with data objects mapped to the first processing partition.

Returning to FIG. 16, if the hardware host determines not to update the processing partition in operation 1612, the hardware host may continue processing the plurality of data objects using the instance of the processing application in operation 1608. If the hardware host determines to update the processing partition in operation 1612, the hardware host may generate an updated processing partition to re-index the data objects associated with the spatial sub-partitions in operation 1614. As discussed above, create, delete, and update operations may add or remove spatial dependencies to organize data objects into processing partitions associated with spatial sub-partitions. The hardware host may generate an updated processing partition by re-indexing the data objects and including any additional data objects that satisfy the spatial relationship with the spatial subdivision.

As discussed above, the update processing partition may additionally be used to inform a cluster or world manager associated with the distributed computing system about the utilization of hardware and software resources associated with the hardware host. Adding new data objects may increase resource utilization because the number of data objects to be processed by the processing application has increased. To load balance the processing of data objects in a distributed computing system, a cluster or world manager may evaluate whether a hardware host has sufficient resources to process additional data objects. Removing data objects from the processing partition may reduce resource utilization because the number of data objects to be processed by the processing application has been reduced. The cluster or world manager may then evaluate whether the hardware host can process the additional data object. Thus, in operation 1616, the hardware host may send the updated processing partition to the world manager.

FIG. 17 is a flow diagram illustrating an example method 1700 for splitting a processing partition organized by spatial location information in accordance with an example of the present technology. The method 1700 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine (e.g., a computer system or information processing device), by hardware components or application specific integrated circuits of an electronic device, or by a combination of software and hardware elements.

In operation 1702, a hardware host in a distributed computing system may process operations associated with a plurality of data objects. Data objects may be grouped using a first spatial index node to a first processing partition in a first position associated with a first spatial subdivision. The operations may include create, read, update, and delete (CRUD) operations handled by a distributed shared memory on a hardware host. In operation 1704, the hardware host may determine to split the first processing partition into a second processing partition and a third processing partition based on the results of the operations. The hardware host may determine to split the first processing partition into the second processing partition and the third processing partition because the number of data objects handled by the instance of the processing application associated with the first processing partition approaches, equals, or exceeds a threshold specified for the processing application. The hardware host may determine a threshold value to use when splitting the first processing partition as a percentage of a capacity defined by the processing application to process the plurality of data objects. For example, when the number of data objects handled by an instance of a processing application meets or exceeds 60% of the capacity defined for the processing application, the hardware host may trigger splitting of the first processing partition into the second processing partition and the third processing partition.

For example, when a new data object is created in the virtual environment and the new data object is mapped to the first processing partition, the hardware host may determine that the number of data objects handled by the instance of the processing application may have increased. When the data object moves from the first location to a second location associated with the first spatial subdivision, the hardware host may determine that the number of data objects handled by the instance of the processing application may have increased. The hardware host may also determine that a motion of a first data object may have triggered a spatial dependency with a second data object mapped to the first processing partition.

In operation 1706, the hardware host may split some of the plurality of data objects to a second processing partition that groups the data objects in a second location associated with the second spatial subdivision. The hardware host may determine that spatial location information associated with the data object is available to group the data object into the second processing partition by the second location. In another example, the hardware host may determine that some of the data objects satisfy a spatial relationship with the second location or have a data dependency with other objects at or near the second location. The hardware host may then allocate the data object to a second processing partition. In operation 1708, the hardware host may generate, determine, or identify a second node representing a second processing partition to index data objects grouped into the second processing partition based on the second spatial subdivision. The hardware host may generate a second spatial index node using metadata that maps some of the data objects to a second processing partition.

In operation 1710, the hardware host may split some of the plurality of data objects into a third processing partition, the third processing partition grouping the data objects in a third location associated with a third spatial subdivision. Thus, the hardware host may assign the data object to the third processing partition. In operation 1712, the distributed shared memory may determine or identify a third spatial index node representing a third processing partition to index data objects associated with the third spatial subdivision.

In operation 1714, the hardware host may update a tree structure associated with the plurality of spatial subdivisions using the second spatial index node and the third spatial index node to reflect the split of the first processing partition. For the first processing partition, the hardware host may update the tree structure to remove metadata (if any) from the first spatial index node for data objects that are now mapped to the second processing partition and the third processing partition. The hardware host may update the tree structure using the second spatial index node to include the metadata of the data object that is now mapped to the second processing partition. The hardware host may further update the tree structure using the third spatial index node to include metadata for the data objects that are now mapped to the third processing partition. The hardware host may generate the second spatial index node and the third spatial index node in the tree structure by updating an existing spatial index node or adding additional spatial index nodes (e.g., children of the first spatial index node or peer spatial index nodes associated with the first processing partition).

Splitting the first processing partition into the second processing partition and the third processing partition may additionally be used to notify a cluster or world manager associated with the distributed computing system of utilization of hardware and software resources associated with the hardware host to which the second processing partition and the third processing partition are currently assigned. As discussed above, the second and third processing partitions may increase resource utilization on the hardware host because the number of data objects to be handled by an instance of the processing application has increased or another instance of the processing application will be launched. To load balance the processing of data objects in the distributed computing system, the cluster or world manager may evaluate whether the hardware host has sufficient resources to process the data objects allocated to the second processing partition and the third processing partition.

FIG. 18 is a flow diagram illustrating an example method 1800 for merging processing partitions organized by spatial location information in accordance with examples of the present technology. The method 1800 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine (e.g., a computer system or information processing device), by hardware components or application specific integrated circuits of an electronic device, or by a combination of software and hardware elements.

In operation 1802, a hardware host in the distributed computing system may process operations associated with a first plurality of data objects grouped into a first processing partition in a first location associated with a first spatial subdivision. In operation 1804, the hardware host may determine to merge the first processing partition and the second processing partition based on the result of the operation. When the number of data objects handled by an instance of a processing application associated with a first processing partition approaches, equals, or exceeds a merge threshold specified for the processing application, the hardware host may determine that the first processing partition is merged with a second processing partition as a result of the operation. The merge threshold may be a percentage of the capacity of the processing application to process the plurality of data objects. For example, the merge threshold may trigger the merging of the first processing partition with the second processing partition when the number of data objects handled by an instance of the processing application meets or is below 10% of the capacity of the processing application.

As discussed above, the number of data objects handled by an instance of a processing application may be reduced, such as when the data objects move in a virtual environment and the data objects may be mapped to different processing partitions based on the new locations of the data objects. In another example, the number of data objects handled by an instance of the processing application may be reduced when removing or deleting data objects from the virtual environment. Removing and updating data objects in the virtual environment may result in a percentage of data objects previously handled by an instance of the processing application subsequently satisfying the merge threshold.

In operation 1806, the hardware host may consolidate the plurality of data objects to a second processing partition that groups the data objects in a second location associated with a second spatial subdivision. For example, the hardware host may determine that spatial location information associated with the plurality of data objects may be used to group data objects from both the first processing partition and the second processing partition into the second processing partition in the second location. In another example, the hardware host may determine that the spatial extent defined for the second processing partition using the second location encompasses locations associated with the plurality of data objects in the first processing partition and the second processing partition. The hardware host may then move or allocate the plurality of data objects from the first processing partition to the second processing partition.

In operation 1808, the hardware host may generate a spatial index node representing the second processing partition to index the plurality of data objects grouped into the second processing partition associated with the second spatial subdivision. The hardware host may generate a spatial index node using metadata mapping the plurality of data objects to the second processing partition.

In operation 1810, the hardware host may update a tree structure associated with the plurality of spatial subdivisions using the spatial index node to reflect the merging of the first processing partition and the second processing partition. The hardware host may update the tree structure using the spatial index node to include the metadata of the data object that is now mapped to the second processing partition. The hardware host may also update the tree structure to remove metadata from the spatial index node associated with the first processing partition. The hardware host may generate the spatial index node in the tree structure by updating an existing spatial index node (e.g., a parent or peer of the spatial index node associated with the first processing partition) or adding additional spatial index nodes.

The merging of the first processing partition into the second processing partition may additionally serve to notify a cluster or world manager associated with the distributed computing system of underutilization of hardware and software resources associated with the hardware host to which the second processing partition is currently assigned. As discussed above, adding new data objects to the second processing partition may increase resource utilization because the number of data objects to be handled by an instance of the processing application has increased. To load balance the processing of data objects in the distributed computing system, the cluster or world manager may evaluate whether the hardware host has sufficient resources to process additional data objects assigned to the second processing partition.

FIG. 19 illustrates a system 1900 and a computing device 1910, upon which modules of the present technology can execute. A computing device 1910 is shown on which high-level examples of the techniques may be executed. The computing device 1910 may include one or more processors 1912 in communication with a memory device 1920. The computing device 1910 may include a local communication interface 1918 for components in the computing device. For example, local communication interface 1918 may be a local data bus and/or any associated address or control bus as may be required.

The memory device 1920 may include a module 1924 executable by the processor 1912 and data for the module 1924. Module 1924 may perform the functions previously described. A data store 1922 may also be located in the memory device 1920 for storing data relating to the module 1924 and other applications, as well as an operating system executable by the processor 1912. Other applications may also be stored in the memory device 1920 and executed by the processor 1912. The components or modules discussed in this specification can be implemented in software using a high level programming language that is compiled, interpreted, or executed using a mix of methods.

The computing device 1910 may also have access to I/O (input/output) devices 1914 that may be used by the computing device 1910. Networking devices 1916 and similar communication devices may be included in the computing device 1910. The networking device 1916 may be a wired or wireless networking device connected to the internet, a LAN, a WAN, or other computing network.

The components or modules 1924 shown stored in the memory device 1920 may be executed by the processor 1912. The term "executable" may mean a program file in a form that may be executed by the processor 1912. For example, a program in a higher level language may be compiled into machine code having a format that can be loaded into the random access portion of the memory device 1920 and executed by the processor 1912, or the source code may be loaded by another executable program and interpreted to generate instructions in the random access portion of memory for execution by the processor. The executable program may be stored in any portion or component of the memory device 1920. For example, the memory device 1920 may be Random Access Memory (RAM), Read Only Memory (ROM), flash memory, solid state drives, memory cards, hard drives, optical disks, floppy disks, magnetic tape, or any other memory component.

The processor 1912 may represent multiple processors, and the memory device 1920 may represent multiple memory units operating in parallel with the processing circuitry. This may provide parallel processing channels for processes and data in the system. The local interface 1918 may serve as a network to facilitate communication between any of a number of processors and a number of memories. The local interface 1918 may use additional systems designed to coordinate communications, such as load balancing, bulk data transfer, and the like.

Although the flow diagrams presented for this technique may imply a particular order of execution, the order of execution may differ from that illustrated. For example, the order of more than two blocks may be rearranged relative to the order shown. Further, two or more blocks shown in succession may be executed concurrently or with partial parallelization. In some configurations, one or more blocks shown in the flowcharts may be omitted or skipped. Any number of counters, state variables, warning semaphores, or messages may be added to the logic flow for purposes of enhanced utility, accounting, performance, measurement, troubleshooting, or the like.

Some of the functional units described in this specification have been labeled as services that can be implemented as modules. A module may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices or the like.

Modules may also be implemented in software for execution by various types of processors. An identified module of executable code may, for instance, comprise one or more blocks of computer instructions which may be organized as an object, procedure, or function. Nevertheless, the executables of an identified module need not be physically located together, but may comprise disparate instructions stored in different locations which comprise the module and which, when joined logically together, achieve the stated purpose for the module.

Indeed, a module of executable code may be a single instruction, or many instructions, and may even be distributed over several different code segments, among different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein within modules, and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed over different locations including over different storage devices. The modules may be passive or active, including agents operable to perform desired functions.

The techniques described herein may also be stored on computer-readable storage media that include volatile and nonvolatile, removable and non-removable media implemented in any technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media includes, but is not limited to, non-transitory machine-readable storage media such as RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other computer storage medium that can be used to store the desired information and the described technology.

The devices described herein may also contain communication connections or networking means and connections that allow the device to communicate with other devices. A communication connection is an example of communication media. Communication media typically embodies computer readable instructions, data structures, program modules and other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. A "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. The term computer readable media as used herein includes communication media.

The foregoing may be better understood in view of the following clauses

1. A method, comprising:

determining a plurality of spatial subdivisions associated with a multidimensional virtual environment hosted by a distributed computing system;

determining a mapping between a plurality of data objects and a plurality of processing partitions associated with the plurality of spatial subdivisions, wherein the mapping is associated with spatial indices of the plurality of spatial subdivisions representing the plurality of data objects grouped into the plurality of processing partitions using spatial location information associated with the plurality of data objects;

Identifying a plurality of hardware hosts in the distributed computing system to process the plurality of data objects;

generating a plurality of processing partition assignments between the plurality of processing partitions and the plurality of hardware hosts in the distributed computing system using the spatial index associated with the mapping, wherein the plurality of processing partition assignments allocate the plurality of processing partitions to the plurality of hardware hosts using the spatial index; and

sending the plurality of processing partition assignments to the plurality of hardware hosts to enable the plurality of hardware hosts to process the assigned plurality of data objects using spatial locality.

2. The method of clause 1, further comprising:

receiving a configuration associated with the multidimensional virtual environment hosted by the distributed computing system, wherein the configuration identifies a tree structure representing the plurality of subdivisions that subdivide the space associated with the multidimensional virtual environment; and

determining the tree structure using the plurality of spatial subdivisions, wherein a plurality of nodes in the tree structure represent the spatial index and include metadata associating the plurality of data objects with the plurality of processing partitions.

3. The method of clause 1 or 2, further comprising:

identifying a first hardware host of the plurality of hardware hosts to process a first data object grouped by a first location in a first spatial subdivision of the plurality of spatial subdivisions, the first data object being mapped to a first processing partition of the plurality of processing partitions; and

identifying a second hardware host of the plurality of hardware hosts to process a second data object grouped by a second location in a second spatial subdivision of the plurality of spatial subdivisions, the second data object being mapped to a second processing partition of the plurality of processing partitions; and

sending the first processing partition and the second processing partition to the first hardware host and the second hardware host.

4. The method of clause 1 or 2, further comprising:

identifying a hardware host of the plurality of hardware hosts to process a first data object grouped by a first location in a first spatial subdivision of the plurality of spatial subdivisions, the first data object being mapped to a first processing partition of the plurality of processing partitions;

identifying the hardware host to process a second data object grouped by a second location in a second spatial subdivision of the plurality of spatial subdivisions, the second data object being mapped to a second processing partition of the plurality of processing partitions, wherein the second data object satisfies a spatial relationship with the first data object; and

Sending the first processing partition and the second processing partition to the hardware host.

5. The method of clauses 1-4, further comprising:

receiving a change to the mapping between the plurality of data objects and the plurality of processing partitions, wherein the change is caused by a split or a merge associated with the spatial index;

determining that the change affects the plurality of processing partition assignments;

updating the plurality of processing partition assignments using the changes to generate the updated plurality of processing assignments; and

sending the updated plurality of processing assignments to the plurality of hardware hosts to update a processing allocation in the distributed computing system.

6. A method, comprising:

determining a plurality of spatial subdivisions in a multidimensional virtual environment hosted by a distributed computing system;

determining a mapping between a plurality of data objects and a plurality of processing partitions associated with the plurality of spatial subdivisions, wherein a processing partition is associated with a processing application that processes data objects grouped using spatial location information associated with the plurality of data objects;

identifying hardware hosts in the distributed computing system that process the data objects grouped by location associated with a spatial subdivision of the plurality of spatial subdivisions, the data objects mapped to the processing partition; and

Sending the processing partition to the hardware host to process the data object, wherein the hardware host includes an application instance of the processing application for processing the data objects grouped by the location.

7. The method of clause 6, further comprising:

receiving the plurality of data objects associated with the multidimensional virtual environment hosted by the distributed computing system, wherein the plurality of data objects includes a plurality of object fields that provide the spatial location information; and

sending the plurality of processing partitions to the hardware host to process the data object.

8. The method of clause 6 or 7, further comprising:

receiving a configuration associated with the multidimensional virtual environment hosted by the distributed computing system, wherein the configuration identifies a tree structure for representing a space associated with the multidimensional virtual environment using the plurality of spatial subdivisions; and

determining the tree structure using the plurality of spatial subdivisions, wherein nodes in the tree structure include metadata that associates the data objects grouped by the location to the processing partition.

9. The method of clause 6 to clause 8, further comprising:

Receiving a configuration associated with the processing application, wherein the configuration provides the processing application with the capacity to process a plurality of data objects at a given location; and

the size of the processing partition is determined using the capacity of the processing application to process a plurality of data objects at a given location.

10. The method of clauses 6-9, further comprising:

identifying a second hardware host in the distributed computing system to process a second data object grouped by a second location, the second data object mapped to a second processing partition; and

sending the second processing partition to the second hardware host to process the second data objects grouped by the second location, wherein the second hardware host includes a second application instance of a second processing application to process the second data objects grouped by the second location.

11. The method of clauses 6-9, further comprising:

identifying the hardware host to process a second data object grouped by a second location, wherein the second data object satisfies a spatial relationship with the data objects grouped by the location; and

sending a second processing partition to the hardware host to process the second data objects grouped by the second location to concatenate related data, wherein the hardware host includes a second application instance of a second processing application to process the second data objects.

12. The method of clauses 6-11, further comprising:

monitoring a plurality of metrics associated with a plurality of hardware hosts in the distributed computing system;

determining a plurality of processing partition assignments between the plurality of processing partitions and the plurality of hardware hosts using the plurality of metrics; and

sending the plurality of processing partition assignments to the hardware host.

13. The method of clauses 6-11, further comprising:

receiving a change to a mapping between the plurality of data objects and the plurality of processing partitions, wherein the change comprises increasing or decreasing a number of data objects in the plurality of sub-partitions; and

determining a plurality of processing partition assignments using the change; and

sending the plurality of processing assignments to a plurality of hardware hosts to update a processing allocation in the distributed computing system.

14. The method of clauses 6-11, further comprising:

receiving a change to a mapping between the plurality of data objects and the plurality of processing partitions, wherein the change includes assigning the data objects from a first processing partition to a second processing partition in response to movement of the data objects in the multi-dimensional virtual environment; and

sending the plurality of processing assignments to a plurality of hardware hosts to enable the plurality of hardware hosts to process the allocated data objects using the spatial locality.

15. The method of clauses 6-11, further comprising:

determining a change to the mapping between the plurality of data objects and the plurality of processing partitions, wherein changing data comprises assigning data objects from a first processing partition to a second processing partition and to a third processing partition; and

updating a plurality of processing partition assignments using the changes to remove the first processing partition and to add the second processing partition and the third processing partition.

16. The method of clauses 6-11, further comprising:

determining a change to the mapping between the plurality of data objects and the plurality of processing partitions, wherein the change includes assigning a data object mapped to a second processing partition and a data object mapped to a third processing to a first processing partition; and

updating a plurality of processing partition assignments with the changes to add the first processing partition and to remove the second and third processing partitions.

17. A system, comprising:

one or more processors; and

one or more memory devices storing instructions that, when executed by the one or more processors, cause the one or more processors to:

receiving a plurality of spatial subdivisions associated with an object-based model representing a multidimensional virtual environment hosted by a distributed computing system;

monitoring a mapping between a plurality of data objects and a plurality of processing partitions associated with the plurality of spatial subdivisions, wherein the plurality of processing partitions use spatial location information associated with the plurality of data objects to group the plurality of data objects in locations associated with the plurality of spatial subdivisions of a processing application that processes the plurality of data objects;

determining, using the mapping, a plurality of processing partition assignments between the plurality of processing partitions and the plurality of hardware hosts in the distributed computing system, wherein the plurality of processing partition assignments allocate the plurality of hardware hosts the plurality of data objects grouped into the plurality of processing partitions associated with the plurality of spatial sub-partitions; and

Sending the plurality of process partitions assignments to the plurality of hardware hosts to manage process allocations for the plurality of data objects.

18. The system of clause 17, wherein the instructions further cause the one or more processors to:

receiving a configuration associated with the object-based model representing the multi-dimensional virtual environment hosted by the distributed computing system, wherein the configuration identifies a tree structure representing the plurality of subdivisions that subdivide the space associated with the multi-dimensional virtual environment; and

determining the tree structure using the plurality of spatial subdivisions, wherein a plurality of nodes in the tree structure represent the plurality of processing partitions and include metadata associating the plurality of data objects with the plurality of processing partitions.

19. The system of clause 17 or 18, wherein the instructions further cause the one or more processors to:

determining a change to the mapping between the plurality of data objects and the plurality of processing partitions, wherein the change includes assigning data objects mapped to a first processing partition grouping the data objects by a first location to a second processing partition grouping data objects by a second location and a third processing partition grouping data objects by a third location;

Updating a plurality of processing partition assignments to remove the first processing partition and to add the second processing partition and the third processing partition; and

sending the plurality of processing assignments to the plurality of hardware hosts to manage processing allocation in the distributed computing system.

20. The system of clause 17 or 18, wherein the instructions further cause the one or more processors to:

determining a change to the mapping between the plurality of data objects and the plurality of processing partitions, wherein the change includes assigning data objects to a first processing partition grouping data objects by a first location, to a second processing partition grouping data objects by a second location, and to a third processing partition grouping data objects by a third location;

updating a plurality of processing partition assignments to add the first processing partition and to remove the second processing partition and the third processing partition; and

21. A method, comprising:

receiving a plurality of data objects associated with an object-based model representing a multidimensional virtual environment hosted by a distributed computing system;

Identifying a plurality of object types associated with the plurality of data objects;

mapping the plurality of data objects to a plurality of processing partitions using the plurality of object types to distribute computing operations by object type within the distributed computing system, wherein the mapping includes a plurality of object type indices representing sets of data objects grouped by object type to processing partitions of the plurality of processing partitions;

identifying a plurality of hardware hosts in the distributed computing system to process the plurality of data objects of the object-based model;

generating a plurality of processing partition assignments between the plurality of processing partitions and the plurality of hardware hosts using the mapping, wherein the plurality of processing partition assignments specify which of the plurality of object type indices are assigned to the plurality of hardware hosts; and

sending the plurality of processing partitions to the plurality of hardware hosts to enable the plurality of hardware hosts to process the plurality of data objects separated using the plurality of processing partitions.

22. The method of clause 21, further comprising:

receiving a configuration associated with a processing application;

determining, using the configuration, a capacity of the processing application to process a plurality of data objects of a first object type;

Determining a data correlation between data objects of the first object type and data objects of a second object type using the configuration;

generating the mapping between the plurality of processing partitions and the plurality of data objects using the capacity of the processing application; and

generating the plurality of processing partition assignments using the data correlations between the data objects of the first object type and the data objects of the second object type.

23. The method of clause 21 or 22, further comprising:

determining changes to the mapping between the plurality of processing partitions and the plurality of data objects to obtain an updated mapping, wherein the changes to the mapping split or merge some of the plurality of object type indices;

determining an updated plurality of processing assignments between the updated plurality of processing partitions and the plurality of hardware hosts using the updated mapping; and

sending the updated plurality of process assignments to the plurality of hardware hosts to modify the processes in the plurality of hardware hosts.

24. The method of clauses 21-23, further comprising:

monitoring a plurality of metrics associated with the plurality of hardware hosts;

Receiving data associated with the plurality of metrics; and

determining a plurality of processing partition assignments between the plurality of processing partitions and the plurality of hardware hosts using the data associated with the plurality of metrics.

25. The method of clauses 21-24, further comprising:

determining, using the plurality of metrics, to migrate at least one of the plurality of processing partitions between a first hardware host and a second hardware host;

determining an updated plurality of processing assignments between the plurality of processing partitions and the plurality of hardware hosts by assigning the at least one of the plurality of processing partitions to the second hardware host; and is

Sending the updated plurality of processing assignments to the plurality of hardware hosts.

26. A method, comprising:

determining a plurality of object types associated with a plurality of data objects in a multidimensional virtual environment hosted by a distributed computing system;

generating a mapping between the plurality of data objects and a plurality of processing partitions using the plurality of object types, wherein a processing partition of the plurality of processing partitions is associated with a processing application that processes data objects of an object type;

identifying hardware hosts in the distributed computing system that process the data objects of the object type, the data objects mapped to the processing partition; and

Sending the processing partition to the hardware host to process the data object of the object type, wherein the hardware host includes an application instance of the processing application for processing the data object of the object type in the processing partition.

27. The method of clause 26, further comprising:

receiving the plurality of data objects from an object data store associated with an object-based model representing the multi-dimensional virtual environment;

determining a plurality of object fields associated with the plurality of data objects, wherein the plurality of object fields specify the plurality of object types; and

generating the mapping between the plurality of data objects and the plurality of processing partitions using the plurality of object types specified by the plurality of object fields.

28. The method of clauses 26-27, further comprising:

determining, using the mapping, a plurality of processing partition assignments between the plurality of processing partitions and a plurality of hardware hosts in the distributed computing system;

identifying the hardware host using the plurality of processing assignments to process the data object mapped to the processing partition; and

sending the plurality of processing partition assignments to the hardware host.

29. The method of clauses 26-28, further comprising:

monitoring a plurality of metrics associated with the plurality of hardware hosts; and

determining the plurality of processing partition assignments between the plurality of processing partitions and the plurality of hardware hosts using the plurality of metrics.

30. The method of clauses 26-29, further comprising:

identifying a second hardware host in the distributed computing system to process a second data object of a second object type mapped to a second processing partition; and

sending the second processing partition to the second hardware host to process the second data object of the second object type, wherein the second hardware host includes a second application instance of a second processing application to process the second data object of the second object type.

31. The method of clauses 26-29, further comprising:

identifying the hardware host to process a related data object of a second object type, wherein the related data object satisfies a data dependency with the data object mapped to the processing partition; and

sending a second processing partition to the hardware host to process the related data objects mapped to the second processing partition to co-locate related data, wherein the hardware host includes a second application instance of a second processing application to process the related data objects of the second object type.

32. The method of clauses 26-31, further comprising:

receiving a change to the mapping between the plurality of data objects and the plurality of processing partitions, wherein data objects are added to or removed from the plurality of data objects;

determining that the change affects a plurality of processing partition assignments between the plurality of processing partitions and a plurality of hardware hosts in the distributed computing system;

updating the plurality of processing partition assignments between the plurality of processing partitions and a plurality of hardware hosts to generate an updated plurality of processing assignments; and

33. The method of clauses 26-32, wherein generating the mapping between the plurality of data objects and the plurality of processing partitions using the plurality of object types further comprises generating the mapping using a plurality of data objects of the object types defined by a capacity of the processing application in a configuration of the processing application.

34. The method of clauses 26-33, further comprising generating a plurality of processing partition assignments between the plurality of processing partitions and a plurality of hardware hosts in the distributed computing system using the capacity of the hardware and software resources of the plurality of hardware hosts.

35. The method of clauses 26-29 or 31-34, further comprising:

monitoring a plurality of metrics associated with the hardware host;

determining to migrate the processing partition to a second hardware host using the plurality of metrics; and

sending the processing partition to the second hardware host to process the data object of the object type, wherein the second hardware host includes a second application instance of the processing application to process the data object of the object type in the processing partition.

36. A system, comprising:

one or more processors; and

identifying a plurality of data objects associated with an object-based model representing a 3D virtual environment hosted by a distributed computing system;

determining a mapping between the plurality of data objects and a plurality of processing partitions using a plurality of object types associated with the plurality of data objects, wherein the plurality of processing partitions group the plurality of data objects by object type;

Determining a plurality of processing partition assignments between the plurality of processing partitions and the plurality of hardware hosts using the mapping, wherein the plurality of processing partition assignments allocate the plurality of data objects grouped by object type in the plurality of processing partitions to the plurality of hardware hosts; and

sending the plurality of processing partitions assignments to the plurality of hardware hosts to manage the plurality of hardware hosts to process the plurality of data objects using the plurality of object types.

37. The system of clause 36, wherein the instructions further cause the one or more processors to:

receiving a configuration associated with a plurality of processing applications that process the plurality of data objects using the plurality of object types;

determining, using the configuration, a capacity for the plurality of processing applications to process a plurality of data objects; and

generating the mapping between the plurality of processing partitions and the plurality of data objects using the capacity of the plurality of processing applications.

38. The system of clause 36 or 37, wherein the instructions further cause the one or more processors to:

monitoring utilization of hardware or software resources by the plurality of hardware hosts in the distributed computing system;

Receiving data associated with utilization of the hardware or software resources by the plurality of hardware hosts; and

determining the processing partition assignment to reallocate utilization of the hardware or software resources of the plurality of hardware hosts using the data associated with utilization of the hardware or software resources by the plurality of hardware hosts.

39. The system of clauses 36-38, wherein the instructions further cause the one or more processors to:

identifying a first hardware host to process a first data object of a first object type, the first data object mapped to a first processing partition of the plurality of processing partitions;

identifying the first hardware host to process a second data object of a second object type, the second data object mapped to a second processing partition of the plurality of processing partitions, wherein the second data object satisfies a data dependency with the first data object; and

allocating the plurality of processing partition assignments to the plurality of hardware hosts using the data dependencies.

40. The system of clauses 36-39, wherein the instructions further cause the one or more processors to:

Receiving a change to the mapping between the plurality of data objects and the plurality of processing partitions, wherein the change results in an update to the plurality of data objects;

modifying the plurality of processing partition assignments between the updated plurality of processing partitions and the plurality of hardware hosts using the update to the plurality of data objects to generate an updated plurality of processing assignments; and

sending the updated plurality of processing assignments to the plurality of hardware hosts to modify allocation of processing partitions.

41. A method, comprising:

receiving, by a distributed shared memory of a computing hub, data objects associated with an object-based model of a virtual environment hosted by a distributed computing system;

writing, using the distributed shared memory, a representation of a data object to a memory location associated with a memory device of the compute hub, wherein the representation of the data object includes an object data segment and a virtual table that maps a plurality of object fields associated with the data object to the object data segment;

receiving a request from an instance of a processing application on the computing hub to modify the data object;

Based on the request, identifying an object field associated with a change to the data object caused by the request;

writing the change to the data object caused by the request to a log record appended to a log section of the data object; and

writing a modified field table to the log record, the modified field table including a reference indicating that the changes to the data object in the log record reflect changes to the object field associated with a portion of an object data segment.

42. The method of clause 41, further comprising:

receiving a plurality of requests to modify data associated with the data object;

determining to aggregate the plurality of requests using a plurality of changes to the data object caused by the plurality of requests during an execution cycle in the distributed computing system; and

appending additional log records to the log segment using the plurality of changes to the data object; wherein the additional log record includes the plurality of changes to the data object.

43. The method of clause 41 or 42, further comprising:

identifying a plurality of log records appended to the log section of the representation of the data object;

Determining to merge the plurality of log records into a single log record based on storage or network conditions; and

writing a further log record to the log segment using the plurality of log records to replace the plurality of log records, wherein the further log record includes a segment indicating which changes to the data object associated with the plurality of log records reflect changes to the representation of the data object.

44. The method of clauses 41-43, further comprising:

receiving an instruction to send a current state of the data object to a second distributed shared memory of a second compute hub in the distributed computing system;

determining whether the second distributed shared memory includes a previous state of the data object;

sending a portion of the log segment of the representation of the data object to the second distributed shared memory when the second distributed shared memory includes the previous state of the data object, wherein the portion of the log segment represents a difference between the current state and the previous state of the data object; and

sending the current state of the data object to the second distributed shared memory using the representation of the data object and a single log record that merges the log segments of the representation of the data object when the second distributed shared memory does not include the previous state of the data object.

45. The method of clauses 41-44, further comprising:

retrieving the data object from an object data store to obtain retrieved data;

determining whether a format of the retrieved data is used by the distributed shared memory;

performing a memory copy of the retrieved data to a memory location when the format of the retrieved data is used by a distributed shared memory; and

when the format of the retrieved data is not used by the distributed shared memory, writing the object data section of the representation of the data object using the retrieved data.

46. A method, comprising:

receiving a data object at a distributed shared memory;

writing a representation of the data object to a memory device using the distributed shared memory;

receiving a request to modify data associated with the data object;

determining a portion of the representation of the data object associated with a change to the data object caused by the request; and

writing a log record to the memory device in a log segment associated with the representation of the data object using the change to the data object, wherein the log record includes a reference indicating that the change to the data object reflects a change associated with the portion of the representation of the data object.

47. The method of clause 46, further comprising:

determining to aggregate the plurality of requests using a plurality of changes to the data object associated with the plurality of requests;

determining a plurality of portions of the representation of the data object associated with the plurality of changes to the data object; and

appending a further log record to the log segment using the plurality of changes, wherein the further log record includes a plurality of references indicating that the plurality of changes to the data object reflect changes to the plurality of portions of the representation of the data object.

48. The method of clause 46 or 47, further comprising:

determining to merge multiple log records into a single log record based on storage or network conditions; and

writing additional log records to the log section using the plurality of log records to replace the plurality of log records.

49. The method of clauses 46-48, further comprising:

receiving an instruction to send a current state of the data object to a second compute hub; and

sending the current state of the data object to the second compute hub using the representation of the data object.

50. The method of clause 49, further comprising:

determining that the second compute hub includes a previous state of the data object; and

sending a portion of the log segment of the representation of the data object to the second compute hub, wherein the portion of the log segment represents a difference between a current state and a previous state of the data object.

51. The method of clause 49, further comprising:

determining that the second compute hub does not include a previous state of the data object;

merging the log segments to generate a single log record representing a current state of the data object; and

sending the representation of the data object and the single log record representing the current state of the data object to the second compute hub.

52. The method of clauses 46-51, further comprising:

identifying an object field associated with the data object, wherein the object field corresponds to a portion of the representation of the data object that is altered;

writing a translation table to the log record, the translation table mapping the object field to a portion of the log record that includes the change to the data object; and

Writing a modified field table to the log record, the modified field table including the reference indicating that the mutation of the data object reflects a change associated with the portion of the representation of the data object.

53. The method of clauses 46-52, further comprising:

receiving the data object from an object data store to obtain a file;

determining that a format of the file is used by a distributed shared memory of the computing hub; and

writing the file to the memory device at a memory location associated with the distributed shared memory.

54. The method of clauses 46-52, further comprising:

receiving the data object from an object data store to obtain retrieved data;

determining that a format of the retrieved data is not used by the distributed shared memory of the compute hub;

determining a plurality of object fields associated with the data object;

writing the retrieved data into an object data section of the memory device associated with the representation of the data object; and

writing to the memory device a virtual table comprising a plurality of references that map the plurality of object fields to the data object portion.

55. The method of clauses 46 to 54, further comprising:

determining whether the request to modify data associated with the data object matches an object identifier and version associated with the data object;

writing the log record to the memory device when the request matches the object identifier and version; and

sending an error message when the request fails to match the object identifier and version.

56. A system, comprising:

at least one processor; and

at least one memory device storing instructions that, when executed by the at least one processor, cause the at least one processor to:

receiving, by a distributed shared memory that manages storage of data objects associated with an object-based model of a virtual environment, an instruction to create a data object in the virtual environment;

writing a representation of the data object to a first memory location associated with the at least one memory device, wherein the representation of the data object comprises an object data segment comprising object data;

receiving a request from an instance of a processing application to modify the object data;

Determining a portion of the object data segment associated with a change to the data object caused by the request;

generating a log record using the change to the data object and a reference indicating that the change to the data object reflects a change associated with the portion of the object data segment; and

appending the log record to a log section associated with the representation of the data object at a second memory location associated with the at least one memory device.

57. The system of clause 56, wherein the instructions further cause the at least one processor to:

determining to aggregate the plurality of requests using a plurality of changes to the data object caused by the plurality of requests; and

appending a further log record to the log segment of the representation of the data object using the plurality of changes, wherein the further log record aggregates the plurality of changes to the data object.

58. The system of clause 56 or 57, wherein the instructions further cause the at least one processor to:

writing additional log records to the log segment using the plurality of log records to replace the plurality of log records and satisfy the storage or network condition.

59. The system of clauses 56-58, wherein the instructions further cause the at least one processor to:

receiving an instruction to send a current state of the data object to a second distributed shared memory;

sending a portion of the log segment to the second distributed shared memory when the second distributed shared memory includes a previous state of the data object, wherein the portion of the log segment represents a difference between a current state and a previous state of the data object;

merging the log segments of the representation of the data object to generate a single log record to represent a current state of the data object when the second distributed shared memory does not include a previous state of the data object; and

sending the representation of the data object to the second distributed shared memory, wherein the representation of the data object includes the single log record.

60. The system of clauses 56-59, wherein the instructions further cause the at least one processor to:

retrieving the data object from an object data store to obtain retrieved data;

copying the retrieved data directly to the memory location when the format of the retrieved data is used by the distributed shared memory; and

61. A method, comprising:

receiving, at a first hardware host in a distributed computing system hosting a multidimensional virtual environment, a plurality of processing partition assignments, wherein a first processing partition assignment of the plurality of processing partition assignments allocates a first processing partition to the first hardware host, and the first processing partition groups a first plurality of data objects in the multidimensional virtual environment by a first object type;

determining a second object type on which processing of the first plurality of data objects by the first processing application depends;

Launching a first instance of the first processing application on the first hardware host to provide processing of the first plurality of data objects mapped to the first processing partition;

determining, using the plurality of processing partition assignments, a second hardware host that is assigned a second processing partition, wherein the second hardware host comprises a second instance of a second processing application that provides processing of a second plurality of data objects of the second object type, the second plurality of data objects mapped to the second processing partition; and

sending, from the first hardware host to the second hardware host, a subscription request, wherein the subscription request instructs the second hardware host to copy the second plurality of data objects to the first instance of the processing application.

62. The method of clause 61, further comprising:

determining a subscription policy that identifies neighboring relationships between spatial subdivisions associated with a plurality of spatial subdivisions for the multi-dimensional virtual environment to filter data objects of the second object type; and

identifying the second processing partition using a neighboring relationship between a first spatial subdivision associated with the first processing partition and a second spatial subdivision associated with the second processing partition that satisfies the subscription policy.

63. The method of clause 61, further comprising:

determining a subscription policy, the subscription policy identifying query criteria associated with a query to filter data objects of the second object type; and

identifying the second processing partition by matching the second plurality of data objects to the query criteria using the subscription policy.

64. The method of clauses 61-63, further comprising:

determining a list of subscribers to the first plurality of data objects; and

sending the first plurality of data objects to the subscriber list, wherein the subscriber list includes a third hardware host or a third instance of a third processing application.

65. The method of clauses 61-63, further comprising:

receiving updates to the plurality of processing partition assignments;

determining, using the update, a migration of the second processing partition between the second hardware host and a third hardware host, wherein the third hardware host includes a third instance of a third processing application that provides processing of the second plurality of data objects; and

sending a second subscription request from the first hardware host to the third hardware host to instruct the third hardware host to copy the second plurality of data objects to the first instance of the first processing application.

66. A method, comprising:

identifying, at a hardware host in a distributed computing system hosting a multidimensional virtual environment, an application library, wherein the application library comprises a plurality of processing applications that process data objects associated with a plurality of object types;

receiving a processing partition assigned to the hardware host, wherein a mapping between the data object and a plurality of processing partitions groups a plurality of data objects by object type into the processing partition;

retrieving from the application library a processing application of the object type associated with the processing partition; and

launching an instance of the processing application to enable the hardware host to process the plurality of data objects.

67. The method of clause 66, further comprising:

determining a second object type on which processing of the plurality of data objects by the instance of the processing application depends;

identifying a second processing partition that groups a second plurality of data objects into the second processing partition by the second object type; and

sending the second plurality of data objects to the instance of the processing application.

68. The method of clause 67, further comprising:

Determining a filter for the data object of the second object type;

identifying the second processing partition based on the second plurality of data objects satisfying the filter; and

sending a subscription request to a second hardware host associated with the second processing partition, wherein the subscription request instructs the second hardware host to copy the second plurality of data objects to the instance of the processing application.

69. The method of clause 66, further comprising:

determining a subscription policy that identifies a neighboring relationship between spatial subdivisions associated with a plurality of spatial subdivisions for the multi-dimensional virtual environment to filter the data objects of the second object type by spatial location;

identifying a second processing partition using a neighboring relationship between a first spatial subdivision associated with the processing partition and a second spatial subdivision associated with the second processing partition that satisfies the subscription policy; and

sending a subscription request to a second hardware host associated with the second processing partition to receive a second plurality of data objects at the instance of the processing application.

70. The method of clause 66, further comprising:

determining a subscription policy that identifies query criteria associated with a query of the data object in the multidimensional virtual environment;

identifying a second processing partition that groups at least one of the data objects that matches the query criteria; and

sending a subscription request to a second hardware host associated with the second processing partition to receive the at least one of the data objects matching the query criteria at the instance of the processing application.

71. The method of clause 66, further comprising:

receiving a change in the mapping between the data object and the plurality of processing partitions, wherein the change is caused by movement of a data object in the multi-dimensional virtual environment;

identifying a second processing partition associated with the data object using the change; and

sending a subscription request to a second hardware host associated with the second processing partition to receive the data object at the instance of the processing application.

72. The method of clauses 66-71, further comprising:

receiving a first data object event stream associated with the plurality of data objects from the processing application; and

The plurality of data objects are sent to a subscriber list using a second data object event stream.

73. The method of clauses 66-71, further comprising:

receiving, from the processing application, a data object event stream associated with the plurality of data objects;

determining, using the data object event stream, a change in the mapping between the data object and the plurality of processing partitions; and

sending the change to a world manager to enable the world manager to manage allocation of the processing partition across the plurality of hardware hosts.

74. The method of clause 66, 72 or 73, further comprising:

identifying at least one of a plurality of data objects grouped by object type into a processing partition;

migrating the at least one of the plurality of data objects between the processing partition and a second processing partition, the second processing partition grouping a second plurality of data objects by the object type; and

launching a second instance of the processing application to enable the hardware host to process the second plurality of data objects.

75. The method of clause 66, 72 or 73, further comprising:

determining a number of data objects in a second processing partition, the second processing partition grouping a second plurality of data objects by the object type;

Assigning a second plurality of data objects to the processing partition based on the number of data objects; and

terminating a second instance of the processing application associated with the second processing partition.

76. A system, comprising:

one or more processors; and

determining, using a plurality of processing partition assignments associated with a distributed computing system hosting a multidimensional virtual environment, a first processing application to be launched from an application library using a first processing partition allocated to a first hardware host for processing, wherein the first processing partition groups a first plurality of data objects by a first object type;

determining, using the plurality of processing partition assignments, a second hardware host associated with a second plurality of data objects associated with the second object type, wherein the second plurality of data objects are grouped by the second object type into a second processing partition assigned in the plurality of processing partition assignments for processing by the second hardware host; and

Sending a subscription request to the second hardware host for the second plurality of data objects to copy changes to the second plurality of data objects to the first instance of the first processing application.

77. The system of clause 76, wherein the instructions further cause the one or more processors to:

determining a subscription policy for filtering data objects of the second object type using spatial locations associated with a plurality of spatial subdivisions for the multi-dimensional virtual environment; and

identifying the second processing partition using a mapping between data objects and processing partitions in the multidimensional virtual environment and the subscription policy to associate the second plurality of data objects with the spatial location.

78. The system of clause 76, wherein the instructions further cause the one or more processors to:

determining a subscription policy for filtering data objects of the second object type using queries; and

identifying the second processing partition using a mapping between data objects and processing partitions in the multidimensional virtual environment and the subscription policy to match the second plurality of data objects to the query.

79. The system of clauses 76-78, wherein the instructions further cause the one or more processors to:

receiving a list of subscribers to the first plurality of data objects;

receiving, from the first processing application, a data object event stream associated with the first plurality of data objects; and

sending the first plurality of data objects to the subscriber list using a second data object event stream.

80. The system of clauses 76 to 79, further comprising:

receiving updates to the plurality of processing partition assignments;

determining, using the update, a third hardware host associated with the second plurality of data objects; and is

Sending a second subscription request for the second plurality of data objects to the third hardware host to copy the second plurality of data objects to the first instance of the first processing application.

Reference will now be made to the examples illustrated in the drawings and specific language will be used herein to describe the same. It will nevertheless be understood that no limitation of the scope of the technology is thereby intended. Alterations and further modifications of the features illustrated herein, and additional applications of the examples as illustrated herein, are contemplated as within the scope of the disclosure.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more examples. In the previous description, numerous specific details were provided, such as examples of various configurations, to provide a thorough understanding of examples of the described technology. It will be recognized, however, that the technology may be practiced without one or more of the specific details, or with other methods, components, devices, etc. In other instances, well-known structures or operations are not shown or described in detail to avoid obscuring aspects of the technology.

Although the subject matter has been described in language specific to structural features and/or operations, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or operations described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. Numerous modifications and alternative arrangements may be devised without departing from the spirit and scope of the described technology.

Claims

1. A method, comprising:

2. The method of claim 1, further comprising:

3. The method of claim 1 or 2, further comprising:

4. The method of claims 1 to 3, further comprising:

5. A system, comprising:

at least one processor;

at least one memory device comprising a data store to store a plurality of data and instructions that, when executed, cause the system to:

6. The system of claim 5, further comprising instructions that, when executed, cause the system to:

7. The system of claim 5 or 6, further comprising instructions that, when executed, cause the system to:

sending the plurality of processing partition assignments to the hardware host.

8. The system of claim 7, further comprising instructions that, when executed, cause the system to:

9. A non-transitory machine-readable storage medium comprising instructions thereon, the instructions being executable by one or more processors, comprising:

10. The non-transitory machine-readable storage medium of claim 9, further comprising:

11. The non-transitory machine-readable storage medium of claim 9 or 10, further comprising:

12. The non-transitory machine-readable storage medium of claims 9 to 11, further comprising:

13. A method, comprising:

14. The method of claim 13, further comprising:

15. The method of claim 13, further comprising: