WO2020198724A1 - Application distribuée, à faible latence, pour mondes interactifs - Google Patents

Application distribuée, à faible latence, pour mondes interactifs Download PDF

Info

Publication number
WO2020198724A1
WO2020198724A1 PCT/US2020/025585 US2020025585W WO2020198724A1 WO 2020198724 A1 WO2020198724 A1 WO 2020198724A1 US 2020025585 W US2020025585 W US 2020025585W WO 2020198724 A1 WO2020198724 A1 WO 2020198724A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
processing
data objects
hardware
partition
Prior art date
Application number
PCT/US2020/025585
Other languages
English (en)
Inventor
Eric Jason EDELMAN
Christopher Denis SCALABRINI
Adam Julio VILLALOBOS
Harold Dillon HICKS
Original Assignee
Amazon Technologies, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US16/368,738 external-priority patent/US11150960B2/en
Priority claimed from US16/368,723 external-priority patent/US11436217B2/en
Priority claimed from US16/368,677 external-priority patent/US11416553B2/en
Priority claimed from US16/368,697 external-priority patent/US11106502B2/en
Application filed by Amazon Technologies, Inc. filed Critical Amazon Technologies, Inc.
Priority to EP20721047.7A priority Critical patent/EP3948539A1/fr
Priority to CN202080038381.6A priority patent/CN114341808A/zh
Publication of WO2020198724A1 publication Critical patent/WO2020198724A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061—Partitioning or combining of resources
    • G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/30—Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
    • A63F13/35—Details of game servers
    • A63F13/352—Details of game servers involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/80—Special adaptations for executing a specific game genre or game mode
    • A63F13/822—Strategy games; Role-playing games

Definitions

  • Virtualization technologies for computing resources have provided benefits with respect to managing large-scale computing resources for many customers with diverse needs, and have allowed various computing resources or computing services to be efficiently and securely shared by multiple customers.
  • virtualization technologies may allow a single physical computing machine to be shared among multiple customers by providing each customer with one or more virtualized compute resources (e.g , computing instances and software containers) hosted by the single physical computing machine using a hypervisor.
  • a user or customer may have access to a large number of dynamic and virtualized compute resources without having to manage the computer hardware upon which those resources execute.
  • a virtualized compute resource may be configured to obtain various additional resources or services via APIs that provide access to the resources and services.
  • Centralized computing resources may be used for the creation of electronic 2D (two- dimensional) virtual environments, 3D (three-dimensional) virtual environments, or multi-dimensional virtual environments such as electronic simulations, electronic worlds, and electronic games, and a large array of centralized computing resources can be used to host such virtual environments.
  • These virtual environments may be accessed by multiple users over a computer network or the internet.
  • virtual worlds, simulations, or electronic games may be accessed by multiple users over the internet. Examples of such virtual worlds may be physics simulators, medical simulations, driving simulators, open adventure worlds, first-person shooter games, sports games, strategy games, or massively multiplayer online (MMO) games, etc.
  • MMO massively multiplayer online
  • Such virtualized worlds may be hosted using a group of virtualized resources in a service provider environment that include distributed applications, virtualized containers, computing instances (i.e., virtual machines), virtualized data stores, virtualized networks, virtualized services and other virtualized computing resources that execute on underlying hardware devices and substrate computer networks
  • virtualized resources in a service provider environment that include distributed applications, virtualized containers, computing instances (i.e., virtual machines), virtualized data stores, virtualized networks, virtualized services and other virtualized computing resources that execute on underlying hardware devices and substrate computer networks
  • Virtualization technologies for computing resources have provided benefits with respect to managing large-scale computing resources for many customers with diverse needs, and have allowed various computing resources or computing services to be efficiently and securely shared by multiple customers.
  • virtualization technologies may allo a single physical computing machine to be shared among multiple customers by providing each customer with one or more virtualized compute resources (e.g., computing instances and software containers) hosted by the single physical computing machine using a hypervisor.
  • a user or customer may- have access to a large number of dynamic and virtualized compute resources without having to manage the computer hardware upon which those resources execute.
  • a virtualized compute resource may be configured to obtain various additional resources or services via APIs that provide access to the resources and services.
  • Centralized computing resources may be used for the creation of electronic 2D (two-dimensional) virtual environments, 3D (three-dimensional) virtual environments, or multi -dimensional virtual environments such as electronic simulations, electronic worlds, and electronic games, and a large array of centralized computing resources can be used to host such virtual environments.
  • These virtual environments may be accessed by multiple users over a computer network or the internet.
  • virtual worlds, simulations, or electronic games may be accessed by multiple users over the internet. Examples of such virtual worlds may he physics simulators, medical simulations, driving simulators, open adventure worlds, first-person shooter games, sports games, strategy' games, or massively multiplayer online (MMO) games, etc.
  • MMO massively multiplayer online
  • Such virtualized worlds may be hosted using a group of virtualized resources in a service provider environment that include distributed applications, virtualized containers, computing instances (i.e., virtual machines), virtualized data stores, virtualized networks, virtualized services and other virtualized computing resources that execute on underlying hardware devices and substrate computer networks.
  • virtualized resources in a service provider environment that include distributed applications, virtualized containers, computing instances (i.e., virtual machines), virtualized data stores, virtualized networks, virtualized services and other virtualized computing resources that execute on underlying hardware devices and substrate computer networks.
  • FIG. 1 illustrates a system that may include a service provider environment hosting computationally intensive distributed computing systems which are used to provide a user accessible virtual environment according to an example of the present technology.
  • FIG. 2 illustrates various example components included in a service provider environment hosting computationally intensive distributed computing systems which are used to provide a user accessible virtual environment according to one example of the present technology.
  • FIG. 3 is a block diagram that illustrates an example computing service that provides computationally intensive distributed computing systems using a distributed computing service manager according to one example of the present technology.
  • FIG. 4A illustrates a system and related operations for assigning
  • computational units in a distributed computing system using processing partitions organized by object domains according to an example of the present technology.
  • FIG. 4B illustrates a graphical example of creating processing partitions to assign to hardware hosts in a distributed computing system using object domains according to an example of the present technology.
  • FIG. 4C illustrates a graphical example of creating processing partitions to assign to hardware hosts in a distributed computing system using object domains and spatial locality according to an example of the present technolog ⁇ .
  • FIG. 5 illustrates various example components included m a service provider environment for managing hardware hosts in a distributed computing system using processing partitions organized by object domains according to one example of the present technolog ⁇ '.
  • FIGS. 6A-6B are flow diagrams that illustrate example methods for managing hardware hosts in a distributed computing system using processing partitions organized by object domains according to examples of the present technology.
  • FIG. 7 is a flow diagram that illustrates an example method for processing data objects assigned to a hardware host using processing partitions organized by object domains according to examples of the present technology.
  • FIG 8 illustrates a hardware host and related operations for storing and replicating states of data objects in a distributed computing system according to an example of the present technology.
  • FIG. 9 illustrates a graphical schematic example of a serialization format for data objects in a distributed computing system that provides low replication cost and thread-safe reads according to an example of the present technology.
  • FIG. 10 is a flow diagram that illustrates an example method for managing storage of data objects m a distributed computing system using a serialization format that provides low replication cost and thread-safe reads according to an example of the present technolog ⁇ .
  • FIG. 1 1 is a flow diagram that illustrates an example method for capturing changes to data objects in a distributed computing system using an ordered, append-only log-based format to provide versioned snapshots of state according to an example of the present technology.
  • FIG 12 is a flow diagram that illustrates an example method for exchanging data objects m a distributed computing system between hardware hosts using an ordered, append-only log-based format to provide versioned snapshots of state according to an example of the present technolog ⁇ .
  • FIG. 13 illustrates a system and related operations for assigning computational units in a distributed computing system using processing partitions with data objects organized by spatial locality according to an example of the present technolog ⁇ .
  • FIG. 14 illustrates a graphical example of creating and assigning processing partitions to hardware hosts in a distributed computing system using spatial location information associated with data objects according to an example of the present technolog ⁇ .
  • FIGS. 15A-15B are flow diagrams that illustrate example methods for managing hardware hosts in a distributed computing system using processing partitions organized by spatial location information associated with data objects according to examples of the present technolog ⁇ 7 .
  • FIG. 16 is a flow diagram that illustrates an example method for processing data objects assigned to a hardware host using processing partitions organized by spatial location information associated with the data objects according to examples of the present technolog ⁇ .
  • FIG. 17 is a flow diagram that illustrates an example method for spliting processing partitions organized by spatial location information according to examples of the present technology.
  • FIG. 18 is a flow diagram that illustrates an example method for merging processing partitions organized by spatial location information according to examples of the present technology.
  • FIG. 19 illustrates an example of physical computer hardware upon which the present technology may execute.
  • This technology provides computationally intensive, distributed computing applications and systems in a service provider environment that may be used for providing virtual worlds, simulations, virtual environments, games, and other multi dimensional (e.g., 2D and 3D) virtual environments that may be viewed by, interacted with, and accessed by users and other computing services.
  • Examples of such distributed virtual environments may be physics simulations, first person games, medical simulations, television or movie sets, driving simulators, airplane simulators, CAD (computer aided design) or CAM (computer aided manufacturing) applications, space simulators, and other virtual worlds.
  • the distributed computing applications may be large-scale, low-latency, distributed applications used with simulations or persistent interactive virtual worlds that can host millions of concurrent users.
  • This technology may manage compute resources in the service provider environment, and may elastically scale to support thousands of applications (i.e., for processing various object types).
  • the technology may load-balance application instances across hardware in response to a virtual world load, so developers may spread the costs of large-scale worlds and simulations across a user base.
  • this technology' may manage compute resources so applications or application instances can effectively utilize the CPU cores (central processing unit cores) of large hardware instances and so that developers can effectively use fewer hardware instances.
  • this technology also enables developers to use a larger number of hardware hosts to run a complex world, even a world with millions of users connecting to a single world or simulation.
  • the technology may automatically scale compute resources based on use (e.g., scale down when not in use or scale up when load increases), and only charge for compute time while a world, simulation, or game is active.
  • the technology may include a state fabric that is a high-throughput compute and data service that may support gigabytes of data writes per second of highly partitioned data from applications or application instances eo-located with the data.
  • Examples of the applications executed using the technology may include a destruction application, which can simulate mass destruction in a game world, simultaneously manipulating hundreds of thousands of physics entities; an AI (artificial intelligence) application, winch enables thousands of competitive AIs in a game; or, a“navmesh’ generation application, which enables developers to recalculate navigation meshes for AI vehicles or characters over a multi-kilometer world changing at 30hz.
  • Custom applications can also have access to the simulation data, data objects, entities, and actors from the state fabric by subscribing to filters for receiving changing data objects and world assets from hardware hosts (e.g., for an area of the world or for an object type).
  • a distributed computing system in the service provider environment may provide the high throughput, low latency compute and data service (or state fabric) to support millions of puts and gets per second of highly partitioned data using one or more hardware hosts.
  • the hardware hosts may include computers, servers, physical hosts or virtual machine instances that execute containerized applications or application instances and manage memory shared the across containerized applications or application instances.
  • a distributed shared memory' (DSM) management system may facilitate client and application interaction with the highly partitioned data as well as enabling distribution of data across many hardware hosts.
  • the DSM enables the highly partitioned data to be read at low-latency and be shared though subscription-based streams.
  • DSM provides fault tolerance by separating the code from the data state. As a result, starting and stopping applications does not affect data stored in the DSM.
  • the DSM can also enable the highly partitioned data to be backed up to long term storage in the service provider environment.
  • the DSM can also replicate the highly partitioned data to a large number (thousands, millions, or tens of millions) of clients. Updates of data to clients may be prioritized using area-of-interest management (e.g., spatial fields or object types) and bandwidth-prioritization systems to maintain client view coherence for massive object worlds or simulations over consumer-grade internet connections.
  • area-of-interest management e.g., spatial fields or object types
  • bandwidth-prioritization systems to maintain client view coherence for massive object worlds or simulations over consumer-grade internet connections.
  • the DSM can also enable customer code (e.g., containerized applications or application instances) to modify the highly partitioned data on a serverless fabric as close to the data as possible for minimum latency and maximum throughput.
  • Applications and clients read data from the DSM by performing one-time queries that subscribe them to updates to data objects that match the query filter.
  • the filter may be a spatial structure (e.g. axis-aligned bounding box), a tag or tag-list (e.g.“bots”) or any other query that can be evaluated on the fields of a data object (e.g. '‘enemy players with low hit points”).
  • a manager service on a hardware host may manage the lifecycles of the applications and the DSM on the hardware host, and may expose an endpoint that allows for linking DSMs on other hardware hosts together.
  • a developer building a game experience in a service provider environment may have been limited by the compute and storage of a single server process on a single hardware instance.
  • gaming consoles paired with a single server process can manipulate a few thousand physics objects per frame, or a few dozen AIs (artificial intelligences) smart enough to challenge a few players.
  • Developers have been able to deploy a single server process to a computing instance on a hardware host or droplet using existing tools, but it may be difficult to utilize additional compute resources to support more users, physics objects, and AIs without building complex clustering technology that enables developers to distribute their virtual world, game, or simulation across servers. Building technology that distributes compute workloads across seivers and is highly stateful, low-latency, and fault tolerant is difficult, risky, and takes years of effort with a large and experienced team.
  • the distributed computing system may use object domains to organize computational units across the hardware hosts.
  • Object domains may define processing partitions (i.e., subdivisions of data objects) using object types of the data objects in an object-based model of a virtual environment.
  • the data objects may be geometric objects (e.g , polygons, spheres, cylinders, etc.), player models, opponent models, buildings, vehicles, monsters, aliens, plants, animals, or any other object that can be modeled or animated in 2D or 3D.
  • An object domain may describe object types for data objects that can be assigned to processing groups or processing partitions.
  • Object types may be data associated with a data object that describes the data object
  • a data object may be of the object type: person, animal, plant, braiding, vehicle, particle, weapon, or another object type.
  • Object domains may use one object type or may use a plurality of object types to subdivide the data objects.
  • Object domains may be associated with one processing partition or with a plurality of processing partitions. Mappings can be created between the object domains and the processing applications that process the data objects subdivided within the object domains.
  • An object type may be used to group data objects of the object type together and the groupings may be treated as the processing partition computational units.
  • Processing applications can be used to process data objects assigned to the processing partition by object type.
  • the object domain can be used to define which application and/or application instances may be used to perform the object domain’s computation and which application and/or application instances may have exclusive mutation permission (e.g., write permission) on data objects for associated object types.
  • data objects with a physics data type may be processed by a physics application.
  • a physics object domain may define which object types belong to the physics object domain, define which physics processing applications have
  • an object type may be owned by an object domain, which has a processing application representing at least one compute process to be applied to the object type owned by the object domain.
  • An object domain thus may provide an indexing process to organize the object data into discrete groups of computational units.
  • An object domain may define which applications have permission to write to specific object types (e.g., exclusive mutation permissions).
  • An object domain may define which applications have permissions to use additional object types (e.g., read-only permissions) during processing.
  • This definition of ownership for data objects enables a single- writer-many- reader model to be enforced for applications and/or application instances.
  • This definition of ownership for data objects also enables replication signaling for the additional object types to be read by object domains, allowing the distributed computing system to calculate what data needs to be made available at which hardware hosts and at what priority.
  • the data objects in the virtual environment may be organized by the distributed computing system into processing partitions according to their object types.
  • the distributed computing system may use the processing partitions to allocate data objects to application instances of the processing applications on the hardware hosts. For example, a number of“particle objects” may be assigned to multiple processing partitions. One or more of the processing partitions may then be assigned to a particle simulation application instance on a first hardware host for processing. Other processing partitions may be assigned to a particle simulation application instance on a second hardware host for processing.
  • a high-performance computing system e.g., a Hadoop cluster
  • the distribution of the data to be processed in parallel has previously been done by dividing the uniform data into parts and sending the data to separate nodes in a computing cluster. Because the data in such cases is uniform, then uniform types of processing is applied to the data. In contrast, dividing data for parallel processing may be problematic where different types of processing may be desired to be applied to separate parts of the data.
  • the present technology ' may divide up a large data set of data objects with various data types for a virtual world (e.g., a game world) into processing partitions for distributed or parallel processing. This technology' enables data objects that are of different types to be separated into groups using the object types.
  • the object types may be assigned to a hardware host that has an application ins tance programmed specifically to process the object type.
  • Use of the object types enables the processing to be divided up into processing partitions distributed among hardware hosts in the distributed computing system based on the processing type that is to be applied to the object type.
  • this technology may also detect and manage dependencies between the data object types, and managing dependencies may overcome bottlenecks in processing of data objects divided into processing partitions.
  • Use of the data dependencies defined by object domains further enables the processing to be organized into processing partitions and be distributed among hardware hosts in the distributed computing system to optimize where the data used by the application instance is located in the distributed computing system.
  • This technology further provides a high throughput, low latency compute and data service that supports millions ofinstall and gets per second of highly partitioned data divided up into processing partitions and distributed among the hardware hosts in the distributed computing system.
  • the distributed computing system may manage storage of the data objects mapped to the processing partitions at the hardware hosts using distributed shared memory (DSM) services on devices.
  • DSM distributed shared memory
  • the distributed shared memory devices may write changes to the data objects using a storage format shared across the distributed shared memory sendees on devices to provide versioned snapshots of state.
  • the distributed shared memory services may also enable the state of the data objects to be read at low-latency though subscription-based streams. Developers using this technology can achieve higher levels of performance, scalability, persistence, and reliability both on a single server, and across on multiple servers, enabling a stable, spatially infinite or spatially dense, interactive experience for players.
  • Tlais technology may write orders of magnitude more simulation state (e.g., 300X+) than past single server database solutions and replicate state across the hardware hosts in the distributed computing system and to users across PCs, gaming consoles, and mobile devices.
  • the distributed computing system may manage storage of the data objects for applications using a serialization format to represent the data objects.
  • the data objects to be stored in the shared memory may be geometric objects (e.g., polygons, spheres, cylinders, etc.), player models, opponent models, buildings, vehicles, monsters, aliens, plants, animals, images, or any other object that can be modeled in 2D or 3D.
  • the distributed computing system may manage state changes to the data objects with an ordered append-only log based storage process using the serialization format.
  • the ordered append-only log based storage process provides low replication cost and thread-safe reads in the distributed computing system. Therefore, data objects may be represented using a format shared across multiple distributed shared memory services on separate devices in the distributed computing system to provide versioned snapshots of state and low replication costs.
  • a data object may be represented in memory of a distributed shared memory in a hardware host using a set of semantics built on top of a byte array.
  • a representation of a data object may be written to a memory device using the byte array broken into a plurality of sections.
  • the sections may describe the in-memory content of the data object and include information on how to read the in-memory content to obtain a current state of the data object.
  • One of the sections may include a log section containing a series of one or more log records. Mutations to the data object caused by processing applications or application instances may be written to a log section using an ordered, log-based format to provide versioned snapshots of state. A log record may be used to represent a version- ordered set of mutations to the data object.
  • the di stributed shared memory' of the hardware host may access the log section from the tail and read log records backwards to get the most up-to-date state of the data object.
  • the distributed shared memory may read the log records backwards to quickly identify the object fields of the data object that may have changed during processing and to collect the most recent version-ordered mutations to the object fields of the data object.
  • the distributed shared memory may also read the representation of the data object originally writen to the memory device when a portion of the data object has not been changed in a log record in the log section.
  • the distributed shared memory may determine the most up-to-date state of the data object from collecting the most recent version-ordered mutations to the object fields and, when needed, the original in memory content of the data object.
  • a high-performance computing system e.g., a Hadoop cluster
  • the format of the data to be processed has generally been a human readable format like ISON or XML, or as a CSV file.
  • These formats generally do not provide an efficient format for distributed computing systems to actually store data, whether in-memory or on disk. Storing data in high level or annotated formats can also be severely inefficient for storage efficiency, over the wire communication, and/or parallel processing.
  • This technology enables data objects to be represented in-memory, on a mass storage device, or transferred over a network connection using a storage process and serialization format that provides a low' replication cost and thread-safe reads.
  • a storage process according to the present technology enables an in-memory representation of a data object to be copied to another memory' location, to another hardware host, or to a storage device without incurring additional serialization processing due to the in-memory
  • the storage process further enables the processing of data objects to be quickly and easily distributed among hardware hosts in the distributed computing system.
  • the storage process may ensure that processing applications perform operations with correct versions of data objects and that updates to the data objects are ordered appropriately.
  • this technology may also manage in-memory changes to the data objects to overcome bottlenecks in processing of the data objects when modified or read by the processing applications.
  • the storage process may further enable the exchange of state between applications and hardware hosts and improve the speed, efficiency, and operation of data storage and replication in the distributed computing system.
  • a spatial analysis of data objects associated with a virtual environment may be used to organize computational units in a distributed computing system.
  • the spatial analysis may determine spatial information about the data objects, such as absolute locations, relative locations, proximal locations, spatial dependencies, and the like.
  • the data objects may be grouped together using spatial location information and collections of data objects may be treated as processing partitions (i.e., sub-divisions of the data objects) for applications or application instances that process data objects assigned to a processing partition.
  • the processing partitions may be defined, in part, according to how the data objects are grouped together spatially.
  • the processing partitions may be load-balanced across hardware hosts in the distributed computing system. Distributing the load for processing data objects in the distributed computing system by using the processing partitions and spatial organization enables large virtual worlds to be processed and presented to clients or users.
  • the technology spatially sub-divides a plurality of data objects into a plurality of processing partitions using one or more object fields that provide spatial location information for the data objects.
  • the spatial location information e.g., an x, y, and z coordinate
  • the processing partition may identify a collection of data objects in the defined locality.
  • the processing partition may be allocated and sent to a hardware host in the distributed computing system for an application to execute code with the data objects.
  • the processing partitions may be sent to the hardware hosts to configure the hardware hosts to allow the application to process the data objects for the identified locality .
  • data objects that are close together spatially can be grouped for one or more processing applications (e.g. a physics application).
  • this technology may use a tree structure to spatially sub-divide the data objects.
  • the tree structure may be an octree or a quad tree that covers three dimensions of a virtual world or two dimensions of a flat environment.
  • Each node in the tree structure may identify a collection of data objects having spatial proximity or spatial groupings to be processed by an application instance configured to process one object type.
  • the nodes in the tree may be used to allocate the data objects to application instances of the processing applications which process the data objects.
  • a number of ‘particle objects” may be assigned to multiple processing partitions by spatially sub-dividing the particle objects by l ocation.
  • One or more of the processing partitions may then be assigned to one or more particle simulation application instances on a hardware host or across multiple hardware hosts for processing.
  • Dividing data for parallel processing may be problematic where different types of processing may be applied to parts of the data.
  • the present technology overcomes this difficulty by dividing a large data set of data objects with various data types for a virtual world (e.g., a game world) into processing partitions for parallel processing.
  • This technolog ⁇ ' considers the spatial relationship of the data objects in order for the data objects to be separated into groups and be more quickly processed.
  • Use of the spatial analysis enables the processing to be divided up into processing partitions distributed among hardware hosts in the distributed computing system in order to group related data as closely to possible.
  • this technology may also detect and manage spatial dependencies between the data object types, and managing spatial dependencies may overcome bottlenecks in processing of data objects divided into processing partitions.
  • Use of the data dependencies defined by object domains further enables the processing to be organized among hardware hosts in the distributed computing system into processing partitions to optimize where the data used by the application instance is located in the distributed computing system.
  • the distributed computing system may use object domains to identify and control processing applications executing on the hardware hosts.
  • the hardware hosts may be preloaded with processing applications using an application library.
  • the application library may be stored by a hardware host and used by an application manager on the hardware host to launch instances of processing applications according to associated object types.
  • an object domain can be used to define which application and/or application instances may be used to perform the object domain’s computation and which application and/or application instances may have exclusive mutation permission (e.g., write permission) on data objects for associated object types.
  • Launching instances of the processing applications m the distributed computing system using the object types enables large virtual worlds to be processed and presented as processing may be allocated to any number of hardware hosts with a copy of the application library.
  • a hardware host may receive an assigned processing partition and identify, from the application library, a processing application for the object type associated with (e.g., matched to or able to perform functions on) the processing partition.
  • the hardware host may launch an instance of the processing application to enable the hardware host to process a plurality of data objects grouped by the object type, which are mapped to the processing partition.
  • the distributed computing system may enable hardware hosts to communicate though subscription-based streams.
  • the processing applications in the application library may define a subscription strategy for obtaining additional data used during the processing of the data objects.
  • the subscription strategy may identify what additional data object types are used during processing and at what sources the additional object types may be located.
  • the hardware hosts may use data dependencies, spatial relationships, object tracking, and queries defined in the subscription strategies to send subscription requests to enable the hardware hosts to communicate and copy the data objects used by the instances of the processing applications during processing.
  • the distributed computing system may enable hardware hosts to share the data objects used by processing applications distributed across the hardware hosts.
  • this technology may enable a first hardware host to receive a plurality of processing partition assignments and to identify a first processing partition allocated to the first hardware host. The first hardware host may then determine a first object type associated with the first processing partition and use the first object type to launch an instance of a corresponding processing application from the application library. The first hardware host may also determine a second object type upon which processing of data objects of the first object type by the processing application is dependent. The first hardware host may use the plurality of processing partition assignments to determine a second hardware host assigned a second processing partition, which groups data objects by the second object type into the second processing partition. The first hardware host may send a subscription request to the second hardware host to instruct the second hardware host to copy (e.g , make copies of the base data objects or changes to the data objects) the second plurality of data objects to the first hardware host and the instance of the processing application.
  • this technology may enable the first hardware host to determine a subscription strategy identifying neighbor relationships between spatial sub divisions associated with a plurality of spatial sub-divisions for a multi-dimensional virtual environment to filter data objects of the second object type satisfying the filter.
  • the first hardware host may identify the second processing partition using a neighbor relationship, winch satisfies the subscription strategy, between a first spatial sub-division associated with the first processing partition and a second spatial sub-division associated with the second processing partition
  • the first hardware host may determine a subscription strategy identifying query' criteria (e.g., vehicles with low' hit points or a particular moving object) associated with a query' to filter data objects of the second object type.
  • a subscription strategy identifying query' criteria e.g., vehicles with low' hit points or a particular moving object
  • the first hardware host may identify the second processing partition by matching the second plurality of data objects to the query' criteria using the subscription strategy.
  • the first hardware host may also determine a list of subscribers to the data objects of the first object type.
  • the first hardware host may send the data objects to the list of subscribers.
  • the first hardware host may further receive an update to the plurality' of processing partition assignments.
  • the first hardware host may determine a migration of the second processing partition between the second hardware host and a third hardware host, and send a second subscription request from the first hardware host to the third hardware host.
  • FIG. 1 illustrates a system 100 that may include a service provider
  • the system 100 may include the service provider environment 102 and one or more client(s) 104 m communication with the sendee provider environment 102 using a network 106
  • the network 106 may include any useful computing network, including an intranet, the Internet, a local area network, a wide area network, a wireless data netw'ork, or any other such network or combination thereof. Components utilized for such a system may depend at least in part upon the type of network and/or environment selected. Communication over the network may be enabled by wired or wireless connections and combinations thereof.
  • the service provider environment 102 may be capable of management and delivery of computing, storage, and networking capacity as a software service for a community of end recipients using the client(s) 104.
  • the service provider environment 102 may include an infrastructure manager 110, one or more infrastructure service(s) 112, and one or more distributed computing system(s) 120.
  • the infrastructure manager 110 may manage and control physical machines and other physical devices for use in supplying computatio ally intensive distributed computing systems.
  • the infrastructure manager 110 may be a service that includes one or more computing systems, such as server computers, configured to control physical and virtualized infrastructure resources of the infrastructure service(s) 112 and the distributed computing system(s) 120.
  • the virtualized infrastructure resources of the infrastructure service(s) 112 and the distributed computing system(s) 120 may include virtualized executable resources, virtualized storage resources, virtual network interfaces, and other virtualized networking components.
  • Some examples of the virtualized executable resources may include computing instances, containers (e.g., Kubemetes), compute functions, hosted applications, and the like.
  • Some examples of the virtualized storage services may include database services, block storage services, content-delivery services, and the like.
  • Some examples of the virtualized networking components may include virtualized networking devices and physical network devices (i.e., routers, firewalls, load- balancers, etc.) configured with logical roles in the infrastructure service(s) 112 and the distributed computing system(s) 120.
  • the infrastructure manager 1 10 may instantiate all or part of the infrastructure service(s) 112 and the distributed computing system(s) 120.
  • the infrastructure manager 110 may identify physical hosts and physical networking devices to be managed by the infrastructure manager 110. Some examples of the physical hosts managed by the infrastructure manager 110 are server computers, embedded computers, computing hosts at the edge of the service provider environment 102, and other physical devices. Some examples of the physical networking devices used by the infrastructure manager 110 are routers, swatches, firewalls, load-balancers, caching services, server computers, embedded devices, and other physical networking devices.
  • the infrastructure service(s) 1 12 may be considered on-demand computing services that are hosted m a server, virtualized machine, grid, cluster, parallel, or distributed computing system.
  • the infrastructure service! s) 1 12 may include one or more compute service(s) 114, one or more storage service(s) 116, networking services, web services, streaming services, network accessible services, software as a sendee, storage as a service, on-demand applications, services for the execution of code functions, and the like in this example, the infrastructure service(s) 112 may include a distributed computing service 118 to provide for building and hosting the distributed computing system(s) 120 as large-scale, low-latency, distributed computing systems.
  • the distributed computing system(s) 120 may be considered on-demand computationally intensive distributed computing systems that are hosted in servers, virtualized machines, grids, clusters, parallel, or distributed computing systems.
  • the distributed computing syslem(s) 120 may be used to provide virtual environments, e.g., 2D (two-dimensional) or 3D (three-dimensional virtual environments), simulations, and “computationally plausible games” that include persistent, interactive virtual worlds that can host millions of concurrent users or game players.
  • the distributed computing system(s) 120 may provide a high throughput, low latency compute and data sendee that supports millions of puts and gets per second for highly partitioned data and data objects.
  • the distributed computing system(s) 120 may include a control plane 122, a data plane 124, and an entry plane 126.
  • the control plane 122 may be responsible for orchestrating the creation of distributed computing systems for simulations, virtual environments, games, and the like, and the control plane 122 may be responsible for load balancing across hardware hosts in the distributed computing systems.
  • a hardware host (such as hardware hosts 142 and 144) may be a physical host, a hardware computing device or server that is able to host one or more virtualized computing resources or services.
  • the control plane 122 may manage aspects of the distributed computing systems, such as running simulations and virtual environments, including deploying code for execution, gathering metrics, uploading logs and monitoring clusters of hardware hosts.
  • the control plane 122 may also provide a dashboard accessible to a user for executing auditable maintenance actions across clusters of hardware hosts, and development teams can use the dashboard to access operating characteristics of running simulations and virtual environments.
  • the control plane 122 may include a system manager 130 that manages resources used m a distributed computing system by simulations and virtual environments, such as a hardware host warming pool 132 and one or more cluster service(s) 134.
  • the hardware host warming pool 132 may include one or more hardware hosts with pre-loaded operating systems, apps, assets, and/or data that are awaiting an assignment from the system manager 130.
  • a warm hardware host may represent a physical host or virtualized infrastructure resource allocated to the simulation for later use.
  • the cluster service(s) 134 may include services that support the operation of the distributed computing system, such as health monitoring, ticketing, logging, metrics, deployment, workflows, and the like.
  • the data plane 124 may be responsible for processing in the distributed computing systems of the simulations, virtual environments, games, and the like, and the data plane 124 may be responsible for communication across hardware hosts in the distributed computing systems.
  • the data plane 124 may include a hardware host acti e pool 140 over which computational uni ts of the simul ation or virtual world are distributed using one or more hardware hosts, such as the hardware hosts 142 and 144.
  • the hardware hosts 142 and 144 include one or more file(s) 150a-b, one or more
  • the file(s) 150 may be data, resources, data objects, 3D models, 2D models, textures, images, animations, web pages, etc.
  • the appiication(s) 152 may include a category of user- computation or executing processes that can execute on data associated with the file(s)
  • an application describes a plurality of application instances in the aggregate.
  • an application may be a physics app that modifies data objects using physically-based numerical methods.
  • the distributed shared memories 154 may manage data in memory that is used and shared across the application(s) 152.
  • the distributed shared memories 154 may also replicate changes to the data in memory across hardware hosts m the hardware host active pool 140.
  • the application(s) 152 may subscribe to streams of data object events by requesting that updates to the data in memory be delivered from the distributed shared memories 154a-b to the appiication(s) 152.
  • the application(s) 152 may create streams of data object events that include modifications to the data in memory, which are sent to the distributed shared memories 154a-h.
  • the distributed shared memories 154a-b may replicate changes in state of the data in the memory between the distributed shared memories (DSMs) of the hardware hosts 142 and 144.
  • DSMs distributed shared memories
  • the entry plane 126 may be responsible to facilitate communication with the distributed computing systems of the simulations, virtual environments, games, and the like. For example, the entry plane 126 may manage connections from outside of the distributed computing system(s) 120.
  • the entry plane 126 may include a front-end service explorer 160, one or more remote entry point(s) 162, and one or more client gateway(s) 164.
  • the client(s) 104 may utilize an application programming interface (API) to connect to the client gateway (s) 164, which authenticate credentials and authorize connections to the remote entry point(s) 162
  • the remote entry point(s) 162 may include a multi-tenant fleet of physical hosts that replicate data from the virtual worlds or simulations to the client(s) 104.
  • the front-end service explorer 160 may provide a graphical user interface for accessing, managing, monitoring, updating, or developing the distributed applications, data objects, and/or assets executing in the service provider environment 102.
  • This technology can provide a serverless virtual world or simulation service for building and operating computationally large worlds, such as massively multiplayer games with millions of connected players and city-scale simulations with millions of persistent objects. Developers can create millions of data objects and launch applications (e.g., compute processes) that modify the data objects at a variable rate (e.g., 10-60 Hz). This technology may automatically manage where and when the millions of data objects and applications are allocated to hardware hosts. The technology may load-balance application instances across the hardware hosts m response to a variety of conditions, such as a load on the virtual world, or collocate data processed by the application to reduce replication costs.
  • Developers can create millions of data objects and launch applications (e.g., compute processes) that modify the data objects at a variable rate (e.g., 10-60 Hz).
  • This technology may automatically manage where and when the millions of data objects and applications are allocated to hardware hosts.
  • the technology may load-balance application instances across the hardware hosts m response to a variety of conditions, such as a load on the virtual
  • FIG. 2 illustrates various example components included in a service provider environment 200 for computationally intensive distributed computing systems which are used to provide a user viewable virtual environment according to one example of the present technology.
  • the service provider environment 200 may be capable of delivery of computing, storage, and networking capacity as a software service to a community of end recipients.
  • the service provider environment 200 may be established for an organization by or on behalf of the organization. That is, the service provider environment 200 may offer a‘"private cloud environment.”
  • the service provider environment 200 may support a multi-tenant environment, wherein a plurality of customers may operate independently (i.e., a public cloud environment).
  • the service provider environment 200 may provide the following models: Infrastructure as a Service (“IaaS”) and/or Software as a Sendee (“SaaS”). Other models may be provided.
  • IaaS Infrastructure as a Service
  • SaaS Software as a Sendee
  • Other models may be provided.
  • the service provider environment 200 may offer computers as physical or virtual machines and other physical devices for use as virtualized infrastructure resources in the virtual infrastructures.
  • the SaaS model allows installation and operation of applications in the service provider environment 200.
  • End customers may access the service provider environment 200 using networked client devices, such as desktop computers, laptops, tablets, smartphones, game consoles, etc., running web browsers or other standalone client applications, for example.
  • the service provider environment 200 may include a number of computing devices that are arranged, for example, in one or more server banks or computer banks or other arrangements.
  • Tire computing devices may support a computing environment using hypervisors, virtual machine managers (VMMs), and other virtualization software.
  • the service provider environment 200 may include one or more server computer(s) 202.
  • the server computer(s) 202 may include a system manager module 210, a cluster manager module 210a, a world manager module 210b, a cluster service(s) module 212, a data store 214, a front-end service explorer module 216, a remote entry point module 218, a client gateway module 220, one or more warm hardware host(s) 224, one or more active hardware host(s) 226, one or more processor(s) 230, and one or more memory module(s) 232
  • the system manager module 210 may include hardware and software configured to create, deploy, and manage high throughput low latency distributed computing systems.
  • the system manager module 210 may use the cluster manager module 210a and the world manager module 210b to orchestrate the creation of the distributed computing systems for simulations, virtual environments, and the like using a cluster of hardware hosts.
  • the cluster manager module 210a may manage aspects of running infrastructures associated with the distributed computing systems, including deploying code for execution, gathering metrics, uploading logs, and monitoring cluster resources.
  • the world manager module 210b may manage aspects of running multi- dimensional virtual environments, worlds, or simulations hosted by the distributed computing systems, including allocating files and applications to hardware hosts and monitoring changes in the processing of data on the hardware hosts by the applications.
  • the system manager module 210 may also use the cluster manager module 210a and the world manager module 210b to load-balance computational operations across the hardware hosts.
  • the cluster service! s) module 212 may include hardware and software elements configured to provide services that support the operation of the distributed computing systems, such as a monitoring service 240, a logging service 242, a ticketing service 244, a workflow service 246, a deployment service 248, and an operations console service 250.
  • the monitoring service 240 may be used to monitor resources used by the distributed computing systems.
  • the monitoring service 240 may collect metrics associated with the acti ve hardware host(s) 226 to determine utilization of one or more hardware or software resources.
  • the logging service 242 may be used to manage and analyze logs and operational data points from the distributed computing system.
  • the ticketing sendee 244 may be used for ticketing of problems or ticketing of check-in or check-out source code, executable code or data.
  • the workflow service 246 may be used manage and execute workflows in the distributed computing systems. For example, the workflow service 246 may initialize, structure, or load virtual computing resources, data objects, hardware hosts, etc.
  • the deployment service 248 may be used to configure and deploy hardware hosts in the distributed computing systems.
  • the operations console service 250 may be used to interact with a control plane, data plane, and entry plane associated with the distributed computing system.
  • the operations console service 250 may provide one or more user interfaces (textural or graphical), application programming interfaces (APIs), and the like through which a user may enter commands or retrieve data associated with the distributed computing systems.
  • the data store 214 may include hardware and software elements configured to provide data services to the distributed computing systems managed by the cluster manager module 210a and the multi-dimensional virtual environments managed by the world manager module 210b.
  • the data store 214 may include one or more world/cluster configuration ⁇ ) 260, one or more file(s) 262, and one or more app!ication(s) 264.
  • the world/cluster configuration(s) 260 may define the distributed computing system used for a world or a simulation.
  • a world may be defined using an object-based model of a virtual environment.
  • the virtual environment may be a 2-dimensional, 3- dimensional, or multi-dimensional virtual world.
  • An object-based model may use data objects to represent entities within the virtual environment.
  • the file(s) 262 may include the data, resources, data objects, 3D models, 2D models, textures, images, animations, web pages, etc. used by the distributed computing systems.
  • data objects in the file(s) 262 may represent entities within the virtual environment. These entities may be characters, animations, geometric objects, actors, non-player characters, vehicles, buildings, plants, rocks, animals, monsters, etc.
  • Data objects in the file(s) 262 may be represented in the data store 214 using attributes having key-value or name-value pairs and object fields having field identifiers and field values.
  • the application(s) 264 may include executable code that processes the file(s) 262. For example, the app!ication(s)
  • 264 may process data objects used to represent entities within the virtual environment. Some examples of the application(s) 264 may include simulations, collision detection, physics engines, rendering, and the like.
  • the front-end sen ice explorer module 216 may include hardware and software elements configured to provide access, management, monitoring, updating, or developing of the world / cluster configurations) 260, the file(s) 262, and the
  • the front-end service explorer module 216 may include one or more graphical user interfaces to construct virtual worlds, define data objects, write applications, and the like.
  • the remote entry point module 218 may include a multi-tenant fleet of physical hosts that replicate data from the data store 214 and/or the active hardware host(s) 226 to remote clients. Further, the clients may utilize an API to connect to the client gateway module 220, which authenticate credentials and authorize connections to the remote entry point module 218.
  • the warmthing pool hardware host(s) 224 may include hardware and software elements configured in a standby mode to execute the application(s) 264 using the file(s) 262.
  • the cluster manager module 210a may allocate a number of hardware hosts to the warming pool hardware hosi(s) 224 in a standby or suspended mode of operation in anticipation of future need.
  • the cluster manager modul e 210a may migrate a hardware host between the warming pool hardware host(s) 224 and the active pool hardware host(s) 226 to scale compute resources when the need arises.
  • the active pool hardware host(s) 226 may include hardware and software elements configured in an active mode to execute the applicatioii(s) 264 using the file(s) 262.
  • the active pool hardware host(s) 226 may include a hardware host manager module 270, an application manager module 272, one or more application instance(s) 284, a distributed shared memory module 274, a file system data store 276, and an in-memory data store 278.
  • the hardware host manager module 270 may include hardware and software elements configured to manage data processing by one or more hardware hosts.
  • the hardware host manager module 270 may include a runtime module that receives instructions from the cluster manager module 210a and the world manager module 210b executes the instructions on the hardware host.
  • the instructions may configure the hardware host in the active mode to execute the application(s) 264 using the file(s) 262.
  • the instructions may identify the file(s) 262 that are assigned to the hardware host and which of the application(s) 264 to use to process the assigned file(s) 262.
  • the hardware host manager module 270 may receive one or more of the file(s) 262 for processing and store the files in the file system data store 276 as local file(s) 280.
  • the hardware host manager module 270 may receive, from the world manager module 210b, one or more index structures that are assigned to the hardware host.
  • the index structures may be used by the world manager module 210b to organize the file(s) 262 into computation units, herein known as processing partitions discussed in more detail below.
  • the index structures may be used by the hardware host manager module 270 to manage storage of the local file(s) 280 in the file system data store 276.
  • the instructions may further identify the application instance(s) 264 to use to process one or more of the local file(s) 280.
  • the hardware host manager module 270 may receive one or more of the application ⁇ ) 264 and store the applications in the file system data store 276 as local application(s) 282. According to one example of the present technolog ⁇ ', the hardware host manager module 270 may receive an application library that includes the application(s) 264.
  • the application library may be stored by the hardware host manager module 270 and used by the application manager module 272 to launch instances of the local application(s) 282 in the file system data store 276
  • the application manager module 272 may include hardware and software elements configured to manage one or more application instance(s) 284 on the hardware host. Examples of the application instance(s) 284 may be instances of physics applications, rendering applications, collision applications, transformation applications, occlusion applications, sound applications, or other types of applications.
  • the application manager module 272 may identify the local file(s) 280 assigned to the hardware host and determine which of the local application(s) 282 to instantiate.
  • the application manager module 272 may determine the number of instances of the local application ⁇ ) 282 to he instantiated as the application instance(s) 284.
  • the application manager module 272 may- scale the number of the application instanee(s) 284 according to demand.
  • the distributed shared memor ' module 274 may include hardware and software elements configured to manage the in-memory data store 278.
  • the in -memory data store 278 of the hardware host may be used for storage by the application instance(s) 284 and changes to the in-memory data store 278 may be shared across the active pool hardware host(s) 226.
  • the distributed shared memory' module 274 may load one or more of the local file(s) 280 into the in-memory data store 278 as shared memory 286.
  • the distributed shared memory' module 274 may receive requests from the application instance(s) 284 to access the shared memory 286.
  • the distributed shared memory' module 274 may send data from the shared memory ' 286 for example by processing the requests from the application instance(s) 284 to read the data.
  • the distributed shared memory' module 274 may also receive requests from the application instance(s) 284 to modify the shared memory 286.
  • the distributed shared memory module 274 may store data to the shared memory 286 for example by processing the requests from the application instance(s) 284 to write the data.
  • the distributed shared memory module 274 may further replicate changes to the shared memory' 286 across the active hardware host(s) 226.
  • the various processes and/or other functionality contained within the service provider environment 200 may be executed on one or more processor(s) 230 that are in communication with one or more memory module(s) 232.
  • the service provider environment 200 may include a number of computing devices that are arranged, for example, in one or more server banks or computer banks or other arrangements.
  • the computing devices may support a computing environment using hypervisors, virtual machine monitors (VMMs), and other virtualization software.
  • VMMs virtual machine monitors
  • the term‘data store” may refer to any device or combination of devices capable of storing, accessing, organizing and/or retrieving data, which may include any combination and number of data servers, relational databases, object-oriented databases, cluster storage systems, data storage devices, data warehouses, flat files, and data storage configuration in any centralized, distributed, or clustered environment.
  • the storage system components of the data stores 214, 276, and 278 may include storage systems such as a SAN (Storage Area Network), cloud storage network, volatile or non-volatile RAM, optical media, or hard-drive type media.
  • the data stores 214, 276, and 278 may be representative of a plurality of data stores as can be appreciated.
  • FIG. 2 illustrates that certain processing modules may be discussed in connection with this technology and these processing modules may be implemented as computing services.
  • a module may be considered a service with one or more processes executing on a server or other computer hardware.
  • Such services may be centrally hosted functionality or a service application that may receive requests and provide output to other services or consumer devices.
  • modules providing services may be considered on-demand computing that are hosted in a server, virtualized service environment, grid or cluster computing system.
  • a API may be provided for each module to enable a second module to send requests to and receive output from the first module.
  • Such APIs may also allow third parties to interface with the module and make requests and receive output from the modules.
  • FIG. 2 illustrates an example of a system that may implement the techniques above, many other similar or different environments are possible. The example environments discussed and illustrated above are merely representative and not limiting.
  • FIG. 3 is a block diagram that illustrates an example computing service 300 that provides computationally intensive distributed computing systems using a distributed computing service manager according to one example of the present technology.
  • the computing service 300 may be used to execute and manage a number of computing instances 304a-d upon which the present technology may execute.
  • the computing service 300 depicted illustrates one environment in which the technology described herein may be used.
  • the computing service 300 may be one type of environment that includes various virtualized service resources that may be used, for instance, to host computing instances 304a-d.
  • the computing service 300 may be capable of delivery of computing, storage, and networking capacity' as a software sendee to a community of end recipients.
  • the computing service 300 may be established for an organization by or on behalf of the organization. That is, the computing service 300 may offer a“private cloud environment.”
  • the computing service 300 may support a multi-tenant environment, wherein a plurality' of customers may operate independently (i.e., a public cloud environment).
  • the computing service 300 may provide the following models: Infrastructure as a Sendee (“laaS”), and/or Software as a Service (“SaaS”). Other models may be provided.
  • laaS model the computing service 300 may offer computers as physical or virtual machines and other resources. The virtual machines may be run as guests by a hypervisor, as described further below.
  • the sendees model delivers computing that may include an operating system, programming language execution environment, database, and web server.
  • the particularly illustrated computing service 300 may include a plurality of server computers 302a-d.
  • the server computers 302a-d may also be known as physical hosts. While four server computers are shown, any number may be used, and large data centers may include thousands of server computers.
  • the computing service 300 may provide computing resources for executing computing instances 304a-d.
  • Computing instances 304a-d may be virtual machines.
  • a virtual machine may be an instance of a software implementation of a machine (i.e. a computer) that executes applications like a physical machine.
  • each of the server computers 302a-d may be configured to execute an instance manager 3Q8a-d capable of executing the instances.
  • the instance manager 308a-d may be a hypervisor, virtual machine manager (VMM), or another type of program configured to enable the execution of multiple computing instances 304a-d on a single server. Additionally, each of the computing instances 304a-d may be configured to execute one or more
  • a server computer 314 may be reserved to execute software components for implementing the present technology or managing the operation of the computing service 300 and the computing instances 304a-d.
  • the server computer 314 may execute a distributed computing service manager 315 to provide a high throughput, low latency compute and data service that supports millions of puts and gets per second of highly partitioned data.
  • a server computer 316 may execute a management component 318.
  • a user may access the management component 318 to configure various aspects of the operation of the computing instances 304a-d purchased by a customer. For example, the user may setup computing instances 3Q4a-d and make changes to the configuration of the computing instances 304a-d.
  • a deployment component 322 may be used to assist customers in the deployment of computing instances 304a-d.
  • the deployment component 322 may have access to account information associated with the computing instances 304a-d, such as the name of an owner of the account, credit card information, country ' of the owner, etc.
  • the deployment component 322 may receive a configuration from a user that includes data describing ho computing instances 3G4a-d may be configured.
  • the configuration may include an operating system, provide one or more applications to be installed in computing instances 304a-d, provide scripts and/or other types of code to be executed for configuring computing instances 304a-d, provide cache logic specify ing how an application cache is to be prepared, and provide other types of information.
  • the deployment component 322 may utilize the user provided configuration and cache logic to configure, prime, and launch computing instances 304a-d.
  • the configuration, cache logic, and other information may be specified by a user accessing the management component 318 or by providing this information directly to the deployment component 322.
  • Customer account information 324 may include any destred information associated with a customer of the multi-tenant environment.
  • the customer account information 324 may include a unique identifier for a customer, a customer address, billing information, licensing information, customization parameters for launching instances, scheduling information, etc.
  • the customer account information 324 may also include security information used in encryption of asynchronous responses to API requests. By“asynchronous,” it is meant that the API response may be made at any time after the initial request and with a different network connection.
  • a network 310 may be utilized to interconnect the computing service 300 and the server computers 302a-d, 316.
  • the network 310 may be a local area network (LAN) and may be connected to a Wide Area Network (WAN) 312 or the Internet, so that end customers may access the computing sendee 300.
  • the network 310 may include a virtual network overlaid on the physical network to provide communications between the server computers 302a-d.
  • the network topology' illustrated in FIG. 3 has been simplified, as many more networks and networking devices may be utilized to interconnect the various computing systems disclosed herein.
  • FIG. 4A illustrates a system 400 and related operations for assigning computational units in a distributed computing system using processing partitions organized by object domains according to an example of the present technology.
  • the system 400 may include a world manager 402.
  • Some examples of the world manager 402 may include the syste manager 130 described with respect to FIG. 1 and the world manager module 210b described with respect to FIG. 2.
  • the world manager 402 may identify one or more data object(s) 404 associated with an object-based model of a multi -dimensional virtual environment (e.g , a 313 simulation, 3D game, etc.).
  • the world manager 402 may retrieve the data object(s) 404 from an object data store.
  • the data object(s) 404 may include one or more object field(s) 420.
  • the object field(s) 420 may include a set of fields, attributes, or properties represented by a field name 422, which may act as an identification label or key, and a field value 424.
  • the field value 424 may include a data value, string, an X, Y, Z coordinate, etc., which may be referenced by a processing application using the object identifier 430 in order to process a data object.
  • the data object(s) 404 may include metadata using one or more of the object field(s) 420 to provide an object identifier (ID) 430, an object ID value 431 , an object version 432, object version value 433, an object type 434, and object type value 425.
  • An example object ID value 431 for the object identifier 430 field may be “forest tree object” and an example object ID value 431 of the object version 432 field may be“version 4.214.”
  • the object type 434 may map to an object domain and have an object type value 435 of“Particle,”“Tree”,“Water”, etc.
  • the object domain describes which object types form part of the domain thereby enforcing permissions on what processing application has exclusive mutation permission, as well as data dependencies (e.g., read-only permission) for object types used in the processing of the object types that form part of the domain.
  • An object type may be owned by one domain, which has a processing application representing its compute.
  • the types of processing may include any types of processing on data objects that may occur for a virtual world, simulation, or game.
  • the world manager 402 may then determine one or more processing partition(s) 406 using the object type 434 of the data objeci(s) 404. In addition to the data object(s) 404 themselves, the world manager 402 may use a world configuration 408 to determine the processing partition(s) 406, as discussed in more detail later.
  • the processing partition(s) 406 may define computational units in a distributed computing system hosting the multi-dimensional environment, which may be allocated to hardware hosts in the distributed computing system.
  • the data ob ect(s) 404 may be organized or grouped by the object type value 435 of the object type 434 into the processing partition(s) 406 to create the computational units for the hardware hosts.
  • the processing partition(s) 406 may group the data object(s) 404 into the computational units according to object domains represented by object types. Metadata in a processing partition can reference a collection or set of data objects, which are mapped to the processing partition, as a computational unit for processing.
  • the world manager 402 may determine the processing parti tion(s) 406 using the object field(s) 420 of the data object(s) 404.
  • the processing parti tion(s) 406 represent groups of data objects organized into computational units to be processed by processing applications
  • the world manager 402 may organize the data object(s) 404 according to one or more of the object field(s) 420.
  • this data object management by the world manager 402 may include the creation of one or more index structures, known as a domain index, which map collections of the data object(s) 404 to the processing partition(s) 406 by object domain using the object type 434, as will be discussed in a later section.
  • the index structures may index the data object(s) 404 into the processing partition(s) 406.
  • this data object management by the world manager 402 may include the creation of one or more index structures, known as a spatial index, which can be used to divide up the processing of the data object(s) 404 by spatial location information (e.g., spatial position, bounding box, etc.) in the object field(s) 420, as will be discussed in a later section.
  • the world manager 402 may use the object field(s) 420 to create multiple tiers in the groupings of the data object(s) 404 into computational units, thereby further organizing the data object(s) 404 by multiple fields, attributes, and properties represented by the object field(s) 420.
  • the world manager 402 may then determine one or more processing partition assignment(s) 410 to assign the processing parti tion(s) 406 to one or more hardware liost(s) 412.
  • the processing partition assignment(s) 410 may be made based on the object type and an application instance that can process the object type, based on data dependencies, and based the metrics for the processing load on the hardware or software (e.g., to avoid overloading, etc.).
  • the world manager 402 may use the processing partition assignment(s) 410 to send the processing partition(s) 410 to the hardware liost(s) 412 to allocate the processing of the data object(s) 404 in the distributed computing system.
  • the processing partition assignments ) 410 may be used to configure the hardware host(s) 412 for processing of the data object(s) 404.
  • the hardware host(s) 412 may process the data object(s) 404 using the processing partition assignment(s) 410 Examples of processing may include data object movements, data object collisions, rendering, explosions, physics simulations, in-world credit transactions, or other types of processing.
  • the data object(s) 404 for a specific processing partition in the processing partition(s) 406 may be loaded at one of the hardware host(s) 412 and then processed by a processing application. For example, an instance of the processing application may be set up on the hardware host to process an explosion of the data object(s) 404.
  • the instance of the processing application can iterate across one or more the data object(s) mapped the processing partitions assigned to the hardware host to simulate what occurs to the data object(s) 404 during an explosion.
  • the changes for the data object(s) 404 may then be written to memory in the hardware host (using the DSM)
  • the changes written to memor ' can also be shared with other hardware hosts.
  • Tlais technology may use multiple tiers of indexing to organize data objects into the processing partition(s) 406 representing computational units in the distributed computing system.
  • the world manager 402 may create the multiple tiers as a hierarchy.
  • a multi-tier index scheme may be used with the object domain grouping being the most-significant tier.
  • FIG. 4B illustrates the separation of a global view of the multi -dimensional virtual environment into more than one group or processing partition according to object domain.
  • the multi-dimensional environment may include data objects with two object domains, a first object domain“Domain A” represented by crosses and a second object domain‘/Domain B represented by solid dots in the global view.
  • the world manager 402 may decompose the global view into a“Domain A Decomposition” and a“Domain B Decomposition,” which become the most-significant tiers grouping data objects by object type associated with“Domain A” and“Domain B.”
  • This decomposition process also allows less-significant tiers in the multi-tier index scheme to use similar semantics, while being different in terms of instance. For example, in FIG. 4C, if each of the“Domain A” and“Domain B” include a large number of data objects densely packed into different spaces, this decomposition process may add another tier, such as a spatial index, to further index the data objects into processing partitions using spatial location information.
  • the world manager 402 may organize the data objects into processing partition(s) 406 and determine whether to allocate data objects m the different states to different hardware hosts.
  • FIG. 4B also illustrates a permission model that may be used with the object domains. More specifically, a domain allows a developer to declare which application owns each of the object types and which additional object types depend on any particular domain. While other applications may read data objects not owned by their domain, they might not mutate them (e.g., may not write to the data objects). As illustrated,“Domain A” permissions for a first application may include read only permissions for data objects associated with“Domain B” and exclusive write permission for data objects associated with“Domain A.”“Domain B” permissions for a second application may include read only permissions for data objects associated with“Domain A” and exclusive write permission for data objects associated with“Domain B.”
  • FIG. 4B further illustrates how the permission model may be used with the object domains for replication signaling. More specifically, as a domain allows a developer to declare which application owns each of the object types and which additional object types depend on any particular domain, the permission model may signal whether data object are to be shared by applications or replicated across the distributed computing system. For example, the first application associated with “Domain A” may declare that the first application owns the data objects associated with the object types for“Domain A.” The first application may further subscribe to streams of data object events by requesting delivery' of updates to the data objects associated with the object types for“Domain B.”
  • This replication signaling mechanism may act as an object filtering mechanism to reduce the number of objects that need to be replicated across hardware hosts. For example, a domain requiring object types of another domain does not imply an inverse relationship.
  • FIG. 5 illustrates various example components included in a service provider environment 500 for managing hardware hosts in a distributed computing system using processing partitions organized by object domains according to one example of the present technology.
  • the service provider environment 500 may include one or more server computer(s) 502 in communication with one or more hardware host(s) 504 using a network 506.
  • the network 506 may include any useful computing network, including an intranet, the Internet, a local area network, a wide area network, a wireless data network, or any other such network or combination thereof.
  • the server computer(s) 502 may include a world manager module 510, a first data store 512, a second data store 514, a third data store 516, one or more processor(s) 518, and one or more memory module(s) 520.
  • the world manager module 510 may be configured to create, deploy, and manage high throughput low latency distributed computing systems.
  • the world manager module 510 may orchestrate the creation of the distributed computing systems for virtual worlds, games, simulations, virtual environments, and the like using the hardware host(s) 504.
  • the world manager module 510 may manage aspects of running the distributed computing systems, including deploying code for execution (e.g., application instances), gathering metrics, uploading logs, and monitoring resources.
  • the world manager module 510 may access the first data store 512 to create, and manage, a world configuration 522 for a virtual world, one or more data object(s) 524 stored for the virtual world, and one or more application(s) 526.
  • the world configuration 522 may identify the virtual world and define the assets and applications used by the virtual world.
  • the world configuration 522 may specify one or more indexes to use in partitioning the assets into processing partitions.
  • One example of the world configuration 522 may use a world schema as follows:
  • the data object(s) 524 may identify the objects and entities in the virtual world and define attributes and properties of the assets.
  • a data object may define a set of object fiel ds and a set of indexable fields that map to its existing fields.
  • the data object(s) 425 may use, for example, an object definition and an object schema definition as follows:
  • some examples of the data object(s) 524 may include:
  • the application(s) 526 may identify and store a copy of the processing applications and application instances that handle the computing and processing in the virtual world.
  • An application may specify the object domain handled during the computing and processing m the virtual world.
  • the application may identify or be linked to an object type to indicate a given object domain.
  • the application may further specify an index that may be used by the world manager module 510 to partition the data object(s) 524 into processing partitions or chunks. For example, the application may specify the maximum number of data objects that can be written to by an application instance in a particular application epoch or processing cycle.
  • the application may also define a subscription strategy for obtaining additional data used during the processing of the data objects.
  • the subscription strategy may identify what additional data object types are used during processing and at what sources the additional object types may be located. Additionally, the subscription strategy may specify a maximum latency that application instances can tolerate for obtaining the additional object types from the specified sources.
  • the application(s) 526 may use, for example, an application definition as follows:
  • Object Type Foo // Name of object type to be updated
  • ARUs ⁇ (String, Int)> // List of types of ARUs (App Resource Units) used and amount
  • Index : Index Type> // Name of index used
  • Max Objects ⁇ Int> // Max number of objects written to supported Subscription Types: ⁇ String> // Object types that will be subscribed to
  • some examples of the application(s) 526 may include:
  • the world manager module 510 may also access the second data store 514 to create and manage one or more hardware host configuration(s) 528 and store one or more hardware host metric(s) 530
  • the world manager module 510 may store a configuration for a computing instance type to be used as a hardware host, the configurations of containers on the hardware host, the capacity of one or more hardware or software resources of the computing instan ce, and a type or types of processing partitions the computing instance can process.
  • the world manager module 510 may obtain and track one or more hardware host metric(s) 530. These hardware host metric(s) 530 may identify and track an ontine/offline status of a hardware host, utilization of hardware or software resources of the hardware host, and other performance metrics associated with a hardware host.
  • the world manager module 510 may access the third data store 516 to create and manage one or more processing partition(s) 532 and one or more processing partition assigmnent(s) 534.
  • the processing partition(s) 532 may include data structures that act as indexes to map the data object(s) 524 into computational units that can be assigned to a hardware host.
  • the processing partition(s) 532 may include metadata that identifies a processing partition and one or more of the data object(s) 524 mapped to the processing partition.
  • object domains may be used to distribute the processing of data objects m a distributed computing system hosting a virtual environment in this example
  • the world manager module 510 may access the first data store 512 to identify the object types for the data object(s) 524.
  • the world manager module 510 may create a list of object types used in the virtual world.
  • the world manager module 510 may then use the list of object types to create the processing parti tion(s) 532 to provide a mapping between the data objectis) 524 and the processing partition(s) 532.
  • a first object type e.g., people
  • a second object type e.g., vehicles
  • the world manager module 510 may also create the processing partition(s)
  • the world manager module 510 may use the world configuration 522 to determine an overall number of processing partitions that may be supported in a virtual environment.
  • the world manager module 510 may use a configuration of the application(s) 526 to determine a maximum or minimum number of data objects that may be allocated to a processing partition.
  • the world manager module 510 may further create the processing partition(s) 532 using the hardware host configuration(s) 528 and the hardware host metric(s) 530
  • the world manager module 510 may use the hardware host configuration(s) 528 to determine a number of hardware hosts available, the type of the hardware hosts, the hardware and software resources allocated to the hardware hosts, and the like, in order to determine the overall number of processing partitions.
  • the world manager module 510 may use the hardware host configuration(s) 528 to determine the capacity of resources defined in the hardware host configuration(s) 528 to determine to a total number of processing partitions in the processing partition(s) 532 or a total number of the data object(s) 524 mapped into one of the processing partition(s) 532.
  • the world manager module 510 may use the hardware host metric(s) 530 to determine whether utilization of hardware or software resources affect the number of processing parti tions in the processing partition(s) 532 or a supported number of the data object(s) 524 mapped into one of the processing partition(s) 532.
  • the world manager module 510 may determine one or more processing partition assignment(s) 534.
  • the processing partition assignments ) 534 may include data structures that act as indexes to assign the processing partition(s) 532 to the hardware host(s) 504.
  • the processing partition assignments) 534 may include metadata that identifies one or more processing partitions and at least one of the hardware liost(s) 504 assigned to the processing partitions.
  • the w ? orld manager module 510 may use the w ? orld configuration 522 and the application(s) 526 to determine processing partition assignments) 534.
  • the world manager module 510 may use the assigned number of hardware hosts defined in the world configuration 522 to determine the processing partition assignments) 534. In another example, the world manager module 510 may use a number of objects supported or a subscription strategy defined for the application(s) 526 to determine the processing partition assignment(s) 534.
  • the world manager module 510 may further use the hardware host configuration(s) 528 and the hardware host metric(s) 530 to determine the processing partition assignments) 534. For example, the world manager module 510 may use the capacity of resources defined in the hardware host configuration(s) 528 to determine the processing partition assignments) 534. In another example, the world manager module 510 may use the hardware host metric(s) 530 to determine whether utilization of hardware or software resources affect the processing partition assignments) 534. [0108] After determining the processing partition(s) 532 using the data object(s) 524, the world manager module 510 may send the processing partition(s) 532 to the hardware host(s) 504 to prepare the hardware host(s) 504 to process the data object(s) 524.
  • the world manager module 510 may send the processing partition assignment(s) 534 to the hardware host(s) 504.
  • a first hardware host in the hardware host(s) 504 may determine which of the processing partition(s) 532 are assigned to the first hardware host.
  • the first hardware host may process one or more of the data object(s) 524 that are mapped to those processing partition(s) 532 assigned to the first hardware host.
  • the hardware liost(s) 504 may include a hardware host manager module 540, an application manager module 542, one or more application container(s) 544a-n, a distributed shared memory (DSM) module 546, a data store 548, one or more processor(s) 550, and one or more memory' module(s) 552.
  • the data store 548 may include one or more local data object(s) 560, an application library 562, one or more local hardware host metric(s) 564, one or more local processing partition(s) 566, and one or more local processing partition assignment(s) 568.
  • the hardware host manager module 540 may be configured to manage operation of a hardware host 504 to process the data object(s) 524.
  • the hardware host manager module 540 may store one or more the data object(s) 524 in the file system data store 548 as the local data object(s) 560.
  • the hardware host manager module 540 may also receive one or more of the appiication(s) 526 for storage in the data store 548 using an application library 562,
  • the hardware host manager module 540 may configure the application library 562 with all or a portion of the application(s) 526. Accordingly, the application library 562 may be used to provide on-demand access to one or more of the application(s) 526 on the hardware host 504 without additional configuration.
  • the hardware host manager module 540 may also monitor utilization of hardware and software resources and store one or more local hardware host metric(s) 564 in the data store 548.
  • the hardware host manager module 540 may monitor the application manager module 542, the application container(s) 544a-n, and the distributed shared memory module 546 or operation of the hardware host 504 overall.
  • the hardware host manager module 540 may report the local hardware host etric(s) 564 back to the world manager module 510 for storage as the hardware host metric(s) 530.
  • the hardware host manager module 540 may manage storage of the local processing partition(s) 566.
  • the hardware host manager module 540 may receive the processing partition(s) 532 from the world manager module 510, and the hardware host manager module 540 may store a copy of the processing partition(s) 532 m the data store 548 using the local processing partition(s) 566.
  • the hardware host manager module 540 may report changes to the local processing partition(s) 566 back to the world manager module 510, winch the world manager module 510 may use to update the processing partition(s) 532 and the processing partition assignment(s) 534.
  • the hardware host manager module 540 may manage storage of the local processing partition assignments) 568.
  • the hardware host manager module 540 may receive one or more of the processing partition assignment(s) 534 from the world manager module 510.
  • the hardware host manager module 540 may store a copy of the processing partition assignments) 534 in the data store 548 using the local processing partition assignments) 568, and the hardware host manager module 540 may use the copy to determine assigned processing partitions. Further, the hardware host manager module 540 may also maintain the copy of the processing partition assignment(s) 534 for other hardware hosts 504 to facilitate communication and data routing with the other hardware host 504.
  • the hardware host manager module 540 may analyze the local processing partition assignment(s) 568 to determine assigned processing partitions in the local processing partition(s) 566.
  • the hardware host manager module 540 may request one or more of the processing partiiion(s) 532 from the world manager module 510 or the hardware host manager module 540 may receive the processing partition(s) 532 from the world manager module 510 together with the processing partition assignment(s) 534.
  • the application manager module 542 may be configured to manage the lifecycl e of the applications on the hardware host 504 using the application library 562. For example, the application manager module 542 may read the local processing partition) s) 566 to determine which of the application) s) 526 are used during processing of the local data object(s) 560 mapped to the local processing partition(s) 566. The application manager module 542 may retrieve one or more of the applieation(s) 526 from the application library 562 and instantiate the applications to process the local data object) s) 560 according to object type. The application manager module 542 may load application instances into the app containers 544a-n and track processing of the local data objects 560 by the application instances. [0116] The distributed shared memory module 546 may manage the data object(s)
  • the distributed shared memory module 546 may read the local processing partition(s) 566 to identify the local data object(s) 560. For example, metadata referencing the local data object(s) 560 may be in the local processing partrtion(s) 566 and may be used to obtain and locally store the data object(s) 524 at the hardware host 504. The distributed shared memory module 546 may then load the local data object(s) 560 into the shared memory' as shared memory data object(s) 570 for use when needed by the application instances executing in the app containers 544a-n.
  • the distributed shared memory module 546 may also process requests to access the shared memory data object(s) 570 from the application instances executing in the app containers 544a-n
  • the requests to access the shared memory data object(s) 570 may include create, read, update, and delete operations by the application instances executing in the app containers 544a-n.
  • the distributed shared memory module 546 may record state changes to the shared memory data object(s) 570 using a thread-safe and low replication cost storage process and a serialization formal understood by the distributed shared memory module 546, as will be discussed in a later section.
  • the distributed shared memory module 546 may further replicate state changes made to the shared memory data objectis) 570 to other application instances and across other hardware hosts in the hardware host(s) 504
  • the distributed shared memory module 546 may determine a list of subscribers, which identify application instances on the hardware host or on other hardware hosts registered to receive streams of data object events associated with the shared memory data object(s) 570.
  • the distributed shared memory' module 546 may send state changes to the shared memory data object(s) 570 to the list of subscribers.
  • the distributed shared memory module 546 may also subscribe to streams of data object events associated with the distributed shared memory modules on other hardware hosts.
  • the distributed shared memory module 546 may receive state changes from the other hardware hosts to reflect in the shared memory data object(s) 570.
  • the distributed shared memory module 546 may determine changes to the local processing partitron(s) 566 as the shared memory data object(s) 570 are processed by the application instances executing in the app containers 544a-n.
  • the distributed shared memory module 546 may change a mapping between the local data object(s) 560 and the local processing partition(s) 566, for example, when a data object is created (i.e , resulting in the addition of a mapping between the data object and one of the local processing partition(s) 566) and when a data object is deleted or removed from the virtual environment (i.e., resulting in the removal of a mapping between the data object and one of the local processing partition(s) 566).
  • the distributed shared memory module 546 may change a mapping between the local data object(s) 560 and the local processing partition(s) 566, for example, as the shared memory data object(s) 570 are processed by the application instances executing in the app containers 544a ⁇ n resulting in changes to object type, spatial location, states and other object attributes used to group the data object(s) 524 into the processing partition(s) 532.
  • the distributed shared memory module 546 may use a change to a mapping between the local data object(s) 560 and the local processing partition(s) 566 to update the local processing partition(s) 566.
  • the distributed shared memory module 546 may send any updates to the local processing partition(s) 566 to the world manager module 510.
  • the world manager module 510 may receive updates from the hardware host(s) 504 reflecting a change to a mapping between one or more of the data object(s) 524 and the processing partition(s) 532. For example, one of the hardware host(s) 504 may create data objects and add the data objects to one of the local processing parti tion(s) 566. The world manager module 510 may obtain the local processing partition(s) 566 from the hardware host(s) for comparison with the processing partition(s) 532 to detect the change. In another example, one of the hardware host(s) 504 may perform operations to update or delete data objects, which cause a change in the mapping between the data object(s) 524 and the processing partition(s) 532.
  • one of the hardware host(s) 504 may split or merge one or more of the local processing partition(s) 566.
  • a processing application may specify a capacity of the processing application to process a number of data objects of a given object type.
  • a threshold may be established for the processing application (e.g., 60%), which once neared, causes the data objects handled by a first instance of the processing application to be split and redistri ubbed among the first instance and a second instance of the processing application.
  • Processing partitions may be merged when there is underutilization of hardware or software resources.
  • the world manager module 510 may use the updates from the hardware liost(s) to update the processing partition(s) 532.
  • the world manager module 510 may also determine whether to update the processing partition assignment(s) 534.
  • the world manager module 510 may update the processing partition assignment(s) 534 to load balance processing of the data object(s) 524 among the hardware host(s) 504.
  • the world manager module 510 may also update the processing partition assignment(s) 534 to ensure that frequently used data may be aggregated or collocated to hardware host.
  • the world manager module 510 may then send the updates to the processing partition assignment(s) 534 to the hardware host(s) 504. Accordingly, the world manager module 510 may control and optimize performance of the hardware host(s) 504 during processing of the data object(s) 524.
  • the world manager module 510 may also monitor metrics associated with the hardware host(s) 504 to manage performance of the hardware host(s) 504.
  • the world manager module 510 may receive data associated with the metrics from the hardware host(s) 504 and use the data to determine an initial configuration for the processing partition assignment(s) 534 or to determine whether to update the processing partition assignments) 534.
  • the world manager module 510 may use the data associated with the metrics to determine whether to migrate a processing partition from a first of the hardware host(s) 504 to a second of the hardware host(s) 504. For example, utilization of a hardware or software resource of the first hardware host may exceed a threshold and affect performance of the first hardware host during processing of the data object(s) 524.
  • the world manager module 510 may determine to change assignments for some of the processing partition(s) 534 to balance processing load, to control hardware or software utilization, to reduce network bandwidth and latency, and the like.
  • the world manager module 510 may send the updated processing partition assignment(s) 534 to the hardware host(s) 504. Again, the world manager module 510 may manage performance of the hardware host(s) 504 during processing of the data object(s) 524 by modifying the all ocations of the processing partition(s) 532.
  • the various processes and/or other functionality contained within the service provider environment 500 may be executed on the processors) 518 and 550 that are in communication with the memory module(s) 520 and 552, respectively.
  • the service provider environment 500 may include a number of computing devices that are arranged, for example, m one or more server banks or computer banks or other arrangements.
  • the computing devices may support a computing environment using hypervisors, virtual machine monitors (VMMs), and other virtualization software.
  • the term‘data store’ may refer to any device or combination of devices capable of storing, accessing, organizing and/or retrieving data, which may include any combination and number of data servers, relational databases, object-oriented databases, cluster storage systems, data storage devices, data warehouses, flat files, and data storage configuration in any centralized, distributed, or clustered environment.
  • the storage system components of the data stores 512, 514, 516, and 548 may include storage systems such as a SAN (Storage Area Network), cloud storage network, volatile or non-volatile RAM, optical media, or hard-drive type media.
  • the data stores 512, 514, 516, and 548 may be representative of a plurality of data stores as can be appreciated.
  • FIG. 5 illustrates that certain processing modules may be discussed in connection with this technolog ⁇ ' and these processing modules may be implemented as computing services.
  • a module may be considered a service with one or more processes executing on a server or other computer hardware.
  • Such services may be centrally hosted functionality or a service application that may receive requests and provide output to other services or consumer devices.
  • modules providing services may be considered on-demand computing that are hosted in a server, virtualized service environment, grid or cluster computing system.
  • An API may be provided for each module to enable a second module to send requests to and receive output from the first module.
  • Such APIs may also allow third parties to interface with the module and make requests and receive output from the modules.
  • FIG. 5 illustrates an example of a system that may implement the techniques above, many other similar or different environments are possible. The example environments discussed and illustrated above are merely representative and not limiting.
  • FIGS. 6A-6B are flow diagrams that illustrate an example method for allocating processing partitions using object domains according to one example of the present technology.
  • the method 600 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine, such as a computer system or information processing device, by hardware components of an electronic device or application-specific integrated circuits, or by combinations of software and hardware elements.
  • software e.g., instructions or code modules
  • CPU or processor central processing unit
  • a logic machine such as a computer system or information processing device
  • hardware components of an electronic device or application-specific integrated circuits or by combinations of software and hardware elements.
  • a world manager may receive a plurality of data objects associated with an object-based model representing a multi-dimensional virtual environment hosted by a distributed computing system.
  • the world manager may determine a plurality of object types associated with the plurality of data objects.
  • An example of object types may be that set for object_l is type“vehicle” and object_2 may be of type“building.” Thus, these two groups of objects may be separated by their types.
  • the world manager may generate a mapping between the plurality of data objects and a plurality of processing partitions using the plurality of object types. As in the example, the objects of type“vehicle” and the objects of type “building” may be mapped to processing partition 1 and processing partition 2, respectively.
  • the world manager may generate data structures that index the plurality of data objects into the plurality of processing partitions using the plurality of object types.
  • the data structures may include metadata about a processing partition and the data objects grouped by an object type, which are mapped to the processing partition.
  • the world manager may identify a plurality of hardware hosts in the distributed computing system to process the object-based model using the plurality of data objects.
  • the world manager may identify hardware hosts from a pool of hardware hosts awaiting assignments, active hardware hosts that have spare computing capacity, and the like.
  • the active hardware hosts may be executing application instances in a container or in a computing instance that may operate on the processing partitions. This means that processing partitions may be allocated for processing to a hardware host that has an application instance for an application that can process the type of data objects mapped to the processing partitions.
  • a first hardware host may have an application instance which can process the“vehicle” type and a second application instance which can process the“building” when a physics event (e.g., a collision or explosion occurs).
  • the world manager may generate a plurality of processing partition assignments between the plurality of processing partitions and the plurality of hardware hosts.
  • the world manager may generate data structures that allocate the plurality of processing parti tion assignments to the plurality of hardware hosts.
  • the data structures may include metadata about a hardware host and the processing partitions allocated for processing to the hardware host.
  • the world manager may send the plurality of processing partition assignments to the plurality of hardware hosts to organize the plurality of hardware hosts to process the plurality of data objects using the plurality of object domains.
  • the data objects may be processed based on the type of the object.
  • the world manager may monitor the plurality of hardware hosts for processing of the plurality of data objects mapped to the plurality of processing partitions.
  • the world manager may receive data associated with utilization of hardware and software resources, changes to the data objects, and changes to the processing partitions.
  • the world manager may receive a change to the processing of the plurality of data objects mapped to the plurality of processing partitions.
  • the change may include an update to a processing partition, which may include mapping a newly created data object to the processing partition, modifying a data object mapped to the processing partition, or removing the mapping between a data ob j ect and the processing partition.
  • the number of data objects mapped to a processing partition may increase for other reasons (e.g., players entering an area) and load balancing may need to occur by revising the assignments.
  • the object type may change from “building” to“exploded building” which may affect the processing partition assignment of the data object.
  • Tire change to the processing of the plurality of processing partitions may also include changes due to overutilization of hardware and software resources on a hardware host.
  • the world manager may determine, using the change, whether to update the plurality of processing partition assignments.
  • the world manager may de termine to update the plurality of processing partitions to manage or optimize performance of the hardware hosts.
  • the world manager may load balance processing of data objects across the plurality of hardware hosts, for example, in response to underutilization or overutilization of a hardware or software resource.
  • the world manager may also redistribute processing partitions due to changes to the data objects or to the processing partitions. If the world manager determines, using the change, not to update the plurality of processing partition assignments in step 620, the method 600 continues in operation 614 where the world manager returns to monitoring the plurality of hardware hosts.
  • the method 600 continues in operation 622 where the world manager updates, using the change, the plurality of processing partition assignments.
  • the world manager may update the plurality of processing partition assignments, for example, to migrate a processing partition assignment from a first hardware host to a second hardware host due to exceeding a threshold for CPU usage associated with the first hardware host.
  • the world manager may migrate a processing partition from a first hardware host to a second hardware host to collocate data more frequently used by the second hardware host.
  • the world manager may also create new processing partition assignments for hardware hosts transferred from a warming pool to an active pool.
  • the method 600 continues using reference“B” from FIG. 6B and returns back to FIG. 6 A where in operation 614 the world manager may send the plurality of processing partition assignments (including any updates) to the plurality of hardware hosts.
  • FIG. 7 is a flow diagram that illustrates an example method 700 for processing data objects grouped by object domains into processing partitions according to one example of the present technology.
  • the method 700 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine, such as a computer system or information processing device, by hardware components of an electronic device or application-specific integrated circuits, or by combinations of software and hardware elements.
  • software e.g., instructions or code modules
  • CPU or processor central processing unit
  • a logic machine such as a computer system or information processing device
  • hardware components of an electronic device or application-specific integrated circuits or by combinations of software and hardware elements.
  • a hardware host may receive a plurality of processing partition assignments, which are assignments between a plurality of processing partitions and a plurality of hardware hosts.
  • the hardware host may receive the plurality of processing partition assignments from a w orld manager.
  • the hardware host may identify, using the processing partition assignments, a processing partition assigned to the hardware host. In one configuration, a single processing partition may be assigned to the hardware host. In another configuration, many processing partitions may be assigned to the hardware host (e.g., dependent, independent, related, or unrelated processing partitions).
  • the hardware host using a distributed shared memory, may load a plurality of data objects mapped by a first object type into the processing partition in a memory device associated with the hardware host.
  • the hardware host may determine a data dependency used by a processing application associated with the processing partition.
  • the data dependency may indicate additional object types used by the processing application upon which the data objects mapped to the processing partition are dependent.
  • the data dependency may indicate that the processing application may subscribe to receive streams of data object events associated with the additional object types.
  • the data dependency may further indicate a subscription strategy used to filter the sources from where the additional object types are obtained.
  • the application instances on the hardware host may communicate, with the distributed shared memory, using the permission model for object types of data objects that are currently being used in processing. The read-only permissions in the permission model may be used to determine data dependencies.
  • the hardware host may subscribe to additional data objects of a second object type using the data dependency.
  • a subscription may be used to identify and filter sources of streams of data object events associated with the additional data objects of the second object type.
  • the distributed shared memories of multiple hardware hosts may be sources to use for streams of data object events associated with the additional data objects of the second object type.
  • Multiple distributed shared memories may communicate with each other to exchange the data objects or exchange updates for the data objects as part of a subscription strategy identified for data dependencies.
  • the hardware host may receive a list of subscribers having data dependencies on the plurality of data objects grouped by the first object type into the processing partition.
  • a list of subscribers may be used to identify and filter destinations for streams of data object events associated with the plurality of data objects of the first object type.
  • Processing applications on the same hardware host and the distributed shared memories of multiple hardware hosts may be destinations for the streams of data object events associated with the plurality of data objects of the first object type.
  • the application instances on the hardware host may communicate with the distributed shared memory for data objects that are currently being used in processing. Additionally, multiple distributed shared memories may communicate with each other to exchange the data objects or exchange updates for the data objects among the list of subscribers.
  • the hardware host may process, using the processing application, the plurality of data objects grouped by the first object type to the processing application using the addition data objects.
  • the application instances may read data associated with the data objects from the distributed shared memory.
  • the application instances may perform operations or calculations on the data objects, some of which may result in changes to the data objects.
  • the hardware host may manage, using the distributed shared memory, storage of the plurality of data objects in the memory devi ce using a plurality of requests from the processing applicati on.
  • the plurality of requests may include requests to modify data associated with the plurality of data objects mapped to the processing partition.
  • the application instances may submit requests to modify data associated with the data objects to the distributed shared memory.
  • the distributed shared memory' may process the requests to record mutations to the data objects caused or implemented by the requests.
  • the hardware host may send, using the distributed shared memory , a plurality of mutations to the plurality of data objects, which were caused by the plurality of requests, to the list of subscribers.
  • the application instances on the hardware host may communicate with the distributed shared memory for data objects that are currently being used in processing. Additionally, multiple distributed shared memories may communicate with each other to exchange the data objects or exchange updates for the data objects among the list of subscribers.
  • FIG. 8 illustrates a system 800 and related operations for storing and replicating state of data objects in a distributed computing system according to an example of the present technology.
  • the system 800 may include a distributed shared memory 802, Some examples of the distributed shared memory 7 802 may include the distributed shared memory module 274 described with respect to FIG. 2 and the distributed shared memory module 546 described with respect to FIG. 5.
  • the distributed shared memory 802 may manage storage of the data object(s) 404 for the system 800.
  • the distributed shared memory 802 may manage storage of the data objects(s) 404 using a storage process to create or load the data object(s) 404 into one or more memory devices using in-memory representations of the data object(s) 404.
  • Tire distributed shared memory 802 may also manage storage of the data object(s) 404 using the storage process to handle requests to read and write to the in -memory representations of the data object(s) 404.
  • the distributed shared memory 802 may use a storage process that includes a serialization format for the data object(s) 404.
  • the distributed shared memory 802 may use the storage process to represent the data object(s) as hierarchical data in-memory and on disk using a plurality of sections.
  • the distributed shared memory 802 may further utilize log records in one of the plurality of sections to represent modifications to the data object(s) 404 to minimize the time cost to reflect state changes in highly distributed, real-time, computations.
  • the distributed shared memory 802 may use the storage process and implement the serialization format for the data object(s) 404 to minimize the time cost to reflect state changes for epoch based simulations, where each epoch provides an execution interval.
  • the epoch-based simulation may use a scenario with 1 Writer /' N Reader processes for the data object(s) 404, and with each process potentially containing multiple threads, where the processes may have epochs or execution intervals lasting ⁇ 30ms.
  • Typical RPC (remote procedure call) frameworks, such as grpc/protobuf. start to fall behind as the epoch frequency increases and the size of messages used to reflect state changes increases. This may be due to CPU time spent de-serializing data objects and the cost of synchronizing the entire data object when an update to the data object is made.
  • the distributed shared memory 802 can meet performance requirements using the storage process according to the present technology because the storage process provides a platform independent and language independent data format specification to use with the data object(s) 404
  • the distributed shared memory 802 may provide a small, predictable amount of overhead because code written to use the storage process with the serialization format should perform as close to a native storage process as possible for CRIJD (create, read, update, delete) operations.
  • the distributed shared memory 802 may include an overhead for using storage process with the serialization format that may maintain an amortized 0(1) cost.
  • the distributed shared memory' 802 also incurs no serialization costs using the storage process by keeping the memory layout the same as the storage/wire format. Additionally, the distributed shared memory 802 may use the storage process to enable updates to the data object(s) 404 in the form of log records.
  • the distributed shared memory' 802 may group one or more modifications to the data object(s) 404 together within an epoch or execution interval as a log record.
  • the distributed shared memory 802 may provide mechanisms for processing applications and other distributed shared memories to retrieve and consume log records for the data object(s) 404. Therefore, the distributed shared memory 802 may be highly concurrent because there is a separation between how the storage process implements the storage of the data object(s) and the storage of metadata used to track changes to the state of the data object(s) 404. Finally, the distributed shared memory 802 is highly efficient because the storage process may use a binary encoding to represent data for the data object(s), thus, reading a data fi eld from the encoding simply uses the binary encoding schema.
  • the distributed shared memory 802 may retrieve the data object(s) 404 from one or more sources (not shown), such a file, an object data store, another distributed shared memory, and the like.
  • the data object(s) 404 may be associated with an object-based model of a virtual environment (e.g., a 3D simulation, 3D game, etc.).
  • the distributed shared memory 802 may manage storage of the data object(s) 404 in memory devices using the storage process according to the present technology during object creation and loading. For example, the distributed shared memory 802 may generate one or more object representation(s) 804 for the data object(s) 404 in a memory' device.
  • the distributed shared memory 802 may copy the data object(s) 404 directly to the memory device when keeping the memory layout the same as the storage/wire format. Alternatively, the distributed shared memory 802 may parse the data object(s) 404 to write the data object(s) to the memory device using the storage process.
  • the object represen tation(s) 804 may include one or more sections m the memory' device. Some examples of a section may include a metadata section, a virtual table section, an object data section, a log section, and the like. At the time of creation, for example, the object representation(s) 804 may include an object metadata section 806, a virtual table 808, and an object data section 810. At other times during the lifecycle of the data object(s) 404, the distributed shared memory 802 may use the storage process to add one or more sections and portions of sections to the object representation(s) 804.
  • the distributed shared memory 802 may use the object metadata section 806 to describe the in-memory /on disk/over the wire content of the data object(s) 404, as well as how- to read the object representation(s) 804.
  • the object metadata section 806 may include one or more metadata fields.
  • the object metadata section 806 may include a length field, which indicates the length of the object metadata section 806 (e.g., m bytes) and an object length field, which indicates a total combined length of the object metadata section 806, the virtual table 808, and the object data section 810.
  • the object metadata section 806 may also include an object identifier (ID) 812 field, which provides a unique identifier to identify a data object, and a base version field, which indicates the version of the data object represented by the object data section 810.
  • ID object identifier
  • the object metadata section 806 may include a virtual table offset field, which provides the memory offset (e.g., from the start of the object metadata section 806) to the start of the virtual table 808, and an object data offset field, which provides the memory offset (from the start of the object metadata section 806) to the start of the object data section 810
  • the distributed shared memory 802 may use the virtual table 808 to describe one or more portions of the object data section 810.
  • the virtual table 808 may include one or more table fields and table entries.
  • the virtual table 808 may include an array of offsets from the start of the object data section 810 that point to the portions of the object data section 810.
  • a desired memory location in the object data section 810 may be fo und by using the offset found in the corresponding table entry of the virtual table 808.
  • the Nth data field of a data object may be found by using the offset found in the Nth index of the virtual table 808.
  • the virtual table 808 may include a length field, which indicates the length of the virtual table 808 (e.g., in bytes), and one or more table entries or field offsets, which provide an array representing the table entries or field offsets for each of the data entries or data fields in the object data section 810.
  • the distributed shared memory 802 may use the object data section 810 to include the data entries, data fields, data values, etc. of the data object(s) 404.
  • the object data section 810 may include a length field, which provides the length of the object data section 810 (e.g., in bytes) and a byte array representing data values in a data object.
  • the distributed shared memory 802 may manage the storage of state changes to the data object(s) 404.
  • the distributed shared memory' 802 may provide mechanisms for processing applications and other distributed shared memories to retrieve and process the data object(s) 404.
  • the distributed shared memory 802 may receive one or more request(s) 820. Hie request(s) 820 may be issued by one or more processing applications.
  • the request(s) 820 may instruct the distributed shared memory 802 to create or delete the object representation(s) 804 in the memory device, read the object representation(s) 804 from the memory device, and write changes to the data object(s) 404 represented by the object representation(s)
  • the distributed shared memory 802 may analyze the requests) 820 to identify a read request associated with the data object(s) 404 to be handled using the object representation(s) 804 in the memory device.
  • the distributed shared memory 802 may read the object metadata section 806 of the object represen tation(s) 804 to determine whether the request(s) 820 match, for example, a particular object ID 812, object version 814 and object type 816.
  • the distributed shared memory 802 may obtain the memory location of the virtual table 808 from the object metadata section 806.
  • the distributed shared memory 802 may read the virtual table 808 to determine the field offsets in the object data section 810 to obtain the in-memory contents of the data object(s) 404.
  • the distributed shared memory 802 may the read the object data section 810 using the field offsets to obtain the in-memory 7 contents of the data objeet(s) 404 from the object representation(s) 804.
  • the distributed shared memory 802 may include the in-memory contents of the data object(s) 404 obtained from the object representation(s) 804 and may send responses to one or more requestors associated with request(s) 820.
  • the distributed shared memory 802 may analyze the request(s) 820 to identify a write request associated with the data object(s) 404 and data to be written to the data object(s) 404 using the object representation(s) 804 in the memory device.
  • the distributed shared memory 7 802 may again read the object metadata section 806 of the object representation(s) 804 to determine whether the request(s) 820 match, for example, a particular object ID and object version.
  • the distributed shared memory 802 may append a version-ordered set of mutations to the object representation(s) 804 in the memory device.
  • the distributed shared memory 802 may use a senes of log records in a log section to reflect state changes to the object representation(s) 804 in the memory 7 device. As illustrated in FIG. 8, the object representation(s) 804 may further include a log section 830. The distributed shared memory 802 may use the log section 830 to represent state changes to the data object(s) 404 represented in the object representations 804. The di stributed shared memory 802 may access a current state of the data object(s) 404 by reading the log section 830 from the tail, and reading backwards to get the most up-to- date information for the data object(s) 404 contained in the object representation(s) 804 m the memory device.
  • the distributed shared memory 802 may manage the current state of the data object(s) 404 by appending one or more log record(s) 832 (e.g., log records 832a- n) to the object representation(s) 804 to reflect the version-ordered set of mutations or writes.
  • the distributed shared memory 802 may use the log record(s) 832 to describe a set of mutations to the data object(s) 404 and to which version of the data object(s) the mutations apply.
  • the distributed shared memor' 802 may append the log record(s) 832 to the object representation(s) 804 in reverse order to facilitate the tail-to-head reading mechanism, as well as single-pass writes and thread-safe reads.
  • FIG. 9 illustrates a graphical example of an object representation 900 used by a distributed shared memory for managing storage of data objects in a distributed computing system that provides low replication cost and thread-safe reads according to an example of the present technology.
  • the distributed shared memory' may implement the object representation 900 using a set of semantics built on top of a byte array.
  • the object representation 900 may provide low replication costs as the byte array can be copied‘'as is” to another memory location, to a file on disk, across a network to another hardware host, and the like, because the serialization format of the object representation 900 reduces the extra processing to format the data for different storage or transport media.
  • the distributed shared memory ' may segment the object representation 900 into one or more sections in-memory', over the wire, and on disk as a file.
  • Tire object representation 900 may include a metadata section 902, a virtual table section 904, an object data section 906, and a log section 908.
  • the object representation 900 may provide thread-safe reads because the organization of the sections in the object representation 900 prevents reading from a section that can be written to at the same.
  • the object representation 900 may include a resizable byte array (RB Array) which provides an abstraction around a resizable array of bytes that allows for atomic“reservations” of bytes.
  • RB Array resizable byte array
  • the object representation 900 may enable a request to resize the array to allocate additional bytes to be completed as an atomic process prior to servicing another pending request to resize the array. Accordingly, the object representation 900 enables atomic reservations to allow a queue of appending writes to the object representation 900 to be thread safe.
  • the distributed shared memory may use the metadata section 902 of the object representation 900 to describe the byte array, as well as to provide information on how to read the byte array.
  • the metadata section 902 may include one or more metadata fields, such as a metadata length 910, an object length 912, an object identifier 914, a base version 916, a virtual table offset 918, and an object data offset 920.
  • the metadata length 910 may specify the length of the metadata section 902 in bytes.
  • the object length 912 may specify a total length of the byte array without the log section 908.
  • the object identifier 914 may specify a unique identifier to identify this object. Some examples of the object identifier 914 may include a UIJID or GUID.
  • the base version 916 may specify' a version of the data object represented by the object data section 906.
  • the virtual table offset 918 may specify' an offset (from the start of the byte array) to the start of the virtual table section 904.
  • the object data offset 920 may specify an (from the start of the byte array) to the start of the object data section 906.
  • the distributed shared memor ' may use the virtual table section 904 of the object representation 900 to describe offsets into specific portions of the object data section 906.
  • the virtual table section 904 may include a virtual table length 930, which specifies a length of a virtual table 932 in bytes.
  • the virtual table 932 may include an array of offsets that begin from the start of the object data section 906.
  • the Nth field of a data object represented by the byte array can be found by using the offset found in the Nth index of the array in the virtual table 932.
  • the distributed shared memory' may use the object data section 906 of the object representation 900 to store a data object.
  • the object data section 906 may include an object data length 940, which specifies a length of the object data 942 in bytes. Data associated with a data object may be retrieved from a file, object store, etc., and copied into the object data 942.
  • the object data 942 may include a byte array representing the data object.
  • the object data 942 may include: integers, floats, characters, strings, blobs, and the like, as well as combinations thereof.
  • the object identifier 914 identifies the data object represented in the object data section 906.
  • the base version 916 specifies a version of the data object represented in the object data section 906.
  • the distributed shared memory' may use the log section 908 of the object representation 900 to represent state changes to the data objects.
  • Tire distributed shared memory' may use the log section 908 to write a version-ordered set of mutations to the data objects whose contents are represented in the object data section 906.
  • the log section 908 may include a series of one or more log records 950 (e.g., log records 950a- d).
  • the distributed shared memory may read the log section 908 starting from the tail and reading backwards to get the current state of the data objects.
  • the distributed shared memory may use the log records 950 to describe a set of mutations to the data objects.
  • the distributed shared memory may write the log records 950 to the log section 908 by appending the log records 950 m reverse order to facilitate the tail-to-head reading mechanism, single-pass writes and thread safe-reads.
  • the log record 95Qd may include one or more log fields, such as a length 960, which specifies a length of the log record 950d m bytes.
  • the log record 950d may include a version field 962.
  • the distributed shared memory may use the version field 962 to specify a version of the data object to which the log record 950d applies.
  • the distributed shared memory also may use the version field 962 to specify a version of a portion of the data records or a particular data field to which the log record 950d applies.
  • the log record 950d may also include an object identifier 964, which provides the object identifier of the data object to which the log record 950d applies, and a modified fields table 966.
  • the distributed shared memory may use the modified fields table 966 to indicate where a change has occurred in the state of the data object.
  • the modified fields table 966 may include a reference indicating that a portion of the log record 95 Od includes a change to the data associated with the data object, which may be different from data currently associated with the data object and stored in a portion of the object data section 906.
  • the distributed shared memory may use the modified fields table 966 to include references indicating which object fields of the data object have been changed.
  • the modified fields table 966 may include a length 972 (in bytes) to describe the size of the modified fields table 966, as well as bitfield content 974 to represent actual fields modified using an index.
  • the distributed shared memory may set one or more bits of the bitfield content 974 to indicate that object fields have changed.
  • Each bit in the byte may correspond to an ordered set of object fields. Setting the bit of a corresponding object field in the bitfield content 974 may provide the reference indicating a change to the object field.
  • the distributed shared memory' may use a translation table 968 to indicate where a change has occurred in the state of the data object.
  • the log record 950d may further include the translation table 968, which provides an inner virtual table that contains a length 976 (in bytes) to describe the size, as well as a table content 978.
  • the distributed shared memory may use the table content 978 as a mapping between offsets that point into a modified data section 970 and the changed object fields, which may also be identified in the modified fields table 966.
  • the distributed shared memory may use the modified data section 970 to store mutations to data objects.
  • the modified data section 970 may include a byte array that represents the changes to the object fields in the log record 950d.
  • the distributed shared memor ' may append the log records 950a-d in memory one after the other or contiguously.
  • the distributed shared memory' may include in the log records 950a-d a previous-record pointer or another type of linked list scheme. The distributed shared memory ' may use the previous-record pointer to provide gap detection to detect an incomplete log.
  • the distributed shared memory may aggregate a plurality of requests to modify data associated with the data object during an epoch or execution interval in the distributed computing system.
  • the distributed shared memory' may use the translation table 968 and/or the modified fields table 966 to include references for multiple portions of the data object affected by the plurality of requests.
  • the distributed shared memory may append a single log record for the epoch to the log section 908 of the object representation 900.
  • the single log record may include a plurality' of mutations to the data object implemented by the plurality of requests, which are referenced as changes in the translation table 968 and/or the modified fields table 966 as affecting multiple portions of the data object.
  • the distributed shared memory may aggregate a plurality of requests to modify data associated with the data object during an epoch or execution interval in the distributed computing system when the requests apply to the same portion of the data object.
  • the distributed shared memory may append a single log record for the epoch to the log section 908 of the object representation 900.
  • the single log record may include the latest in series of mutations to the data object implemented by the plurality of requests, which is referenced as a single change in the translation table 968 and/or the modified fields table 966 as affecting the portion of the data object.
  • the distributed shared memory may append a plurality of log records to the log section 908 of the object representation 900 during an epoch or execution interval in the distributed computing system.
  • the distributed shared memory may determine to coalesce the plurality of log records into a single log record (e.g., at the end of an epoch, when a read is made, or at another point in time) in order to save storage space or improve later reading times to determine the current state of the data object.
  • the distributed shared memory may write the single log record to the log section 908 to replace the plurality of log records.
  • the single log record may include a plurality of references indicating that a plurality of mutations to the data object aggregated from the plurality of log records reflect changes to a plurality of portions of the data object.
  • the distributed shared memory may use the object representation 900 to communicate state changes associated with the data object.
  • the distributed shared memory may receive an instruction to send a current state of the data object to a recipient, such as a processing application, another distributed shared memory, an object data store, and the like.
  • the distributed shared memory may determine whether the recipient includes a prior state of the data object. For example, the distributed shared memory may track whether all or part of the object representation 900 had been previously sent to the recipient.
  • the distributed shared memory may send the recipient one or more of the log records 950 (e.g., all of the log records) in the log section 950 when distributed shared memory determines the recipient includes the prior state of the data object.
  • the distributed shared memory may send the recipient one or more delta log records that represent a difference between the current state of the data object and the prior state held at the recipient.
  • the distributed shared memory may determine that the recipient does not include the prior state of the data object and second either the object representation 900 or the entirety of the log section 908.
  • the distributed shared memory' may coalesce the log section 908 into a single log record as discussed above to represent the current state based on storage or network conditions. For example, the distributed shared memory may coalesce the log section 908 into a single log record to reduce storage used by the object representation 900. In another example, the distributed shared memory may coalesce the log section 908 into a single log record to reduce the netw ork bandwidth used for data transfer.
  • FIG. 10 is a flow diagram that illustrates an example method 1000 performed by a distributed shared memory for managing storage of data objects in a distributed computing system that provides low replication cost and thread-safe reads according to an example of the present technology .
  • the method 1000 may be performed by software (e.g , instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine, such as a computer system or information processing device, by hardware components of an electronic device or application-specific integrated circuits, or by combinations of software and hardware elements
  • a distributed shared memory of a hardware host may receive a data object associated with an object-based model of a virtual environment hosted by the distributed computing system.
  • the distributed shared memory of the hardware host may retrieve data associated with the data object from an object data store.
  • the distributed shared memory may receive data associated with the data object from another hardware host, a world manager, a data service, or a client.
  • the distributed shared memory may determine a memory- location for the data object in a memory device associated with the hardware host.
  • the memory- device may include volatile and non-volatile storage devices.
  • the distributed shared memory- may set aside a portion of memory' in the memory ' - device to manage a shared storage space for representations of data objects.
  • the distributed shared memory- may allocate memory at the memory location for the data object and share the memory with a processing application on the hardware host and with other distributed shared memories on other hardware hosts in the distributed computing system.
  • the distributed shared memory- may share a portion of random access memory as the shared storage space.
  • the distributed shared memory may determine a format of the data object.
  • the format of the data object may include an on-disk format and an in- memory- format.
  • the on-disk format may be the same or may be different from the in memory format.
  • the distributed shared memory may use the same format for file storage and memory- storage to reduce the need to perform serialization prior to storage or transfer over a netw ork. If the distributed shared memory determines in operation 1008 that the format of the data object is used by the distributed shared memory, the distributed shared memory may write the data object to the memory location in the memory ' - device in operation 1010.
  • the distributed shared memory- may copy data associated w-ith the data object directly to the memory device, irrespecti ve of whether the data w3 ⁇ 4s obtained from an object data store, a file, in a network packet, and the like.
  • the distributed shared memory' may perform a memcopy to allocate the data object to the shared memory space.
  • the distributed shared memory may identify metadata associated with the data object, as in operation 1012.
  • the distributed shared memory' may parse data associated with the data object to determine an object identifier that uniquely identifies the data object and an object version for the data associated with the data object.
  • the hardware host may write the metadata to an in-memory representation of the data object at the memory location using a metadata section.
  • the distributed shared memory may identify ' object data associated with the data object.
  • the distributed shared memory ' may parse data associated with the data object to determine numerical data, strings, binary blobs, and the like.
  • the distributed shared memory may apply a number of data transformation rules to the data.
  • the distributed shared memory may determine a schema for the data object, which identifies one or more object fields and the format of data contained in the object fields.
  • the distributed shared memory may apply the data transformation rules to the data contained in the object fields to convert and prepare the data for storage in the memory device.
  • the distributed shared memory may write the object data to the representation of the data object at the memory location using an object data section.
  • the distributed shared memory may use the object data section to store the data contained in the object fields associated with the data object in a byte array.
  • the distributed shared memory may identify 7 one or more object fields associated with the data object.
  • the distributed shared memory may identify one or more portions of data associated with the data object to determine one or more object fields and parse the object data into values for the object fields.
  • the distributed shared memory may detect the schema of the data object and use the schema to identify the object fields and structure of the data object.
  • the distributed shared memory may write a mapping between an object field and a portion of the data object section that contains the value for the object field using a virtual table. The distributed shared memory may use the virtual table to describe an array of offsets from the start of the object data section.
  • Hie distributed shared memory may access the memory location in the object data section of the Nth field of a data object by using the offset found m the Nth index of the virtual table.
  • FIG 11 is a flow diagram that illustrates an example method 1100 performed by a distributed shared memory for managing the storage of state of data objects in a distributed computing system using an ordered, append-only log-based format to provide versioned snapshots of state according to an example of the present technology.
  • the method 1100 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine, such as a computer system or information processing device, by hardware components of an electronic device or application-specific integrated circuits, or by combinations of software and hardware elements
  • a distributed shared memor' of a hardware host may receive a request to modify data associated with a data object.
  • the distributed shared memory may receive the request from a processing application configured to process the data object on the hardware host according to an object type of the data object.
  • the distributed shared memory may receive the request in response to the processing application issuing the request via an API used to access data objects whose storage is managed by the distributed shared memory.
  • the distributed shared memory may determine whether the request to modify data associated with the data object matches an object identifier and version associated with a representation of the data object whose storage is managed by the distributed shared emoiy.
  • the distributed shared memory may analyze the request to determine whether the request specifically identifies the data object.
  • the distributed shared memoiy' may further analyze the request to determine whether the request is directed to a version of the data object whose storage is being managed by the distributed shared memory.
  • the distributed shared memory may generate an error message using the request in operation 1108.
  • the distributed shared memoiy may include in the error message details associated with object identifiers and versions referenced by the request.
  • the error message may also include data associated with the processing application, memory' contents, stacks, heaps, and other debugging information.
  • the distributed shared memory may identify a mutation to the data object that is in the request, as in operation 1110.
  • a mutation may include any change to the data object, such as a change in a field identifier, a change in a field value, or any other write to the data object.
  • the distributed shared memory may analyze the request to identify the mutation.
  • the distributed shared memory may interpret the request according to an API used to modify data objects whose storage is managed by the distributed shared memory. In one example, the request may indicate to modify an object field using a supplied value.
  • the distributed shared memory may determine the object field and the supplied value from the request.
  • the distributed shared memory may write the mutation to a memory' location associated with the representation of the data object using a log record.
  • the distributed shared memory' may w rite the log record to a log section of the representation of the data object whose storage is managed by the distributed shared memor .
  • the distributed shared memory may append the log record to a series of log records in the log section.
  • the distributed shared memory may identify an object field associated with the mutation.
  • the distributed shared memory may analyze the request to identify the object field in the request.
  • the request may indicate one or more object fields to be modified and new' corresponding values for the object fields, and the distributed shared memory may track which object fields are being modified.
  • the distributed shared memory' may write, to the log record, a translation table mapping the object field to a portion of the log record that includes the mutation.
  • the distributed shared memory ' may generate a reference indicating that the mutation reflects a change to the object field.
  • the reference may include: a bit, byte, flag, resource identifier, or other data that identifies the object field.
  • the distributed shared memory may write, to the log record, a modified fields table that includes the reference.
  • the distributed shared memory' may use the modified fields table to indicate which portions of the data object have a change in state.
  • the hardware host may set one or more bits of a byte or group of bytes to indicate that object fields corresponding to the bits have changed.
  • Each bit in the byte may be mapped to or associated with an object field in an ordered set of object fields. Setting the bit of a corresponding object field may provide the reference indicating that the mutation reflects a change to the object field.
  • the distributed shared memory may write the byte or group of bytes, with one or more bits set, to the log record as the modified fields table.
  • the distributed shared memory may perform the method 1100 with multiple requests to modify data associated with the data object during an epoch or execution interval in the distributed computing system to aggregate or coalesce the requests into a single log record.
  • the distributed shared memory may use the modified fields table to include references for multiple portions of the data object affected by the plurality of requests.
  • the distributed shared memory' may append a single log record for the epoch to the log section of the representation of the data object whose storage is managed by the distributed shared memory .
  • the distributed shared memory may aggregate a plurality of requests to modify data associated with the data object during an epoch or execution interval in the distributed computing system when the requests apply to the same portion of the data object.
  • the distributed shared memory' may append a single log record for the epoch to the log section of the representation of the data object whose storage is managed by the distributed shared memory'.
  • the single log record may include the latest in series of mutations to the data object implemented by the plurality of requests, which is referenced as a single change in the modified fields table.
  • FIG. 12 is a flow' diagram that illustrates an example method 1200 performed by a distributed shared memory' for exchanging state of data objects in a distributed computing system between hardware hosts according to an example of the present technology .
  • the method 1200 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine, such as a computer system or information processing device, by hardware components of an electronic device or application-specific integrated circuits, or by combinations of software and hardware elements.
  • software e.g., instructions or code modules
  • CPU or processor central processing unit
  • a logic machine such as a computer system or information processing device
  • hardware components of an electronic device or application-specific integrated circuits or by combinations of software and hardware elements.
  • a first hardware host using a distributed shared memory to store a representation of a data object may receive an instruction to send a current state of the data object to a second hardware host.
  • the first hardware host may receive the instruction from the second hardware host as a data request.
  • the first hardware host may receive the instruction in the form of a list of subscribers.
  • the first hardware host may use the list of subscribers to publish changes made to the data object.
  • the list of subscribers may include processing applications on the first hardware host, the second hardware host, and other hardware hosts and services in the distributed computing system.
  • the first hardware host may publish the changes directly to a subscriber or the first hardware host may notify the subscriber that changes are available. The subscriber then may retrieve or request the changes from the first hardware host.
  • the first hardware host may determine whether the second hardware host includes a prior state of the data object. For example, the first hardware host may use the distributed shared memory to determine whether data associated with the representation of the data object has previously been sent to or obtained by the second hardware host. The first hardware host may be in communication with the second hardware host to determine whether the second hardware host includes a prior state of the data object. The first hardware host may analyze the instruction for an indication of whether the second hardware host includes a prior state of the data object. In another example, the first hardware host may use the distributed shared memory' to track communications sent to the second hardware host indicating whether the second hardware host includes a prior state of the data object.
  • the first hardware host may use the distributed shared memory to send the representation of the data object to the second hardware host in operation 1208.
  • the distributed shared memory of the first hardware host may initiate a memory' copy that sends the in-memory' representation of the data object to a distributed shared memory of the second hardware host.
  • the distributed shared memory of the first hardware host incurs no serialization costs in sending the in- memory' representation of the data object to the distributed shared memory of the second hardware host because the in-memory layout is the same as the storage/wire format according to the present technology.
  • the distributed shared memory of the second hardware host may then load the memory' copy directly into a memory' device.
  • the first hardware host may use the distributed shared memory to identify a portion of a log section associated with the representation of the data object which represents a difference between the current state and the prior state of the data object m operation 1210.
  • the distributed shared memory of the first hardware host may determine that a single log record represents the difference between the current state and the prior state of the data object.
  • the distributed shared memory of the first hardware host may determine that multiple log records represent the difference between the current state and the prior state of the data object.
  • the first hardware host may use the distributed shared memory to send the portion of the log section to the second hardware host.
  • the distributed shared memory' of the first hard ware host may send a single log record to the second hardware host, which was identified as representing the difference between the current state and the prior state of the data object.
  • the distributed shared memory of the first hardware host may send multiple log records to the second hardware host, which was identified as representing the difference between the current state and the prior state of the data object.
  • the distributed shared memory' of the first hardware host may merge one or more log records, which are identified as representing the difference between the current state and the prior state of the data object, into a single log record to be sent to the second hardware host.
  • the first hardware host may use the distributed shared memory to send the single log record to the second hardware host to reduce network bandwidth for data transfers.
  • the distributed shared memory may use the method 1200 to preserve or archive the current state of the data object in an object data store or m a file.
  • the distributed shared memory may determine to coalesce a plurality of log records that represent the current state into a single log record in order to save storage space or improve later reading times to determine the current state of the data object.
  • the distributed shared memory may write the single log record to the log section to replace the plurality of log records, for example, prior to storage on disk as a file.
  • the single log record may include a pl urality of references indicating that a plurality of mutations to the data object aggregated from the plurality of log records reflect changes to a plurality' of portions of the data object which represent the current state of the data object.
  • FIG. 13 illustrates a system 1300 and related operations for assigning computational units in a distributed computing system using processing partitions with data objects organized by spatial locality' according to an example of the present technology.
  • the system 1300 may include a world manager 1302 and one or more hardware liost(s) 1304.
  • the world manager 1302 may assign the computational units to the hardware host(s) 1304 in the distributed computing system using processing partitions organized by spatial locality according to an example of the present technology.
  • the world manager 1302 may include one or more server computers configured to manage processing of data objects in the distributed computing system. Some examples of the world manager 1302 may include the cluster manager 130 described with respect to FIG. 1 and the cluster manager module 210 described with respect to FIG. 2. To manage processing, according to one example of the technology, the world manager 1302 may determine one or more spatial sub-division(s) 1306 associated with an object-based model of a virtual environment hosted by the distributed computing system.
  • the spatial sub-division(s) 1306 may include spatial regions that divide a world space into two or more subsets (e.g., disjoint subsets that are non overlapping). The spatial regions may be arbitrary and/or uniform divisions of the space.
  • the spatial regions may define a spatial unit as square or rectangular, or may define spatial units as: arbitrary points, lines, areas, polytopes (e.g., polygons and polyhedrons), and conic sections such as ellipses, circles, and parabolas.
  • the space may be a 2- dimensional space, a 3-dimensional space, or any multi-dimensional space that may be sub-divided.
  • the spatial sub-division(s) 1306 may be hierarchical, meaning that the space (or a region of the space) may be divided into several spatial regions, and then the spatial regions may be again recursively sub-divided to create additional spatial regions.
  • the spatial sub-division(s) 1306 may also include multiple layers or hierarchies of sub divisions, which may overlay each other and which may include the different layers or hierarchies providing a different metric, criteria, and granularity for sub-dividing the virtual world.
  • the spatial sub-division(s) 1306 may be represented using one or more tree structures, for example, as discussed below' with respect to FIG. 14.
  • a world space associated with the virtual environment may be sub-divided using octants and the spatial sub-division(s) 1306 may be represented using an octree, which is a tree structure in which each internal node has eight children.
  • the octree may be used to sub divide a three-dimensional space associated with the virtual world by recursively subdividing regions into eight octants.
  • the world space associated with the virtual environment may be sub-divided using quadrants and the spatial sub- division(s) 1306 may be represented using a quadtree, which is a tree structure m which each internal node has four children.
  • the quadtree may be used to partition a two- dimensional space associated with the virtual world by recursively subdividing regions into four quadrants.
  • Other methods may be used to sub-divide the world space associated with the virtual environment and to represent the spatial sub-division(s) 1306 using a data structure in order to provide nodes to represent a particular unit of spatial information and nodes to which data (e.g., data objects) may associated with the particular unit of spatial information.
  • Some additional examples may include: grids, Voronoi diagrams, space filling curves, KD-Trees, bounding volume hierarchies, and the like.
  • the world manager 1302 may also determine one or more processing partition(s) 1308 associated with the spatial sub-division(s) 1306. For example, the world manager 1302 may create the processing partition(s) 1308 using the spatial sub division ⁇ ) 1306. In one example, a processing partition in the processing partition(s) 1308 may be associated with a single spatial sub-division in the spatial sub-division(s) 1306. In another example, a single processing partition may be associated with multiple spatial sub-divisions in the spatial sub-division(s) 1306. In a further example, multiple processing partitions may be associated with one or more spatial sub-divisions in the spatial sub-division(s) 1306. In addition, multiple processing partitions may be associated with a single spatial sub-division.
  • the world manager 1302 may determine the processing partition(s) 1308 by mapping the data object(s) 404 to the processing partition(s) 1308 associated with the spatial sub-division(s) 1306.
  • the world manager 1302 may index the data object(s) 404 using spatial location information into the processing partition(s) 1308.
  • the world manager 1302 may instruct the hardware host(s) 1304 to create the processing partition(s) 1308 using the spatial sub-division(s) 1306.
  • the world manager 1302 may define the world space associated with the virtual environment and instruct the hardware host(s) 1304 to create the processing partition(s) 1308 during processing of the object-based model of the virtual environment.
  • the hardware host(s) 1304 may be responsible for indexing the data object(s) 404 into the processing partition(s) 1308 using spatial location information.
  • the hardware host(s) 1304 may index the data object(s) 404 upon creation, deletion, and modification of data objects 404 by instances of processing applications on the hardware host(s) 1304. Such operations by the instances of the processing applications may cause additions, deletions, splits, mergers, and other modifications by the hardware host(s) 1304 to the processing partition(s) 1308.
  • the world manager 1302 may monitor the hardware host(s) 1304 to receive additions, deletions, splits, mergers, and other modifications to the processing partition(s) 1308 associated with the spatial sub-division(s) 1306.
  • the processing partition(s) 1308 may define computational units in the distributed computing system, organized by spatial location information associated with the data object(s) 404, which may be allocated to the hardware host(s) 1304.
  • the data object(s) 404 may he associated with an object-based model of a virtual environment (e.g., a 3D simulation, 3D game, etc.).
  • the data object(s) 404 may include one or more object field(s) 420.
  • the data object(s) 404 may include one or more object field(s) 420 to provide the object identifier 430, the object version 432, and the object type 434 as metadata for the data object(s) 404.
  • the object field(s) 420 may further include one or more object properties or atributes defined using field-value or key-value pairs, such as a location field 1316, which may specify a first spatial attribute, and a location field 1318, which may specify a second spatial atribute.
  • the location fields 1316 and 1318 may include: spatial location information, such as a data value, string, an X, Y, Z coordinate, a vector, a boundary, a mesh, a range, etc., which may be referenced by a processing application in order to process the data object(s) 404.
  • spatial location information included in the data object(s) 404 may include: position, location, and orientation information associated with points, lines, polytopes (e.g., polygons and polyhedrons), conic sections such as ellipses, circles, parabolas, or other types of data object models.
  • polytopes e.g., polygons and polyhedrons
  • conic sections such as ellipses, circles, parabolas, or other types of data object models.
  • the processing partition(s) 1308 may be determined using the object field(s) 420 of the data object(s) 404.
  • the processing parti tion(s) 1308 represent groups of data objects organized into computational units to be processed by processing applications
  • the data object(s) 404 may be organized according to spatial location information in one or more of the object field(s) 420.
  • this data object management by the world manager 1302 and the hardware host(s) 1304 may include the creation of one or more index structures, known as a spatial index, which map the data object( s) 404 to the processing parti tion(s) 1308 using spatial location information from the object field(s) 420.
  • the world manager 402 may use the object fieid(s) 420 to create multiple tiers in the groupings, thereby further organizing the data object(s) 404 by multiple atributes such as object domains as discussed previously.
  • the processing partition(s) 1308 may be represented by one or more spatial index structures that group the data object(s) 404 at or near spatial units defined by the spatial sub-division(s) 1306 using spatial location information associated with the data object(s) 404.
  • a spatial index element or node may represent a processing partition and include data or metadata identifying the processing partition and a collection or group of data objects organized into the processing partition by a spatial unit (e.g., location or locality) associated with a spatial sub-division.
  • the spatial index representing the processing parti tion(s) 1308 may include a tree structure representing the spatial sub division ⁇ ) 1306.
  • Nodes of the tree structure representing the spatial sub-division(s) 1306 may map or index the data object(s) 404 to the processing partition(s) 1308 using the spatial location information.
  • the spatial index representing the processing partition(s) 1308 may also contain metadata information about related areas of the spatial sub division ⁇ ) 1306, such as parents, neighbors (special siblings), children, level, and the like.
  • the world manager 402 may use the processing partiiion(s) 1308 to determine one or more processing partition assignment(s) 1312.
  • the processing partition assignment(s) 1312 allocate the processing partition(s) 1308 to the hardware host(s) 1304.
  • the processing partition assignment(s) 1312 may be represented by a data structure, such as a list, index, tree, etc. that identifies a hardware host in the hardware hosi(s) 1304 and the processing partition(s) 1308 assigned to the hardware host.
  • the processing partition assignment(s) 1312 may be made based on a number of spatial sub-divisions in the spatial sub-division(s) 1306, the object type of the data object(s) 404 at a given locality, an application instance that can process the object type, spatial relationships (e.g., between the spatial sub-division(s) 1306 or between the data object(s) 404, data dependencies between the data object(s) 404, and metrics associated with the hardware host(s) 1304, such as the processing load on hardware or software resources (e.g., to avoid overloading, etc.).
  • the world manager 1302 may send the processing partition assignment(s)
  • the world manager 1302 may send the processing partition assignment(s) 1312 directly to the hardware host(s) 1304. In another example, the world manager 1302 may notify the hardware host(s) 1304 of the processing partition assignment(s) 1312 and the hardware host(s) 1304 may retrieve the processing partition assignment(s) 1312 from another source. The hardware host(s) 1304 may then process the data object(s) 404 using the processing partition assigmnent(s)
  • processing may include: data object movements, data object deletions, data object insertions, data object collisions, rendering, explosions, physics simulations, in-world credit transactions, or other types of processing.
  • the hardware host(s) 1304 may use the processing partition assignment(s) 1312 to identify allocations of the processing partition(s) 1308.
  • the world manager 1302 may send the processing partition(s) 1308 to the hardware host(s) 1304 with the processing partition assignment(s) 1312.
  • the hardware host(s) 1304 may retrieve allocations of the processing parti tion(s) 1308 from another source.
  • FIG. 14 illustrates a graphical example of organizing computational units to assign to hardware hosts in a distributed computing system 1400 using spatial location information associated with data objects according to an example of the present technology.
  • FIG. 14 illustrates the separation of a world space into more than one group or processing partition, first using object domains (e.g.,“Domain A” and“Domain B”) as discussed above and then using spatial indexing. If the world space of a virtual environment hosted by the distributed computing system includes the two domains, each of which can have a large number of data objects densely packed into different spaces, the computational units of the two domains may be in different states, as depicted.
  • object domains e.g.,“Domain A” and“Domain B”
  • FIG. 14 illustrates the further separation of the world space into more than one group or processing partition, using a plurality of spatial sub-divisions (e.g.,“Sub- Division A” and“Sub-Division B”).
  • a plurality of spatial sub-divisions e.g.,“Sub- Division A” and“Sub-Division B”.
  • the distributed computing system 1400 may include a spatial indexing manager 1402, which may include one or more server computers configured to index data objects in a virtual world hosted by the distributed computing system 1400 into computational units or processing partitions using spatial location information.
  • Some examples of the spatial indexing manager 1402 may include the world manager 1302 and the hardware host(s) 1304 described with respect to FIG. 13.
  • the spatial indexing manager 1402 may determine a plurality of processing partitions associated with the plurality of spatial sub-divisions of the domains of Sub-Division A and Sub-Division B
  • the spatial indexing manager 1402 may use one or more object field(s) 1404 and an indexing configuration 1406 (as described later) to initially create and manage one or more spatial index(es) 1408.
  • the spatial indexing manager 1402 may analyze spatial location information associated with the object field(s) 1404 to determine collections of data objects that may be organized into processing partitions by position, proximity , size, shape, spatial relationships, spatial dependencies, spatial causalities, and the like. For example, the spatial indexing manager 1402 may analyze the spatial location information to determine whether a data object can be grouped into a processing partition by position or location.
  • the spatial indexing manager 1402 may further analyze the spatial location information to determine whether a spatial relationship exists for a data object. For example, a first data object may have a spatial relationship to a second data object by being positioned near the second data object. In another example, the first data object and the second data object may have a spatial relationship with a bounding box by being positioned within a plane or volume associated with the bounding box. The spatial indexing manager 1402 may determine whether some of the data objects satisfy spatial causality by being within a predefined distance of each other.
  • Other examples may include the data objects satisfying a spatial relationship with a gravitational or electromagnetic field, located within a defined geometry, positioned on a defined topology, approaching or intersecting with a set of vertices and edges describing a polygon, mesh, collision box, or other shape, a range, a camera view, and the like.
  • the spatial indexing manager 1402 may use the indexing configuration 1406 to determine how to represent the processing partitions associated with the spatial index(es) 1408.
  • the spatial indexing manager 1402 may determine a capacity of a processing partition using a number of data objects that may be allocated to the processing partition.
  • the indexing configuration 1406 may specify a number of data objects that may be indexed by the spatial index(es) 1408.
  • the number of data objects that may indexed by the spatial index(es) 1408 may be determined based on the configurations of processing applications and a capacity of the processing applications to process a number of data objects.
  • the number of data objects that may indexed by the spatial index(es) 1408 may also be determined based on a configuration of a virtual yvorld, which may define whether there is a one-to-one, one-to-many, and many-to-one correspondence between processing partitions and the spatial index(es) 1408, the structure of the spatial index(es) 1408, one or more functions used to index the data objectis) 404, the configurations of the hardware and software resources, the number of hardware hosts in the distributed computing systems, and the like.
  • the indexing configuration 1406 may further include index specific configurations that are set as part of a world definition (e.g., to use an octree with a max le vel and a predefined range).
  • the spatial indexing manager 1402 creates, updates, and manages the spatial index(es) 1408 using the object field(s) 1404 and the indexing configuration 1406.
  • the spatial indexing manager 1402 may organize the data objects according to spatial location information in the object field(s) 1404.
  • the spatial indexing manager 1402 may create the spatial index(es) 1408 to map collections of the data objects to the processing partition by spatial location information using the object field(s) 1404.
  • the spatial index(es) 1408 may use the object field(s) 1404 to create multiple tiers in the groupings, thereby further organizing the data object by multiple attributes such as object domain and spatial locality.
  • the spatial index(es) 1408 may be represented by a tree structure having nodes 1410, 1412, 1414, 1416, 1418, 1420, and 1422.
  • the node 1410 may represent a root node and the nodes 1412, 1414, 1416, 1418, 1420, and 1422 may represent child nodes.
  • the root node 1410 may include metadata 1430 that provides information associated with a processing partition represented by the root node 1430 and a collection of the data objectis) 404 organized into the processing partition. Metadata 1430 may also include information about related areas of the spatial index(es) 1408, such as parent nodes, neighbor nodes (special siblings), child nodes, and the like.
  • the metadata 1430 may identify indexing tiers and a first tier used in sub-divid g a space.
  • the metadata 1430 may specify that the first tier for partitioning the world space is the object type and the second tier is the spatial locality.
  • the children nodes 1412, 1414, 1416, 1418, 1420, and 1422 may also include similar metadata (not shown).
  • the spatial indexing manager 1402 may be responsible for indexing data objects into processing partitions using the spatial index(es) 1408.
  • the spatial indexing manager 1402 may allocate a single node in the spatial index(es) 1408 for a data object based on an object type or spatial sub-division.
  • the spatial indexing manager 1402 may receive an instruction to index a data object, and once positioned within the world space, the spatial indexing manager 1402 may map the data object to a processing partition by reading the object Heidis) 1404 and by adding metadata associated with the data object to a node in the spatial index(es) 1408, which is associated with the processing partition.
  • the spatial indexing manager 1402 may maintain an atomic reference to a working copy of the spatial index(es) 1408 and only the spatial indexing manager 1402 can access the working copy.
  • the spatial indexing manager 1402 may index the data object using the working copy and then flip the atomic reference to the spatial index(es) 1408 so other threads (such as a world manager or application manager) can access a current state of the processing partitions.
  • the spatial indexing manager 1402 may use a bash code octree implementation to calculate the location of the data object using a normalized range.
  • the spatial indexing manager 1402 may compute a hash for data objects by normalizing spatial location information associated with the data object to the normalized range.
  • the computed hash may identify the node in the spatial index(es) 1408 that corresponds to a spatial sub-division of the world space that encompasses the location of the data object.
  • the spatial indexing manager 1402 may normalize positional dimensions associated with the data object to a floating-point number from 0 to 1.
  • the spatial indexing manager 1402 may- then multiply the normalized positions by 2 to a power associated with a number of levels specified for the octree.
  • the spatial indexing manager 1402 may effectively hash the data object into the deepest possible level of the spatial index(es) 1408.
  • the spatial indexing manager 1402 may then use the calculated hash to traverse the spatial index(es) 1408 from the root node 1410 in order to find an internal or child node that encloses the data object.
  • the spatial indexing manager 1402 may compare a hash bit or hash bits matching the“next” level to the children of the current node.
  • the spatial indexing manager 1402 may use the hash to determine the index of the next child in the array of children.
  • requests to modify data associated with the data object may be issued from instances of processing applications associated with the object type of the data object.
  • the spatial indexing manager 1402 may re-index the data object into the spatial index(es) 1408 using mutations to the data object caused by the requests. For example, during re-indexing, the spatial indexing manager 1402 may determine to update the map between the data object and a processing partition in response to the requests to modify data. The spatial indexing manager 1402 may also determine to update the mapping in response to an instruction from a world manager or another process to re-index the data object.
  • the spatial indexing manager 1402 may re-index the data object using mutations to the data object caused by the requests and update the metadata associated with the data object in the spatial index(es) 1408. Re-indexing may occur when the ownership of a data object is transferred from a first hardware host to a second hardware host. In this case, the spatial indexing manager 1402 may take ownership of the data object and update metadata associated with the data object in the spatial index(es) 1408.
  • the spatial indexing manager 1402 may also use spatial location information associated with the data object to determine whether to change or maintain the mapping between the data object and the processing partition.
  • the spatial indexing manager 1402 may re-index the data object by adding metadata associated with the data object at a different node in the spatial index(es) 1408, for example, because mo vement of the data object reflected in the spatial location information causes the data object to be at a position associated with a different spatial sub-division of the world space.
  • the spatial indexing manager 1402 may also re-index the data object to remove the metadata associated with the data object from a node in the spatial index(es) 1408 because the data object is no longer located in the world space (e.g , from de-spawning or being destroyed).
  • the spatial indexing manager 1402 may also determine whether to split or merge nodes in the spatial index(es) 1408.
  • the spatial indexing manager 1402 may determine to split a first node into a second node and a third node when a number of data objects mapped into a first processing partition represented by the first node approaches, meets, or exceeds a first defined threshold.
  • the first defined threshold may be determined as a first percentage of the capacity of a processing application to process a number of data objects (e.g., 60%).
  • the spatial indexing manager 1402 may determine to merge a first node and a second node to create a third node when a number of data object mapped into a first and second processing partitions represented by the first and second nodes approaches, meets, or falls blow a second defined threshold.
  • the second defined threshold may be determined as a second percentage of the capacity of a processing application to process a number of data objects (e.g., 10%).
  • the spatial indexing manager 1402 may read the spatial index(es) 1408 and perform any splits or merges as needed.
  • the spatial indexing manager 1402 may split and add nodes to the spatial index(es) 1408 (or update metadata of existing nodes to reflect a split) when the first defined threshold, determined as a first percentage of the capacity of a processing application to process a number of data objects (e.g., 60%), is met or exceeded.
  • the first defined threshold determined as a first percentage of the capacity of a processing application to process a number of data objects (e.g., 60%)
  • the world space region associated with the child node 1412 may be sub-divided into four sub-divided regions, which may be associated with the child node 1414 and three siblings.
  • the spatial indexing manager 1402 may allocate some of the data objects indexed by the child node 1412 across one or more of the child nodes 1414 and three siblings, or combinations thereof, when the first threshold is met or exceeded.
  • the spatial indexing manager 1402 may use spatial location information associated with the data objects, which positions the data objects into corresponding sub-divided regions, to index the data objects into the child node 1414 and three siblings.
  • data objects outside of the normalized range that hashes into child nodes of the spatial index(es) 1408 may be mapped to the root node 1410. If there are too many data objects at the root node 1410, an“upwards” split may be performed.
  • An additional node may be added to spatial index(es) 1408 as siblings of the root node 1408 representing an additional quadrant or octant that expands the world space.
  • Each additional quadrant or octant added on the side of the root node 1410 may be considered for the upward split to become the new root node.
  • a completely new root node may be created.
  • data objects at the root 1410 may be classified into super-regions encompassing the existing world space. Using, for example four super-regions, the best super-region to use for expansion upwards may be determined by which split removes the most objects from the root node 1410 and keeps the size of the nodes below' the limit.
  • the object hashes may be recalculated by adding bits to represent the expanded space to the current object hashes.
  • range changes that are powers of 2 (doubling, quadrupling)
  • the conversion can be done on existing location hashes by bit shifting each dimension.
  • An actual range change may also be performed by re-hashing all the data objects and re- creating the spatial index(es) 1408 A“shadow” octree may be created by the spatial indexing manager 1402 and then published when complete.
  • a level addition may be performed.
  • the level addition may include adding a bit to the end of data object hashes.
  • the level addition may occur when a node is created within two levels of the bottom.
  • the spatial indexing manager 1402 may merge and remove nodes from the spatial index(es) 1408 (or update metadata of existing nodes to reflect a merge) when the second defined threshold, determined as a second percentage of the capacity of a processing application to process a number of data objects (e.g., 10%), is met or exceeded.
  • the second defined threshold determined as a second percentage of the capacity of a processing application to process a number of data objects (e.g., 10%)
  • the four sub-divided regions that include a world space region associated with the child node 1422 and three siblings may be merged into a world space region associated with the child node 1420. Any child nodes associated with the child node 1422 may also be merged into the world space region associated with the child node 1420.
  • the spatial indexing manager 1402 may combine the data objects indexed by one or more of the child nodes 1422, the three siblings, and any children of the child node 1422 and the three siblings, or combinations thereof, with the child node 1420. To merge nodes together, the spatial indexing manager 1402 may take ownership of a child node 1420 in an Nth level, the child node 1422 in an N+l level, and the three siblings to the child node 1420 in N+ 1 level. If the total number of data objects indexed by the child node 1422 and the three siblings approaches a lower threshold or minimum number of data objects, the spatial indexing manager 1402 may merge the child node 1422 at the Nth level and the three siblings at the N+l level into the node 1420. The spatial indexing manager 1402 may then delete the child node 1422 and the three siblings.
  • FIGS. 15A-15B are flow diagrams that illustrate an example method 1500 for managing hardware hosts in a distributed computing system using processing partitions organized by spatial location information associated with data objects according to examples of the present technology.
  • the method 1500 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine, such as a computer system or information processing device, by hardware components of an electronic device or application-specific integrated circuits, or by combinations of software and hardware elements.
  • a world manager may determine a plurality of spatial sub divisions associated with an object-based model representing a virtual environment hosted by a distributed computing system.
  • the world manager may determine the plurality of spatial sub-divisions using a configuration associated with the object-based model representing the virtual environment.
  • the world manager may parse the configuration to determine a size of a world space associated with the virtual
  • a data structure to use to represent the plurality of spatial sub-divisions e.g., an octree or quadtree
  • a configuration of the data structure e.g., a number of levels, a number of nodes within a level, and a size of each node terms of a minimum or maximum number of data objects that may be indexed by a node.
  • the world manager may also determine the plurality of spatial sub-divisions using configurations associated with processing applications that process the data objects in the object-based model.
  • the world manager may read the configurations to determine a plurality of object types associated with the data objects processed by the processing applications, appli cation resources used by the processing applications, the type of the data structures that the processing applications use to access the data objects, a minimum and maximum number of data objects that may be processed by the processing app!ication(s), subscription types which specify what additional object types are subscribed to in order to process the data objects, and a subscription strategy for obtaining the additional data.
  • the subscription strategy may identify what additional data object types are used during processing and at what source locations the additional object types may be located.
  • the subscription strategy of processing applications may be used to determine what additional data objects to synchronize to the hardware hosts on which processing instances of the processing applications are located.
  • the subscription strategy may specify for a processing application to subscribe to the neighbor nodes in the data structure (e.g., index) that is used to reference the data objects.
  • a hardware host may find the neighbors of a node into which data objects being processed by the processing application are indexed (e.g., using spatial location information) and the hardware host may subscribe the processing application to the data from those neighbor nodes.
  • the subscription strategy may specify object tracking, which identifies a data object, and subscribes to the data from the node into which the data object is indexed, the node’s neighbors, parents, children, and the like.
  • the subscription strategy can be used to start subscriptions to data objects from different hardware hosts in the distributed computing system.
  • the processing application may be instantiated on a first hardware host and the neighbor nodes may be all ocated to a second hardware host.
  • the first and second hardware hosts may communicate to share changes in state to the data objects indexed into the neighbor nodes.
  • the first hardware host may identify the neighbor nodes as being allocated to the second hardware hosts using processing partition assignments.
  • the first hardware host may send an instruction, message, subscription request, etc, to the second hardware host to subscribe and obtain data associated with data objects mapped to the neighbor nodes (e.g. for dependencies between the first hardware host and second hardware host).
  • the second hardware host may add the first hardware host to a list of subscribers to receive data associated with data object mapped into the neighbor nodes.
  • the second hardware host may determine a change to the data objects mapped to the neighbor nodes and send all or part of the data associated with data object mapped to the nei ghbor nodes to the first hardware host.
  • a processing application may issue a subscription for data associated with all players near an enemy, for use by AI to control enemy units. This may also be useful for processing applications or clients that perform rendering.
  • the subscription strategy may specify a maximum latency that application instances can tolerate for obtaining the additional object types from the specified sources.
  • the world manager may establish the plurality of spatial sub-divisions as spatial regions that divide a space into two or more subsets.
  • the plurality of spatial sub-divisions may be hierarchical, meaning that the space (or a region of the space) may be divided into several spatial regions, and then the spatial regions may be again recursively sub-divided to create additional spatial regions.
  • the plurality of spatial sub-divisions may also include multiple layers or hierarchies of sub-divisions, which may overlay each other and which may include the different layers or hierarchies providing a different metric, criteria, and granularity for sub-dividing the virtual world.
  • the plurality of spatial sub-divisions may be represented using one or more tree structures, for example, as discussed above with respect to FIG.
  • a world space associated with the virtual environment hosted by the distributed computing system may be sub-divided using octants and the plurality of spatial sub divisions may be represented using an octree to sub-divide a three-dimensional space associated with the virtual world.
  • the virtual world may be sub divided using quadrants and the plurality of spatial sub-divisions may be represented using a quadtree to partition a two-dimensional space associated with the virtual world.
  • Other methods may be used to sub-divide the virtual world and to represent the plurality of spatial sub-divisions using a data structure in order to provide nodes to represent a particular unit of spatial information and to which data may associated with the particular unit of spatial information.
  • the world manager may determine a mapping between a plurality of data objects and a plurality of processing partitions associated with the plurality of spatial sub-divisions. For example, the world manager may initialize a spatial index using an octree associated with the plurality of spatial sub-divisions.
  • the world manager may represent a processing partition with a node in the octree m order to associate the processing partition with a single spatial sub-division in the plurality of spatial sub-divisions.
  • the w'orld manager may also represent multiple processing partitions with a singl e node in the octree and associate a plurality of processing partitions with a single spatial sub-division.
  • the world manager may represent a processing partition with multiple nodes in the octree and associate a single processing partition with a plurality of spatial sub-divisions.
  • the world manager may identify a plurality of hardware hosts in the distributed computing system to process the object-based model using the plurality of data objects.
  • the world manager may identify hardware hosts from a pool of hardware hosts waiting for assignments, or the wOrld manager may identify active hardware hosts that have spare computing capacity, and the like.
  • the active hardware hosts may be executing application instances in a container or in a computing instance that may operate on the processing partitions.
  • the world manager may allocate the plurality of processing partitions to the plurality of hardware hosts using a plurality of spatial indexes. More specifically, the world manager may assign one or more spatial indexes to a hardware host.
  • the world manager may also assign one or more hardware hosts to a spatial index.
  • the world manager may allocate a processing partition to a hardware host based on: spare computing capacity, the hardware host having specialized hardware (e.g., a GPU), load balancing, collocating data, and the like.
  • the world manager may generate a plurality of processing partition assignments between the plurality of processing partitions and the plurality of hardware hosts.
  • processing partitions may be allocated to a hardware host that has an application instance for an application that can process the type of data objects mapped to the processing partitions and organized using spatial locality.
  • the world manager may assign a spatial index to a hardware host having a physics app that modifies data objects using: physics simulations, a collision detection app that determines collisions between data objects, a rendering app that renders the data objects into one or more images or videos, and the like.
  • the world manager may send the plurality of processing partition assignments to the plurality of hardware hosts to organize the plurality of hardware hosts to process the plurality of data objects using spatial locality.
  • the world manager may monitor spatial indexing of the plurality of data objects by the plurality of hardware hosts using the mapping to process the plurality of data objects.
  • the world manager may monitor the plurality of hardware hosts to track object changes, to determine whether changes occur to the plurality of processing partitions, and the determine metrics associated with the hardware hosts.
  • a data object may be modified as a result of processing by a processing application, which may change the spatial location information associated with the data object.
  • a processing partition may also be modified as a result of changes in the spatial location information due to the processing of the data object by the processing applications. Changes m spatial location information may introduce re-indexing of data objects by the hardware hosts. Changes in spatial location information may also cause the spliting or merging of processing partitions.
  • the world manager may determine a change to the plurality of processing partitions that is associated with the plurality of spatial sub-divisions.
  • An update to a processing partition with the change may include mapping a newly created data object to the processing partition, modifying a data object mapped to the processing partition, or removing the mapping between a data object and the processing partition.
  • An update to a processing partition with the change may include spliting a first processing partition into a second processing partition and a third processing partition, which allocates the data objects indexed into the first processing partition to the second and third processing partitions.
  • an update to a processing partition with the change may include merging a first processing partition and a second processing partition into a third processing partition, which allocates the data objects indexed into the first and second processing partitions to the third processing partition.
  • the world manager may determine, using the change, whether to update the plurality of processing partition assignments.
  • the world manager may determine to update the plurality of processing partitions to manage or optimize performance of the hardware hosts.
  • the world manager may load balance processing of data objects across the plurality of hardware hosts.
  • the world manager may redistribute processing partitions due to changes to the data objects or the hardware hosts. If the world manager determines, using the change, not to update the plurality of processing partition assignments in step 1520, the method 1500 continues in operation 1514 where the world manager returns to monitoring the plurality of hardware hosts.
  • the method 1500 continues in operation 1522 where the world manager updates the plurality of processing partitions using updates from the hardware hosts.
  • the world manager may update the plurality of processing partitions, for example, to change a processing partition assignment of a first hardware host and allocate spatial indexes owned by the first hardware host to a second hardware host.
  • the world manager may thereby migrate the processing partition from the first hardware host to the second hardware host.
  • the method 1500 continues using reference“B” from FIG. 15B and returns back to FIG. 15A where m operation 1512 the world manager may send the plurality of processing partition assignments to the plurality of hardware hosts.
  • FIG. 16 is a flow diagram that illustrates an example method 1600 for processing data objects assigned to a hardware host using processing partitions organized by spatial location information associated with the data objects according to examples of the present technology.
  • the method 1600 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine, such as a computer system or information processing device, by hardware components of an electronic device or application-specific integrated circuits, or by combinations of software and hardware elements.
  • software e.g., instructions or code modules
  • CPU or processor central processing unit
  • a logic machine such as a computer system or information processing device
  • hardware components of an electronic device or application-specific integrated circuits or by combinations of software and hardware elements.
  • a distributed shared memor' associated with a hardware host in a distributed computing system may receive a plurality of processing partition assignments representing assignments between a plurality of processing partitions and a plurality of hardware hosts.
  • the distributed shared memory may receive the plurality of processing partition assignments from a cluster or world manager associated with the distributed computing system.
  • the plurality of processing partition assignments may identify the plurality of hardware hosts in the distributed computing system and nodes of a tree structure assigned to the plurality 7 of hardware hosts.
  • a node may include a spatial index structure that represents a set of data objects grouped or indexed by spatial location information.
  • the set of data objects may additionally be organized by object type.
  • the node may represent a collection of data objects of the same object type and that satisfy a spatial dependency.
  • the distributed shared memory may identify, using the plurality of processing partition assignments, a processing partition assigned to the hardware host that is associated with a spatial sub-division.
  • the distributed shared memory may analyze a list providing the plurality of processing partition assignments between the plurality of hardware hosts and the plurality of processing partitions to find allocations to the hardware host.
  • the hardware host may filter the list to identify the processing partitions assigned to the hardware host.
  • the distributed shared memory may load a plurality of data objects grouped by a location associated with the spatial sub-division, which are mapped to the processing partition, to a memory device associated with the hardware host. For example, the distributed shared memory may read a node of the tree structure corresponding to the processing partition to obtain metadata about the data objects mapped to the processing partition. The distributed shared memory may retrieve the data objects from a file, an object data store, across a network, from another distributed shared memory', and the like.
  • the hardware host may process the plurality of data objects using an instance of a processing application on the hardware host.
  • the hardware host may determine an object type of the data objects to identify a processing application configured to execute code on data objects of the object type.
  • the hardware host may also determine locations of the data objects to identify a processing application configured to execute code on data objects according to location.
  • the hardware host may create an instance of the processing application as a containerized app to process the data objects.
  • the hardware host may determine whether to update the processing partition using a change associated with the spatial sub-division.
  • the hardware host may determine to update the processing partition in response to the processing of the data ob j ects in operation 1608, which causes changes in the spatial location information associated with the data objects.
  • the processing application may create a new' data object located within the virtual environment, move a data object, change data associated with an existing data object, and remove a data object from the virtual environment. Creation, deletion, and update operations may add or remove the spatial dependencies between the data objects used to organize the data objects into the processing partition.
  • a new data object may be created at the given location and trigger a spatial dependency with one or more of the data objects.
  • the hardware host may determine to update the processing partition to add the new data object to the processing partition because the data object satisfies the spatial dependency.
  • new data objects introduced to the given location which trigger the spatial dependency may cause the number of data objects to be handled by the processing application to increase and approach a set upper threshold value. As the number of data objects to be handled by the processing application approaches the threshold value, the hardware host may determine to split a first processing partition into two processing partitions, thereby allocating some of the data objects to remain mapped to the first processing partition and allocating some of the data objects to be mapped to a new second processing partition. The hardware host may also create a second instance of the processing application to handle the data objects mapped to the second processing partition.
  • some of the data objects may move during processing to a different location than the remaining data objects and remove a spatial dependency with the rest of the data objects.
  • some of the data objects may be destroyed in the virtual during processing.
  • the hardware host may determine to update the processing partition to remove the data objects from the processing partition that no longer satisfy the spatial dependency or that no longer exist in the virtual environment.
  • the removal of data objects from the given location may cause the number of data objects to be handled by the processing application to decrease.
  • the number of data objects handled by the processing application may approach or exceed a defined threshold value.
  • the hardware host may determine to merge a first processing partition with a second processing partition, thereby combining the data objects mapped to the second processing position with the data objects mapped to the first processing partition.
  • the hardware host may continue to process the plurality of data objects using the instance of the processing application in operation 1608. If the hardware host determines to update the processing partition in operation 1612, the hardware host may generate an updated processing partition to re-index the data objects associated with the spatial sub-division, in operation 1614. As discussed above, creation, deletion, and update operations may add or remove the spatial dependencies used to organize the data objects into the processing partition associated with the spatial sub-division. The hardware host may generate the updated processing partition by re indexing the data objects and including any additional data objects that satisfy spatial relationships with the spatial sub-division.
  • updating the processing partition may additionally be used to notify the cluster or world manager associated with the distributed computing system about utilization of hardware and software resources associated with the hardware host.
  • Adding new data objects may increase resource utilization because the number of data objects to be handled by the processing application has increased.
  • the cluster or w'orld manager may evaluate whether the hardware host has sufficient resources to process the additional data objects. Removing data objects from a processing partition may decrease resource utilization because the number of data objects to be handled by the processing application has decreased.
  • the cluster or world manager may then evaluate whether the hardware host can process additional data objects. Accordingly, in operation 1616, the hardware host may send the updated processing partition to the world manager.
  • FIG. 17 is a flow diagram that illustrates an example method 1700 for splitting processing partitions organized by spatial location information according to examples of the present technology.
  • the method 1700 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine, such as a computer system or information processing device, by hardware components of an electronic device or application-specific integrated circuits, or by combinations of software and hardware elements.
  • software e.g., instructions or code modules
  • CPU or processor central processing unit
  • a logic machine such as a computer system or information processing device
  • hardware components of an electronic device or application-specific integrated circuits or by combinations of software and hardware elements.
  • a hardware host in a distributed computing system may process an operation associated with a plurality of data objects.
  • the data objects may be grouped, using a first spatial index node, into a first processing partition for a first location associated with a first spatial sub-division.
  • the operation may include a create, read, update, and delete (CRIJD) operation processed by a distributed shared memory on the hardware host in operation 1704, the hardware host may determine, based on a result of the operation, to split the first processing partition into a second processing partition and a third processing partition.
  • CLIJD create, read, update, and delete
  • the hardware host may determine to split the first processing partition into the second and third processing partitions because the number of data objects handled by an instance of a processing application associated with the first processing partition approaches, equals, or exceeds a threshold specified for the processing application.
  • the hardware host may determine the threshold to use when splitting the first processing partition as a percentage of a capacity defined by the processing application to process a number of data objects. For example, the hardware host may trigger the splittin of the first processing partition into the second and third processing partitions when the number of data objects handled by the instance of the processing application meets or exceeds 60% of the capacity defined for the processing application.
  • the hardware host may determine that the number of data objects handled by the instance of the processing application may have increased, for example, when a new data object is created in a virtual environment and the new data object is mapped to the first processin partition.
  • the hardware host may determine that the number of data objects handled by the instance of the processing application may have been increased when a data object moves from a first location to a second location associated with the first spatial sub-division.
  • the hardware host may also determine that motion of a first data object may have triggered a spatial dependency with a second data object that is mapped to the first processing partition.
  • the hardware host may split some of the plurality of data objects to a second processing partition that groups data objects by a second location associated with a second spatial sub-division.
  • the hardware host may determine that the spatial location information associated with the data objects can be used to group the data objects into the second processing partition by the second location.
  • the hardware host may determine that some of the data objects satisfy a spatial relationship with the second location or have a data dependency with other objects at or near the second location.
  • the hardware host may then allocate the data objects to the second processing partition.
  • the hardware host may generate, determine, or identify a second node that represents the second processing partition in order to index data objects grouped into the second processing partition based on a second spatial sub-division. '
  • the hardware host may generate the second spatial index node using metadata that maps some of the data objects to the second processing partition.
  • the hardware host may split some of the plurality of data objects to the third processing partition that groups data objects by a third location associated with a third spatial sub-division. Accordingly, the hardware host may allocate the data objects to the third processing partition.
  • the distributed shared memory may determine, or identify' a third spatial index node that represents the third processing partition to index data objects associated with the third spatial sub-division.
  • the hardware host may update a tree structure associated with a plurality of spatial sub-divisions using the second and third spatial index nodes to reflect the split of the first processing partition.
  • the hardware host may, for the first processing partition, update the tree structure to remove metadata, if present, for the data objects now mapped to the second and third processing partitions from the first spatial index node.
  • the hardware host may update the tree structure using the second spatial index node to include metadata for the data objects now mapped to the second processing partition.
  • the hardware host may further update the tree structure using the third spatial index node to include metadata for the data objects now mapped to the third processing partition.
  • the hardware host may generate the second and third spatial index nodes in the tree structure by updating existing spatial index nodes or adding additional spatial index nodes (e.g., child or sibling spatial index nodes of the first spatial index node associated with the first processing partition)
  • Splitting the first processing partition into the second and third processing partitions may additionally be used to notify the cluster or world manager associated with the distributed computing system about utilization of hardware and software resources associated with a hardware host to which the second and third processing partitions are currently assigned.
  • the second and third processing partitions may increase resource utilization on the hardware host because the number of data objects to be handled by an instance of a processing application has increased or another instance of the processing application is to be launched.
  • the cluster or world manager may evaluate whether the hardware host has sufficient resources to process the data objects being allocated to the second and third processing partitions.
  • FIG. 18 is a flow diagram that illustrates an example method 1800 for merging processing partitions organized by spatial location information according to examples of the present technology'.
  • the method 1800 may be performed by software (e.g., instructions or code modules) when executed by a central processing unit (CPU or processor) of a logic machine, such as a computer system or information processing device, by hardware components of an electronic device or application-specific integrated circuits, or by combinations of software and hardware elements.
  • software e.g., instructions or code modules
  • CPU or processor central processing unit
  • a logic machine such as a computer system or information processing device
  • hardware components of an electronic device or application-specific integrated circuits or by combinations of software and hardware elements.
  • a hardware host in a distributed computing system may process an operation associated with a first plurality of data objects grouped into a first processing partition by a first location associated with a first spatial sub-division.
  • the hardware host may determine, based on a result of the operation, to merge the first processing partition and a second processing partition.
  • the hardware host may determine to merge the first processing partition with the second processing partition as a result of the operation when the number of data objects handled by an instance of a processing application associated with the first processing partition approaches, equals, or exceeds a merge threshold specified for the processing application.
  • the merge threshold may be a percentage of the capacity' of the processing application to process a number of data objects.
  • the merge threshold may trigger the merging of the first processing partition with the second processing partition when the number of data objects handled by the instance of the processing application satisfies or falls below 10% of the capacity of the processing application.
  • the number of data objects handled by the instance of the processing application may decrease, for example, when a data object is moved in a virtual environment and the data object may be mapped to a different processing partition based on the new location of the data object.
  • number of data objects handled by the instance of the processing application may decrease when a data object is removed or deleted from the virtual environment. Removing and updating data objects m the virtual environment may cause the percentage of data objects previously handled by the instance of the processing application to then satisfy the merge threshold.
  • the hardware host may merge the plurality of data objects to the second processing partition that groups data objects by a second location associated with a second spatial sub-division. For example, the hardware host may determine that spatial location information associated with the plurality of data objects can be used to group the data objects from both the first processing partition and the second processing partition into the second processing partition by the second location. In another example, the hardware host may determine that a spatial range defined for the second processing partition using the second location encompasses positions associated with the plurality of data objects in the first and second processing partitions. The hardware host may then move or allocate the plurality of data objects from the first processing partition to the second processing partition.
  • the hardware host may generate a spatial index node that represents the second processing partition to index the plurality of data objects grouped into the second processing partition associated with the second spatial-subdivisions.
  • the hardware host may generate the spatial index node using metadata that maps the plurality of data objects to the second processing partition.
  • the hardware host may update a tree structure associated with a plurality of spatial sub-divisions using the spatial index node to reflect the merger of the first processing partition and the second processing partition.
  • the hardware host may update the tree structure using the spatial index node to include metadata for the data objects now' mapped to the second processing partition.
  • the hardware host may also update the tree structure to remove metadata from a spatial index node associated with the first processing partition.
  • the hardware host may generate the spatial index node in the tree structure by updating existing spatial index nodes (e.g., a parent or sibling of a spatial index node associated with the first processing partition) or adding additional spatial index nodes.
  • Merging the first processing partition into the second processing partition may additionally be used to notify the cluster or world manager associated with the distributed computing system about underutilization of hardware and software resources associated with a hardware host to which the second processing partition is current assigned.
  • adding new data objects to the second processing partition may increase resource utilization because the number of data objects to be handled by an instance of a processing application has increased.
  • the cluster or world manager may evaluate whether the hardware host has sufficient resources to process the additional data objects being allocated to the second processing partition.
  • FIG. 19 illustrates a system 1900 and a computing device 1910 on which modules of this technology may execute.
  • a computing device 1910 is illustrated on which a high-level example of the technology may be executed.
  • the computing device 1910 may include one or more processors 1912 that are in communication with memory devices 1920.
  • the computing device 1910 may include a local communication interface 1918 for the components in the computing device.
  • the local communication interface 1918 may be a local data bus and/or any related address or control busses as may be desired.
  • the memory device 1920 may contain modules 1924 that are executable by the processor(s) 1912 and data for the modules 1924.
  • the modules 1924 may execute the functions described earlier.
  • a data store 1922 may also be located in the memory device 1920 for storing data related to the modules 1924 and other applications along with an operating system that is executable by the processors) 1912.
  • Other applications may also be stored in the memory device 1920 and may be executable by the processor(s) 1912.
  • Components or modules discussed in this description that may be implemented in the form of software using high-level programming languages that are compiled, interpreted, or executed using a hybrid of the methods.
  • the computing device 1910 may also have access to I/O (input/output) devices 1914 that are usable by the computing device 1910.
  • I/O (input/output) devices 1914 may be included in the computing device 1910.
  • Networking devices 1916 and similar communication devices may be included in the computing device 1910.
  • the netw orking devices 1916 may be w ired or wireless networking devices that connect to the internet, a LAN, WAN, or other computing network.
  • the components or modules 1924 that are shown as being stored in the memory device 1920 may be executed by the processor(s) 1912.
  • the term“executable” may mean a program file that is a form that may be executed by a processor 1912.
  • a program in a higher-level language may be compiled into machine code in a format that may be l oaded into a random-access portion of the memor device 1920 and executed by the processor 1912, or source code may be loaded by another executable program and interpreted to generate instructions in a random-access portion of the memory to be executed by a processor.
  • the executable program may be stored in any portion or component of the memory de vice 1920.
  • the memory device 1920 may be random access memory (RAM), read only memory (ROM), flash memory, a solid-state drive, memory card, a hard drive, optical disk, floppy disk, magnetic tape, or any other memory' components.
  • the processor 1912 may represent multiple processors and the memory device 1920 may represent multiple memory' units that operate in parallel to the processing circuits. This may provide parallel processing channels for the processes and data in the system.
  • the local interface 1918 may be used as a network to facilitate communication between any of the multiple processors and multiple memories.
  • the local interface 1918 may use additional systems designed for coordinating communication such as load balancing, bulk data transfer and similar systems.
  • one or more blocks shown the flow chart may be omitted or skipped. Any number of counters, state variables, warning semaphores, or messages might be added to the logical flo for purposes of enhanced utility, accounting, performance, measurement, troubleshooting, or for similar reasons.
  • a module may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components.
  • a module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, or the like.
  • Modules may also be implemented in software for execution by various types of processors.
  • An identified module of executable code may, for instance, comprise one or more blocks of computer instructions, which may be organized as an object, procedure, or function. Nevertheless, the executables of an identified module need not be physically located together, but may comprise disparate instructions stored in different locations which comprise the module and achieve the stated purpose for the module when joined logically together.
  • a module of executable code may be a single instruction, or many instructions and may even be distributed over several different code segments, among different programs and across several memory devices.
  • operational data may be identified and illustrated herein within modules and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed over different locations including over different storage devices.
  • the modules may be passive or active, including agents operable to perform desired functions.
  • Computer readable storage medium includes volatile and non-volatile, removable and non-removable media implemented with any technology for the storage of information such as computer readable instructions, data structures, program modules, or other data.
  • Computer readable storage media include, but is not limited to, a non-transitory machine readable storage medium, such as RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other computer storage medium which may be used to store the desired information and described technology.
  • the devices described herein may also contain communication connections or networking apparatus and networking connections that allow the devices to communicate with other devices.
  • Communication connections are an example of communication media.
  • Communication media typically embodies computer readable instructions, data structures, program modules, and other data in a modulated data signal such as a earner wave or other transport mechanism and includes any information delivery media.
  • a “modulated data signal’' means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
  • communication media includes wired media such as a wired network or direct-wired connection and wireless media such as acoustic, radio frequency, infrared and other wireless media.
  • the term computer readable media as used herein includes communication media.
  • mapping determining a mapping between a plurality of data objects and a plurality of processing partitions associated with the plurality of spatial sub-divisions, wherein the mapping is associated with a spatial index of the plurality of spatial sub-divisions that represents the plurality of data objects grouped into the plurality of processing partitions using spatial location information associated with the plurality of data objects;
  • the configuration identifies a tree structure for representing the plurality of sub-divisions subdividing a space associated with the multi-dimensional virtual environment;
  • a method comprising:
  • determining a mapping between a plurality of data objects and a plurality of processing partitions associated with the plurality of spatial sub-divisions wherein a processing partition is associated with a processing application that processes data objects grouped using spatial location information associated with the plurality of data objects; identifying a hardw'are host in the distributed computing system to process the data objects grouped by a location associated with a spatial sub-division in the plurality 7 of spatial sub-divisions, wiiich are mapped to the processing partition;
  • the plurality of data objects include a plurality of object fields that provide the spatial location information
  • the configuration identifies a tree structure for representing a space associated with the multi-dimensional virtual environment using the plurality of spatial sub-divisions;
  • determining the tree structure using the plurality of spatial sub-divisions wherein a node in the tree structure includes metadata associating the data objects grouped by the location to the processing partition.
  • change data includes allocating data objects from a first processing partition to a second processing partition and to a third processing partition
  • a system comprising:
  • one or more memory devices that store instructions which when executed by the one or more processors cause the one or more processors to:
  • the configuration identifies a tree structure for representing the plurality of sub divisions subdividing a space associated with the multi-dimensional virtual environment
  • a plurality of nodes in the tree structure represent the plurality of processing partitions and include metadata associating the plurality of data objects with the plurality of processing partitions.
  • the change includes allocating data objects mapped to a first processing partition that groups the data objects by a first location to a second processing partition that groups data objects by a second location and to a third processing partition that groups data objects by a third location;
  • the change includes allocating, to a first processing partition that groups data objects by a first location, data objects mapped to a second processing partition grouped by a second location and data objects mapped to a third processing partition grouped by a third location;
  • a method comprising:
  • mapping using the plurality of object types, the plurality of data objects to a plurality of processing partitions to distribute computational operations by object type within the distributed computing system, wherein the mapping includes a plurality of object type indexes that represent a set of data objects grouped, by object type, into a processing partition in the plurality' of processing partitions;
  • a method comprising:
  • determining a plurality' of object types associated with a plurality' of data objects in a multi -dimensional virtual environment hosted by a distributed computing system generating a mapping between the plurality of data objects and a plurality of processing partitions using the plurality' of object types, wherein a processing partition in the plurality of processing partitions is associated with a processing application that processes data objects of an object type;
  • mapping between the plurality of data objects and the plurality of processing partitions using the plurality of object types further comprises generating the mapping using a number of data objects of the object type as defined by a capacity of the processing application in a configuration of the processing application.
  • a system comprising:
  • one or more memory devices that store instructions which when executed by the one or more processors cause the one or more processors to:
  • the processing partition assignments determine, using the data associated with the utilization of the hardware or software resource by the plurality of hardware hosts, the processing partition assignments to re-allocate the utilization of the hardware or software resource of the plurality of hardware hosts.
  • a method comprising:
  • a representation of a data object to a memory' l ocation associated with a memory device of the computing hub, wherein the representation of the data object includes an object data section and a virtual table mapping a plurality of object fields associated with the data object to the object data section;
  • an addi tional log record to the log section to replace the plurality' of log records, wherein the additional log record includes a section indicating which mutations to the data object associated with the plurality of log records reflect changes to the representation of the data object.
  • a method comprising:
  • a virtual table that includes a plurality of references mapping the plurality of object fields to the data object section.
  • a system comprising:
  • At least one memory device storing instructions that, when executed by the at least one processor, cause the at least one processor to:
  • the log record to a log section associated with the representation of the data object.
  • a method comprising:
  • the second hardware host includes a second instance of a second processing application that provides processing of a second plurality of data objects of the second object type, which are mapped to the second processing partition;
  • determining a list of subscribers to the first plurality of data objects determining a list of subscribers to the first plurality of data objects; and sending the first plurality of data objects to the list of subscribers, wherein the list of subscribers includes a third hardware host or a third instance for a third processing application.
  • the third hardware host includes a third instance of a third processing application that provides processing of the second plurality of data objects
  • a method comprising:
  • an application library' at a hardware host in a distributed computing system hosting a multi-dimensional virtual environment, wherein the application library comprises a plurality' of processing applications that process data objects associated with a plurality of object types;
  • a system comprising:
  • one or more memor devices that store instructions which when executed by the one or more processors cause the one or more processors to:
  • a first processing application to launch from an application library using a first processing partition allocated for processing to a first hardware host using a plurality of processing partition assignments associated with a distributed computing system hosting a multi-dimensional virtual environment, wherein the first processing partition groups a first plurality of data objects by a first object type;

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

L'invention concerne une technologie qui fournit des applications et des systèmes informatiques distribués, à forte intensité de calcul, dans un environnement de fournisseur de services qui peut être utilisé pour fournir des mondes virtuels, des simulations, des environnements virtuels, des jeux, et d'autres environnements virtuels multidimensionnels (par exemple, 2D et 3D) qui peuvent être visualisés par des utilisateurs, auxquels des utilisateurs peuvent accéder et avec lesquels ils peuvent interagir, ainsi que d'autres services informatiques. Les applications informatiques distribuées peuvent être des applications distribuées, à grande échelle, à faible latence, utilisées avec des simulations ou des mondes virtuels interactifs persistants qui peuvent héberger des millions d'utilisateurs simultanés. Au moyen d'un partitionnement, cette technologie peut gérer des ressources de calcul de telle sorte que des applications ou des instances d'application peuvent utiliser efficacement les cœurs de CPU (cœurs d'unité de traitement central) de grandes instances de matériel et de telle sorte que des développeurs peuvent utiliser efficacement moins d'instances de matériel.
PCT/US2020/025585 2019-03-28 2020-03-28 Application distribuée, à faible latence, pour mondes interactifs WO2020198724A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP20721047.7A EP3948539A1 (fr) 2019-03-28 2020-03-28 Application distribuée, à faible latence, pour mondes interactifs
CN202080038381.6A CN114341808A (zh) 2019-03-28 2020-03-28 用于交互世界的低延迟、分布式应用

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
US16/368,677 2019-03-28
US16/368,697 2019-03-28
US16/368,738 US11150960B2 (en) 2019-03-28 2019-03-28 Distributed application allocation and communication
US16/368,723 US11436217B2 (en) 2019-03-28 2019-03-28 Ordered append-only log based data storage
US16/368,677 US11416553B2 (en) 2019-03-28 2019-03-28 Spatial indexing
US16/368,697 US11106502B2 (en) 2019-03-28 2019-03-28 Object domains
US16/368,738 2019-03-28
US16/368,723 2019-03-28

Publications (1)

Publication Number Publication Date
WO2020198724A1 true WO2020198724A1 (fr) 2020-10-01

Family

ID=70416560

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2020/025585 WO2020198724A1 (fr) 2019-03-28 2020-03-28 Application distribuée, à faible latence, pour mondes interactifs

Country Status (3)

Country Link
EP (1) EP3948539A1 (fr)
CN (1) CN114341808A (fr)
WO (1) WO2020198724A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114979164A (zh) * 2022-04-14 2022-08-30 网易（杭州）网络有限公司 虚拟房间的分配方法、装置和电子设备
WO2023244568A1 (fr) * 2022-06-17 2023-12-21 Luminary Cloud, Inc. Cadre en nuage pour analyse utilisant des accélérateurs

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080140762A1 (en) * 2006-10-05 2008-06-12 Holt John M Job scheduling amongst multiple computers

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080140762A1 (en) * 2006-10-05 2008-06-12 Holt John M Job scheduling amongst multiple computers

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
PAUL B BESKOW ET AL: "Latency reduction in massively multi-player online games by partial migration of game state", 18 September 2007 (2007-09-18), pages 153 - 163, XP002659986, ISBN: 978-0-946881-54-3, Retrieved from the Internet <URL:http://heim.ifi.uio.no/~paalh/publications/files/ita07.pdf> [retrieved on 20110926] *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114979164A (zh) * 2022-04-14 2022-08-30 网易（杭州）网络有限公司 虚拟房间的分配方法、装置和电子设备
CN114979164B (zh) * 2022-04-14 2023-11-17 网易（杭州）网络有限公司 虚拟房间的分配方法、装置和电子设备
WO2023244568A1 (fr) * 2022-06-17 2023-12-21 Luminary Cloud, Inc. Cadre en nuage pour analyse utilisant des accélérateurs

Also Published As

Publication number Publication date
EP3948539A1 (fr) 2022-02-09
CN114341808A (zh) 2022-04-12

Similar Documents

Publication Publication Date Title
US11150960B2 (en) Distributed application allocation and communication
US11416553B2 (en) Spatial indexing
US11436217B2 (en) Ordered append-only log based data storage
US10700991B2 (en) Multi-cluster resource management
US11816402B2 (en) Simulation systems and methods
Xin et al. Graphx: A resilient distributed graph system on spark
Docan et al. Dataspaces: an interaction and coordination framework for coupled simulation workflows
US11416364B2 (en) Methods and systems that identify dimensions related to anomalies in system components of distributed computer systems using clustered traces, metrics, and component-associated attribute values
Wang et al. Research and implementation on spatial data storage and operation based on Hadoop platform
US11533367B2 (en) Simulation systems and methods using query-based interest
Chen et al. Federation in cloud data management: Challenges and opportunities
CN106453618A (zh) 基于G‑Cloud云计算的遥感图像处理服务云平台系统
EP3948539A1 (fr) Application distribuée, à faible latence, pour mondes interactifs
Zhong et al. A distributed geospatial data storage and processing framework for large-scale WebGIS
US11113174B1 (en) Methods and systems that identify dimensions related to anomalies in system components of distributed computer systems using traces, metrics, and component-associated attribute values
Musial et al. Implementing distributed shared memory for dynamic networks
CN113032356A (zh) 一种客舱分布式文件存储系统及实现方法
Yang et al. K-Means Method for Grouping in Hybrid MapReduce Cluster.
AU2015276830B2 (en) Dynamic n-dimensional cubes for hosted analytics
Tudoran et al. Adaptive file management for scientific workflows on the azure cloud
US11106502B2 (en) Object domains
Silva et al. An efficient communication aware heuristic for multiple cloud application placement
KR101772955B1 (ko) 맵리듀스 기반의 분산 처리 시스템에서 인덱스를 이용하여 레코드를 처리하는 방법
Cao et al. Programming platforms for Big Data analysis
El Jamiy et al. A new software architecture style for hadoop systems

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20721047

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020721047

Country of ref document: EP

Effective date: 20211028