JP2022522320A - Reconfigurable Computation Pod Using Optical Network - Google Patents

Abstract

Methods, systems and appliances for creating clusters of building blocks of compute nodes using optical networks. In one aspect, the method comprises receiving request data specifying a compute node requested to perform a compute workload. The request data specifies the n-dimensional target configuration of the compute node. From a superpod containing a set of building blocks, each containing a compute node in an m-dimensional configuration, select a subset of the building blocks that, when combined, match the target configuration specified by the request data. A set of building blocks is connected to an optical network containing one or more optical circuit switches. Generate a workload cluster of compute nodes that contains a subset of building blocks. Creating a workload cluster involves configuring the routing data for each of one or more optical circuit switches for each dimension of the workload cluster.

Description

Background Some computational workloads, such as machine learning training, require many processing nodes to process the workload efficiently. Processing nodes can communicate with each other via an interconnect network. For example, in the case of machine learning training, processing nodes can converge to an optimal deep learning model by communicating with each other. The interconnect network is important for the speed and efficiency with which the processing unit achieves convergence.

Due to the changing size and complexity of machine learning and other workloads, a fixed configuration of a supercomputer with multiple processing nodes can limit the availability, scalability and performance of the supercomputer. .. For example, if some processing node of a supercomputer with a fixed interconnect network connecting a particular configuration consisting of multiple processing nodes fails, the supercomputer cannot replace the failed processing node. Availability and performance are reduced. Also, the performance of some specific configurations may be higher than that of other configurations, regardless of the failed node.

Overview This specification describes a technique that can reconfigure the superports of compute nodes that generate workload clusters using optical networks.

In general, one invention aspect of the subject matter described herein can be embodied in a method comprising receiving request data specifying a compute node required to perform a computational workload. .. The request data specifies the n (n is 2 or more) dimensional target configuration of the compute node. A portion of the building block that, when combined, matches the n-dimensional target configuration specified by the request data, from a superpod containing a set of building blocks, each containing a compute node of the m (m is 2 or more) dimension configuration. Select a set. The set of building blocks described above is connected to an optical network containing one or more optical circuit switches for each of the n dimensions. Generate a workload cluster of compute nodes that contains a subset of building blocks. A workload cluster is a cluster of compute nodes dedicated to the computation or execution of a particular compute workload. Generating involves configuring the routing data for each dimension of one or more optical circuit switches for that dimension for each dimension of the workload cluster. The routing data corresponding to each dimension of the workload cluster specifies how to route the data of the compute workload between the compute nodes along each dimension of the workload cluster. Computational nodes in a workload cluster run computational workloads.

These and other implementations may optionally include one or more of the following features: In some embodiments, the request data specifies different types of compute nodes. Selecting a subset of building blocks can include selecting a building block that contains one or more compute nodes of the specified type for each type of compute node specified by the request data.

In some embodiments, the routing data for each dimension of the superpod comprises an optical circuit switch routing table for one of one or more optical circuit switches. In some embodiments, the optical network comprises one or more optical circuit switches of the optical network that route data between computational nodes along that dimension for each dimension of n dimensions. Each building block can contain multiple segments of compute nodes along each dimension of the building block. An optical network can include an optical circuit switch of the optical network that routes data between the compute node segments corresponding to each building block in the workload cluster for each segment of each dimension.

In some embodiments, each building block comprises one of a three-dimensional torus-like compute node or a mesh-like compute node. In some embodiments, the superpod contains multiple workload clusters, each workload cluster containing a different set of building blocks and capable of running different workloads than other workload clusters.

Some embodiments include receiving data indicating that a particular building block of a workload cluster has failed and replacing the particular building block with an available building block. Replacing a particular building block with an available building block is like stopping data routing between a particular building block in a workload cluster and one or more other building blocks. One of the optical networks to update the data routing of one or more optical circuit switches and route data between the available building blocks of the workload cluster and one or more other building blocks. It can include updating the data routing of the above optical circuit switch.

In some embodiments, selecting a subset of building blocks that, when combined, matches the n-dimensional target configuration specified by the request data means that the n-dimensional configuration specified by the request data is available within the superpod. And determining that a first quantity of building blocks is needed that exceeds a healthy second quantity of building blocks, and the n-dimensional configuration specified by the request data, is available and healthy in the superpod. One or more that has a lower priority than the computational workload and is being executed by other Superpod building blocks, depending on the determination that a first quantity of building blocks is needed that exceeds two quantities of building blocks. Includes identifying a second computational workload in and reassigning one or more building blocks of one or more second computational workloads to a workload cluster of computational workloads. Creating a compute node workload cluster containing a subset of building blocks can have one or more building blocks of one or more second computational workloads included in the building block subset. ..

In some embodiments, generating a workload cluster of compute nodes containing a subset of building blocks is one or more building blocks of one or more second computational workloads for each dimension of the workload cluster. Each of the one or more optical circuit switches for that dimension so that each building block in the Includes reconstructing the routing data of.

The subject matter described herein may be implemented in a particular embodiment to achieve one or more of the following advantages: Computational nodes are available because you can easily replace failed or offline compute nodes on other compute nodes by dynamically configuring a cluster of compute nodes to run workloads using an optical network. The sex becomes higher. Flexible configuration of compute nodes allows for better performance of compute nodes, more efficient allocation of the appropriate number of compute nodes to run each workload, and more compute nodes to run each workload. The configuration can be optimized (or improved). Limited to compute nodes that are physically close to each other (eg, connected and / or adjacent to each other in the same rack), for example, in a data center or elsewhere, using a superpod that contains multiple types of compute nodes. Instead, you can generate a workload cluster that contains the appropriate number and configuration of compute nodes, as well as the appropriate type of compute node to run each workload. Instead, optical networks allow workload clusters of various shapes. In these workload clusters, compute nodes operate so that they are adjacent to each other, even if they are placed at arbitrary physical locations.

Also, by configuring the pod with an optical network, fault isolation and better security of the workload are provided. For example, some traditional supercomputers route traffic between the various computers that make up a supercomputer. If one computer fails, the communication path is interrupted. Data can be rapidly rerouted using an optical network and / or a failed compute node can be replaced with an available compute node. Also, the physical separation between workloads provided by an optical circuit switching (OCS) switch, such as the physical separation of different optical paths, is the same superport as compared to managing the separation using vulnerable software. Provides better security between various workloads running on.

Further, by connecting the building blocks using an optical network, it is possible to reduce the delay in transmitting data between the building blocks as compared with the packet switching network. For example, in the case of packet switching, the delay is long because the switch must receive the packet, buffer it, and send it again on another port. By connecting building blocks with OCS switches, it is possible to provide a true end-to-end optical path without packet switching or buffering along the way.

Hereinafter, various features and advantages of the above-mentioned subjects will be described with reference to the drawings. Further features and advantages are evident from the subject matter described herein and the claims.

FIG. 6 is a block diagram showing an environment in which an exemplary processing system creates a workload cluster of compute nodes and executes computational workloads using the workload cluster. It is a diagram showing an exemplary logical superpod, and an exemplary workload cluster generated from some building blocks within the superpod. It is a figure which shows the exemplary building block, and the exemplary workload cluster generated using the building block. FIG. 6 shows an exemplary optical link from a compute node to an optical circuit switching (OCS) switch. It is a figure which shows the logical calculation tray for forming a building block. It is a figure which shows the sub-block of an exemplary building block which omitted one dimension. It is a figure which shows an exemplary building block. It is a figure which shows the OCS fabric topology of a superpod. It is a figure which shows the component of an exemplary superpod. It is a flow chart which shows an exemplary process for generating a workload cluster and executing a computational workload using the workload cluster. FIG. 6 is a flow chart illustrating an exemplary process for reconstructing an optical network to replace a failed building block.

Detailed Description In various drawings, similar reference numbers and names refer to similar elements.

In general, the systems and techniques described herein can generate a workload cluster of compute nodes from a superpod by configuring an optical network fabric. A superpod is a group of multiple building blocks consisting of computational nodes connected to each other via an optical network. For example, a superpod can include a set of interconnected building blocks. Each building block can contain multiple compute nodes in an m-dimensional configuration, eg, a two-dimensional configuration or a three-dimensional configuration.

The user can specify the compute nodes of the goal configuration to run a particular workload. For example, the user can provide a machine learning workload and specify computational nodes for goal configurations to perform machine learning operations. The target configuration can define the number of computational nodes along each dimension of n (n is, for example, 2 or more) dimensions. That is, the goal configuration can define the size and shape of the workload cluster. For example, some machine learning models and computations work better in non-square topologies.

Bandwidth cross-sectional areas can also limit computations across compute nodes that are waiting for data transfer or exit idle computation cycles, for example. Depending on how the work is allocated across the compute nodes and how much data needs to be transferred over the network in different dimensions, the shape of the workload cluster depends on the performance of the compute nodes in the workload cluster. May affect.

For workloads that use all compute nodes to compute all compute node data traffic, a cubic workload cluster can minimize the number of hops between compute nodes. A workload is more specific than any other dimension when it has many local communications and transfers data along a particular dimension to a set of adjacent compute nodes and many of these adjacent communications are chained together. A configuration with more compute nodes along the dimension of is advantageous. Therefore, by allowing the user to specify the configuration of the compute nodes in the workload cluster, the user can specify the configuration that provides better performance to run the workload.

If different types of compute nodes are included in the superpod, the request can also specify the number of compute nodes of each type in the workload cluster. This allows the user to specify a configuration of compute nodes that works better to run a particular workload.

The workload scheduler is run, for example, by building block availability, building block health (eg, running or failing), and / or workload priority within the superpod (eg, the superpod compute node). The building blocks of the workload cluster can be selected based on the workload priority). The workload scheduler can provide the optical circuit switching (OCS) manager with data that identifies the selected building block and the target configuration of the building block. OCS managers can create workload clusters by configuring one or more OCS switches in an optical network to connect building blocks to each other. The workload scheduler can then run the compute workload on the compute nodes in the workload cluster.

If one of the building blocks in the workload cluster fails, the failed building block can be quickly replaced with another building block by simply reconfiguring the OCS switch. For example, the workload scheduler can select available building blocks from the superpod to replace the failed building block. The workload scheduler can instruct the OCS manager to replace the failed building block with the selected building block. The OCS manager will then reconfigure the OCS switch to connect the selected building block to other building blocks in the workload cluster and not to connect the failed building block to the building block in the workload cluster. Can be done.

FIG. 1 is a block diagram showing an environment 100 in which an exemplary processing system 130 creates a workload cluster of compute nodes and executes a compute workload using the workload cluster. The processing system 130 receives a computational workload 112 from a user apparatus 110 via a data communication network 120, such as a local area network (LAN), wide area network (WAN), internet, mobile network, or a combination thereof. be able to. By way of example, workload 112 includes software applications, machine learning models, such as training and / or use of machine learning models, video coding and decoding, and digital signal processing workloads.

The user can specify the cluster 114 of compute nodes required to run workload 112. For example, the user can specify the target shape and target size of the required cluster of compute nodes. That is, the user can specify the quantity of the calculation node and the shape of the calculation node along a plurality of dimensions. For example, if the compute nodes are arranged along three dimensions x, y, and z, the user can specify the number of compute nodes in each dimension. The user can also specify one or more types of compute nodes included in the cluster. As described below, the processing system 130 can include different types of compute nodes.

As described below, the processing system 130 can use building blocks to generate workload clusters that match clusters of target shape and target size. Each building block can contain a plurality of compute nodes arranged in m dimensions, eg, 3 dimensions. Therefore, the user can specify the target shape and the target size by specifying the number of building blocks in each dimension of the plurality of dimensions. For example, the processing system 130 can provide a user interface to the user equipment 110. This user interface allows the user to select the maximum number of building blocks along each dimension.

The user apparatus 110 can provide the processing system 130 with the workload 112 and the data designating the requested cluster 114. For example, the user apparatus 110 can provide the processing system 130 with request data including the workload 112 and data specifying the requested cluster 114 via the network 120.

The processing system 130 includes a cell scheduler 140 and one or more cells 150. Cell 150 is a group consisting of one or more superpods. For example, the illustrated cell 150 includes four superpods 152-158. Each superpod 152-158 includes a set of building blocks 160, also referred to herein as building block pools. In this example, each superpod 152-158 contains 64 building blocks 160. However, the superpods 152-158 can include other numbers of building blocks 160, such as 20, 50, 100, or another suitable number of building blocks 160. Also, the superpods 152-158 can include a different number of building blocks 160. For example, the superpod 152 can contain 64 building blocks and the superpod 152 contains 100 building blocks.

As will be described in more detail below, each building block 160 can include a plurality of compute nodes arranged in two or more dimensions. For example, the building block 160 can include 64 compute nodes arranged along three dimensions, specifically four compute nodes arranged along each dimension. In the present specification, the configuration of such a calculation node is four calculation nodes arranged along the x dimension, four calculation nodes arranged along the y dimension, and four calculation nodes arranged along the z dimension. Called as a 4x4x4 building block containing compute nodes. Other dimensions, such as 2 dimensions, and other numbers of compute nodes along each dimension, such as 3x1, 2x2x2, 6x2, 2x3x4, are also possible.

Also, the building block may contain only one compute node. However, as described below, in order to generate a workload cluster, the optical links between the building blocks are configured to connect the building blocks to each other. Therefore, smaller building blocks, such as building blocks containing only one compute node, can generate workload clusters with greater flexibility, but with more OCS switch configurations and more optical network elements ( For example, cables and switches) are required. The number of compute nodes in a building block is a trade-off between the desired workload cluster flexibility and the number of building blocks that need to be connected to each other to create a workload cluster and the number of OCS switches required. May be selected based on.

Each compute node in building block 160 may include application specific integrated circuits (ASICs), such as tensor processing units (TPUs), graphics processing units (GPUs), or other types of processing units for machine learning workloads. can. For example, each compute node may be a single processor chip containing a processing unit.

In some implementations, all building blocks 160 in the superpod have similar compute nodes. For example, the superpod 152 may include 64 building blocks to perform machine learning workloads, and each building block may have 64 TPUs in a 4x4x4 configuration. Superpods can also contain different types of compute nodes. For example, the superpod 154 can include 60 building blocks with TPUs and 4 building blocks with dedicated processing units that perform tasks other than machine learning workloads. In this way, the workload cluster for running the workload can contain different types of compute nodes. The superpod can contain multiple building blocks of each type of compute node for redundancy and / or to allow the execution of multiple workloads within the superpod.

In some implementations, all building blocks 160 in the superpod have similar configurations, eg, similar sizes and shapes. For example, each building block 160 in the superpod 152 can have a 4x4x4 configuration. Superpods can also contain building blocks of different configurations. For example, the superpod 154 can include 32 4x4x4 building blocks and 32 16x8x16 building blocks. Different building block configurations can contain similar or different compute nodes. For example, a building block containing a TPU may have a different configuration than a building block containing a GPU.

Superpods can contain building blocks at different levels. For example, the superpod 152 can include a basic building block having a 4x4x4 configuration. Also, the superpod 152 can include an intermediate building block consisting of more compute nodes. For example, an intermediate building block can have an 8x8x8 configuration formed from, for example, eight basic building blocks. In this way, intermediate building blocks can be used to generate larger workload clusters with fewer links than by connecting basic building blocks to generate larger workload clusters. Also, by including the basic building block in the superpod, it is possible to flexibly form a smaller workload cluster that does not require the number of compute nodes in the intermediate building block.

Superpods 152-158 in cell 150 can include building blocks consisting of similar or different types of compute nodes. For example, cell 150 can include one or more superpods consisting of TPU building blocks and one or more superpods consisting of GPU building blocks. The sizes and shapes of the building blocks of the different superpods 152-158 in the cell 150 may be similar or different.

In addition, each cell 150 includes a shared data storage 162 and a shared auxiliary calculation element 164. Each superpod 152-158 in cell 150 can use shared data storage 162 to store data generated by a workload running in, for example, superpods 152-158. Shared data storage 162 can include hard drives, solid state drives, flash memory, and / or other suitable data storage devices. The shared auxiliary calculation element can include a CPU (eg, a general purpose CPU device), a GPU, and / or other accelerators (eg, a video decoding accelerator or an image decoding accelerator) shared in the cell 150. Auxiliary computational elements 164 can also include storage devices, memory devices, and / or other computational elements that can be shared by compute nodes via the network.

The cell scheduler 140 can select cells 150 and / or superpods 152-158 of cell 150 to execute each workload received from the user apparatus 110. The cell scheduler 140 sets the superpods based on the goal configuration specified to run the workload, the availability of the building blocks 160 in the superpods 152-158, and the health of the building blocks in the superpods 152-158. You can choose. For example, the cell scheduler 140 can select a superpod that contains at least a sufficient number of available and healthy building blocks to generate a workload cluster with a target configuration to run the workload. If the request data specifies a type of compute node, the cell scheduler 140 may select a superpod that contains at least a sufficient number of available and healthy building blocks that include the specified type of compute node.

As described below, each superpod 152-158 may include a workload scheduler and an OCS manager. If the cell scheduler 140 selects the superpod of cell 150, the cell scheduler 140 may provide the workload scheduler of that superpod 150 with data specifying the workload and the requested cluster. As described in more detail below, the workload scheduler from the Superpod building blocks to the workload clusters based on the availability and health of the building blocks and, if necessary, the workload priorities within the Superpod. You can select a set of building blocks to connect to generate. For example, if the workload scheduler receives a request requesting a workload cluster that contains more building blocks than the number of available and healthy building blocks in the superpod, as described below, the workload scheduler will Building blocks for running lower priority workloads can be reassigned to the requested workload cluster. The workload scheduler can provide the OCS manager with data that identifies the selected building block. The OCS manager can create a workload cluster by configuring one or more OCS switches to connect the building blocks to each other. The workload scheduler can then run the workload on the compute nodes in the workload cluster.

In some implementations, the cell scheduler 140 balances the load between the various cells 150 and the superpods 152-158, for example, when selecting the superpods 152-158 to run the workload. For example, if one superpod is selected between two or more superpods containing a building block capable of handling the workload, the cell scheduler 140 will be the most capable superpod, eg, the most available and healthy. You can choose the building block, or the superpod of the cell with the highest overall capacity. An available and healthy building block is a building block that is not running another workload or is not part of a running workload cluster and has not failed. The cell scheduler can store the index of the building block. The index for each building block is that the building block is healthy (eg, not failed) and / or is available (eg, part of a workload cluster that is not running or running another workload). Can include data indicating whether or not.

In some implementations, the cell scheduler 140 can determine the goal configuration for running the workload. For example, the cell scheduler 140 can determine the target configuration of building blocks based on the estimated computational demand of the workload and the throughput of one or more types of compute nodes available. In this example, the cell scheduler 140 can provide the determined goal configuration to the superpod workload scheduler.

FIG. 2 shows an exemplary logical superpod 210 and exemplary workload clusters 220, 230, and 240 generated from some building blocks within the superpod 210. In this example, the superpod 210 contains 64 building blocks, each building block having a 4x4x4 configuration. Although many examples herein have described building blocks in a 4x4x4 configuration, the technique may be applied to building blocks in other configurations.

As will be described later, the shaded building blocks in the superpod 210 are the building blocks assigned to the workload. White building blocks are available and healthy building blocks. Black building blocks are unhealthy building blocks that cannot be used to generate workload clusters, for example due to a failure.

The workload cluster 220 is an 8x8x4 pod containing four 4x4x4 building blocks among the building blocks of the superpod 210. That is, the workload cluster 220 has eight calculation nodes along the x-dimensional, eight calculation nodes along the y-dimension, and four calculation nodes along the z-dimension. Because each building block has four compute nodes along each dimension, the workload cluster 220 has two building blocks along the x-dimension, two building blocks along the y-dimension, and two along the z-dimension. Includes one building block.

The four building blocks of the workload cluster 220 are shaded to indicate their location within the superpod 210. As shown, the building blocks of workload cluster 220 are not adjacent to each other. As described in more detail below, using an optical network to generate a workload cluster from any combination of building blocks in the superpod 210, regardless of the relative location of the building blocks in the superpod 210. Can be done.

The workload cluster 230 is an 8 × 8 × 8 pod containing eight building blocks among the building blocks of the superpod 210. Specifically, the workload cluster 230 contains two building blocks along each dimension. Thereby, the workload cluster 230 includes eight compute nodes along each dimension. Building blocks within the workload cluster 230 are indicated by vertical lines to indicate their location within the superpod 210.

The workload cluster 240 is a 16 × 8 × 16 pod containing 32 building blocks among the building blocks of the superpod 210. Specifically, the workload cluster 240 includes four building blocks along the x-dimension, two building blocks along the y-dimension, and four building blocks along the z-dimension. Thereby, this workload cluster includes 16 compute nodes along the x-dimension, 8 compute nodes along the y-dimension, and 16 compute nodes along the z-dimension. The building blocks within the workload cluster 240 are shown in mesh to indicate their location within the superpod 210.

Workload clusters 220, 230 and 240 are just a few examples of superpod 210 clusters that can be generated to simply run a workload. The workload cluster may have many other configurations. The exemplary workload clusters 220, 230 and 240 have a rectangular shape, but may have other shapes.

The shape of the workload cluster including the workload clusters 220, 230 and 240 is not a physical shape but a logical shape. The optical network is configured so that the building blocks communicate with each other along each dimension so that the workload clusters are physically connected in a logical configuration. However, physical building blocks and corresponding compute nodes may be physically located within the data center in various ways. The building blocks of workloads 220, 230 and 240 are of any use, regardless of the physical relationship between the building blocks in the superpod 210, except that all the building blocks in the superpod 210 are connected to the optical network. You can choose from possible and healthy building blocks. For example, as described above and illustrated in FIG. 2, workload clusters 220, 230 and 240 include building blocks that are not physically adjacent.

Moreover, the logical configuration of the workload cluster is not limited by the physical configuration of the building blocks in the superpod. For example, you may place 8 rows and 8 columns of building blocks and only one building block along the z dimension. However, a workload cluster can be configured by configuring the optical network to create a logical configuration containing a plurality of building blocks along the z dimension.

FIG. 3 is a diagram showing an exemplary building block 310 and exemplary workload clusters 320, 330 and 340 generated using the building block 310. Building block 310 is a 4x4x4 building block containing four compute nodes along each dimension. In this example, each dimension of the building block 310 in each dimension contains 16 segments, each segment containing 4 compute nodes. For example, at the top of the building block 310, there are 16 compute nodes. Of the 16 compute nodes, the segment along the y dimension includes one compute node and three other compute nodes including the corresponding last compute node at the bottom of the building block 310. For example, one segment along the y dimension includes compute nodes 301-304.

Computational nodes in the building block 310 can be connected to each other via internal links 318 made of conductive material, such as copper cables. Computational nodes within each segment of each dimension can be connected via internal link 318. For example, one internal link 318 connects compute node 301 to compute node 302. Also, one internal link 318 connects the compute node 302 to the compute node 303. Another internal link 318 connects compute node 303 to compute node 304. Similarly, by connecting compute nodes in other segments, internal data communication between compute nodes in building block 310 can be provided.

The building block 310 also includes external links 311 to 316 for connecting the building block 310 to the optical network. The optical network connects the building block 310 to another building block. In this example, the building block 310 includes 16 external input links 311 in x-dimensional. That is, the building block 310 includes an external input link 311 for each of the 16 segments along the x-dimension. Similarly, the building block 310 has an external output link 312 for each segment along the x-dimension, an external input link 313 for each segment along the y-dimension, and an external output link 314 for each segment along the y-dimension. It includes an external input link 315 for each segment along the z dimension and an external output link 316 for each segment along the z dimension. Since some building blocks can have a configuration of four or more dimensions, such as a torus, the building block 310 can include similar external links for each dimension of the building block 310.

Each external link 311 to 316 may be an optical fiber link for connecting a compute node on a segment of the corresponding compute node to an optical network. For example, each external link 311-316 can connect the corresponding compute node to an OCS switch in the optical network. As described below, the optical network can include one or more OCS switches for each dimension of the segment of building block 310. That is, the x-dimensional external links 311 and 312 are connected to an OCS switch different from the external links 313 and 314. As described in detail below, OCS switches may be configured to generate workload clusters by connecting building blocks to other building blocks.

The building block 310 has a 4x4x4 mesh configuration. The 4x4x4 (or other size) building block may have other configurations. For example, the building block 310 may have a three-dimensional torus configuration that includes a wraparound torus link, similar to the workload cluster 320. The workload cluster 320 may be generated from a single mesh building block 310 by configuring the optical network to form wraparound torus links 321 to 323.

The torus links 321 to 323 provide wraparound data communication between one end of each segment and the other end of the segment. For example, the torus link 321 connects a compute node located at each end of each segment along the x-dimension to a compute node located at the other end of the segment. The torus link 321 may include a link connecting the compute node 325 to the compute node 326. Similarly, the torus link 322 may include a link connecting the compute node 325 to the compute node 327.

The torus links 321 to 323 may be a conductive cable, for example, a copper cable, or may be an optical link. For example, the optical links of the torus links 321 to 323 can connect the corresponding compute node to one or more OCS switches. The OCS switch can be configured to route data from one end of each segment to the other end of each segment. The building block 310 may include an OCS switch for each dimension. For example, a torus link 321 can be connected to a first OCS switch, the first OCS switch between one end of each segment along the x-dimension and the other end of each segment along the x-dimension. Data can be routed. Similarly, the torus link 322 can be connected to a second OCS switch, which is between one end of each segment along the y dimension and the other end of each segment along the y dimension. Data can be routed to. The torus link 322 can be connected to a third OCS switch, which switches data between one end of each segment along the z dimension and the other end of each segment along the z dimension. Can be routed.

The workload cluster 330 contains two building blocks 338 and 339 that generate 4x8x4 pods. Each of the building blocks 338 and 339 may be similar to the building block 310 or the workload cluster 320. The two building blocks are connected along the y dimension via an external link 337. For example, one or more OCS switches may be configured to route data between the y-dimensional segment of building block 338 and the y-dimensional segment of building block 339.

Further, the one or more OCS switches may be configured to form wraparound links 331 to 333 between one end of each segment and the other end of each segment along all three dimensions. In this example, the wraparound link 333 is generated by combining two building blocks 338 and 339 by connecting one end of the y-dimensional segment of building block 338 to one end of the y-dimensional segment of building block 339. Provides full wraparound communication for dimensional segments.

The workload cluster 340 contains eight building blocks (one not shown) that produces an 8x8x8 cluster. Each building block 348 may be similar to the building block 310. Building blocks connected along the x-dimension are connected via external links 345A-354C. Similarly, building blocks connected along the y dimension are connected via external links 344A to 344C, and building blocks connected along the z dimension are connected via external links 346A to 346C. For example, one or more OCS switches may be configured to route data between x-dimensional segments, and one or more OCS switches may be configured to route data between y-dimensional segments. The OCS switch may be configured to route data between z-dimensional segments. Along each dimension, there are additional external links connecting building blocks not shown in FIG. 3 to adjacent building blocks. Further, the one or more OCS switches may be configured to form wraparound links 341 to 343 between one end of each segment and the other end of each segment along all three dimensions.

FIG. 4 is a diagram showing an exemplary optical link 400 from the compute node to the OCS switch. The compute node of the superpod may be installed in the tray of the data center rack. Each compute node can include six high speed electrical links. Two of these electrical links may be connected to the circuit board of the compute node, and the four electrical links are external electrical connectors connected to port 410, eg OSFP (Octal Small Form Factor Pluggable) ports, eg OSFP connectors. May be routed to. In this example, the port 410 is connected to the optical module 420 via an electrical link 412. The optical module 420 extends the electrical link by the length of the external link, eg, 1 km (km), to provide data communication between computing nodes located in large data centers, as needed. It can be converted into an optical link that extends beyond. The type of optical module may be varied based on the required length between the building block and the OCS switch and the desired speed and bandwidth of the link.

The optical module 420 is connected to the circulator 430 via fiber optic cables 422 and 424. The fiber optic cable 422 can include one or more fiber optic cables for transmitting data from the optical module 420 to the circulator 430. The fiber optic cable 424 can include one or more fiber optic cables for receiving data from the circulator 430. For example, fiber optic cables 422 and 424 can include bidirectional fiber optic or unidirectional TX / RX fiber optic pairs. The circulator 430 can reduce the number of optical fibers (eg, by converting a unidirectional optical fiber to a bidirectional optical fiber, the pair of optical fiber cables 432 can be reduced to a pair). This typically corresponds to a single OCS port 445 of the OCS switch 440, which accommodates a pair of integrally converted optical paths (two fibers). In some implementations, the circulator 430 may be integrated into the optical module 420 or may be omitted from the optical link 400.

FIGS. 5-7 show a method of forming a 4 × 4 × 4 building block using a plurality of calculation trays. Similar techniques can be used to form building blocks of other sizes and shapes.

FIG. 5 shows a logical calculation tray 500 for forming a 4x4x4 building block. The basic hardware block of a 4x4x4 building block is a single compute tray 500 with a 2x2x1 topology. In this example, the compute tray 500 includes two compute nodes along the x-dimension, two compute nodes along the y-dimension, and one compute node along the z-dimension. For example, compute nodes 501 and 502 form an x-dimensional segment, and compute nodes 503 and 504 form an x-dimensional segment. Similarly, compute nodes 501 and 503 form a y-dimensional segment, and compute nodes 502 and 504 form a y-dimensional segment.

Each compute node 501-504 is connected to two other compute nodes via an internal link 510, eg, a copper cable or trace on a printed circuit board. Also, each compute node is connected to four external ports. The compute node 501 is connected to the external port 521. Similarly, the compute node 502 is connected to the external port 522, the compute node 503 is connected to the external port 523, and the compute node 504 is connected to the external port 524. As mentioned above, the external ports 521-524 may be OSFP ports or other ports that connect the compute node to the OCS switch. These ports can accommodate fiber optic modules attached to copper or fiber optic cables.

Each of the external ports 521 to 524 of the compute nodes 501 to 504 includes one x-dimensional port, one y-dimensional port, and two z-dimensional ports. The reason is that each compute node 501-504 is already connected to another compute node along the x-dimensional and y-dimensions via the internal link 510. By including two z-dimensional external ports, each compute node 501-504 can be connected to two compute nodes along the z-dimension.

FIG. 6 is a diagram showing a sub-block 600 of an exemplary building block omitting one dimension (z dimension). Specifically, the sub-block 600 is a 4 × 4 × 1 block formed by a calculation tray having a 2 × 2 configuration, for example, a calculation tray 500 having a 2 × 2 configuration in FIG. The sub-block 600 includes four calculation trays 620A to 620D having a 2 × 2 configuration. Each calculation tray 620A to 620D may be similar to the calculation tray 500 of FIG. 5 including four 2 × 2 × 1 configuration calculation nodes 622.

Computation nodes 622 of the calculation trays 620A to 620D may be connected via internal links 631 to 634, for example, a copper cable. For example, the two compute nodes 622 of the compute tray 620A are connected to the two compute nodes 622 of the compute tray 620B along the y dimension via the internal link 632.

Further, the two calculation nodes 622 of each calculation tray 620A to 620D are connected to the external link 640 along the x-dimension. Similarly, the two compute nodes of each compute tray 620A-620D are connected to the external link 641 along the y dimension. Specifically, the compute node arranged at the end of each x-dimensional segment and the compute node arranged at the end of each y-dimensional segment are connected to the external link 640. These external links 640 may be fiber optic cables that connect a compute node, i.e., a building block containing the compute node, to an OCS switch via, for example, the optical link 400 of FIG.

The 4x4x4 building blocks may be formed by connecting four of the subblocks 600 to each other along the z dimension. For example, the compute node 622 of each compute tray 620A-620A may be connected via an internal link to one or two compute nodes of the corresponding compute tray on the other subblock 600 located in the z dimension. can. Computational nodes located at the ends of each z-dimensional segment can include external links 640 connected to OCS switches, as well as compute nodes located at the ends of the x-dimensional and y-dimensional segments. ..

FIG. 7 is a diagram showing an exemplary building block 700. The building block 700 includes four sub-blocks 710A-710D connected along the z dimension. Each sub-block 710A to 710D may be the same as the sub-block 600 of FIG. FIG. 7 shows some of the connections between the subblocks 710A-710D along the z dimension.

Specifically, the building block 700 includes internal links 730-733 between the corresponding compute nodes 716 of the compute trays 715 of subblocks 710A-710D along the z dimension. For example, the internal link 730 connects the segments of compute node 0 along the z dimension. Similarly, the internal link 731 connects the segments of the compute node 1 along the z dimension, the internal link 732 connects the segments of the compute node 8 along the z dimension, and the internal link 733 connects the segments of the compute node 8 along the z dimension. Connect the segments of the compute node 9. Although not shown, similar internal links connect the segments of compute nodes 2-7 and AF.

Also, the building block 700 includes an external link 720 located at the end of each segment along the z dimension. Although the illustration shows only the external link 720 of the segments 0, 1, 8 and 9, each of the other segments of compute nodes 2-7 and Af also includes the external link 720. External links can connect segments to OCS switches as well as external links located at the ends of x-dimensional and y-dimensional segments.

FIG. 8 is a diagram showing the OCS fabric topology 800 of the superpod. In this example, the OCS fabric topology contains 64 building blocks 805, i.e., a separate OCS switch for each segment along each dimension of the 4x4x4 building blocks of the superpod containing building blocks 0-63. .. The 4x4x4 building block 805 includes 16 segments along the x-dimension, 16 segments along the y-dimension, and 16 segments along the z-dimension. In this example, the OCS fabric topology includes 16 x-dimensional OCS switches, 16 y-dimensional OCS switches, and 16 z-dimensional OCS switches, for a total of 48 OCS switches for various workpieces. It can be configured to form a load cluster.

In the x-dimensional case, the OCS fabric topology 800 includes 16 OCS switches, including an OCS switch 810. Each building block 805 includes an external input link 811 and an external output link 812 connected to OCS switches 810 for each segment along the x-dimension. These external links 811 and 812 may be similar or similar to the optical link 400 of FIG.

For the y-dimension, the OCS fabric topology 800 includes 16 OCS switches, including an OCS switch 820. Each building block 805 includes an external input link 821 and an external output link 822 connected to the OCS switch 810 of each segment along the y dimension. These external links 821 and 822 may be similar to or similar to the optical link 400 of FIG.

For the z dimension, the OCS fabric topology 800 includes 16 OCS switches, including an OCS switch 830. Each building block 805 includes an external input link 821 and an external output link 822 connected to the OCS switch 810 of each segment along the y dimension. These external links 821 and 822 may be similar to or similar to the optical link 400 of FIG.

In another example, multiple segments may share the same OCS switch, for example, depending on the number of OCS radix and / or number of building blocks in the superpod. For example, if one OCS switch has a sufficient number of ports for all x-dimensional segments of all building blocks in the superpod, then all x-dimensional segments can be connected to the OCS switch. In another example, if one OCS switch has a sufficient number of ports, the two segments of each dimension can share the OCS switch. However, by connecting all the building block segments of the superpod to the same OCS switch, a single routing table can be used for data communication between the compute nodes in these segments. Failure response and diagnosis can also be simplified by using separate OCS switches for each segment or dimension. For example, if there is a problem with a particular segment or dimension of data communication, it is easier to identify a potentially failed OCS than if you used multiple OCS for a particular segment or dimension. Will.

FIG. 9 is a diagram showing components of an exemplary Superpod 900. For example, the superpod 900 may be one of the superpods of the processing system 130 of FIG. An exemplary Superpod 900 contains 64 4x4x4 building blocks 960 and uses these building blocks 960 to form a workload cluster for running computational workloads, such as machine learning workloads. can do. As mentioned above, each 4x4x4 building block 960 contains 32 compute nodes consisting of 4 compute nodes arranged along each of the 3 dimensions. For example, the building block 960 may be similar to or similar to the building block 310, workload cluster 320, or building block 700 described above.

An exemplary Superpod 900 includes an optical network 970, wherein the optical network 970 has 48 OCS switches 930 connected to the building blocks via 96 external links 931, 932 and 933 of each building block 960. Includes 940 and 950. Each external link may be an optical fiber link similar to or similar to the optical link 400 of FIG.

The optical network 970 includes an OCS switch for each segment of each dimension of each building block, similar to the OCS fabric topology 800 of FIG. In the x-dimensional case, the optical network 970 includes 16 OCS switches 950, one in each segment along the x-dimensional. The optical network 970 also includes, for each building block 960, an input external link and an output external link corresponding to each segment of the building block 960 along the x-dimension. These external links connect compute nodes on a segment to OCS switches 950 for that segment. Since each building block 960 contains 16 segments along the x-dimension, the optical network 970 has 32 external links to connect the x-dimensional segments of each building block 960 to the OCS switch 950 corresponding to that segment. 933 (ie, 16 input links and 16 output links) is included.

For the y-dimension, the optical network 970 includes 16 OCS switches 930, one for each segment along the y-dimension. The optical network 970 also includes, for each building block 960, an input external link and an output external link for each segment of the building block 960 along the y dimension. These external links connect compute nodes on a segment to OCS switches 930 for that segment. Since each building block 960 contains 16 segments along the y dimension, the optical network 970 has 32 external links to connect the y dimension segment of each building block 960 to the OCS switch 930 corresponding to that segment. Includes 931 (ie, 16 input links and 16 output links).

For the z dimension, the optical network 970 includes 16 OCS switches 932 arranged one for each segment along the y dimension. The optical network 970 also includes, for each building block 960, an input external link and an output external link corresponding to each segment of the building block 960 along the z dimension. These external links connect compute nodes on a segment to OCS switches 940 on that segment. Since each building block 960 contains 16 segments along the z dimension, the optical network 970 has 32 external links to connect the z dimension segment of each building block 960 to the OCS switch 940 corresponding to that segment. Includes 932 (ie, 16 input links and 16 output links).

The workload scheduler 910 can receive request data including the workload and data specifying a cluster of building blocks 960 requested to execute the workload. The request data may include workload priorities. Priority can be expressed in high, medium or low levels and can be expressed numerically, for example in the range 1-100 or another suitable range. For example, the workload scheduler 910 can receive request data from a user device or a cell scheduler, for example, the user device 110 or the cell scheduler 140 of FIG. As described above, the request data can specify the n-dimensional goal configuration of the compute node, for example, the goal configuration of the building block containing the compute node.

The workload scheduler 910 can select a set of building blocks 960 to generate a workload cluster that matches the goal configuration specified by the request data. For example, the workload scheduler 910 can identify a set of healthy building blocks available in the Superpod 900. As mentioned above, an available and healthy building block is a building block that is not running another workload or is not part of a running workload cluster and has not failed.

For example, the workload scheduler 910 can store and update state data indicating the state of each building block 960 in the superpod, for example in the form of a database. The availability of building block 960 can indicate whether building block 960 is assigned to a workload cluster. The healthy state of the building block 960 can indicate whether the building block is in operation or out of order. The workload scheduler 910 can identify building blocks 960 with an available state indicating that they are not assigned to a workload and a healthy state indicating that they are in operation. If building block 960 is assigned to a workload, for example, if it is used to create a workload cluster to run the workload, or if the health state changes from running to failing and vice versa. If so, the workload scheduler can update the state data of building block 960 accordingly.

The workload scheduler 910 can select a number of building blocks 960 that matches the number defined by the target configuration from the plurality of identified building blocks 960. If the request data specifies one or more types of compute nodes, the workload scheduler 910 can select from the identified building blocks 960 a building block containing the requested types of compute nodes. For example, if the request data specifies a 2x2 building block consisting of two TPU building blocks and two GPU building blocks, the workload scheduler 910 will have two available and healthy TPU building blocks and two. You can choose between two available and healthy GPU building blocks.

Further, the workload scheduler 910 can select the building block 960 based on the priority of each workload running in the superpod and the priority of the workload included in the request data. If the Superpod 900 does not have enough available and healthy building blocks to generate a workload cluster to run the requested workload, the workload scheduler 910 will have a lower priority than the requested workload. It is possible to determine if a workload with a degree is running within the Superpod 900. If a workload with a lower priority than the requested workload is running, the workload scheduler 910 requests a building block of the workload cluster to run one or more lower priority workloads. Can be reassigned to a workload cluster to run the workload to be executed. For example, the workload scheduler 910 terminates lower priority workloads, delays lower priority workloads, or reduces the size of the workload cluster for lower priority workloads. You can free up building blocks for higher priority workloads.

The workload scheduler 910 reassigns building blocks from one workload cluster to another by simply reconfiguring the optical network (eg, configuring an OCS switch as described below). Can be done. This connects this building block to the building block for running the higher priority workload, not the building block for running the lower priority workload. Similarly, if a building block to run a higher priority workload fails, the workload scheduler 910 works to run a lower priority workload by reconfiguring the optical network. Building blocks in a load cluster can be reassigned to a workload cluster to run higher priority workloads.

The workload scheduler 910 can generate configuration data 912 for each job and provide it to the OCS manager 920 of the superpod 900. The per-job configuration data 912 can specify the building block 960 and the configuration of the building blocks selected to run the workload. For example, if the configuration is a 2x2 configuration, it will include four spots for arranging building blocks. The configuration data for each job can specify that the selected building blocks 960 are to be placed in each of the four spots.

The configuration data 912 for each job can identify the selected building block 960 by using the logical identifier of each building block. For example, each building block 960 can include a unique logical identifier. In a particular embodiment, 64 building blocks 960 can be assigned numbers 0-63, which may be unique logical identifiers.

The OCS manager 920 configures the OCS switches 930, 940 and / or 950 with the per-job configuration data 912 to generate a workload cluster that matches the configuration specified by the per-job configuration data. Each OCS switch 930, 940 and 950 contains a routing table used when routing data between the physical ports of the OCS switch. For example, assume that the output external link of the x-dimensional segment of the first building block is connected to the input external link of the x-dimensional segment of the corresponding second building block. In this case, the routing table of the OCS switch 950 for the x-dimensional segment indicates that the data between the physical ports of the OCS switch to which these segments are connected is routed between these physical ports.

The OCS manager 920 can store port data that maps each port of each OCS switch 920, 930, and 940 to each logical port of each building block. The port data for each x-dimensional segment of the building block can specify which physical port of the OCS switch 950 the external input link is connected to and which physical port of the OCS switch 950 the external output link is connected to. can. The port data of each dimension of each building block 960 of the superpod 900 can include similar data.

The OCS manager 920 can generate a workload cluster to run the workload by configuring the routing table of the OCS switches 930, 940 and / or 950 with the port data. For example, suppose you want to connect the first building block to the second building block in a 2x1 configuration where the first building block is located to the left of the second building block along the x-dimension. The OCS manager 920 updates the routing table of the x-dimensional OCS switch 950 to route data between the x-dimensional segment of the first building block and the x-dimensional segment of the second building block. Since it is necessary to connect each x-dimensional segment of the building block, the OCS manager 920 can update the routing table of each OCS switch 950.

The OCS manager 920 can update the routing table of the OCS switch 950 for each x-dimensional segment. Specifically, the OCS manager 920 updates the routing table to connect the physical port of the OCS switch 950 to which the segment of the first building block is connected to the segment of the second building block. It can be mapped to the physical port of the OCS switch. Since each x-dimensional segment contains an input link and an output link, OCS Manager 920 connects the input link of the first building block to the output link of the second building block, and the output link of the first building block is the first. The routing table can be updated to connect to the input link of the 2 building blocks.

The OCS manager 920 can update the routing table by acquiring the current routing table from each OCS switch. In another example, the OCS manager 920 can update the appropriate routing table and send the updated routing table to the appropriate OCS switch. In another example, the OCS manager 920 can send update data specifying updates to the OCS switch, which can update the routing table according to the update data.

After configuring the OCS switch with the updated routing table, a workload cluster is created. The workload scheduler 910 can then cause the compute nodes in the workload cluster to execute the workload. For example, the workload scheduler 910 can provide and execute the workload to the compute nodes in the workload cluster.

After the workload finishes running, the workload scheduler 910 may update the state of each building block to return the state of each building block used to create the workload cluster to the available state. can. The workload scheduler 910 can also instruct OCS Manager 920 to disconnect between the building blocks used to create the workload cluster. Therefore, OCS Manager 920 can update the routing table to break the mapping between the physical ports of the OCS switch used to route data between building blocks.

In this way, by configuring the optical fabric topology with OCS switches to create a workload cluster to run the workload, the superpod can dynamically and safely host multiple workloads. Can be done. The workload scheduler 920 can generate a workload cluster immediately when a new workload is received and release the workload cluster immediately when the workload is processed. The routing between segments provided by the OCS switch provides better security between different workloads running on the same superpod than traditional supercomputers. For example, OCS switches use the air gap between workloads to physically separate the workload. Traditional supercomputers use software to separate workloads. Therefore, information is easily leaked.

FIG. 10 is a flow chart illustrating an exemplary process 1000 for creating a workload cluster and executing a computational workload using the workload cluster. The operation of process 1000 may be performed by a system that includes one or more data processing devices. For example, the operation of process 1000 may be executed by the processing system 130 of FIG.

The system receives request data specifying the cluster of compute nodes requested (1010). The request data may be received from the user device. The request data can include computational workloads and data that specify the n-dimensional goal configuration of the compute node. For example, the request data can specify an n-dimensional goal configuration of a building block containing compute nodes.

In some implementations, the request data can specify the type of compute node for generating building blocks. Superpods can contain building blocks consisting of different types of compute nodes. For example, a superpod can include 90 building blocks, each containing a 4x4x4 TPU, and 10 dedicated building blocks, each containing a 2x1 dedicated compute node. The request data can specify the number of building blocks consisting of each type of compute node and the configuration of these building blocks.

The system selects a subset of building blocks to generate the requested cluster from a superpod containing a set of building blocks (1020). As described above, a superpod can include a set of building blocks containing compute nodes in a three-dimensional configuration, eg, a 4x4x4 configuration. The system can select a quantity of building blocks that matches the quantity defined by the target configuration. As mentioned above, the system can select building blocks that are available to generate healthy and requested clusters.

The subset of building blocks may be the appropriate subset of building blocks. A suitable subset is a subset that does not include all members of a set of building blocks. For example, fewer building blocks than all building blocks may be needed to generate workload clusters that match the compute nodes in the target configuration.

The system creates a workload cluster containing a subset of the selected compute nodes (1030). This workload cluster can contain building blocks with configurations that match the target configuration specified by the request data. For example, if the request data specifies a compute node with a 4x8x4 configuration, the workload cluster can include two building blocks arranged as in the workload cluster 330 of FIG.

To generate a workload cluster, the system can configure routing data for each dimension of the workload cluster. For example, as mentioned above, a superpod can include an optical network that includes one or more OCS switches for each dimension of the building block. For a dimension, the routing data can include a routing table for one or more OCS switches. As mentioned above with reference to FIG. 9, the OCS switch routing table may be configured to route data between the appropriate segments of compute nodes along each dimension.

The system causes a compute node in the workload cluster to execute the compute workload (1040). For example, the system can provide a compute workload to compute nodes in a workload cluster. While the computational workload is running, the configured OCS switch can route data between the building blocks of the workload cluster. The configured OCS switch can route data between the compute nodes of the building block as if the compute nodes were physically connected even if they were not physically connected in the target configuration.

For example, a compute node in each segment of a dimension sends data to different building blocks through an OCS switch so that it is physically connected to a single physical segment with other compute nodes in the segment in different building blocks. It can communicate with other compute nodes in the segment within. A workload cluster in this configuration differs from a packet switching network because it can provide a true end-to-end optical path without packet switching or buffering along the way. In the case of packet switching, the delay is high because the switch has to receive the packet, buffer it, and send it again on another port.

After the computational workload finishes running, the system updates the state of the building blocks, for example, to the available state to run other workloads, and does not route data between the building blocks of the workload cluster. You can free the building blocks by updating the data routing so that.

FIG. 11 is a flow chart illustrating an exemplary process 1100 for reconfiguring an optical network to replace a failed building block. The operation of process 1100 may be performed by a system that includes one or more data processing devices. For example, the operation of process 1100 may be performed by the processing system 130 of FIG.

The system causes a compute node in the workload cluster to execute the compute workload (1110). For example, the system can create a workload cluster and have the compute node run the compute workload according to process 1000 in FIG.

The system receives data indicating that the building block of the workload cluster has failed (1120). For example, if one or more compute nodes in a building block fail, another element, such as a monitoring element, may determine that the building block has failed and send data to the system indicating that the building block has failed. can.

The system identifies available building blocks (1130). For example, the system can identify available and healthy building blocks in a superpod similar to other building blocks in a workload cluster. The system can identify available and healthy building blocks, for example, based on the building block state data stored by the system.

The system replaces the failed building block with the identified available building block (1140). The system can update the data routing of one or more OCS switches in the optical network connecting the building blocks to replace the failed building block with the identified available building blocks. For example, the system can update the routing table of one or more OCS switches to disconnect between other building blocks in the workload cluster and the failed building block. The system can also update the routing table of one or more OCS switches to connect the identified building block to other building blocks in the workload cluster.

The system can logically place the identified building block in the logical spot of the failed building block spot. As mentioned above, the OCS switch's routing table can map the physical port of an OCS switch connected to a segment of one building block to the physical port of an OCS switch connected to a segment of the corresponding building block. can. In this case, the system can make the replacement by updating the mapping with the corresponding segment of the identified available building block rather than the failed building block.

For example, the input external link for a particular x-dimensional segment of a failed building block is connected to the first port of the OCS switch, and the input external link for the corresponding x-dimensional segment of the identified available building block is the OCS switch. Suppose you are connected to a second port. Further assume that the routing table maps the first port to the third port of the OCS switch connected to the corresponding x-dimensional segment of another building block. To make the substitution, the system can update the routing table mapping by mapping the second port to the third port instead of mapping the first port to the third port. .. The system can do the same for each segment of the failed building block.

Embodiments of the subject matter and operation described herein are digital electronic circuits, computer software, firmware, hardware, or a combination thereof, including the structures and their equivalents disclosed herein. It may be realized by. The embodiments of the subject described herein are one or more computer programs, i.e., computer programs encoded on a computer storage medium and executed by or controlling the operation of the data processing apparatus. It may be implemented as one or more modules of instructions. Alternatively or additionally, the program instruction is generated by a propagating signal, eg, a machine, that is artificially generated to encode the information sent to the appropriate receiver for execution by the data processor. It may be encoded into an electrical signal, an optical signal, or an electromagnetic signal. The computer storage medium may be, or may be contained in, a computer readable storage device, a computer readable storage board, a random access memory array or device or a serial access memory array or device, or a combination thereof. good. Further, the computer storage medium may be a source or installation destination of computer program instructions encoded in an artificially generated propagating signal, although it is not a propagating signal. The computer storage medium may also be, or may be contained in, one or more separate physical elements or media (eg, multiple CDs, discs, or other storage devices).

The operations described herein may be implemented as operations performed by a data processing device on data stored in one or more computer-readable storage devices or received from other sources.

The term "data processing appliance" includes all types of equipment, appliances, and machines for processing data, including, for example, programmable processors, computers, system-on-chips, or combinations thereof. The device can include dedicated logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). In addition to the hardware, the device also contains code that creates the execution environment for the computer program, such as processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or a combination thereof. Can include constituent code. Equipment and execution environments can implement a variety of different computing model infrastructures, such as web services, distributed computing infrastructure, and grid computing infrastructure.

Computer programs (also known as programs, software, software applications, scripts, or code) may be written in any programming language, including compiled or interpreted languages, declarative or procedural languages, and are stand-alone. It may be deployed in any form, including programs, modules, components, subroutines, objects, or other units that can be used appropriately in a computing environment. The computer program may, but does not necessarily, support the files in the file system. A program is a portion of a file that holds other programs or data (eg, one or more scripts stored in a markup language document), a single file dedicated to that program, or multiple collaborative files (eg,). It may be stored in one or more modules, subprograms, or files that store parts of the code). The computer program may be deployed and executed on one computer, or a plurality of computers arranged in one place or distributed in a plurality of places and interconnected by a communication network.

The processes and logical flows described herein may be performed by one or more programmable processors. One or more programmable processors execute one or more computer programs by processing input data and producing outputs to perform operations. Further, the process and the logic flow may be executed by a dedicated logic circuit, for example, FPGA (field programmable gate array) or ASIC (application specific integrated circuit), and the device may be implemented as a dedicated logic circuit.

Suitable processors for running computer programs include, for example, general purpose microprocessors, dedicated microprocessors, and one or more processors of any type of digital computer. Generally, the processor receives instructions and data from read-only memory, random access memory, or both. Essential elements of a computer include a processor for performing operations according to instructions and one or more memory devices for storing instructions and data. In general, a computer also includes or receives data from one or more mass storage devices for storing data, such as magnetic disks, optomagnetic disks or optical disks, or from one or more mass storage devices. , Or transfer data to one or more mass storage devices, or both, and are operably coupled to one or more mass storage devices. However, the computer does not need to have such a device. In addition, the computer may be another device, such as a mobile phone, personal digital assistant (PDA), mobile audio player or video player, game console, Global Positioning System (GPS) receiver, or portable storage device (eg, USB flash drive). ) Can be incorporated. Suitable devices for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices, such as magnetic disks such as internal hard disks or removable disks, magneto-optical disks, CD ROMs and DVDs. Includes all non-volatile memory, media, and memory devices, including ROM disks. The processor and memory may be supplemented by dedicated logic circuits or may be incorporated into dedicated logic circuits.

To provide interaction with the user, embodiments of the subject matter described herein are display devices for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and the user. May be implemented on a computer that includes a keyboard and pointing device that can provide input to the computer, such as a mouse or trackball. Other types of devices can also be used to provide user interaction. The feedback provided to the user may be, for example, any sensory feedback, such as visual feedback, auditory feedback, or tactile feedback. The input received from the user may include acoustic input, voice input, or tactile input. Further, the computer sends a document to the device used by the user and receives the document from the device, for example, in response to a request received from a web browser, a web browser on the user's client device. You can interact with the user by sending a web page to.

The embodiments of the subject matter described herein may be implemented in a computing system that includes back-end components, such as a data server, or may be implemented in a computing system that includes middleware components, such as an application server. , Or a computing system that includes front-end components, such as a graphical user interface or web browser that allows the user to interact with the implementations of the subject matter described herein, or the back-end components, middleware components or front described above. It may be implemented on a client computer with any combination of end components. The components of the system can be interconnected by any digital data communication medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), internetworks (eg, the Internet), and peer-to-peer networks (eg, ad hoc peer-to-peer networks).

The computing system can include clients and servers. Clients and servers are generally remote from each other and generally interact over a communication network. The client-server relationship runs on each computer and is formed by computer programs that have a client-server relationship with each other. In some embodiments, the server sends data (eg, an HTML page) to the client device (eg, to display data to a user interacting with the client device and receive user input). The server can receive data generated by the client device (eg, the result of user interaction) from the client device.

Although the present specification includes many specific implementation details, these implementation details should not be construed as limitations to the scope or claims of the invention, but rather are specific to a particular embodiment of a particular invention. Should be interpreted as an explanation of the characteristics of. The particular features described herein with respect to separate embodiments may be implemented in any combination in a single embodiment. Conversely, the various features described for a single embodiment may be implemented separately or in any suitable subcombination in multiple embodiments. Furthermore, even if a characteristic combination works and it is claimed to work with that combination, one or more features can be removed from the claimed combination as needed, and the claimed combination is divided into sub-combinations. be able to.

Similarly, the drawings show the operations in a particular order, which requires or all of these operations to be performed in the specific order shown or in sequence to achieve the desired result. Should not be understood as having to perform the actions of. Depending on the particular situation, multitasking and parallel processing may be advantageous. Moreover, the separation of various system elements in the embodiments described above should not be understood as requiring such separation in all embodiments, and the program elements and systems described are generally single. It should be understood that it may be integrated into one software product or packaged into multiple software products.

Therefore, specific embodiments of the present invention have been described. Other embodiments are included in the claims. In some cases, the actions described in the claims can be performed in different orders to achieve the desired result. Moreover, the processes shown in the accompanying drawings do not necessarily have to be performed in the particular order or sequence shown to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous.

Claims (20)

A method performed by one or more data processing devices, wherein the method is performed.
The request data comprises receiving request data that specifies the compute node requested to perform the compute workload, which specifies the n (n is greater than or equal to 2) dimensional goal configuration of the compute node. ,
The building that, when combined, matches the n-dimensional target configuration specified by the request data, from a superpod containing a set of building blocks, each containing a compute node with an m (m is 2 or more) dimension configuration. A set of building blocks comprising selecting a subset of blocks is connected to an optical network containing one or more optical circuit switches for each of the n dimensions.
Including creating a workload cluster of compute nodes containing a subset of the building blocks
The above-mentioned generation is
For each dimension of the workload cluster, the routing data corresponding to each dimension of the workload cluster comprises configuring the routing data of each of the one or more optical circuit switches for that dimension. Specifying how to route the data of the compute workload between the compute nodes along the dimension of the workload cluster.
A method comprising causing the compute node of the workload cluster to execute the compute workload. The request data specifies different types of compute nodes.
Choosing a subset of the building block comprises selecting a building block containing one or more compute nodes of the specified type for each type of compute node specified by the request data. Item 1. The method according to Item 1. The method of claim 1, wherein the routing data for each dimension of the superpod comprises an optical circuit switch routing table for one of the one or more optical circuit switches. The method of claim 1, wherein the optical network comprises, for each of the n dimensions, one or more optical circuit switches of the optical network that route data between computing nodes along the dimension. Each building block contains multiple segments of compute nodes along each dimension of the building block.
4. The optical network of claim 4, wherein the optical network includes an optical circuit switch of the optical network that routes data between the compute node segments corresponding to each building block in the workload cluster for each segment of each dimension. Method. The method of claim 1, wherein each building block comprises one of a three-dimensional torus-like calculation node or a mesh-like calculation node. The superpod contains multiple workload clusters.
The method of claim 1, wherein each workload cluster comprises a different subset of said building blocks and performs a different workload than the other workload clusters. Receiving data indicating that a particular building block in the workload cluster has failed,
The method of claim 1, further comprising replacing the particular building block with an available building block. Replacing the particular building block with an available building block
Updating the data routing of one or more optical circuit switches in the optical network to stop the data routing between the particular building block of the workload cluster and one or more other building blocks. ,
Update the data routing of the one or more optical circuit switches in the optical network to route data between the available building blocks of the workload cluster and the one or more other building blocks. The method according to claim 8, including the above. When combined, selecting a subset of the building blocks that matches the n-dimensional goal configuration specified by the request data
Determining that the n-dimensional configuration specified by the request data requires a first quantity of building blocks that exceeds the available and sound second quantity of building blocks in the superpod.
In response to the determination that the n-dimensional configuration specified by the request data requires the first quantity of building blocks that exceeds the available and sound second quantity of building blocks in the superpod.
Identifying one or more second computational workloads that have a lower priority than the computational workload and are being executed by other building blocks of the superpod, and
Including reallocating one or more building blocks of the one or more second compute workloads to the workload cluster for the compute workload.
Creating the workload cluster of the compute node containing a subset of the building blocks includes the one or more building blocks of the one or more second compute workloads in the building block subset. The method according to claim 1. Creating said workload clusters of compute nodes containing a subset of said building blocks means that for each dimension of said workload cluster, said said one or more building blocks of said one or more second computational workloads. The one or more optical circuit switches for the dimension so that each building block communicates with the other building blocks of the workload cluster rather than the building blocks of the one or more second computational workloads. 10. The method of claim 10, comprising reconstructing each routing data of. It ’s a system,
Data processing equipment,
Equipped with a computer storage medium that encodes a computer program
The program includes a data processing device instruction that causes the data processing device to perform the following operations when executed by the data processing device.
The request data comprises receiving request data that specifies the compute node requested to perform the compute workload, which specifies the n (n is greater than or equal to 2) dimensional goal configuration of the compute node. ,
The building that, when combined, matches the n-dimensional target configuration specified by the request data, from a superpod containing a set of building blocks, each containing a compute node with an m (m is 2 or more) dimension configuration. A set of building blocks comprising selecting a subset of blocks is connected to an optical network containing one or more optical circuit switches for each of the n dimensions.
Including creating a workload cluster of compute nodes containing a subset of the building blocks
The above-mentioned generation is
The routing data corresponding to each dimension of the workload cluster comprises configuring the routing data of each of the one or more optical circuit switches for that dimension for each dimension of the workload cluster. Specifying how to route the data for the compute workload between the compute nodes along the dimension of the workload cluster.
A system comprising having the compute node of the workload cluster execute the compute workload. The request data specifies different types of compute nodes.
Choosing a subset of the building block comprises selecting a building block containing one or more compute nodes of the specified type for each type of compute node specified by the request data. Item 12. The system according to item 12. 12. The system of claim 12, wherein the routing data for each dimension of the superpod comprises an optical circuit switch routing table for one of the one or more optical circuit switches. 12. The system of claim 12, wherein the optical network comprises one or more optical circuit switches of the optical network that route data between computing nodes along the n-dimensions for each dimension. Each building block contains multiple segments of compute nodes along each dimension of the building block.
15. The optical network of claim 15, wherein the optical network includes an optical circuit switch of the optical network that routes data between the compute node segments corresponding to each building block in the workload cluster for each segment of each dimension. system. 12. The system of claim 12, wherein each building block comprises one of a three-dimensional torus-like compute node or a mesh-like compute node. The superpod contains multiple workload clusters.
12. The system of claim 12, wherein each workload cluster comprises a different subset of said building blocks and performs a different workload than the other workload clusters. The above operation is
Receiving data indicating that a particular building block in the workload cluster has failed,
12. The system of claim 12, comprising replacing the particular building block with an available building block. A non-temporary computer storage medium in which a computer program is encoded, wherein the program comprises a data processing device instruction that causes the data processing device to perform the following operations when executed by the data processing device.
The request data comprises receiving request data that specifies the compute node requested to perform the compute workload, which specifies the n (n is greater than or equal to 2) dimensional goal configuration of the compute node. ,
The building that, when combined, matches the n-dimensional target configuration specified by the request data, from a superpod containing a set of building blocks, each containing a compute node with an m (m is 2 or more) dimension configuration. A set of building blocks comprising selecting a subset of blocks is connected to an optical network containing one or more optical circuit switches for each of the n dimensions.
Including creating a workload cluster of compute nodes containing a subset of the building blocks
The above-mentioned generation is
For each dimension of the workload cluster, the routing data corresponding to each dimension of the workload cluster comprises configuring the routing data of each of the one or more optical circuit switches for that dimension. Specifying how to route the data of the compute workload between the compute nodes along the dimension of the workload cluster.
A non-temporary computer storage medium comprising having the compute node of the workload cluster execute the compute workload.

Images