DE102018005850A1 - Technologies for automated network congestion management - Google Patents

Abstract

A resource allocation server for bandwidth allocation, the resource manager server comprising one or more processors and one or more storage devices storing a plurality of instructions that, when executed by the one or more processors, cause the resource manager server to discover a topology of a plurality of sleds coupled to a layer of switches communicatively coupled to the resource manager server, constructing a model of network connectivity between the plurality of sleds and the layer of switches based on the topology, determining an overubscription of a network based on the model of the network Network connectivity, where the overubscription is based on available bandwidth for the layer of switches and a maximum bandwidth of the plurality of carriages, determining a bandwidth limit for each of the plurality of carriages based on the overubscription and programming of each sled of the plurality of sledges with the appropriate bandwidth limit.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of US Provisional Patent Application No. 201741030632, filed on Aug. 30, 2017, and US Provisional Application No. 62 / 584,401, filed on Nov. 10, 2017.

BACKGROUND

Data centers and other large computer networks typically include multiple layers of switches. For example, servers may be installed in racks and each server in a rack may be connected to a top-of-rack switch. Several top-of-rack switches can be connected to an upstream switch and so on. Therefore, communication between servers or other nodes in different racks may require traversing multiple switch layers. Traversing each layer of switches can introduce queuing latency.

list of figures

• 1 ist eine vereinfachte Darstellung von wenigstens einer Ausführungsform eines Datenzentrums zum Ausführen von Arbeitslasten mit disaggregierten Ressourcen;
• 2 ist eine vereinfachte Darstellung von wenigstens einer Ausführungsform eines Pods des Datenzentrums aus 1;
• 3 ist eine perspektivische Ansicht von wenigstens einer Ausführungsform eines Racks, das im Pod aus 2 eingeschlossen sein kann;
• 4 ist eine seitliche Höhenansicht des Racks aus 3;
• 5 ist eine perspektivische Ansicht des Racks aus 3 mit einem darin montierten Schlitten;
• 6 ist ein vereinfachtes Blockschaltbild von wenigstens einer Ausführungsform einer Oberseite des Schlittens aus 5;
• 7 ist ein vereinfachtes Blockschaltbild von wenigstens einer Ausführungsform einer Unterseite des Schlittens aus 6;
• 8 ist ein vereinfachtes Blockschaltbild von wenigstens einer Ausführungsform eines Rechenschlittens, der im Datenzentrum aus 1 verwendbar ist;
• 9 ist eine perspektivische Draufsicht auf wenigstens eine Ausführungsform des Rechenschlittens aus 8;
• 10 ist ein vereinfachtes Blockschaltbild von wenigstens einer Ausführungsform eines Beschleunigerschlittens, der im Datenzentrum aus 1 verwendbar ist;
• 11 ist eine perspektivische Draufsicht auf wenigstens eine Ausführungsform des Beschleunigerschlittens aus 10;
• 12 ist ein vereinfachtes Blockschaltbild von wenigstens einer Ausführungsform eines Speichereinrichtungsschlittens, der im Datenzentrum aus 1 verwendbar ist;
• 13 ist eine perspektivische Draufsicht auf wenigstens eine Ausführungsform des Speichereinrichtungsschlittens aus 12;
• 14 ist ein vereinfachtes Blockschaltbild von wenigstens einer Ausführungsform eines Speicherschlittens, der im Datenzentrum aus 1 verwendbar ist;
• 15 ist ein vereinfachtes Blockschaltbild eines Systems, das innerhalb des Datenzentrums aus 1 eingerichtet werden kann, um Arbeitslasten mit verwalteten Knoten, die aus disaggregierten Ressourcen bestehen, auszuführen.
• 16 ist ein vereinfachtes Blockschaltbild von wenigstens einer Ausführungsform eines Systems zur Bandbreitenzuweisung;
• 17 ist ein vereinfachtes Blockschaltbild von wenigstens einer Ausführungsform einer Rechenvorrichtung aus 16;
• 18 ist ein vereinfachtes Blockschaltbild von wenigstens einer Ausführungsform einer Umgebung des Ressourcenmanagers aus 16 und 17;
• 19 ist ein vereinfachtes Blockschaltbild von wenigstens einer Ausführungsform einer Umgebung eines Schlittens aus 16 und 17;
• 20 ist ein vereinfachtes Flussdiagramm von wenigstens einer Ausführungsform eines Verfahrens zur Bandbreitenzuweisung, das vom Ressourcenmanagerserver aus 16-18 ausgeführt werden kann; und
• 21 ist ein vereinfachtes Flussdiagramm von wenigstens einer Ausführungsform eines Verfahrens zur Bandbreitenzuweisung, das vom Schlitten aus 16-17 und 19 ausgeführt werden kann.

The concepts described herein are presented by way of example and not by way of limitation in the accompanying figures. For the sake of simplicity and clarity of illustration, the elements shown in the figures are not necessarily drawn to scale. When appropriate, reference numerals in the figures are repeated to indicate corresponding or analogous elements.

• 1 Figure 10 is a simplified illustration of at least one embodiment of a data center for performing workloads with disaggregated resources;
• 2 Figure 5 is a simplified illustration of at least one embodiment of a data center pod 1 ;
• 3 FIG. 12 is a perspective view of at least one embodiment of a rack made in the pod. FIG 2 can be included;
• 4 is a side elevation view of the rack off 3 ;
• 5 is a perspective view of the rack 3 with a carriage mounted therein;
• 6 is a simplified block diagram of at least one embodiment of a top of the carriage 5 ;
• 7 is a simplified block diagram of at least one embodiment of a bottom of the carriage 6 ;
• 8th Figure 4 is a simplified block diagram of at least one embodiment of a compute slide deployed in the data center 1 is usable;
• 9 is a perspective top view of at least one embodiment of the calculation slide 8th ;
• 10 Figure 4 is a simplified block diagram of at least one embodiment of an accelerator carriage deployed in the data center 1 is usable;
• 11 is a perspective top view of at least one embodiment of the accelerator carriage 10 ;
• 12 Figure 4 is a simplified block diagram of at least one embodiment of a storage device sled located in the data center 1 is usable;
• 13 FIG. 12 is a top perspective view of at least one embodiment of the storage device carriage. FIG 12 ;
• 14 FIG. 5 is a simplified block diagram of at least one embodiment of a storage sled that is located in the data center 1 is usable;
• 15 is a simplified block diagram of a system operating within the data center 1 can be set up to handle workloads with managed nodes that consist of disaggregated resources.
• 16 FIG. 10 is a simplified block diagram of at least one embodiment of a bandwidth allocation system; FIG.
• 17 Figure 4 is a simplified block diagram of at least one embodiment of a computing device 16 ;
• 18 FIG. 4 is a simplified block diagram of at least one embodiment of a resource manager environment 16 and 17 ;
• 19 FIG. 4 is a simplified block diagram of at least one embodiment of an environment of a carriage 16 and 17 ;
• 20 FIG. 10 is a simplified flowchart of at least one embodiment of a bandwidth allocation method that originates from the resource manager server 16-18 can be executed; and
• 21 FIG. 10 is a simplified flowchart of at least one embodiment of a method of bandwidth allocation from the carriage 16-17 and 19 can be executed.

DETAILED DESCRIPTION OF THE DRAWINGS

Although the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will be described in detail herein. However, it should be understood that it is not intended to limit the concepts of the present disclosure to the particular forms disclosed, but on the contrary, it is intended to cover all modifications, equivalents and alternatives consistent with the present disclosure and the appended claims ,

References in the specification to "a single embodiment," "an embodiment," "an illustrative embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily requires the particular feature, the particular structure or the specific property. Moreover, such terms do not necessarily refer to the same embodiment. Further, when describing a particular feature, structure, or characteristic in connection with an embodiment, it is anticipated that it will be within the knowledge of those skilled in the art to associate such feature, structure, or trait to effect with other embodiments, whether they are now explicitly described or not. In addition, it should be noted that items included in a list are in the form of "at least one of A, B and C" (A); (B); (C); (A and B); (A and C); (B and C) or (A, B and C) may mean. Similarly, elements listed in the form of "at least one of A, B or C" may be (A); (B); (C); (A and B); (A and C); (B and C) or (A, B and C).

The disclosed embodiments may in some cases be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions that are transmitted or stored on a transitory or non-transitory machine-readable (eg, computer readable) storage medium that may be read and executed by one or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a machine-readable form (eg, a volatile or nonvolatile memory, a storage disk, or other media device). ,

In the drawings, some structural or procedural features may be displayed in specific arrangements and / or orders. It should be noted, however, that such specific arrangements and / or orders may not be required. Rather, in some embodiments, such features may be arranged in a different manner and / or order than shown in the illustrative figures. In addition, inclusion of a structural or procedural feature in a particular figure is not intended to imply that such a feature is required in all embodiments, and may not be included in some embodiments or may be combined with other features.

With reference to 1 has a data center 100 in which disaggregated resources can cooperatively perform one or more workloads (for example, applications on behalf of customers), multiple pods 110 . 120 . 130 . 140 each having one or more rows of racks. As described in greater detail herein, each rack receives multiple carriages, each of which may be implemented as a computing device, such as a server, that is primarily equipped with a particular type of resource (eg, storage devices, data storage devices, accelerator devices, general-purpose processors). In the illustrative embodiment, the carriages are in each pod 110 . 120 . 130 . 140 with multiple pod switches (for example, switches that route data communications to and from sleds within the pod). The pod switches in turn are with spine switches 150 connected to the communications between pods (such as pods 110 . 120 . 130 . 140 ) in the data center 100 switch. In some embodiments, the carriages may be connected to a fabric using Intel Omni-Path technology. As described in greater detail herein, resources within sleds may be in the data center 100 a group (referred to herein as a "managed node") that contains resources from one or more other carriages to be used collectively in the execution of a workload. The workload can be performed as if the resources associated with the managed node were located on the same sled. The resources in a managed node may even belong to sleds belonging to different racks, and even to different pods 110 . 120 . 130 . 140 , Some resources of a single sled may be assigned to a managed node while other resources of the same sled are assigned to another managed node (e.g. one processor assigned to a managed node and another processor of the same carriage assigned to another managed node). By disaggregating resources into sleds that are predominantly of a single resource type (e.g., compute sleds that primarily include compute resources, storage sleds that primarily contain storage resources) and selectively allocate and disassociate the disaggregated resources to form a managed node, which is assigned to execute a workload, represents the data center 100 provide more efficient resource utilization over typical data centers consisting of hyper-converged servers that contain compute, storage, storage, and possibly additional resources. Thus, the data center 100 provide greater performance (e.g., throughput, operations per second, latency, etc.) than a typical data center having the same number of resources.

With reference to 2 the pod has 110 in the illustrative embodiment, a set of rows 200 . 210 . 220 . 230 from racks 240 on. Every rack 240 may accommodate a plurality of carriages (eg, sixteen carriages) and provide power and data connections to the received carriages, as described in greater detail herein. In the illustrative embodiment, the racks are in each row 200 . 210 . 220 . 230 with several pod switches 250 . 260 connected. The pod switch 250 has a set of ports 252 with which the carriages of the racks of the pod 110 connected, and another set of ports 254 that the pod 110 with the spine switches 150 connect to connectivity to other pods in the data center 100 provide. Similarly, the pod switch points 260 a set of ports 262 with which the carriages of the racks of the pod 110 connected, and a set of ports 264 that the pod 110 with the spine switches 150 connect, up. Thus, the use of the pair of switches represents 250 . 260 the pod 110 a certain redundancy ready. For example, if one of the switches 250 . 260 fails, the sledges in the pod 110 continue the data communication with the rest of the data center 100 (eg sleds of other pods) through the other switch 250 . 260 maintained. Furthermore, the switches can 150 . 250 . 260 in the illustrative embodiment may be implemented as dual-mode optical switches carrying both Ethernet protocol communications carrying Internet Protocol (IP) packets and second high-performance link layer protocol communications (e.g. Intel's Omni-Path architecture, Infiniband) via optical signaling media of an optical fabric.

It should be noted that each of the other pods 120 . 130 . 140 (as well as any additional data center pods 100 ) can be similarly structured as the pod 110 who in 2 shown and referred to, and may have similar components to it (eg, each pod may have rows of racks holding a plurality of carriages as described above). Although two pod switches 250 . 260 In addition, it should be understood that in other embodiments, each pod 110 . 120 . 130 . 140 can be connected to a different number of pod switches (for example, for even more failover capacity).

With reference to 3-5 has each illustrative rack 240 of the data center 100 two elongated support posts 302 . 304 on which are arranged vertically. For example, the elongated support posts 302 . 304 from a floor of the data center 100 extend upward when they are provided. The rack 240 also has one or more horizontal pairs 310 of elongated support arms 312 on (in 3 indicated by a dashed ellipse), which are designed to be a sled of the data center 100 support, as discussed below. An elongated support arm 312 the pair of elongated support arms 312 extends from the elongated support post 302 to the outside, and the other elongated support arm 312 extends from the elongated support post 304 outward.

In the illustrative embodiments, each carriage is the data center 100 designed as a subcarrier carrier sled. That is, each carriage has a subrackless circuit board substrate on which physical resources (eg, processors, memory, accelerators, memory devices, etc.) are mounted, as discussed in greater detail below. Thus, the rack 240 Designed to accommodate the subcarrier carriage. For example, each pair defines 310 of elongated support arms 312 a sled slot 320 of the rack 240 , which is designed to accommodate a corresponding subcarrier carriage. For this purpose, each illustrative elongated support arm 312 a circuit board guide 330 , which is designed to accommodate the subrackless printed circuit board substrate of the carriage. Each PCB guide 330 is at a top 332 the corresponding elongated support arm 312 attached or otherwise mounted on this. For example, in the illustrative embodiment, each circuit board guide 330 at a distal end of the corresponding elongate support arm 312 relative to the corresponding elongated support post 302 . 304 assembled. For clarity of the figures, not every circuit board guide 330 be referenced in each figure.

Each PCB guide 330 has an inner wall that has a PCB slot 380 defined to be the subrackless printed circuit board substrate of a carriage 400 to pick up when the sled 400 in the corresponding slide slot 320 of the rack 240 is included. For this purpose, a user (or robot), as in 4 1, the subrackless printed circuit board substrate of an illustrative subrackless carriage 400 on a sled slot 320 out. The user or robot can then move the subrackless circuit board substrate forward into the carriage slot 320 slide, leaving each side edge 414 the subrackless printed circuit substrate in a corresponding PCB slot 380 the PCB guides 330 of the couple 310 of elongated support arms 312 is recorded, the corresponding carriage slot 320 define how in 4 shown. With the availability of robotically accessible and robotic manipulable carriages that include disaggregated resources, each resource type can be independently updated at its own optimized refresh rate. Further, the carriages are designed to blind them with power and data communication cables in each rack 240 which improves their ability to be quickly removed, upgraded, reinstalled and / or replaced. Thus, the data center 100 in some embodiments, working without human involvement in the data center (e.g., performing workloads, performing maintenance and / or upgrades, etc.). In other embodiments, a human may perform one or more maintenance or upgrade operations in the data center 100 enable.

It should be noted that each PCB guide 330 is double-sided. That is, every circuit board guide 330 includes an inner wall that has a PCB slot 380 on each side of the PCB guide 330 Are defined. In this way, every circuit board guide 330 carry a subrackless PCB substrate on each side. Thus, a single additional elongated support post can be added to the rack 240 be added to the rack 240 Turn into a two-rack solution that has twice as much sled slots 320 as in 3 can show shown. The illustrative rack 240 has seven pairs 310 of elongated support arms 312 on, the corresponding seven sled slots 320 define, each designed to a corresponding slide 400 to pick up and carry, as discussed above. Of course, the rack can 240 in other embodiments, additional or fewer pairs 310 of elongated support arms 312 (ie additional or less sled slots 320 ) exhibit. It should be noted that, since the sled 400 subrack is the slide 400 can have a total height that is different from typical servers. Thus, in some embodiments, the height of each carriage slot 320 shorter than the height of a typical server (for example, shorter than a single rack unit "1U"). That is, the vertical distance between each pair 310 of elongated support arms 312 may be smaller than a standard rack unit "1U". Additionally, due to the relative reduction in the height of the carriage slots 320 the overall height of the rack 240 shorter in some embodiments than the height of conventional rack housings. For example, in some embodiments, each of the elongate support posts 302 . 304 have a length of six feet or less. In other embodiments, the rack 240 again have different dimensions. It should also be noted that the rack 240 has no walls, housing or the like. The rack 240 Rather, it's a rack-less rack that's open to the local environment. Of course, in some cases, an end plate may be attached to one of the elongate support posts 302 . 304 be appropriate in such situations where the rack 240 a row-end rack in the data center 100 formed.

In some embodiments, various interconnects may be through the elongated support posts 302 . 304 be routed up or down. To enable such routing, each has elongated support posts 302 . 304 an inner wall defining an inner chamber in which the interconnect may be disposed. The through the elongated support posts 302 . 304 routed interconnects may be embodied as any type of interconnect, including, but not limited to, data or communication interconnects to communication links to each carriage slot 320 provide power interconnections to power to each slide slot 320 and / or other types of interconnects.

The rack 240 In the illustrative embodiment, has a support platform on which a corresponding optical data connector (not shown) is mounted. Each optical data connector is with a corresponding carriage slot 320 and is designed to work with an optical data connector of a corresponding carriage 400 Fits when the carriage 400 in the corresponding slide slot 320 is recorded. In some embodiments, optical connections between components (e.g., carriages, racks, and switches) in the data center 100 made with a dummy optical connector. For example, a door on each cable can prevent dust from entering the fiber inside the cable contaminated. During the process of connecting to a dummy optical connector mechanism, the door is opened when the end of the cable enters the connector mechanism. Subsequently, the optical fiber within the cable enters a gel within the connector mechanism and the optical fiber of one cable comes into contact with the optical fiber of another cable within the gel within the connector mechanism.

The illustrative rack 240 also has a fan assembly 370 on top of that with the cross support arms of the rack 240 is coupled. The fan arrangement 370 has one or more rows of cooling fans 372 in a horizontal line between the elongated support posts 302 . 304 are aligned. In the illustrative embodiment, the fan assembly 370 a series of cooling fans 372 for each sled slot 320 of the rack 240 on. As discussed above, not everyone has a sled 400 in the illustrative embodiment, an on-board cooling system and thus provides the fan assembly 370 One cooling for everyone in the rack 240 recorded slide 400 ready. Every rack 240 In the illustrative embodiment, also includes a power supply connected to each carriage slot 320 is associated. Each power supply is on one of the elongated support arms 312 of the couple 310 of elongated support arms 312 attached to the corresponding carriage slot 320 define. For example, the rack 240 Have a power supply with each elongated support arm 312 coupled or attached to, extending from the elongated support post 302 extends. Each power supply has a power connector designed to mate with a power connector of the carriage 400 Fits when the carriage 400 in the corresponding slide slot 320 is included. In the illustrative embodiment, the carriage 400 no on-board power supply and thus supply those in the rack 240 provided power supplies the corresponding sledges 400 with power when on the rack 240 are mounted.

With reference to 6 is the sled 400 in the illustrative embodiment designed to be in a corresponding rack 240 of the data center 100 mounted as discussed above. In some embodiments, each carriage may 400 be optimized or otherwise designed to perform certain tasks, such as computational tasks, acceleration tasks, data storage tasks, etc. For example, the carriage 400 as a calculating slide 800 as described below with reference to 8-9 discussed as an accelerator sled 1000 as described below with reference to 10-11 discussed as a storage device sled 1200 as described below with reference to 12-13 discussed, or as a sled that is optimized or otherwise designed to perform other specialized tasks, such as a storage sled 1400 as described below with reference to 14 be discussed, executed.

As discussed above, the illustrative carriage 400 a subrackless printed circuit board substrate 602 carrying various physical resources (eg, electrical components) mounted thereon. It should be noted that the PCB substrate 602 "Subrackless" insofar is that the slide 400 has no enclosure or housing. Rather, the subrackless printed circuit board substrate 602 open to the local area. The subrackless printed circuit board substrate 602 may be formed of any material that can support the various electrical components mounted thereon. For example, the subrackless printed circuit substrate is 602 formed in an illustrative embodiment of a glass-reinforced FR-4-Epoxidlaminatmaterial. Of course, other materials may be used to support the subrackless circuit substrate 602 in other embodiments.

As discussed in more detail below, the subrackless printed circuit board substrate 602 several features that improve the thermal cooling properties of the various electrical components on the subrackless PCB substrate 602 are mounted. As discussed, the subrackless printed circuit board substrate has 602 no enclosure or enclosure on which the airflow over the electrical components of the carriage 400 can improve by reducing those structures that can inhibit airflow. Since the subrackless printed circuit board substrate 602 For example, if it is not located in a single enclosure or a single enclosure, there is no backplane (eg, a backplane of the chassis) to the subrackless circuit board substrate 602 that could inhibit the flow of air through the electrical components. In addition, the subrackless printed circuit board substrate has 602 a geometric shape, which is designed to the length of the air flow path via the electrical components, on the subrackless printed circuit board substrate 602 are mounted, reduce. For example, the illustrative subrackless printed circuit board substrate 602 a width 604 on, which is greater than a depth 606 of the subrackless printed circuit board substrate 602 is. In a specific embodiment, the subrackless printed circuit board substrate 602 for example one Width of about 21 inches and a depth of about 9 inches, compared to a typical server having a width of about 17 inches and a depth of about 39 inches. Thus, an air flow path 608 that is from a leading edge 610 of the subrackless printed circuit board substrate 602 to a trailing edge 612 extends a shorter distance relative to typical servers, which enhances the thermal cooling characteristics of the carriage 400 can improve. Furthermore, although in 6 does not illustrate the various physical resources on the subrackless PCB substrate 602 mounted, so that two substantially heat-generating electrical components do not shadow each other, as discussed in more detail below. That is, no two electrical components that produce significant heat during operation (ie, greater than a rated heat sufficient to adversely affect the cooling of another electrical component) are on the subrackless circuit substrate 602 along the direction of the air flow path 608 mounted linearly inline to each other (ie along a direction extending from the leading edge 610 to the trailing edge 612 of the subrackless printed circuit board substrate 602 extends).

As discussed above, the illustrative carriage 400 one or more physical resources 620 on that at a top 650 of the subrackless printed circuit board substrate 602 are mounted. Although two physical resources 620 in 6 It should be noted that the sled 400 in other embodiments, one, two or more physical resources 620 can have. The physical resources 620 may be embodied as any type of processor, controller, or other arithmetic circuit capable of performing various tasks such as calculating functions and / or controlling the functions of the carriage 400 depending on, for example, the type or intended functionality of the carriage 400 perform. For example, the physical resources 620 as discussed in more detail below as high performance processors in embodiments in which the carriage 400 as a computing slide, as accelerator coprocessors or circuits in embodiments in which the carriage 400 as an accelerator carriage, as a storage device controller in embodiments in which the carriage 400 as a storage device carriage, or as a set of storage devices in embodiments in which the carriage is embodied 400 be executed as a storage carriage, executed.

The sled 400 also has one or more additional physical resources 630 on top of that 650 of the subrackless printed circuit board substrate 602 are mounted. In the illustrative embodiment, the additional physical resources comprise a network interface controller (NIC), as discussed in greater detail below. Of course, the physical resources 630 depending on the type and functionality of the sled 400 additional or other electrical components, circuits and / or devices in other embodiments.

The physical resources 620 are communicative with the physical resources via an input / output (I / O) subsystem 622 630 coupled. The I / O subsystem 622 may be implemented as circuits and / or components to perform input / output operations on the physical resources 620 , the physical resources 630 and / or other components of the carriage 400 to enable. For example, the I / O subsystem 622 as memory controller hubs, input / output control hubs, integrated sensor hubs, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, optical fibers, printed circuit board traces, etc.) and or other components and subsystems, or otherwise having these to facilitate the input / output operations. In the illustrative embodiment, the I / O subsystem is 622 as a data bus at twice the data rate 4 (DDR4) or as a DDR5 data bus or otherwise.

In some embodiments, the carriage may 400 also a resource-to-resource interconnect 624 exhibit. The resource-to-resource interconnect 624 may be implemented as any type of communication interconnect capable of facilitating resource-to-resource communications. In the illustrative embodiment, the resource-to-resource interconnect is 624 as a high-speed point-to-point interconnect (e.g., faster than the I / O subsystem 622 ). For example, the resource-to-resource interconnect 624 as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI, UltraPath Interconnect) or other high-speed point-to-point interconnect dedicated to resource-to-resource communications.

The sled 400 also has a power connector 640 on, which is designed to be connected to a corresponding power connector of the rack 240 Fits when the carriage 400 in the corresponding rack 240 is mounted. The sled 400 receives power from a rack power supply 240 over the power connector 640 to the various electrical components of the carriage 400 to supply electricity. That is, the sled 400 has no local power supply (ie on-board power supply) to the electrical components of the carriage 400 to supply electricity. The exclusion of a local or on-board power supply allows the reduction of the total footprint of the subrackless PCB substrate 602 , which can increase the thermal cooling properties of the various electrical components on the subrackless printed circuit board substrate 602 are mounted as discussed above. In some embodiments, power over vias is directly under the processors 820 to the processors 820 provided (eg by the bottom 750 of the subrackless printed circuit board substrate 602 ), providing an increased thermal budget, additional current and / or voltage, and better voltage control compared to typical boards.

In some embodiments, the carriage may 400 also assembly features 642 , which are adapted to mate with a mounting arm or other structure of a robot to the arrangement of the carriage 600 in a rack 240 to enable through the robot. The mounting features 642 can be embodied as any type of physical structures that enable the robot, the slide 400 to grab, without the subrackless PCB substrate 602 or to damage the electrical components mounted on it. For example, the mounting features 642 in some embodiments, may be implemented as non-conductive pads attached to the subrackless printed circuit board substrate 602 are attached. In other embodiments, the mounting features may be implemented as brackets, brackets, or other similar structures that may be present on the subrackless circuit board substrate 602 are attached. The specific number, shape, size and / or composition of the assembly feature 642 may depend on the design of the robot, which is designed for the carriage 400 manage.

With reference to 7 shows the sled 400 in addition to the physical resources 630 that on the top 650 of the subrackless printed circuit board substrate 602 are mounted, also one or more storage devices 720 on that at a bottom 750 of the subrackless printed circuit board substrate 602 are mounted. That is, the subrackless printed circuit board substrate 602 is designed as a double-sided circuit board. The physical resources 620 are communicative via the I / O subsystem 622 with the storage devices 720 coupled. For example, the physical resources 620 and the storage devices 720 communicatively coupled by one or more vias extending through the subrackless printed circuit board substrate 602 extend. In some embodiments, each physical resource 620 communicatively with another set of one or more storage devices 720 be coupled. Alternatively, any physical resource 620 in other embodiments, communicatively with each storage device 720 be coupled.

The storage devices 720 can be implemented as any type of storage device, the data for the physical resources 620 during operation of the carriage 400 such as any type of volatile (eg dynamic random access memory (DRAM), etc.) or non-volatile memory. A volatile memory may be a storage medium that requires power to maintain the state of data being stored by the medium. Non-limiting examples of volatile memory may include various types of Random Access Memory (RAM), such as dynamic random access memory (DRAM) or static random access memory (SRAM). One particular type of DRAM that may be used in a memory module is synchronous dynamic random access memory (SDRAM). In certain embodiments, DRAM of a memory component may conform to a standard published by the JEDEC, such as JESD79F for DDR SDRAM, JESD79-2F for DDR2 SDRAM, JESD79-3F for DDR3 SDRAM, JESD79-4A for DDR4 SDRAM, JESD209 for Low Power DDR (LPDDR), JESD209-2 for LPDDR2, JESD209-3 for LPDDR3 and JESD209-4 for LPDDR4 (these standards are available at www.jedec.org). Such standards (and similar standards) may be referred to as DDR-based standards, and communication interfaces of the memory devices implementing such standards may be referred to as DDR-based interfaces.

In one embodiment, the memory device is a block addressable memory device, such as those based on NAND or NOR technologies. A memory device may also include next generation non-volatile devices such as Intel 3D XPoint ™ memory or other byte addressable non-volatile write-in-place memory devices. In one embodiment The memory device may include memory devices using chalcogenide glass, multi-threshold NAND flash memory, NOR flash memory, single or multi-level phase change memory (PCM), a resistive memory, nanowire memory, ferroelectric transistor random access memory (US Pat. FeTRAM, ferroelectric transistor random access memory), anti-ferroelectric memory, magnetoresistive random access memory (MRAM) incorporating memristor technology, resistive memory, including metal oxide base, oxygen vacancy base, and bridge-conductor random access memory (CB-RAM, Conductive Bridge Random Access Memory), or Spin-Transfer-Torque (STT) MRAM, a Spintronic Magnetic Transfer Memory-based device, a Magnetic Tunneling Junction (MTJ) -based device, a Domain Wall (DW), and Spin Orbit Transfer (SOT) -based device, a thyristor-based Memory device or a combination of any of the above or another memory or have. The storage device may refer to the die itself and / or to a storage product in a package. In some embodiments, the memory device may include a transistorless stackable crosspoint architecture in which memory cells are located at the intersection of word lines and bit lines and are individually addressable, and wherein bit storage is based on a change in bulk resistance.

With reference to 8th can the sled 400 in some embodiments as a calculation slide 800 be executed. The calculating slide 800 is optimized or otherwise designed to perform computational tasks. As discussed above, the calculation slide can 800 Of course, it also relies on other carriages, such as acceleration slides and / or storage sled, to perform such computational tasks. The calculating slide 800 has various physical resources (eg, electrical components) similar to the physical resources of the carriage 400 on that in 8th were identified using the same reference numerals. The description of such components as described above with respect to 6 and 7 has been provided, applies to the corresponding computational slide components 800 and is herein for clarity of description of the calculation slide 800 not repeated.

In the illustrative calculating slide 800 are the physical resources 620 as processors 820 executed. Although only two processors 820 in 8th should be noted that the calculation slide 800 in other embodiments additional processors 820 can have. Illustrative are the processors 820 as high performance processors 820 and can be designed to operate at a relatively high rating. Although the processors 820 Operating at rated powers greater than that of typical processors (operating at about 155-230W) will provide additional heat, enabling the improved thermal cooling characteristics of the sub-rackless printed circuit board substrate discussed above 602 the operation with higher performance. For example, the processors 820 in the illustrative embodiment to operate at a rated power of at least 250W. In some embodiments, the processors may 820 be designed to operate at a rated power of at least 350W.

In some embodiments, the calculation slide 800 also a processor-to-processor interconnect 842 exhibit. Similar to the resource-to-resource interconnect 624 of the carriage discussed above 400 may be the processor-to-processor interconnect 842 be executed as any type of communication interconnect, the communications through the processor-to-processor interconnect 842 can allow. In the illustrative embodiment, the processor-to-processor interconnect is 842 as a high-speed point-to-point interconnect (e.g., faster than the I / O subsystem 622 ). For example, the processor-to-processor interconnect 842 as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI, UltraPath Interconnect) or other high-speed point-to-point interconnect dedicated to processor-to-processor communications.

The calculating slide 800 also has a communication circuit 830 on. The illustrative communication circuit 830 has a network interface controller (NIC) 832, which may also be referred to as a host fabric interface (HFI). The NIC 832 can be considered any type of integrated circuit, discrete circuits, controller chips, chipsets, add-in boards, daughter cards, network interface cards or other devices that come from the computing slide 800 for connecting to another computing device (eg other carriages 400 ) can be used, be executed or have this otherwise. In In some embodiments, the NIC 832 as part of a system-on-a-chip (SoC) having one or more processors, or included in a multi-chip package that also includes one or more processors. In some embodiments, the NIC 832 a local processor (not shown) and / or a local memory (not shown), both for the NIC 832 are local. In such embodiments, the local processor of the NIC 832 to be able to do one or more of the functions of the processors 820 perform. Additionally or alternatively, the local memory of the NIC 832 in such embodiments, be integrated into one or more components of the rake at the board level, the socket level, the chip level, and / or other levels.

The communication circuit 830 is communicative with an optical data connector 834 coupled. The optical data connector 834 is designed to connect with a corresponding optical data connector of the rack 240 to match when the calculating slide 800 in the rack 240 is mounted. Illustratively, the optical data connector 834 a plurality of optical fibers received from a mating surface of the optical data connector 834 to an optical transceiver 836 to lead. The optical transceiver 836 is configured to convert incoming optical signals from the rack-side optical data connector into electrical signals and to convert electrical signals in outgoing optical signals to the rack-side optical data connector. Although in the illustrative embodiment, as a part of the optical data connector 834 shown, the optical transceiver 836 in other embodiments, a part of the communication circuit 830 form.

In some embodiments, the calculation slide 800 also an expansion connector 840 exhibit. In such embodiments, the expansion connector is 840 adapted to mate with a corresponding connector of a subrackless expansion board substrate to the computing slide 800 to provide additional physical resources. For example, the additional physical resources may be provided by the processors 820 during the operation of the calculation slide 800 be used. The subrackless extender board substrate may be substantially similar to the subrackless circuit board substrate discussed above 602 and may have various electrical components mounted thereto. The particular electrical components mounted on the subrackless printed circuit board substrate may depend on the intended functionality of the subrackless add-on board substrate. For example, the subrackless extender board substrate may provide additional computational resources, memory resources, and / or storage resources. Thus, the additional physical resources of the subrackless extender board substrate may include processors, memory devices, memory devices, and / or accelerator circuits, including, for example, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), security coprocessors, graphics processing units (GPUs, Graphics Processing Units), machine learning circuits or other specialized processors, controllers, devices and / or circuits are, but are not limited thereto.

With reference to 9 is an illustrative embodiment of the calculation slide 800 shown. As shown, the processors are 820 , the communication circuit 830 and the optical data connector 834 at the top 650 of the subrackless printed circuit board substrate 602 assembled. Any suitable mounting or mounting technology may be used to calculate the physical resources of the calculation slide 800 on the subrackless PCB substrate 602 to assemble. For example, the various physical resources may be mounted in corresponding sockets (eg, a processor socket), brackets, or brackets. In some cases, some of the electrical components may be directly connected to the subrackless circuit board substrate 602 be mounted by soldering or similar techniques.

As discussed above, the individual processors are 820 and the communication circuit 830 at the top 650 of the subrackless printed circuit board substrate 602 mounted such that no two heat-generating electrical components shade each other. In the illustrative embodiment, the processors are 820 and the communication circuit 830 in appropriate places on the top 650 of the subrackless printed circuit board substrate 602 so mounted that no two of these physical resources linearly inline with others along the direction of the airflow path 608 lie. It should be noted that although the optical data connector 834 inline with the communication circuit 830 is, the optical data connector 834 no or only nominal heat generated during operation.

The storage devices 720 of the calculation slide 800 are at the bottom 750 of the subrackless printed circuit board substrate 602 mounted as above with respect to the carriage 400 discussed. Although she is at the bottom 750 are mounted, the storage devices 720 communicative with the processors 820 that on the top 650 are arranged through the I / O subsystem 622 coupled. Because the subrackless PCB substrate 602 is designed as a double-sided circuit board, the memory devices 720 and the processors 820 communicatively coupled by one or more vias, connectors, or other mechanisms passing through the subrackless printed circuit board substrate 602 extend. Of course, every processor can 820 in some embodiments, communicatively with another set of one or more storage devices 720 be coupled. Alternatively, every processor 820 in other embodiments, communicatively with each storage device 720 be coupled. In some embodiments, the memory devices 720 on one or more memory mezzanines on the underside of the subrackless circuit board substrate 602 be mounted and can with a corresponding processor 820 be connected by a ball grid arrangement.

Each of the processors 820 has a heat sink 850 which is attached to it. Due to the mounting of the storage devices 720 on the bottom 750 of the subrackless printed circuit board substrate 602 (as well as the vertical distance of the carriages 400 in the corresponding rack 240 ) has the top 650 of the subrackless printed circuit board substrate 602 an additional "free" area or clearance on which the use of heat sinks 850 with a larger size compared to conventional heatsinks used in typical servers. Due to the improved thermal cooling properties of the subrackless printed circuit board substrate 602 In addition, none of the processor heat sink 850 attached cooling fans. That is, each of the heat sinks 850 is designed as a fanless heat sink.

With reference to 10 can the sled 400 in some embodiments as an accelerator carriage 1000 be executed. The accelerator carriage 1000 is optimized or otherwise designed to perform specialized computing tasks, such as machine learning, encryption, hashing, or other compute-intensive tasks. For example, in some embodiments, a calculation slide 800 during operation tasks on the accelerator carriage 1000 unload. The accelerator carriage 1000 has various components, the components of the carriage 400 and / or the calculation slide 800 similar to those in 10 be identified using the same reference numerals. The description of such components as described above with respect to 6 . 7 and 8th has been provided, applies to the corresponding components of the accelerator carriage 1000 and is herein for clarity of description of the accelerator carriage 1000 not repeated.

In the illustrative accelerator carriage 1000 are the physical resources 620 as accelerator circuits 1020 executed. Although only two accelerator circuits 1020 in 10 should be noted that the accelerator carriage 1000 in other embodiments additional accelerator circuits 1020 can have. For example, the accelerator carriage 1000 , as in 11 4, in some embodiments, four accelerator circuits 1020 exhibit. The accelerator circuits 1020 may be embodied as any type of processor, coprocessor, arithmetic circuit, or other device capable of performing computational or processing operations. For example, the accelerator circuits 1020 as field-programmable gate arrays (FPGA), application-specific integrated circuits (ASICs), security coprocessors, graphics processing units (GPUs), machine learning units or other specialized processors, controllers, devices and / or circuits are executed, but are not limited to this.

In some embodiments, the accelerator may be sled 1000 also an accelerator-to-accelerator interconnect 1042 exhibit. Similar to the resource-to-resource interconnect 624 of the carriage discussed above 600 may be the accelerator-to-accelerator interconnect 1042 be implemented as any type of communication interconnect that may enable accelerator-to-accelerator communications. In the illustrative embodiment, the accelerator-to-accelerator interconnect is 1042 as a high-speed point-to-point interconnect (e.g., faster than the I / O subsystem 622 ). For example, the accelerator-to-accelerator interconnect 1042 as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI, UltraPath Interconnect) or other high-speed point-to-point interconnect dedicated to processor-to-processor communications. In some embodiments, the accelerator circuits 1020 with a primary accelerator circuit 1020 via the I / O subsystem 622 with the NIC 832 and the memory 720 is connected, and a secondary accelerator circuit 1020 that has a primary accelerator circuit 1020 with the NIC 832 and the memory 720 connected in series.

With reference to 11 is an illustrative embodiment of the accelerator carriage 1000 shown. As discussed above the accelerator circuits 1020 , the communication circuit 830 and the optical data connector 834 at the top 650 of the subrackless printed circuit board substrate 602 assembled. Again, the individual accelerator circuits 1020 and the communication circuit 830 at the top 650 of the subrackless printed circuit board substrate 602 mounted so that no two heat-generating electrical components shadow each other, as discussed above. The storage devices 720 of the accelerator carriage 1000 are at the bottom 750 of the subrackless printed circuit board substrate 602 mounted as above with respect to the carriage 600 discussed. Although she is at the bottom 750 are mounted, the storage devices 720 communicative with the accelerator circuits 1020 that on the top 650 are arranged through the I / O subsystem 622 coupled (eg by means of vias). Furthermore, each of the accelerator circuits 1020 a heat sink 1070 larger than a conventional heat sink used in a server. As above with reference to the heatsink 870 discussed, the heat sinks can 1070 due to the "free" area provided by the storage devices 750 on the bottom 750 of the subrackless printed circuit board substrate 602 instead of on top 650 are arranged to be larger than conventional heat sink.

With reference to 12 can the sled 400 in some embodiments, as a storage device sled 1200 be executed. The storage device sled 1200 is optimized or otherwise designed to store data in a data store 1250 local to the storage device sled 1200 save. For example, a calculation slide 800 or an accelerator carriage 1000 during operation, data from the data storage device 1250 of the storage device carriage 1200 save and retrieve. The storage device sled 1200 has various components, the components of the carriage 400 and / or the calculation slide 800 similar to those in 12 be identified using the same reference numerals. The description of such components as described above with respect to 6 . 7 and 8th has been provided, applies to the corresponding components of the storage device carriage 1200 and is herein for clarity of description of the storage device carriage 1200 not repeated.

In the illustrative storage device sled 1200 are the physical resources 620 as a storage device controller 1220 executed. Although only two storage device controllers 1220 in 12 should be noted that the storage device carriage 1200 in other embodiments, additional storage device controllers 1220 can have. The storage device controllers 1220 may be embodied as any type of processor, controller, or control circuit that supports the storage and retrieval of data in the data storage device 1250 based on requirements made via the communication circuit 830 can be received, control. In the illustrative embodiment, the storage device controllers are 1220 running as processors or controllers with relatively low power. In some embodiments, the storage device controllers 1220 For example, be designed to operate at a rated power of about 75 watts.

In some embodiments, the storage device carriage may 1200 also a controller-to-controller interconnect 1242 exhibit. Similar to the resource-to-resource interconnect 624 of the carriage discussed above 400 may be the controller-to-controller interconnect 1242 may be implemented as any type of communication interconnect that may enable controller-to-controller communications. In the illustrative embodiment, the controller-to-controller interconnect is 1242 as a high-speed point-to-point interconnect (e.g., faster than the I / O subsystem 622 ). For example, the controller-to-controller interconnect 1242 as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI, UltraPath Interconnect) or other high-speed point-to-point interconnect dedicated to processor-to-processor communications.

With reference to 13 is an illustrative embodiment of the storage device carriage 1200 shown. In the illustrative embodiment, the data storage device is 1250 as a storage device cage 1252 designed to carry one or more solid-state drives (SSDs) 1254 take or otherwise assign this. For this purpose, the storage device cage 1252 a number of mounting slots 1256 on, each of which is designed to be a corresponding solid-state drive 1254 take. Each of the mounting slots 1256 has a number of drive guides 1258 on, working together to create an access opening 1260 the corresponding mounting slot 1256 define. The storage device cage 1252 is on the subrackless PCB substrate 602 fixed so that the access openings from the subrackless PCB substrate 602 are turned away (ie towards the front). Thus, solid state drives 1254 accessible while the storage device carriage 1200 in a corresponding rack 204 is mounted. For example, a solid-state drive 1254 from a rack 240 (eg via a robot) while the storage device carriage is being replaced 1200 in the corresponding rack 240 remains mounted.

The storage device cage 1252 illustratively has sixteen mounting slots 1256 on and is capable of sixteen solid-state drives 1254 to assemble and store. Of course, the storage device cage 1252 be designed to additional or less solid-state drives in other embodiments 1254 save. In addition, in the illustrative embodiment, the solid state drives are in the memory device cage 1252 mounted vertically, but in other embodiments may be in a different orientation in the storage device cage 1252 be mounted. Any solid state drive 1254 can be implemented as any type of data storage device that can store long-term data. For this purpose, the solid-state drives 1254 include volatile and non-volatile memory devices discussed above.

As in 13 shown are the storage device controllers 1220 , the communication circuit 830 and the optical data connector 834 Illustrating at the top 650 of the subrackless printed circuit board substrate 602 assembled. Again, as discussed above, any suitable mounting or mounting technology may be used to secure the electrical components of the storage device carriage 1200 on the subrackless PCB substrate 602 including, for example, sockets (eg, processor sockets), brackets, brackets, solder joints, and / or other mounting or mounting techniques.

As discussed above, the individual storage device controllers are 1220 and the communication circuit 830 at the top 650 of the subrackless printed circuit board substrate 602 mounted such that no two heat-generating electrical components shade each other. For example, the storage device controllers 1220 and the communication circuit 830 in appropriate places on the top 650 of the subrackless printed circuit board substrate 602 mounted so that no two of these electrical components linearly inline with each other along the direction of the air flow path 608 lie.

The storage devices 720 of the storage device carriage 1200 are at the bottom 750 of the subrackless printed circuit board substrate 602 mounted as above with respect to the carriage 400 discussed. Although she is at the bottom 750 are mounted, the storage devices 720 communicatively with the storage device controllers 1220 that on the top 650 are arranged through the I / O subsystem 622 coupled. Since the subrackless printed circuit board substrate 602 As a double-sided circuit board is executed, in turn, the memory devices 720 and the storage device controllers 1220 communicatively coupled by one or more vias, connectors, or other mechanisms passing through the subrackless printed circuit board substrate 602 extend. Each of the storage device controllers 1220 has a heat sink 1270 which is attached to it. As discussed above, due to the improved thermal cooling characteristics of the sub-rackless printed circuit substrate 602 of the storage device carriage 1200 none of the heatsinks 1270 attached cooling fans. That is, each of the heat sinks 1270 is designed as a fanless heat sink.

With reference to 14 can the sled 400 in some embodiments as a storage sled 1400 be executed. The storage sled 1400 is optimized or otherwise designed to other sledges 400 (eg calculating slide 800 , Accelerator carriage 1000 etc.) Access to a storage pool (eg in two or more sets 1430 . 1432 of storage devices 720 ) locally to the storage sled 1200 provide. For example, a calculation slide may be used during operation 800 or an accelerator carriage 1000 removed into one or more of the memory sets 1430 . 1432 of the storage sled 1200 write and / or read from, using a logical address space, which refers to physical addresses in the memory sets 1430 . 1432 maps. The storage sled 1400 has various components, the components of the carriage 400 and / or the calculation slide 800 similar to those in 14 be identified using the same reference numerals. The description of such components as described above with respect to 6 . 7 and 8th has been provided applies to the corresponding components of the storage carriage 1400 and is herein for clarity of description of the storage sled 1400 not repeated.

In the illustrative storage sled 1400 are the physical resources 620 as a memory controller 1420 executed. Although only two memory controllers 1420 in 14 should be noted that the storage sled 1400 in other embodiments, additional memory controllers 1420 can have. The memory controller 1420 can be implemented as any type of processor, controller, or control circuit that handles the writing or reading of data in the memory packs 1430 . 1432 based on requirements made via the communication circuit 830 can be received, control. In the Illustrative embodiment is any storage device controller 1220 with a corresponding memory set 1430 . 1432 connected to the storage devices 720 within the corresponding memory set 1430 . 1432 to write and read from these and enforce any permissions (such as reading, writing, etc.) with the sled 400 associated with a request to the storage sled 1400 to perform a memory access operation (e.g., read or write).

In some embodiments, the storage carriage may 1400 also a controller-to-controller interconnect 1442 exhibit. Similar to the resource-to-resource interconnect 624 of the carriage discussed above 400 may be the controller-to-controller interconnect 1442 may be implemented as any type of communication interconnect that may enable controller-to-controller communications. In the illustrative embodiment, the controller-to-controller interconnect is 1442 as a high-speed point-to-point interconnect (e.g., faster than the I / O subsystem 622 ). For example, the controller-to-controller interconnect 1442 as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI, UltraPath Interconnect) or other high-speed point-to-point interconnect dedicated to processor-to-processor communications. Thus, a memory controller 1420 in some embodiments, via the controller-to-controller interconnect 1442 access memory within the memory set 1432 lies with another memory controller 1420 is associated. In some embodiments, a scalable memory controller is comprised of a plurality of smaller memory controllers, referred to herein as "chiplets," on a storage sled (eg, the storage sled 1400 ). The chiplets may be interconnected (eg, using EMIB (Embedded Multi-The Interconnect Bridge)). The combined chiplet memory controller can scale up to a relatively large number of memory controllers and I / O ports (eg, up to 16 memory channels). In some embodiments, the memory controllers 1420 implement a memory entanglement (for example, a memory address is placed on the memory set 1430 imaged, the next memory address will be on the memory set 1432 pictured and the third address will be on the memory set 1430 pictured, etc.). The entanglement can be within the memory controller 1420 or CPU sockets (such as the calculation slide) 800 ) via network connections to the memory sets 1430 . 1432 and can improve the latency associated with performing memory access operations as compared to accessing contiguous memory addresses from the same memory device.

Furthermore, the storage sled 1400 in some embodiments with one or more other carriages 400 (eg in the same rack 240 or in a neighboring rack 240 ) through a waveguide using the waveguide connector 1480 be connected. In the illustrative embodiment, the waveguides are 64 millimeter waveguides providing 16 Rx tracks (ie, receiving) and 16 Rt tracks (ie, transmitting). Each track is either 16 Ghz or 32 Ghz in the illustrative embodiment. In other embodiments, the frequencies may be different. The use of a waveguide can provide high throughput access to the storage pool (e.g., the storage sets 1430 . 1432 ) to another sled (eg a sled 400 in the same rack 240 or a neighboring rack 240 as storage sled 1400 ) without burdening the optical data connector 834 to increase.

With reference to 15 may be a system for executing one or more workloads (eg, applications) in accordance with the data center 100 be implemented. In the illustrative embodiment, the system 1510 an orchestrator server 1520 which may be implemented as a managed node comprising a computing device (eg, a compute sled 800 ) that executes management software (eg, a cloud operating environment such as OpenStack) communicatively with multiple carriages 400 coupled, including a large number of calculation slides 1530 (eg in each case similar to the calculation slide 800 ), Storage sled 1540 (eg, each similar to the storage sled 1400 ), Accelerator carriage 1550 (eg, each similar to the storage sled 1000 ) and storage sled 1560 (eg, each similar to the storage device sled 1200 ). One or more of the sledges 1530 . 1540 . 1550 . 1560 can be in a managed node 1570 be grouped, such as by the Orchestrator server 1520 to a workload (for example, an application 1532 performed in a virtual machine or in a container). The managed node 1570 can be considered an arrangement of physical resources 620 such as processors 820 , Memory resources 720 , Accelerator circuits 1020 or data storage devices 1250 , the same or different sledges 400 be executed. Further, the managed node may be serviced by the Orchestrator server at the time the managed node is to be assigned a workload or at any other time 1520 set up, defined or "booting up" and may exist regardless of whether any workloads are currently assigned to the managed node. In the illustrative embodiment, the orchestrator server 1520 selectively physical resources 620 from the sledges 400 assign and / or cancel the assignment and / or one or more sleds 400 from the managed node 1570 add or remove as a function of quality of service (QoS) goals (eg, performance goals associated with throughput, latency, instructions per second, etc.) that are associated with a service level agreement for the service Workload (eg the application 1532 ) are associated. That way, the Orchestrator server can 1520 Receive telemetry data indicative of performance conditions (eg, throughput, latency, instructions per second, etc.) in each sled 400 of the managed node 1570 compare the telemetry data with the Quality of Service targets to determine if the Quality of Service goals are met. If so, the Orchestrator server can 1520 additionally determine whether the allocation of one or more physical resources from the managed node 1570 can be canceled while the QoS targets are still satisfied, thereby releasing these physical resources for use in another managed node (eg, to perform a different workload). If the QoS targets are currently not met, the Orchestrator server can 1520 alternatively, determine that additional physical resources are dynamically allocated to the execution of the workload (eg, the application 1532 ) while the workload is running.

In addition, the Orchestrator server 1520 in some embodiments, resource utilization trends of the workload (e.g., the application 1532 ), such as by identifying phases of execution (e.g., periods of time in which different operations are performed with different resource usage characteristics) of the workload (e.g., the application 1532 ) and preventively identifying available resources in the data center 100 and their assignment to the managed node 1570 (eg within a predefined time period of the associated phase start). In some embodiments, the Orchestrator server 1520 Model performance based on different latencies and a distribution scheme to workloads between calculation slides and other resources (such as accelerator sleds, storage sleds, storage sleds) in the data center 100 to place. For example, the Orchestrator server 1520 use a model that reduces the power of resources on the sled 400 (eg, FPGA power, memory access latency, etc.) and the power (eg, congestion, latency, bandwidth) of the path through the network to the resource (eg, FPGA). Thus, the Orchestrator server 1520 based on the overall latency, with everyone in the data center 100 determine which resource (s) should be used with which workloads (eg, the latency associated with the performance of the resource itself in addition to the latency associated with the path through the network between the compute carriage that carries out the workload, and the sled 400 that the resource is associated with).

In some embodiments, the Orchestrator server 1520 a map of heat production in the data center 100 using telemetry data (eg, temperatures, fan speeds, etc.) generated by the slides 400 and allocate resources to managed nodes as a function of the map of heat generation and predicted heat generation associated with different workloads to a target temperature and heat distribution in the data center 100 maintain. Additionally or alternatively, the Orchestrator server 1520 in some embodiments, organize telemetry data received into a hierarchical model that is indicative of a relationship between the managed nodes (eg, a spatial relationship, such as the physical locations of the managed node resources within the data center 100 , and / or a functional relationship, such as customer groupings of the managed nodes for which the managed nodes provide services, the types of functions typically performed by the managed nodes, managed nodes that typically share or exchange workloads , etc.). Based on differences in the physical locations and resources in the managed nodes, a given workload may have different resource usages over the resources of different managed nodes (eg, cause a different internal temperature, use a different percentage of the processor or memory capacity). The Orchestrator server 1520 can determine the differences based on the telemetry data stored in the hierarchical model and incorporate the differences in a prediction of the future resource utilization of a workload, if the workload is reassigned from one managed node to another managed node to the resource usage in the data center 100 to balance exactly.

To the workload on the Orchestrator server 1520 and to reduce the data transfer load on the network, the Orchestrator server 1520 in some embodiments, self-test information about the carriages 400 send to everyone carriage 400 to enable. locally (eg on the sled 400 ) to determine if telemetry data coming from the sled 400 one or more conditions (eg, an available capacity that meets a predefined threshold, a temperature that meets a predefined threshold, etc.). Every sled 400 can then return a simplified result (such as Yes or No) to the Orchestrator server 1520 Report back to the Orchestrator server 1520 in determining the allocation of resources to managed nodes.

With reference to 16 can a system 1600 for bandwidth allocation in accordance with the data center 100 be implemented, with reference to above 1-15 has been described. As shown, the system points 1600 a resource manager server 1602 and several sleds 1610 . 1612 . 1614 . 1616 . 1618 . 1620 on that over a network using multiple switches 1604 . 1606 . 1608 communicate. The resource manager server 1602 and the sledges 1610 . 1612 . 1614 . 1616 . 1618 . 1620 may each be configured as server computers, rack servers, blade servers, compute nodes, and / or carriages in a data center, such as a sled 400 as above in connection with 1-15 described, or another sled of the data center.

The network elements of the system 1600 are organized in a network topology. As shown, the slides can 1610 . 1612 . 1614 be organized in a rack and each with the switch 1606 which may be implemented as a top-of-rack switch, middle-of-rack switch, end-of-row switch, or other switch device. Similarly, the sledges 1616 . 1618 . 1620 with the switch 1608 connected. The switches 1606 . 1608 are in turn with the switch 1604 which may be implemented as a data center domain switch or other upstream switch. The resource manager server 1602 is illustrated with the upstream switch 1604 however, in other embodiments may be connected to any other location in the network topology. As in 16 shown are the switches 1604 . 1606 . 1608 thus, organized in multiple layers, and sending data across layers can add switching latency, queue latency, or other latencies.

Although the resource manager server 1602 is illustrated as a single server computing device, the resource manager server 1602 in some embodiments, may be implemented as a "virtual server" formed of a plurality of distributed computing devices passing through the system 1600 are distributed and / or working in a public or private cloud. Although the resource manager server 1602 as a single-server computing device executed in 16 Accordingly, it should be noted that the resource manager server 1602 may be implemented as a plurality of devices cooperating to provide the functionality described below.

In use, the resource manager server 1602 as described below, the network topology of the system 1600 discover and model a resource overubscription in the system 1600 to construct. A resource oversubscription may include a network uplink overubscription, a storage resource overubscription, or any other circumstance where the aggregate bandwidth or other demand generated by the carriages is the available bandwidth or other capacity of the system 1600 exceeds. The resource manager server 1602 can set bandwidth limits for each sled or other network element of the system 1600 and program these bandwidth limits for the network elements. Each network element enforces the programmed bandwidth limits, which can reduce network congestion. By reducing the congestion, the system can 1600 Eliminate or reduce queue latency for each layer of switches. For example, bandwidth limits may be for a storage device sled 1610 Ensure non-volatile Memory Express (NVMe) over Ethernet traffic from the storage device sled 1610 (and from other storage device carriages) that does not exceed available upstream bandwidth. Accordingly, the system can 1600 improve network latency and reduce network congestion without implementing expensive switch-level bandwidth reservation mechanisms.

With reference to 17 is an illustrative computing device 1700 shown. The computing device 1700 can be the resource manager server 1602 , a sleigh 400 , a storage device sled 1200 . 1610 . 1614 . 1616 . 1620 , an accelerator sled 1000 , a storage sledge 1400 , a calculating slide 800 . 1612 . 1618 and / or a similar server computing device. As in 17 shown has the computing device 1700 Illustrates a processor 1720 , an input / output subsystem 1722 , a store 1724 , a data storage device 1726 , a communication subsystem 1728 , and / or other components and devices commonly found in a carriage 400 , a storage device sled 1200 . 1610 . 1614 . 1616 . 1620 , an accelerator carriage 1000 , a storage sledge 1400 , a calculating slide 800 . 1612 . 1618 and / or a similar server computing device. Of course, the computing device can 1700 in other embodiments have other or additional components as commonly found in a server computer (eg, various input / output devices). In addition, in some embodiments, one or more of the illustrative components may be integrated with or otherwise form part of another component. For example, the memory 1724 or parts thereof, in some embodiments in the processor 1720 be integrated.

The processor 1720 may be embodied as any type of processor capable of performing the functions described herein. For example, the processor 1720 be implemented as a single or multi-core processor (s), digital signal processor, microcontroller or other processor or processing / control circuit. Similarly, the memory can 1724 as any type of volatile or non-volatile memory or data storage device capable of performing the functions described herein. During operation, the memory can 1724 store various data and software during operation of the computing device 1700 such as operating systems, applications, programs, libraries and drivers. The memory 1724 is communicative with the processor 1720 via the I / O subsystem 1722 coupled, which may be implemented as circuits and / or components to the input / output operations with the processor 1720 the store 1724 and other components of the computing device 1700 to enable. For example, the I / O subsystem 1722 as memory controller hubs, input / output control hubs, sensor hubs, firmware devices, communication links (ie, point-to-point links, bus links, wires, cables, optical fibers, printed circuit board traces, etc.) and / or other components and subsystems to enable, execute, or otherwise have the input / output operations. In some embodiments, the I / O subsystem 1722 form part of a system-on-a-chip (SoC) and, together with the processor 1720 the store 1724 and other components of the computing device 1700 be integrated on a single integrated circuit chip.

The data storage device 1726 may be embodied as any type of device or devices designed for short-term or long-term storage of data, such as memory devices and circuits, memory cards, hard disk drives, solid-state drives, non-volatile flash memory, or other data storage devices. The computing device 1700 can also be a communication subsystem 1728 which may be embodied as any communication circuit, device or collection thereof, the communications between the computing device 1700 and other remote devices via a computer network (not shown). The communication subsystem 1728 may be designed to accommodate one or more communication technologies (eg, wired or wireless communications) and associated protocols (eg, Ethernet, InfiniBand®, Bluetooth®, Wi-Fi®, WiMAX, 3G, 4G LTE, etc.) use to effect such communication. As shown, the communication subsystem 1728 a network interface controller (NIC) 1330.

The illustrative communication subsystem 1728 has a network interface controller (NIC) 1330. The NIC 1730 may be implemented as one or more add-in boards, daughter cards, controller chips, chipsets, circuits, or other devices provided by the computing device 1700 can be used for network communications with remote devices. For example, the NIC 1730 as an expansion card that connects to the I / O subsystem via an expansion bus, such as PCI Express 1722 is coupled. As another example, the NIC 1730 In some embodiments, it may be implemented as a network controller, a host fabric interface, or another component that interfaces with the I / O subsystem 1722 , the processor 1720 , a SoC and / or one or more other components of the computing device 1700 is integrated.

As shown, the computing device 1700 also one or more peripheral devices 1732 exhibit. The peripheral devices 1732 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. In some embodiments, the peripheral devices 1732 For example, a display, a touch screen, graphics circuits, a keyboard, a mouse, a speaker system, a microphone, a network interface and / or other input / output devices, interface devices and / or peripheral devices.

With reference to 18 sets up the resource manager server 1602 an environment in one illustrative embodiment 1800 during operation. The illustrative environment 1800 has a topology manager 1802 , a model constructor 1804 , one Bandbreitengrenzenbestimmer 1806 , a bandwidth limit programmer 1808 and a usage manager 1810 on. The different components of the environment 1800 may be implemented as hardware, firmware, software or a combination thereof. In some embodiments, therefore, one or more of the components of the environment may 1800 as circuits or collection of electrical devices (e.g., topology manager circuits 1802 Model Constructor Circuits 1804 , Bandwidth limiters circuits 1806 , Bandwidth Limit Programmer Circuits 1808 and / or usage manager circuits 1810 ). It should be noted that in such embodiments, one or more of the topology manager circuits 1802 , the model constructor circuits 1804 , bandwidth bandwidth estimator circuits 1806 , the bandwidth limit programmer circuits 1808 and / or the usage manager circuits 1810 a part of the processor 1720 , the I / O subsystem 1722 , the NIC 1730 and / or other components of the resource manager server 1602 can train. Additionally, in some embodiments, one or more of the illustrative components may form part of another component, and / or one or more of the illustrative components may be independent of each other.

The topology manager 1802 is designed to discover the topology of the sleds coupled to a layer of switches communicatively with the resource manager server 1602 are coupled. The model constructor 1804 is designed to construct a model of network connectivity between the plurality of carriages and the layer of switches based on the topology. Constructing the model may include identifying which carriages of the plurality of carriages are connected to a particular switch of the layer of switches.

The bandwidth limit determiner 1806 is designed to determine an over-subscription of the network based on the network connectivity model. The oversubscription is based on available bandwidth for the layer of switches and a maximum bandwidth of the carriages. The bandwidth limit determiner 1806 may determine a network uplink overubscription for the layer of switches or may determine a storage resource over-subscription of the sleds. The bandwidth limit determiner 1806 is further configured to determine a bandwidth limit for each carriage based on the overubscription. The bandwidth limit programmer 1808 is designed to program each carriage with the appropriate bandwidth limit. The bandwidth limit programmer 1808 can limit the bandwidth to the NIC 1730 communicate the corresponding carriage.

The usage manager 1810 is designed to monitor a bandwidth utilization of the carriages. Bandwidth usage monitoring may include receiving telemetry data from the carriages indicative of the bandwidth utilized by each carriage. The usage manager 1810 is further configured to determine whether the network is congested based on the bandwidth utilization of the plurality of carriages. For example, determining whether the network is congested may include determining whether a network queuing depth exceeds a predetermined queue depth limit for a predetermined period of time. In some embodiments, determining whether the network is congested may include monitoring bandwidth per class of network traffic, such as NVMe over Ethernet traffic, field programmable gate array (FPGA) over Ethernet traffic, storage device traffic, and / or other traffic classes. Each traffic class can be independently monitored with its own queue depth controls. Further, the bandwidth at the source or at the destination or in some embodiments may be limited based on combinations of source and destination pairs. The usage manager 1810 is further configured to modify a bandwidth limit in response to determining that the network is overloaded. The bandwidth limit can be reduced for each carriage associated with a high input rate flow.

With reference to 19 aligns a storage device sled 1610 an environment in one illustrative embodiment 1900 during operation. It should be understood that the environment 1900 also by other sleds of the system 1600 can be set up. The illustrative environment 1900 has a bandwidth limit manager 1904 , a bandwidth programmer 1906 and a telemetry data manager 1908 on. The different components of the environment 1900 may be implemented as hardware, firmware, software or a combination thereof. In some embodiments, therefore, one or more of the components of the environment may 1900 as circuits or collection of electrical devices (e.g., bandwidth limit manager circuits 1904 , Bandwidth Programmer Circuits 1906 and / or telemetry data manager circuits 1908 ). It should be noted that in such embodiments, one or more of the bandwidth limit manager circuits 1904 , the bandwidth programmer circuits 1906 and / or the Telemetry data manager circuits 1908 a part of the processor 1720 , the I / O subsystem 1722 , the NIC 1730 and / or other components of the storage device carriage 1610 can train. Additionally, in some embodiments, one or more of the illustrative components may form part of another component, and / or one or more of the illustrative components may be independent of each other.

The bandwidth programmer 1906 is designed to set a bandwidth limit for the sled from the resource manager server 1602 to receive and the bandwidth limit for the NIC 1730 to program the slide. The bandwidth limit manager 1904 is designed to set the bandwidth limit in response to the programming of the bandwidth limit by the NIC 1730 to enforce.

The telemetry data manager 1908 is designed to provide telemetry data indicative of the use of the NIC 1730 are to the resource manager server 1602 to send. The telemetry data can be obtained from the NIC 1730 be sent. The telemetry data may be indicative of a NIC queue depth and / or a network stack queue depth.

With reference to 20 can be the resource manager server 1602 in use a procedure 2000 for bandwidth allocation. It should be noted that the operations of the procedure 2000 in some embodiments by one or more components of the environment 1800 of the resource manager server 1602 can be performed as in 18 shown. The procedure 2000 starts in block 2002 in which the resource manager server 1602 the network topology of the components of the system 1600 discovered. The topology may be at the design time of the system 1600 is predetermined or may be discovered in some embodiments using a topology discovery protocol or other discovery technique. In some embodiments, the resource manager server 1602 in block 2004 Slides, racks, switches and network connections of the system 1600 discover. For example, the resource manager server 1602 identify that a slide is connected to a specific port of a switch. As another example, the resource manager server 1602 identify that a port of a switch is connected to a specific port of an upstream switch.

In block 2006 constructs the resource manager server 1602 a model of network connectivity between the components of the system 1600 , The model can handle network connections and the associated available bandwidth between sleds, switches, and other network elements of the system 1600 identify.

In block 2008 determines the resource manager server 1602 an over-subscription of the system 1600 based on the model of network connectivity. As described above, the system can 1600 be organized in layers, and each layer may have a maximum amount of available bandwidth or other resources. An over-subscription may exist if the total maximum bandwidth or other resource requirement of a layer exceeds the available bandwidth or other resources of a higher layer. In some embodiments, the resource manager server 1602 in block 2010 determine a network uplink overubscription. For example, the sledges 1610 . 1612 . 1614 , as in 16 shown with the switch 1606 connected. An overubscription may exist if the maximum bandwidth used by the sled 1610 . 1612 . 1614 used in combination the available bandwidth of the uplink from the switch 1606 to the switch 1604 exceeds. Similarly, an overubscription may exist if the maximum bandwidth used by the sled 1616 . 1618 . 1620 is used, the available bandwidth of the uplink from the switch 1608 to the switch 1604 exceeds. In some embodiments, the resource manager server 1602 in block 2012 determine a storage resource over-subscription. For example, an oversubscription may exist if the need for storage resources of a carriage (eg, the storage device sled 1610 ) exceeds the available bandwidth or other resources of this sled.

In block 2014 determines the resource manager server 1602 Bandwidth limits for each slide in the system 1600 based on the overubscription. The bandwidth limits can be determined to be a network congestion in the system 1600 to prevent or reduce. For example, again with reference to 16 , the bandwidth limits for the carriages 1610 . 1612 . 1614 be set so that the combined bandwidth limits are less than or equal to the uplink bandwidth of the switch 1606 to the switch 1604 are.

In block 2016 programs the resource manager server 1602 each slide with the appropriate bandwidth limit. After programming, each carriage sets the bandwidth limits as discussed below 21 described. The resource manager server 1602 can use any technique for programming the carriage. In some embodiments, the resource manager server 1602 in block 2018 the NIC 1730 of the sled with the Program bandwidth limit. For example, the resource manager server 1602 communicate out-of-band or otherwise with the NIC 1730 communicate without invoking the operating system or other software network stack of the carriage.

In block 2020 can be the resource manager server 1602 Bandwidth telemetry from the carriages of the system 1600 receive. The bandwidth telemetry can indicate the current bandwidth utilization of the carriage and / or whether the associated network connection is overloaded. For example, bandwidth telemetry may be the queuing depth of the NIC 1730 , the associated switch port and / or the network stack of the carriage.

In block 2022 identifies the resource manager server 1602 a network congestion based on telemetry. The resource manager server 1602 can use any suitable algorithm to otherwise identify lost packets, increased latency, or network congestion. In some embodiments, the resource manager server 1602 in block 2024 determine if any queuing depth in the system exceeds a predetermined threshold queue depth for more than a predetermined time. For example, the resource manager server 1602 the queue depth of a NIC 1730 , a switch port and / or a network stack of a sled. In block 2026 determines the resource manager server 1602 whether an overload has been detected. If not, the procedure returns 2000 to block 2020 back to continue monitoring network utilization. If an overload is detected, the procedure moves 2000 to block 2028 in front.

In block 2028 modifies the resource manager server 1602 one or more bandwidth limits to reduce or eliminate the congestion. In some embodiments, the resource manager server 1602 in block 2030 one or more high input rate flows in the system 1600 identify. For example, one or more storage device carriages 1610 that generate NVMe over Ethernet data, generate high-input-rate flows. The resource manager server 1602 may reduce the input rate bandwidth limit associated with the high input rate flows. Additionally or alternatively, the resource manager server 1602 in some embodiments, generate one or more congestion warnings, and a network administrator may provide modified bandwidth limits. In some embodiments alternative network routes may be possible. If alternate routes are possible, different bandwidth limits can be set for different routes based on the overload telemetry data to reduce the congestion rate. After modifying the bandwidth limits, the method returns 2000 to block 2016 back to program the carriages with the modified bandwidth limits and to continue monitoring network usage.

With reference to 21 can be a storage device sled 1610 in use a procedure 2100 for bandwidth allocation. It should be noted that the operations of the procedure 2100 in some embodiments by one or more components of the environment 1900 of the storage device carriage 1610 can be performed as in 19 shown. The procedure 2100 starts in block 2102 in which the storage device sled 1610 determines whether an update of a bandwidth limit from the resource manager server 1602 was received. As above in connection with 20 described, the resource manager server 1602 the bandwidth limit for the storage device sled 1610 in response to modeling the network connectivity and determining an overubscription of the system 1600 and / or in response to detecting network congestion based on telemetry data. If an update of the bandwidth limit has not been received, the method branches 2100 to block 2108 as described below. If an update of the bandwidth limit has been received, the procedure moves 2100 to block 2104 in front.

In block 2104 programs the storage device slide 1610 one or more Network Interface Controllers (NICs) 1330 of the storage device sled with the new bandwidth limit. After programming, the NIC 1730 through the storage device sled 1610 reduce or otherwise limit the bandwidth used below the bandwidth limit. In particular, the storage device carriage may 1610 in some embodiments in block 2106 a maximum input bandwidth for the NIC 1730 establish. Thus, the bandwidth limits may limit the amount of data (e.g., NVMe over Ethernet data) that is being dropped from the storage device sled 1610 generated and sent to the switch 1606 be transmitted. Although through the NIC 1730 as enforced, it should be noted that the bandwidth limits in some embodiments are limited by other components of the storage device sled 1610 can be enforced, such as an operating system, software network stack, NVMe over Ethernet subsystem or other component.

In block 2108 determines the storage device carriage 1610 whether telemetry data to the Resource Manager Server 1602 to be sent. For example, the storage device carriage 1610 be configured by an administrator to send telemetry data. In some embodiments, the storage device carriage may 1610 Send telemetry data in response to certain events, for example in response to a detected network congestion. If the storage device carriage 1610 determined not to send telemetry data, the procedure returns 2100 to block 2102 back to continue monitoring for updated bandwidth limits. If the storage device carriage 1610 destined to send telemetry data, moves the procedure 2100 to block 2110 in front.

In block 2110 sends the storage device sled 1610 Bandwidth telemetry data to the resource manager server 1602 , As described above, the bandwidth telemetry may indicate the current bandwidth utilization of the carriage and / or whether the associated network connection is overloaded. For example, bandwidth telemetry may be the queuing depth of the NIC 1730 , the associated switch port and / or the network stack of the carriage. In some embodiments, the storage device carriage may 1610 in block 2112 the telemetry data from the NIC 1730 of the storage device carriage 1610 recall. For example, an operating system, software network stack, or other component of the storage device sled 1610 Telemetry data from the NIC 1730 recall. In some embodiments, the storage device carriage may 1610 in block 2114 the telemetry data from the NIC 1730 to the resource manager server 1602 send. The NIC 1730 The telemetry data may be off-band or otherwise without the participation of the operating system, the software network stack, or other components of the storage device sled 1610 send. After sending the telemetry data, the procedure returns 2100 to block 2102 back to continue monitoring for updated bandwidth limits.

EXAMPLES

• Beispiel 1 umfasst einen Ressourcenmanagerserver zur Bandbreitenzuweisung, der Ressourcenmanagerserver umfassend: einen oder mehrere Prozessoren; und eine oder mehrere Speichervorrichtungen, in denen eine Mehrzahl von Anweisungen gespeichert sind, die, wenn sie durch den einen oder die mehreren Prozessoren ausgeführt werden, den Ressourcenmanagerserver veranlassen zum: Entdecken einer Topologie einer Mehrzahl von Schlitten, die mit einer Schicht von Switches gekoppelt sind, die kommunikativ mit dem Ressourcenmanagerserver gekoppelt sind; Konstruieren eines Modells der Netzkonnektivität zwischen der Mehrzahl von Schlitten und der Schicht von Switches basierend auf der Topologie; Bestimmen einer Oversubscription eines Netzes basierend auf dem Modell der Netzkonnektivität, wobei die Oversubscription auf einer verfügbaren Bandbreite für die Schicht von Switches und einer maximalen Bandbreite der Mehrzahl von Schlitten basiert; Bestimmen einer Bandbreitengrenze für jeden Schlitten der Mehrzahl von Schlitten basierend auf der Oversubscription; und Programmieren von jedem Schlitten der Mehrzahl von Schlitten mit der entsprechenden Bandbreitengrenze.
• Beispiel 2 umfasst den Gegenstand von Beispiel 1, und wobei das Konstruieren des Modells der Netzkonnektivität das Identifizieren umfasst, welche Schlitten der Mehrzahl von Schlitten mit einem bestimmten Switch der Schicht von Switches verbunden sind.
• Beispiel 3 umfasst den Gegenstand eines der Beispiele 1 und 2, und wobei das Bestimmen der Oversubscription umfasst, eine Netz-Uplink-Oversubscription für die Schicht von Switches zu bestimmen.
• Beispiel 4 umfasst den Gegenstand eines der Beispiele 1-3, und wobei das Bestimmen der Oversubscription umfasst, eine Speichereinrichtungsressourcen-Oversubscription der Mehrzahl von Schlitten zu bestimmen.
• Beispiel 5 umfasst den Gegenstand eines der Beispiele 1-4, und wobei das Programmieren der Bandbreitengrenze für jeden Schlitten umfasst, die Bandbreitengrenze an einen Netzschnittstellencontroller des entsprechenden Schlittens zu kommunizieren.
• Beispiel 6 umfasst den Gegenstand eines der Beispiele 1-5, und wobei die eine oder mehreren Speichervorrichtungen eine Mehrzahl von Anweisungen darin gespeichert haben, die, wenn sie durch den einen oder die mehreren Prozessoren ausgeführt werden, den Ressourcenmanagerserver ferner veranlassen zum: Überwachen einer Bandbreitennutzung der Mehrzahl von Schlitten; Bestimmen, ob das Netz überlastet ist, basierend auf der Bandbreitennutzung der Mehrzahl von Schlitten; und Modifizieren einer Bandbreitengrenze in Reaktion auf eine Bestimmung, dass das Netz überlastet ist.
• Beispiel 7 umfasst den Gegenstand eines der Beispiele 1-6, und wobei das Überwachen der Bandbreitennutzung der Mehrzahl von Schlitten umfasst, Telemetriedaten von der Mehrzahl von Schlitten zu empfangen, die indikativ für die Bandbreite sind, die von jedem Schlitten genutzt wird.
• Beispiel 8 umfasst den Gegenstand eines der Beispiele 1-7, und wobei das Bestimmen, ob das Netz überlastet ist, das Bestimmen umfasst, ob eine Warteschlangentiefe des Netzes eine vorbestimmte Warteschlangentiefengrenze für eine vorbestimmte Zeitdauer überschreitet.
• Beispiel 9 umfasst den Gegenstand eines der Beispiele 1-8, und wobei die Warteschlangentiefe eine Switch-Port-Warteschlangentiefe, eine Netzschnittstellencontroller-Warteschlangentiefe oder eine Netzstapel-Warteschlangentiefe umfasst.
• Beispiel 10 umfasst den Gegenstand eines der Beispiele 1-9, und wobei das Modifizieren der Bandbreitengrenze umfasst: Identifizieren eines ersten Schlittens der Mehrzahl von Schlitten, die mit einem Fluss mit hoher Eingangsrate assoziiert sind; und Reduzieren einer Eingangsrate der Bandbreitengrenze für den ersten Schlitten.
• Beispiel 11 umfasst einen Schlitten zur Bandbreitenzuweisung, wobei der Schlitten kommunikativ mit einer Schicht von Switches gekoppelt ist, die kommunikativ mit einem Ressourcenmanagerserver in einem Netz gekoppelt sind, der Schlitten umfassend: einen oder mehrere Prozessoren; und eine oder mehrere Speichervorrichtungen, in denen eine Mehrzahl von Anweisungen gespeichert sind, die, wenn sie durch den einen oder die mehreren Prozessoren ausgeführt werden, den Schlitten veranlassen zum: Empfangen einer Bandbreitengrenze für den Schlitten vom Ressourcenmanagerserver; Programmieren der Bandbreitengrenze für einen Netzschnittstellencontroller des Schlittens; und Durchsetzen, durch den Netzschnittstellencontroller, der Bandbreitengrenze in Reaktion auf das Programmieren der Bandbreitengrenze.
• Beispiel 12 umfasst den Gegenstand von Beispiel 11, und wobei die eine oder mehreren Speichervorrichtungen eine Mehrzahl von Anweisungen darin gespeichert haben, die, wenn sie durch den einen oder die mehreren Prozessoren ausgeführt werden, den Schlitten ferner veranlassen, Telemetriedaten, die indikativ für eine Nutzung des Netzschnittstellencontrollers sind, an den Ressourcenmanagerserver des Netzes zu senden.
• Beispiel 13 umfasst den Gegenstand eines der Beispiele 11 und 12, und wobei das Senden der Telemetriedaten umfasst, die Telemetriedaten durch den Netzschnittstellencontroller zu senden.
• Beispiel 14 umfasst den Gegenstand eines der Beispiele 11-13, und wobei die Telemetriedaten indikativ für eine Netzschnittstellencontroller-Warteschlangentiefe oder eine Netzstapel-Warteschlangentiefe sind.
• Beispiel 15 umfasst ein Verfahren zur Bandbreitenzuweisung, das Verfahren umfassend: Erkennen, durch einen Ressourcenmanagerserver eines Netzes, einer Topologie einer Mehrzahl von Schlitten, die mit einer Schicht von Switches gekoppelt sind, die kommunikativ mit dem Ressourcenmanagerserver gekoppelt sind; Konstruieren, durch den Ressourcenmanagerserver, eines Modells der Netzkonnektivität zwischen der Mehrzahl von Schlitten und der Schicht von Switches basierend auf der Topologie; Bestimmen, durch den Ressourcenmanagerserver, einer Oversubscription des Netzes basierend auf dem Modell der Netzkonnektivität, wobei die Oversubscription auf einer verfügbaren Bandbreite für die Schicht von Switches und einer maximalen Bandbreite der Mehrzahl von Schlitten basiert; Bestimmen, durch den Ressourcenmanagerserver, einer Bandbreitengrenze für jeden Schlitten der Mehrzahl von Schlitten basierend auf der Oversubscription; und Programmieren, durch den Ressourcenmanagerserver, von jedem Schlitten der Mehrzahl von Schlitten mit der entsprechenden Bandbreitengrenze.
• Beispiel 16 umfasst den Gegenstand von Beispiel 15, und wobei das Konstruieren des Modells der Netzkonnektivität das Identifizieren umfasst, welche Schlitten der Mehrzahl von Schlitten mit einem bestimmten Switch der Schicht von Switches verbunden sind.
• Beispiel 17 umfasst den Gegenstand eines der Beispiele 15 und 16, und wobei das Bestimmen der Oversubscription umfasst, eine Netz-Uplink-Oversubscription für die Schicht von Switches zu bestimmen.
• Beispiel 18 umfasst den Gegenstand eines der Beispiele 15-17, und wobei das Bestimmen der Oversubscription umfasst, eine Speichereinrichtungsressourcen-Oversubscription der Mehrzahl von Schlitten zu bestimmen.
• Beispiel 19 umfasst den Gegenstand eines der Beispiele 15-18, und wobei das Programmieren der Bandbreitengrenze für jeden Schlitten umfasst, die Bandbreitengrenze an einen Netzschnittstellencontroller des entsprechenden Schlittens zu kommunizieren.
• Beispiel 20 umfasst den Gegenstand eines der Beispiele 15-19, und ferner umfassend: Überwachen, durch den Ressourcenmanagerserver, einer Bandbreitennutzung der Mehrzahl von Schlitten; Bestimmen, durch den Ressourcenmanagerserver, ob das Netz überlastet ist, basierend auf der Bandbreitennutzung der Mehrzahl von Schlitten; und Modifizieren, durch den Ressourcenmanagerserver, einer Bandbreitengrenze in Reaktion auf das Bestimmen, dass das Netz überlastet ist.
• Beispiel 21 umfasst den Gegenstand eines der Beispiele 15-20, und wobei das Überwachen der Bandbreitennutzung der Mehrzahl von Schlitten umfasst, Telemetriedaten von der Mehrzahl von Schlitten zu empfangen, die indikativ für die Bandbreite sind, die von jedem Schlitten genutzt wird.
• Beispiel 22 umfasst den Gegenstand eines der Beispiele 15-21, und wobei das Bestimmen, ob das Netz überlastet ist, das Bestimmen umfasst, ob eine Warteschlangentiefe des Netzes eine vorbestimmte Warteschlangentiefengrenze für eine vorbestimmte Zeitdauer überschreitet.
• Beispiel 23 umfasst den Gegenstand eines der Beispiele 15-22, und wobei die Warteschlangentiefe eine Switch-Port-Warteschlangentiefe, eine Netzschnittstellencontroller-Warteschlangentiefe oder eine Netzstapel-Warteschlangentiefe umfasst.
• Beispiel 24 umfasst den Gegenstand eines der Beispiele 15-23, und wobei das Modifizieren der Bandbreitengrenze umfasst: Identifizieren eines ersten Schlittens der Mehrzahl von Schlitten, die mit einem Fluss mit hoher Eingangsrate assoziiert sind; und Reduzieren einer Eingangsrate der Bandbreitengrenze für den ersten Schlitten.
• Beispiel 25 umfasst ein Verfahren zur Bandbreitenzuweisung, das Verfahren umfassend: Empfangen, durch einen Schlitten einer Mehrzahl von Schlitten, die kommunikativ mit einer Schicht von Switches gekoppelt sind, die kommunikativ mit einem Ressourcenmanagerserver in einem Netz gekoppelt sind, einer Bandbreitengrenze für den Schlitten vom Ressourcenmanagerserver; Programmieren, durch den Schlitten, der Bandbreitengrenze für einen Netzschnittstellencontroller des Schlittens; und Durchsetzen, durch den Netzschnittstellencontroller des Schlittens, der Bandbreitengrenze in Reaktion auf das Programmieren der Bandbreitengrenze.
• Beispiel 26 umfasst den Gegenstand von Beispiel 25, und ferner umfassend das Senden, durch den Schlitten, von Telemetriedaten, die indikativ für eine Nutzung des Netzschnittstellencontrollers sind, an den Ressourcenmanagerserver des Netzes.
• Beispiel 27 umfasst den Gegenstand eines der Beispiele 25 und 26, und wobei das Senden der Telemetriedaten umfasst, die Telemetriedaten durch den Netzschnittstellencontroller zu senden.
• Beispiel 28 umfasst den Gegenstand eines der Beispiele 25-27, und wobei die Telemetriedaten indikativ für eine Netzschnittstellencontroller-Warteschlangentiefe oder eine Netzstapel-Warteschlangentiefe sind.
• Beispiel 29 umfasst eine Rechenvorrichtung, umfassend: einen Prozessor; und einen Speicher, in dem eine Mehrzahl von Anweisungen gespeichert sind, die, wenn sie durch den Prozessor ausgeführt werden, die Rechenvorrichtung veranlassen, das Verfahren nach einem der Beispiele 15-28 durchzuführen.
• Beispiel 30 umfasst ein oder mehrere nicht-transitorische, computerlesbare Speichermedien, umfassend eine Mehrzahl von darauf gespeicherten Anweisungen, die in Reaktion auf das Ausführen darin resultieren, dass eine Rechenvorrichtung das Verfahren nach einem der Beispiele 15-28 durchführt.
• Beispiel 31 umfasst eine Rechenvorrichtung umfassend Mittel zum Durchführen des Verfahrens nach einem der Beispiele 15-28.
• Beispiel 32 umfasst einen Ressourcenmanagerserver zur Bandbreitenzuweisung, der Ressourcenmanagerserver umfassend: Topologiemanagerschaltungen zum Entdecken einer Topologie einer Mehrzahl von Schlitten, die mit einer Schicht von Switches gekoppelt sind, die kommunikativ mit dem Ressourcenmanagerserver gekoppelt sind; Modellkonstruktorschaltungen zum Konstruieren eines Modells der Netzkonnektivität zwischen der Mehrzahl von Schlitten und der Schicht von Switches basierend auf der Topologie; Bandbreitengrenzen-Programmierschaltungen zum (i) Bestimmen einer Oversubscription eines Netzes basierend auf dem Modell der Netzkonnektivität, wobei die Oversubscription auf einer verfügbaren Bandbreite für die Schicht von Switches und einer maximalen Bandbreite der Mehrzahl von Schlitten basiert, und (ii) Bestimmen einer Bandbreitengrenze für jeden Schlitten der Mehrzahl von Schlitten basierend auf der Oversubscription; und Bandbreitengrenzen-Programmierschaltungen zum Programmieren von jedem Schlitten der Mehrzahl von Schlitten mit der entsprechenden Bandbreitengrenze.
• Beispiel 33 umfasst den Gegenstand von Beispiel 32, und wobei das Konstruieren des Modells der Netzkonnektivität das Identifizieren umfasst, welche Schlitten der Mehrzahl von Schlitten mit einem bestimmten Switch der Schicht von Switches verbunden sind.
• Beispiel 34 umfasst den Gegenstand eines der Beispiele 32 und 33, und wobei das Bestimmen der Oversubscription umfasst, eine Netz-Uplink-Oversubscription für die Schicht von Switches zu bestimmen.
• Beispiel 35 umfasst den Gegenstand eines der Beispiele 32-34, und wobei das Bestimmen der Oversubscription umfasst, eine Speichereinrichtungsressourcen-Oversubscription der Mehrzahl von Schlitten zu bestimmen.
• Beispiel 36 umfasst den Gegenstand eines der Beispiele 32-35, und wobei das Programmieren der Bandbreitengrenze für jeden Schlitten umfasst, die Bandbreitengrenze an einen Netzschnittstellencontroller des entsprechenden Schlittens zu kommunizieren.
• Beispiel 37 umfasst den Gegenstand eines der Beispiele 32-36, und ferner umfassend Nutzungsmanagerschaltungen zum: Überwachen einer Bandbreitennutzung der Mehrzahl von Schlitten; Bestimmen, ob das Netz überlastet ist, basierend auf der Bandbreitennutzung der Mehrzahl von Schlitten; und Modifizieren einer Bandbreitengrenze in Reaktion auf eine Bestimmung, dass das Netz überlastet ist.
• Beispiel 38 umfasst den Gegenstand eines der Beispiele 32-37, und wobei das Überwachen der Bandbreitennutzung der Mehrzahl von Schlitten umfasst, Telemetriedaten von der Mehrzahl von Schlitten zu empfangen, die indikativ für die Bandbreite sind, die von jedem Schlitten genutzt wird.
• Beispiel 39 umfasst den Gegenstand eines der Beispiele 32-38, und wobei das Bestimmen, ob das Netz überlastet ist, das Bestimmen umfasst, ob eine Warteschlangentiefe des Netzes eine vorbestimmte Warteschlangentiefengrenze für eine vorbestimmte Zeitdauer überschreitet.
• Beispiel 40 umfasst den Gegenstand eines der Beispiele 32-39, und wobei die Warteschlangentiefe eine Switch-Port-Warteschlangentiefe, eine Netzschnittstellencontroller-Warteschlangentiefe oder eine Netzstapel-Warteschlangentiefe umfasst.
• Beispiel 41 umfasst den Gegenstand eines der Beispiele 32-40, und wobei das Modifizieren der Bandbreitengrenze umfasst: Identifizieren eines ersten Schlittens der Mehrzahl von Schlitten, die mit einem Fluss mit hoher Eingangsrate assoziiert sind; und Reduzieren einer Eingangsrate der Bandbreitengrenze für den ersten Schlitten.
• Beispiel 42 umfasst einen Schlitten zur Bandbreitenzuweisung, wobei der Schlitten kommunikativ mit einer Schicht von Switches gekoppelt ist, die kommunikativ mit einem Ressourcenmanagerserver in einem Netz gekoppelt sind, der Schlitten umfassend: Bandbreitenprogrammierschaltungen zum: (i) Empfangen einer Bandbreitengrenze für den Schlitten vom Ressourcenmanagerserver, und (ii) Programmieren der Bandbreitengrenze für einen Netzschnittstellencontroller des Schlittens; und Bandbreitengrenzenmanager-Schaltungen zum Durchsetzen, durch den Netzschnittstellencontroller, der Bandbreitengrenze in Reaktion auf das Programmieren der Bandbreitengrenze.
• Beispiel 43 umfasst den Gegenstand von Beispiel 42, und ferner umfassend Telemetriedatenmanager-Schaltungen zum Senden von Telemetriedaten, die indikativ für eine Nutzung des Netzschnittstellencontrollers sind, an den Ressourcenmanagerserver des Netzes.
• Beispiel 44 umfasst den Gegenstand eines der Beispiele 42 und 43, und wobei das Senden der Telemetriedaten umfasst, die Telemetriedaten durch den Netzschnittstellencontroller zu senden.
• Beispiel 45 umfasst den Gegenstand eines der Beispiele 42-44, und wobei die Telemetriedaten indikativ für eine Netzschnittstellencontroller-Warteschlangentiefe oder eine Netzstapel-Warteschlangentiefe sind.
• Beispiel 46 umfasst einen Ressourcenmanagerserver zur Bandbreitenzuweisung, der Ressourcenmanagerserver umfassend: Mittel zum Entdecken einer Topologie einer Mehrzahl von Schlitten, die mit einer Schicht von Switches gekoppelt sind, die kommunikativ mit dem Ressourcenmanagerserver in einem Netz gekoppelt sind; Mittel zum Konstruieren eines Modells der Netzkonnektivität zwischen der Mehrzahl von Schlitten und der Schicht von Switches basierend auf der Topologie; Mittel zum Bestimmen einer Oversubscription des Netzes basierend auf dem Modell der Netzkonnektivität, wobei die Oversubscription auf einer verfügbaren Bandbreite für die Schicht von Switches und einer maximalen Bandbreite der Mehrzahl von Schlitten basiert; Mittel zum Bestimmen einer Bandbreitengrenze für jeden Schlitten der Mehrzahl von Schlitten basierend auf der Oversubscription; und Mittel zum Programmieren von jedem Schlitten der Mehrzahl von Schlitten mit der entsprechenden Bandbreitengrenze.
• Beispiel 47 umfasst den Gegenstand von Beispiel 46, und wobei die Mittel zum Konstruieren des Modells der Netzkonnektivität Mittel zum Identifizieren umfassen, welche Schlitten der Mehrzahl von Schlitten mit einem bestimmten Switch der Schicht von Switches verbunden sind.
• Beispiel 48 umfasst den Gegenstand eines der Beispiele 46 und 47, und wobei die Mittel zum Bestimmen der Oversubscription Mittel zum Bestimmen einer Netz-Uplink-Oversubscription für die Schicht von Switches umfassen.
• Beispiel 49 umfasst den Gegenstand eines der Beispiele 46-48, und wobei die Mittel zum Bestimmen der Oversubscription Mittel zum Bestimmen einer Speichereinrichtungsressourcen-Oversubscription der Mehrzahl von Schlitten umfassen.
• Beispiel 50 umfasst den Gegenstand eines der Beispiele 46-49, und wobei die Mittel zum Programmieren der Bandbreitengrenze für jeden Schlitten Schaltungen zum Kommunizieren der Bandbreitengrenze an einen Netzschnittstellencontroller des entsprechenden Schlittens umfassen.
• Beispiel 51 umfasst den Gegenstand eines der Beispiele 46-50, und ferner umfassend: Mittel zum Überwachen einer Bandbreitennutzung der Mehrzahl von Schlitten; Mittel zum Bestimmen, ob das Netz überlastet ist, basierend auf der Bandbreitennutzung der Mehrzahl von Schlitten; und Mittel zum Modifizieren einer Bandbreitengrenze in Reaktion auf eine Bestimmung, dass das Netz überlastet ist.
• Beispiel 52 umfasst den Gegenstand eines der Beispiele 46-51, und wobei die Mittel zum Überwachen der Bandbreitennutzung der Mehrzahl von Schlitten Schaltungen zum Empfangen von Telemetriedaten von der Mehrzahl von Schlitten umfassen, die indikativ für die Bandbreite sind, die von jedem Schlitten genutzt wird.
• Beispiel 53 umfasst den Gegenstand eines der Beispiele 46-52, und wobei die Mittel zum Bestimmen, ob das Netz überlastet ist, Mittel zum Bestimmen umfassen, ob eine Warteschlangentiefe des Netzes eine vorbestimmte Warteschlangentiefengrenze für eine vorbestimmte Zeitdauer überschreitet.
• Beispiel 54 umfasst den Gegenstand eines der Beispiele 46-53, und wobei die Warteschlangentiefe eine Switch-Port-Warteschlangentiefe, eine Netzschnittstellencontroller-Warteschlangentiefe oder eine Netzstapel-Warteschlangentiefe umfasst.
• Beispiel 55 umfasst den Gegenstand eines der Beispiele 46-54, und wobei die Mittel zum Modifizieren der Bandbreitengrenze umfassen: Mittel zum Identifizieren eines ersten Schlittens der Mehrzahl von Schlitten, die mit einem Fluss mit hoher Eingangsrate assoziiert sind;und Mittel zum Reduzieren einer Eingangsrate der Bandbreitengrenze für den ersten Schlitten.
• Beispiel 56 umfasst einen Schlitten zur Bandbreitenzuweisung, wobei der Schlitten kommunikativ mit einer Schicht von Switches gekoppelt ist, die kommunikativ mit einem Ressourcenmanagerserver in einem Netz gekoppelt sind, der Schlitten umfassend: Schaltungen zum Empfangen einer Bandbreitengrenze für den Schlitten vom Ressourcenmanagerserver; Mittel zum Programmieren der Bandbreitengrenze für einen Netzschnittstellencontroller des Schlittens; und Mittel zum Durchsetzen, durch den Netzschnittstellencontroller des Schlittens, der Bandbreitengrenze in Reaktion auf das Programmieren der Bandbreitengrenze.
• Beispiel 57 umfasst den Gegenstand von Beispiel 56, und ferner umfassend Mittel zum Senden von Telemetriedaten, die indikativ für eine Nutzung des Netzschnittstellencontrollers sind, an den Ressourcenmanagerserver des Netzes.
• Beispiel 58 umfasst den Gegenstand eines der Beispiele 56 und 57, und wobei die Mittel zum Senden der Telemetriedaten Mittel zum Senden der Telemetriedaten durch den Netzschnittstellencontroller umfassen.
• Beispiel 59 umfasst den Gegenstand eines der Beispiele 56-58, und wobei die Telemetriedaten indikativ für eine Netzschnittstellencontroller-Warteschlangentiefe oder eine Netzstapel-Warteschlangentiefe sind.

Illustrative examples of the technologies disclosed herein are provided below. One embodiment of the technologies may include one or more and any combination of the examples described below.

• Example 1 includes a resource allocation server for bandwidth allocation, the resource manager server comprising: one or more processors; and one or more memory devices storing a plurality of instructions that, when executed by the one or more processors, cause the resource manager server to: discover a topology of a plurality of sleds coupled to a layer of switches communicatively coupled with the resource manager server; Constructing a model of network connectivity between the plurality of sleds and the layer of switches based on the topology; Determining an overubscription of a network based on the network connectivity model, wherein the overubscription is based on an available bandwidth for the layer of switches and a maximum bandwidth of the plurality of carriages; Determining a bandwidth limit for each of the plurality of carriages based on the overubscription; and programming each carriage of the plurality of carriages with the corresponding bandwidth limit.
• Example 2 includes the subject matter of Example 1, and wherein constructing the model of network connectivity comprises identifying which carriages of the plurality of carriages are connected to a particular switch of the layer of switches.
• Example 3 includes the subject matter of any of Examples 1 and 2, and wherein determining the overubscription comprises determining a network uplink overubscription for the layer of switches.
• Example 4 includes the subject matter of any of Examples 1-3, and wherein determining the overubscription comprises determining a storage resource overubscription of the plurality of carriages.
• Example 5 includes the subject matter of any of Examples 1-4, and wherein programming the bandwidth limit for each carriage comprises communicating the bandwidth limit to a network interface controller of the corresponding carriage.
• Example 6 includes the subject matter of any of Examples 1-5, and wherein the one or more storage devices have stored therein a plurality of instructions that, when executed by the one or more processors, further cause the resource manager server to: monitor bandwidth usage the plurality of carriages; Determining whether the network is congested based on the bandwidth utilization of the plurality of carriages; and modifying a bandwidth limit in response to a determination that the network is overloaded.
• Example 7 includes the subject matter of any of Examples 1-6, and wherein monitoring the bandwidth utilization of the plurality of A sled comprises receiving telemetry data from the plurality of sledges indicative of the bandwidth used by each sled.
• Example 8 includes the subject matter of any of Examples 1-7, and wherein determining whether the network is congested comprises determining whether a network queuing depth exceeds a predetermined queue depth limit for a predetermined period of time.
• Example 9 includes the subject matter of any of Examples 1-8, and wherein the queue depth includes a switch port queue depth, a network interface controller queue depth, or a network stack queue depth.
• Example 10 includes the subject matter of any of Examples 1-9, and wherein modifying the bandwidth limit comprises: identifying a first carriage of the plurality of carriages associated with a high input rate flow; and reducing an input rate of the bandwidth limit for the first carriage.
• Example 11 includes a carriage for bandwidth allocation, wherein the carriage is communicatively coupled to a layer of switches communicatively coupled to a resource manager server in a network, the carriage comprising: one or more processors; and one or more memory devices storing a plurality of instructions that, when executed by the one or more processors, cause the carriage to: receive a bandwidth limit for the carriage from the resource manager server; Programming the bandwidth limit for a network interface controller of the carriage; and enforcing, by the network interface controller, the bandwidth limit in response to programming the bandwidth limit.
• Example 12 includes the subject matter of Example 11, and wherein the one or more storage devices have stored therein a plurality of instructions that, when executed by the one or more processors, further cause the carriage to provide telemetry data indicative of usage of the network interface controller are to be sent to the resource manager server of the network.
• Example 13 includes the subject matter of any of Examples 11 and 12, and wherein transmitting the telemetry data comprises transmitting the telemetry data through the network interface controller.
• Example 14 includes the subject matter of any of Examples 11-13, and wherein the telemetry data is indicative of a network interface controller queue depth or a network stack queue depth.
• Example 15 includes a method of bandwidth allocation, the method comprising: detecting, by a resource manager server of a network, a topology of a plurality of carriages coupled to a layer of switches communicatively coupled to the resource manager server; Construct, by the resource manager server, a model of network connectivity between the plurality of carriages and the layer of switches based on the topology; Determining, by the resource manager server, an over-subscription of the network based on the network connectivity model, wherein the over-subscription is based on available bandwidth for the layer of switches and a maximum bandwidth of the plurality of carriages; Determining, by the resource manager server, a bandwidth limit for each of the plurality of carriages based on the overubscription; and programming, by the resource manager server, each carriage of the plurality of carriages with the corresponding bandwidth limit.
• Example 16 includes the subject matter of Example 15, and constructing the model of network connectivity includes identifying which carriages of the plurality of carriages are connected to a particular switch of the layer of switches.
• Example 17 includes the subject matter of one of Examples 15 and 16, and wherein determining the overubscription comprises determining a network uplink overubscription for the layer of switches.
• Example 18 includes the subject matter of any of Examples 15-17, and wherein determining the overubscription comprises determining a storage resource over-subscription of the plurality of carriages.
• Example 19 includes the subject matter of any of Examples 15-18, and wherein programming the bandwidth limit for each carriage comprises communicating the bandwidth limit to a network interface controller of the corresponding carriage.
• Example 20 includes the subject matter of any of Examples 15-19, and further comprising: monitoring, by the resource manager server, a bandwidth usage of the plurality of carriages; Determine by the Resource manager server, whether the network is congested, based on the bandwidth usage of the plurality of carriages; and modifying, by the resource manager server, a bandwidth limit in response to determining that the network is congested.
• Example 21 includes the subject matter of any of Examples 15-20, and wherein monitoring the bandwidth utilization of the plurality of carriages comprises receiving telemetry data from the plurality of carriages indicative of the bandwidth utilized by each carriage.
• Example 22 includes the subject matter of any of Examples 15-21, and wherein determining whether the network is congested comprises determining whether a network queuing depth exceeds a predetermined queue depth limit for a predetermined period of time.
• Example 23 includes the subject matter of any of Examples 15-22, and wherein the queue depth includes a switch port queue depth, a network interface controller queue depth, or a network stack queue depth.
• Example 24 includes the subject matter of any of Examples 15-23, and wherein modifying the bandwidth limit comprises: identifying a first carriage of the plurality of carriages associated with a high input rate flow; and reducing an input rate of the bandwidth limit for the first carriage.
• Example 25 includes a method of bandwidth allocation, the method comprising: receiving, by a carriage of a plurality of carriages communicatively coupled to a layer of switches communicatively coupled to a resource manager server in a network, a bandwidth limit for the carriage from the resource manager server ; Programming, by the carriage, the bandwidth limit for a network interface controller of the carriage; and enforcing, by the network interface controller of the carriage, the bandwidth limit in response to programming the bandwidth limit.
• Example 26 includes the subject matter of Example 25, and further comprising transmitting, by the carriage, telemetry data indicative of usage of the network interface controller to the resource manager server of the network.
• Example 27 includes the subject matter of any of Examples 25 and 26, and wherein transmitting the telemetry data comprises transmitting the telemetry data through the network interface controller.
• Example 28 includes the subject matter of any of Examples 25-27, and wherein the telemetry data is indicative of a network interface controller queue depth or a network stack queue depth.
• Example 29 includes a computing device comprising: a processor; and a memory storing a plurality of instructions which, when executed by the processor, cause the computing device to perform the method of any one of Examples 15-28.
• Example 30 includes one or more non-transitory computer-readable storage media comprising a plurality of instructions stored thereon that result in response to execution in that a computing device performs the method of any one of Examples 15-28.
• Example 31 includes a computing device comprising means for performing the method of any one of Examples 15-28.
• Example 32 includes a resource allocation server for bandwidth allocation, the resource manager server comprising: topology manager circuits for discovering a topology of a plurality of carriages coupled to a layer of switches communicatively coupled to the resource manager server; Model constructor circuits for constructing a model of network connectivity between the plurality of carriages and the layer of switches based on the topology; Bandwidth limit programming circuits for (i) determining an overubscription of a network based on the network connectivity model, wherein the overubscription is based on an available bandwidth for the layer of switches and a maximum bandwidth of the plurality of carriages, and (ii) determining a bandwidth limit for each Carriages of the plurality of carriages based on the overubscription; and bandwidth limit programming circuits for programming each of the plurality of carriages with the corresponding bandwidth limit.
• Example 33 includes the subject matter of Example 32, and wherein constructing the model of network connectivity comprises identifying which carriages of the plurality of carriages are connected to a particular switch of the layer of switches.
• Example 34 includes the subject matter of any of Examples 32 and 33, and wherein determining the overubscription comprises, a network uplink Oversubscription for the layer of switches to determine.
• Example 35 includes the subject matter of any of Examples 32-34, and wherein determining the overubscription comprises determining a storage resource over-subscription of the plurality of carriages.
• Example 36 includes the subject matter of any of Examples 32-35, and wherein programming the bandwidth limit for each carriage comprises communicating the bandwidth limit to a network interface controller of the corresponding carriage.
• Example 37 includes the subject matter of any of Examples 32-36, and further comprising usage manager circuits for: monitoring bandwidth usage of the plurality of carriages; Determining whether the network is congested based on the bandwidth utilization of the plurality of carriages; and modifying a bandwidth limit in response to a determination that the network is overloaded.
• Example 38 includes the subject matter of any of Examples 32-37, and wherein monitoring the bandwidth utilization of the plurality of carriages comprises receiving telemetry data from the plurality of carriages indicative of the bandwidth utilized by each carriage.
• Example 39 includes the subject matter of any of Examples 32-38, and wherein determining whether the network is congested includes determining whether a network queuing depth exceeds a predetermined queue depth limit for a predetermined period of time.
• Example 40 includes the subject matter of any of Examples 32-39, and wherein the queue depth includes a switch port queue depth, a network interface controller queue depth, or a network stack queue depth.
• Example 41 includes the subject matter of any of Examples 32-40, and wherein modifying the bandwidth limit comprises: identifying a first carriage of the plurality of carriages associated with a high input rate flow; and reducing an input rate of the bandwidth limit for the first carriage.
• Example 42 includes a carriage for bandwidth allocation, wherein the carriage is communicatively coupled to a layer of switches communicatively coupled to a resource manager server in a network comprising carriages: bandwidth programming circuits for: (i) receiving a bandwidth limit for the carriage from the resource manager server; and (ii) programming the bandwidth limit for a network interface controller of the carriage; and bandwidth limit manager circuits for enforcing, by the network interface controller, the bandwidth limit in response to programming the bandwidth limit.
• Example 43 includes the subject matter of Example 42, and further comprising telemetry data manager circuitry for sending telemetry data indicative of usage of the network interface controller to the resource manager server of the network.
• Example 44 includes the subject matter of any of Examples 42 and 43, and wherein transmitting the telemetry data comprises transmitting the telemetry data through the network interface controller.
• Example 45 includes the subject matter of one of Examples 42-44, and wherein the telemetry data is indicative of a network interface controller queue depth or a network stack queue depth.
• Example 46 includes a resource allocation server for bandwidth allocation, the resource manager server comprising: means for discovering a topology of a plurality of carriages coupled to a layer of switches communicatively coupled to the resource manager server in a network; Means for constructing a network connectivity model between the plurality of carriages and the layer of switches based on the topology; Means for determining an overubscription of the network based on the network connectivity model, wherein the overubscription is based on an available bandwidth for the layer of switches and a maximum bandwidth of the plurality of carriages; Means for determining a bandwidth limit for each of the plurality of carriages based on the overubscription; and means for programming each carriage of the plurality of carriages with the corresponding bandwidth limit.
• Example 47 includes the subject matter of Example 46, and wherein the means for constructing the network connectivity model comprises means for identifying which carriages of the plurality of carriages are connected to a particular switch of the layer of switches.
• Example 48 includes the subject matter of any of Examples 46 and 47, and wherein the means for determining the overubscription comprises means for determining a network uplink overubscription for the layer of switches.
• Example 49 includes the subject matter of any of Examples 46-48, and wherein the means for determining overubscription comprises means for determining a storage resource over-subscription of the plurality of sleds.
• Example 50 includes the subject matter of any of Examples 46-49, and wherein the means for programming the bandwidth limit for each carriage includes circuitry for communicating the bandwidth limit to a network interface controller of the corresponding carriage.
• Example 51 includes the subject matter of any of Examples 46-50, and further comprising: means for monitoring a bandwidth usage of the plurality of carriages; Means for determining whether the network is congested based on the bandwidth usage of the plurality of carriages; and means for modifying a bandwidth limit in response to a determination that the network is overloaded.
• Example 52 includes the subject matter of any of Examples 46-51, and wherein the means for monitoring the bandwidth utilization of the plurality of carriages includes circuitry for receiving telemetry data from the plurality of carriages indicative of the bandwidth utilized by each carriages.
• Example 53 includes the subject matter of any of Examples 46-52, and wherein the means for determining whether the network is congested comprises means for determining whether a network queuing depth exceeds a predetermined queue depth limit for a predetermined period of time.
• Example 54 includes the subject matter of any of Examples 46-53, and wherein the queue depth includes a switch port queue depth, a network interface controller queue depth, or a network stack queue depth.
• Example 55 includes the subject matter of any of Examples 46-54, and wherein the means for modifying the bandwidth limit comprises: means for identifying a first carriage of the plurality of carriages associated with a high input rate flow; and means for reducing an input rate of Bandwidth limit for the first slide.
• Example 56 includes a carriage for bandwidth allocation, wherein the carriage is communicatively coupled to a layer of switches communicatively coupled to a resource manager server in a network comprising carriages: circuitry for receiving a bandwidth limit for the carriage from the resource manager server; Means for programming the bandwidth limit for a network interface controller of the carriage; and means for enforcing, by the network interface controller of the carriage, the bandwidth limit in response to programming the bandwidth limit.
• Example 57 includes the subject matter of Example 56, and further comprising means for transmitting telemetry data indicative of utilization of the network interface controller to the resource manager server of the network.
• Example 58 includes the subject matter of any of Examples 56 and 57, and wherein the means for transmitting the telemetry data comprises means for transmitting the telemetry data through the network interface controller.
• Example 59 includes the subject matter of one of Examples 56-58, and wherein the telemetry data is indicative of a network interface controller queue depth or a network stack queue depth.

Claims (25)

Resource manager server for bandwidth allocation, the resource manager server comprising: one or more processors; and one or more memory devices storing a plurality of instructions that, when executed by the one or more processors, cause the resource manager server to: Detecting a topology of a plurality of sleds coupled to a layer of switches communicatively coupled to the resource manager server; Constructing a model of network connectivity between the plurality of sleds and the layer of switches based on the topology; Determining an overubscription of a network based on the network connectivity model, wherein the overubscription is based on an available bandwidth for the layer of switches and a maximum bandwidth of the plurality of carriages; Determining a bandwidth limit for each of the plurality of carriages based on the overubscription; and Program each carriage of the plurality of carriages with the appropriate bandwidth limit. Resource manager server Claim 1 wherein constructing the model of network connectivity comprises identifying which carriages of the plurality of carriages are connected to a particular switch of the layer of switches. Resource manager server Claim 1 wherein determining the overubscription comprises determining a network uplink overubscription for the layer of switches. Resource manager server Claim 1 wherein determining the overubscription comprises determining a storage resource over-subscription of the plurality of carriages. Resource manager server Claim 1 wherein programming the bandwidth limit for each carriage comprises communicating the bandwidth limit to a network interface controller of the corresponding carriage. Resource Manager server after one of Claims 1 - 5 wherein the one or more memory devices have stored therein a plurality of instructions that, when executed by the one or more processors, further cause the resource manager server to: monitor a bandwidth usage of the plurality of carriages; Determining whether the network is congested based on the bandwidth utilization of the plurality of carriages; and modifying a bandwidth limit in response to a determination that the network is overloaded. Resource manager server Claim 6 wherein monitoring the bandwidth usage of the plurality of carriages comprises receiving telemetry data from the plurality of carriages indicative of the bandwidth utilized by each carriages. Resource manager server Claim 7 wherein determining whether the network is overloaded comprises determining whether a network queuing depth exceeds a predetermined queue depth limit for a predetermined period of time. Resource manager server Claim 8 where the queue depth includes a switch port queue depth, a network interface controller queue depth, or a network stack queue depth. Resource manager server Claim 6 wherein modifying the bandwidth limit comprises: identifying a first carriage of the plurality of carriages associated with a high input rate flow; and reducing an input rate of the bandwidth limit for the first carriage. A sled for sled assignment, the sled being communicatively coupled to a layer of switches communicatively coupled to a resource manager server in a network, the sled comprising: one or more processors; and one or more memory devices storing a plurality of instructions which, when executed by the one or more processors, cause the carriage to: Receiving a bandwidth limit for the carriage from the resource manager server; Programming the bandwidth limit for a network interface controller of the carriage; and Enforce, by the network interface controller, the bandwidth limit in response to programming the bandwidth limit. Sledges behind Claim 11 wherein the one or more storage devices have stored therein a plurality of instructions that, when executed by the one or more processors, further cause the carriage to send telemetry data indicative of usage of the network interface controller to the resource manager server of the network to send. Sledges behind Claim 12 wherein sending the telemetry data comprises transmitting the telemetry data through the network interface controller. Bandwidth allocation method, the method comprising: Detecting, by a resource manager server of a network, a topology of a plurality of carriages coupled to a layer of switches communicatively coupled to the resource manager server; Construct, by the resource manager server, a model of network connectivity between the plurality of carriages and the layer of switches based on the topology; Determining, by the resource manager server, an over-subscription of the network based on the network connectivity model, wherein the over-subscription is based on available bandwidth for the layer of switches and a maximum bandwidth of the plurality of carriages; Determining, by the resource manager server, a bandwidth limit for each of the plurality of carriages based on the overubscription; and Programming, by the resource manager server, from each carriage of the plurality of carriages with the corresponding bandwidth limit. Method according to Claim 14 wherein constructing the model of network connectivity comprises identifying which carriages of the plurality of Slides are connected to a specific switch of the layer of switches. Method according to Claim 14 wherein programming the bandwidth limit for each carriage comprises communicating the bandwidth limit to a network interface controller of the corresponding carriage. Method according to Claim 14 , further comprising: monitoring, by the resource manager server, a bandwidth usage of the plurality of carriages; Determining, by the resource manager server, whether the network is congested based on the bandwidth usage of the plurality of carriages; and modifying, by the resource manager server, a bandwidth limit in response to determining that the network is congested. Method according to Claim 17 wherein monitoring the bandwidth usage of the plurality of carriages comprises receiving telemetry data from the plurality of carriages indicative of the bandwidth utilized by each carriages. Method according to Claim 18 wherein determining whether the network is overloaded comprises determining whether a network queuing depth exceeds a predetermined queue depth limit for a predetermined period of time. Method according to Claim 17 wherein modifying the bandwidth limit comprises: identifying a first carriage of the plurality of carriages associated with a high input rate flow; and reducing an input rate of the bandwidth limit for the first carriage. Bandwidth allocation method, the method comprising: Receiving, by a carriage of a plurality of carriages communicatively coupled to a layer of switches communicatively coupled to a resource manager server in a network, a bandwidth limit for the carriage from the resource manager server; Programming, by the carriage, the bandwidth limit for a network interface controller of the carriage; and Enforce, by the network interface controller of the carriage, the bandwidth limit in response to programming the bandwidth limit. Method according to Claim 21 further comprising transmitting, by the carriage, telemetry data indicative of usage of the network interface controller to the resource manager server of the network. A computing device, comprising: a processor; and a memory storing a plurality of instructions which, when executed by the processor, cause the computing device to perform the method according to any one of Claims 14 - 22 perform. One or more non-transitory computer-readable storage media comprising a plurality of instructions stored thereon that, in response to execution, result in a computing device following the method of any one of Claims 14 - 22 performs. A computing device comprising means for performing the method according to any one of Claims 14 - 22 ,

Images