RU2754605C1 - Computer network of a high-performance computer system - Google Patents

Abstract

FIELD: computer technology; communication systems.SUBSTANCE: computer network is organized in such a way that computing nodes, including an adapter block and one or several computing modules with arithmetic processors, are arranged orthogonally from one to four dimensions. Communication between computing nodes located in the first dimension occurs using adapter blocks connected by bidirectional communication channels over a fully connected topology without the use of switch blocks. In other dimensions, computing nodes and their corresponding adapter blocks are combined using switch blocks or using switch assemblies, which are a set of switch blocks.EFFECT: technical result is an increase in the bandwidth of the computer network by reducing its diameter.1 cl, 5 dwg

Description

The invention relates to the field of computer technology and communication systems and can be used to create high-performance computing systems (VVS).

Air Force computing networks are usually divided into two groups: with direct topology and indirect topology (Dally W., Towles B. Principles and Practices of Interconnection Networks. San Francisco. Morgan Kaufmann Publishers, 2004. Chapter 3.1.2. ).

Computing networks with an indirect topology have a small diameter, depending on the number of levels in the topology graph, and, accordingly, high communication characteristics. However, a further reduction in the diameter of a computer network is provided only by a decrease in the number of layers, which is achieved by an increase in the number of network ports in switches, and, consequently, a significant increase in their cost and the use of expensive technologies. Indirect topologies include, for example, the thickened tree topology, often used in existing air forces, both foreign and domestic (Fat Tree [Internet resource] 07/10/2020 https://sites.google.com/site/exemsenko/6-topologia -kommunikacionnyh-setej-multiprocessornyh-sistem-sovremennye-superkomputery-i-vs-primenenie-ih-v-socialno-sfere).

Computing networks with a direct topology, on the other hand, are distinguished by the cheapness of components (switches, as a rule, have a small number of network ports) and ease of expansion, largely independent of the number of ports. But this causes a significant increase in the diameter of the computer network and the average length of the communication path, increases the probability of message conflicts and, accordingly, reduces the throughput, i.e. degrades the communication characteristics of the entire computer network as a whole. Computer networks with direct topology include the well-known topologies of computer networks Mesh (Mesh, [Internet resource] 07/10/2020 https://ru.wikipedia.org/wiki/%D0%AF%Dl%87%D0%B5%D0 % B8% Dl% 81 ⁰ / oD1% 82% D0% B0% D1% 82% D1% 82% D0% BF% D0% BF% D0% BE% D0% BB% D 0% BE% D0% B3% D0 % B8% Dl% 8F) , Top (Torus), Hypercube (Hypercub) ([Internet resource] 07/10/2020 https://sites.google.com/site/exemsenko/6-topologia-kommunikacionnyh-setej-multiprocessornyh-sistem -sovremennye-superkomputery-i-vs-primenenie-ih-v-socialno-sfere) and TOFU (TOFU [Internet resource] 07/10/2020 https://www.osp.ru/os/2009/10/11180288/).

Crispin

Maria E.

Pedro

& Jose Duato// The K-ary N-direct S-indirect family of topologies for large-scale interconnection network, опубликовано 05.02.2016, The Journal of Supercomputing volume 72, pages 1035-1062, [ресурс интернет] http://doi.org/10.1007/s11227-016-1640-z), образующая гибридное семейство топологий K-ary N-direct S-indirect.The Air Force computer network (Roberto

Crispin

Maria E.

Pedro

& Jose Duato // The K-ary N-direct S-indirect family of topologies for large-scale interconnection network, published 02/05/2016, The Journal of Supercomputing volume 72, pages 1035-1062, [Internet resource] http: // doi.org/10.1007/s11227-016-1640-z), forming a hybrid family of K-ary N-direct S-indirect topologies.

The family of topologies K-ary N-direct S-indirect is an attempt to combine the best features of direct and indirect topologies in a computer network, i.e. to achieve significant improvement in the communication characteristics of computer networks, such as diameter, cost and power consumption, while ensuring high bandwidth. In a computer network with a K-ary N-direct S-indirect topology, two types of communication devices are used: a router (adapter) and a switch.

The known Air Force computer network contains computing nodes, each of which contains a corresponding router and one or more computing modules including one or more arithmetic processors. Compute nodes are positioned (numbered) orthogonally along dimensions (coordinate directions), similar to straight Mesh or Torus topologies. Communication between computing nodes located in the same dimension occurs either using single switches or using several switches combined according to some topology, for example, Fat Tree. The router provides communication of computing modules with the network and change of measurements when transmitting transit messages. The family of topologies K-ary N-direct S-indirect got its name from its three main parameters: K - the number of computing nodes in one dimension, N - the number of measurements in the direct topology and S - the number of switching levels in the indirect section of the topology. This Air Force computer network has been chosen as the closest analogue.

The main disadvantage of the closest analogue is that the Air Force computer network, created according to the K-ary N-direct S-indirect topology, has a low connectivity equal to the number of network ports of the router involved, and, consequently, low reliability. For example, when organizing a computer network with a two-dimensional topology K-ary N-direct S-indirect, each router uses only two network ports and, therefore, the connectivity value is two. In case of failure of any communication channel, a "hanging peak" appears in the computer network. Although the compute node and router are in good working order, the router loses the ability to transmit transit messages. In an Air Force computer network with a K-ary N-direct S-indirect topology, the algorithm for bypassing such faults is rather complicated and difficult to implement in practice. An increase in connectivity with a fixed number of computational nodes is achieved by increasing the number of measurements in the computational network. This leads to an increase in the diameter of the computer network, since with each new measurement, the diameter of the computer network increases by two "transit sections" ("transit section" is a section of the computer network located between two communication devices), and, consequently, this leads to an increase in the delay at transmission of messages and the increase in the cost of the computer network.

Another disadvantage of the known computer network is that the minimum route length is equal to two "transit sections", which means that a message passes at least three communication devices, which increases the average distance traveled by messages, which also leads to an increase in the delay in message transmission.

A technical problem is the creation of a computer network of a high-performance computing system with high bandwidth and reliability, which provides lower costs for communication of messages between computing nodes.

The technical results to be achieved by the invention are an increase in throughput, an increase in reliability and a decrease in the cost of a computer network.

Technical results are achieved by the fact that in the Air Force computer network containing computing nodes, each of which includes an adapter block and Μ computing modules including arithmetic processors, where Μ = 1, 2, ..., while the computing nodes are located (numbered) orthogonally in N measurements, where N = 1, 2, ..., in each dimension are located up to K computational nodes, where K = 2, 3, ..., computational nodes located in one dimension, except for the first dimension, are interconnected by connecting the corresponding adapter blocks through the corresponding switch unit or the corresponding switch assembly, what is new is that in the first dimension the computational nodes are interconnected in a fully connected topology by connecting the corresponding network ports of the corresponding adapter units with bidirectional communication channels.

In the claimed Air Force computer network, two types of communication devices are used: an adapter unit (AB) and a switching unit (KB), each having n network ports, where n = 1, 2…. The claimed computer network forms the MKNS topology, which means the modernized KNS. At the same time Μ in the name - determines the number of computational modules in a computational node. K - determines the number of computational nodes in the MKNS topology dimensions (in the first dimension the value of the maximum number of computational nodes is n-2, in other dimensions the number of computational nodes is unlimited) in different dimensions the number of computational nodes can be different, the numbering of computational nodes and, accordingly, AB in each dimension starts from 0. N - defines the number of used dimensions (coordinate directions) in MKNS topology (up to four dimensions). S - defines the number of levels in the indirect section of the MKNS topology (one level when using a switch unit, two or three levels when using a switch assembly (KSb)). Each computing node is identified by its coordinate in the computing network.

In the claimed Air Force computer network, due to the connection of computing nodes in the first dimension to each other in a fully connected topology ("each with each"), the diameter of the computing network is reduced by one "transit section", which leads to an increase in throughput. At the same time, the connectivity of the computer network increases, leading to an increase in its reliability, which is provided by a large number of alternative routes. The minimum connectivity SV _min in the described scheme is calculated by the following formula:

SV _min = (K ₁ -l) + (Nl);

where K ₁ is the number of computing nodes in the first dimension,

N is the number of measurements.

A decrease in the number of switching units used leads to a decrease in the cost of the Air Force computer network.

To organize a fully connected topology in the first dimension, ports of each AB are involved from the first to n-3, so the maximum number of computational nodes in this dimension is n-2. To organize the transfer of information packets between the adapter blocks of computing nodes located in the first dimension, (n-3) × (n-2) / 2 bidirectional communication channels are used, connected in a certain way.

Port n-2 of each unit is used for communication with the corresponding KB or KSb, which provide message transfer in the second dimension of the MKNS topology. Port n-1 of each unit is used for communication with the corresponding KB or KSb, which provide the transfer of messages in the third dimension of the MKNS topology. Port n of each AU is used for communication with the corresponding KB or KSb, which ensure the transmission of messages in the fourth dimension of the MKNS topology.

In one dimension of MKNS topology, either KB or KSB can be used. In different dimensions of the MKNS topology, both KB and KSb can be used. The use of MKNS topology in one dimension instead of KSB KSb adds to the diameter of the computer network topology two "hops" for each dimension in which KSb was used.

FIG. 1 shows a block diagram of the Air Force computer network for Μ = 2, where Μ is the number of computing modules in one VU; N = 2, where N is the number of measurements; K = 4, where K is the number of computational nodes in one dimension; S = 1, where S is the number of switching levels in the indirect section of the topology, in Fig. 2 shows an adapter block; FIG. 3 - switching unit, in Fig. 4 shows possible switching options in the indirect section of the topology, FIG. 5 shows a variant of the switch assembly.

The computing network of the Air Force (Fig. 1) includes computing nodes VU1, VU2, ..., VU16 located (numbered) orthogonally in two dimensions and switching units 17, 18, 19, 20.

Each computing node (VU) VU1 (VU2, VU3, ..., VU16) contains a corresponding adapter unit 21 and two corresponding computing modules (VM) 24 and 25. Computing modules 24 and 25 are connected to the corresponding adapter unit 21 via PCI Express ports 22 (PCI0) and 23 (PC11), respectively. Each computing node VU1 (VU2, VU3, ..., VU16) is identified by its coordinate in the computer network.

Computing nodes VU1, VU 2, ..., VU16 communicate with each other using two communication devices: the corresponding adapter unit (AB) and the corresponding switching unit (KB).

Each adapter block 21 (Fig. 2) contains a full-matrix switch providing switching of m PCI Express controllers and n network ports. Network ports in the adapter block are numbered starting with one. The adapter block is designed to implement the interface between the computing modules included in the computing node and the computing network; the adapter block also provides measurement change when transmitting transit messages between computing nodes. The functionality of the adapter block corresponds to the functionality of the router in the computer network of the closest analogue.

Each switching unit 17 (18, 19, 20) (Fig. 3) contains a full-matrix switch providing switching of n network ports. The numbering of network ports in a switching unit starts from one. Switching unit 17 (18, 19, 20) provides transit of messages between computing nodes located in one dimension (except for the first dimension). The functionality of the switching unit corresponds to the functionality of the switch in the computer network of the closest analogue.

In the first dimension 26 (Fig. 1), the computing nodes VU5, VU6, VU7, VU8 are respectively assigned coordinates (identifiers) (0,1); (1,1); (2.1); (3,1), which have the same second coordinate value.

Computing nodes VU5, VU6, VU7, VU8 are interconnected in a fully connected topology "each with each" by connecting the corresponding network ports of the corresponding adapter blocks by bidirectional communication channels.

In the second dimension (Fig. 1), the computing nodes VU2, VU6, VU10, VU14 are assigned the corresponding coordinates (1,0); (1,1); (1,2); (1,3), which have the same value of the first coordinate. Computing nodes VU2, VU6, VU10, VU14 are interconnected by connecting the corresponding adapter blocks through the switching unit 18.

In the second, third and fourth dimensions, adapter blocks 21 of computing nodes VU1 (VU2, VU3, ..., VU16) are combined by either one switch unit or using a switch assembly (Fig. 4). The number of network ports of the switch unit (equal to n) limits the number of computational nodes used in the second, third and fourth dimensions. To remove this limitation, switch assemblies are used, which make it possible not to limit the number of computational nodes in these measurements.

FIG. 5 shows a variant of the implementation of a two-level switch assembly using the Fat Tree topology (thickened tree). Switch units 27 form the first level of the switch assembly. Their network ports numbered 6-10 are intended for connection with adapter blocks of computing nodes, and network ports numbered 1-5 are intended for connection with the corresponding network ports of switch blocks 28 of the second level in an indirect Fat Tree topology. Such a switch assembly, consisting of switch units each having n network ports, allows connecting up to 2 × n computing nodes in the second, third and fourth dimensions of a computer network. The advantage of the commutator assembly is that it does not reduce such a communication characteristic as the width of the bisection.

The computing network of a high-performance computing system operates as follows.

Previously, each VM is assigned a serial number in the VC and each VU is assigned a corresponding identifier (coordinates by measurements), which uniquely determine the positions of the transmitting and receiving devices in the computer network.

The transfer of information between computational nodes with the same coordinates means the transfer of information between computational modules connected to one computational node, only through the AB of this computational node. For example, the transfer of information from VM 25 located in VU1 to VM 24 located in VU1, having the same coordinates, is carried out as follows. From VM 25 (VU1) through the corresponding port 23 (PCI 1), information is transferred to AB 21 (VU1) and then through port 22 (PCI 0) to VM 24 (VU1).

The transfer of information between computational nodes, whose coordinates differ only in the first dimension, occurs as follows. Information from the VM of the source of one VU through the corresponding PCI port 23 or 22 is transferred to its AB, then through the corresponding network port of this AB the information is transmitted to the corresponding network port of the AB of another VU, and then through the corresponding PCI port 22 or 23 to the VM receiver. For example, the transfer of information from VM 25 located in VU1 to VM 24 located in VU3, having coordinates that differ only in the first dimension, is carried out as follows. From VM 25 (VU1) through the corresponding port 23 (PCI 1) information is transferred to AB 21 (VU1), then through the network port 2 of this AB21 (VU1) information is transmitted to the network port 1 AB21 (VU3) and then through port 22 (PCI 0) in VM 24 (VU3).

The transfer of information between computational nodes, whose coordinates differ either only in the second, or only in the third, or only in the fourth dimensions, proceed as follows. Information from the VM of the source of one VU through the corresponding PCI port 23 or 22 is transferred to its AB, then through the network port of this AB corresponding to this measurement, the information is transferred to the corresponding network port of the corresponding KB, from which through the corresponding network port it is transmitted to the corresponding network port of the AB of another VU , and then through the corresponding PCI port 22 or 23 in the VM receiver. For example, the transfer of information from VM 24 (VU1) to VM 25 (VU9), having coordinates that differ only in the second dimension, is carried out as follows. From VM 24 (VU1) through the corresponding port 22 (PCI0) information will be transferred to AB21 (VU1), then through the network port 8 of this AB21 (VU1) information is transmitted to the network port 1 KB 17, then through its network port 3 to the network port 8 AB21 (VU9), and then through the corresponding port 23 (PCI21) to VM 25 (VU9).

The transfer of information between computational nodes, whose coordinates differ in several dimensions, proceed as follows. Information from the VM of the source of one WU through the corresponding PCI port 23 or 22 is transmitted to its AB, in which the coordinates of the current WU and the WU of the receiver are compared. The comparison starts with the first measurement in order. If the values of coordinates in the first dimension do not match, then through the corresponding network port of this AB the information is transmitted to the corresponding network port of the AB of another VU. Otherwise, if the coordinate values for the second, or third, or fourth dimension did not match, then through the network port of this AB corresponding to this measurement, the information is transferred to the corresponding network port of the corresponding KB, from which, through the corresponding network port, it is transmitted to the corresponding network port of the AB of the other VU. In the achieved AB, the process of comparing the coordinates of this current WA and the WA of the receiver takes place again. The process of transferring information from the VU to the VU is repeated until the coordinate of the reached VU does not coincide with the coordinate of the VU of the receiver, then through the corresponding PCI port 22 or 23 of the adapter unit, the information will be sent to the VM receiver. For example, the transfer of information from VM 24 (VU1) to VM 24 (VU11), having coordinates that differ in two dimensions, is carried out as follows. From VM 24 (VU1) through the corresponding port 22 (PCI0) information is transmitted to AB21 (VU1), then port 2 of this AB21 (VU1) information is transmitted to network port 1 AB21 (V3). In AB21 (VU3), it occurs through a network change in the measurement of information transfer, through the network port 8 of this AB21 (VU3) information is transmitted to the network port 1 KB 19, then through its network port 3 to the network port 8 AB 21 (VU11) and then through the port 22 corresponding to PCI0 in VM 24 (VU11).

Claims (1)

A computing network of a high-performance computing system containing computing nodes, each of which includes an adapter unit and Μ computing modules including arithmetic processors, where Μ = 1, 2, ..., while the computing nodes are located orthogonally along N dimensions, where N = 1, 2 , ..., up to K computational nodes are located in each dimension, where K = 2, 3, ..., computational nodes located in one dimension, except for the first dimension, are interconnected by connecting the corresponding network ports of the corresponding adapter blocks through the corresponding network ports of the corresponding switch unit or corresponding switch assembly, characterized in that in the first dimension the computing nodes are interconnected in a fully connected topology by connecting the corresponding network ports of the corresponding adapter units by bidirectional communication channels.

Images