FR3091764A1 - Modification of the characteristics of fast network interconnection systems: Simulation and Application to predict the performance of parallel applications - Google Patents

Abstract

.

Description

Modification of the characteristics of the interconnection systems of fast networks: Simulation and Application to the prediction of the performance of parallel applications

1) Name of editor(s) Stephan Jaure, Nourredine Taguelmint

2) Proposed title Modification of the characteristics of fast network interconnection systems: Simulation and Application to the prediction of the performance of parallel applications.

3) Technical area

This work has been implemented in the field of Benchmark studies of parallel applications in the context of High Performance Computing "HPC". A particular interest is focused on the analysis and prediction of the performance of applications on computing machines with different characteristics.

4) Technical problem raised

High Performance Computing is increasingly affecting scientific fields such as the environment, weather and physics. In these areas, the need to solve ever more complex physical problems, whether at the level of models or their resolution, leads to a strong demand for computing capacity.

• nœud de calcul
• réseau d’interconnexion
• systèmes de stockage

These "machines" consist of a set of calculation servers called "calculation nodes". These are connected via a fast network called "interconnect", or "Infiniband network". Note that Infiniband actually designates one of the different technologies making it possible to interconnect the computing nodes (there is also an Intel technology called OmniPath). We will hereinafter call a “computing cluster” or “computing machine” the set made up of (see figure 1 ):

• compute node
• interconnection network
• storage systems

General architecture of a "Cluster" of calculation.

illustration of communications between two processes.

Possibilities of implementing the solution.

• Maximiser la performance des applications scientifiques (codes de calculs)
• Optimiser le prix de la machine proposée.

One of the great difficulties in designing a “computing cluster” is its sizing. Indeed, in the context of a call for tenders, for example, this one (Comment [JS1]: something is too much here) is generally accompanied by a set of scientific applications, following up on a companion analysis, testing and performance projection. The final objective of this campaign is to size the “computing cluster”. This is done in two steps:

• Maximize the performance of scientific applications (calculation codes)
• Optimize the price of the proposed machine.

One of the recurring problems in analyzing the performance of scientific applications is the unavailability of the target computing machine, that is to say, the one that will be offered to the bearer of the call for tenders. This is mainly due to two reasons. On the one hand, with the variety of architectures and computing machines, it is impossible to have all the existing models available. On the other hand, the target machine may simply be under development and only released to market after months or even years from the date of the offer proposal. The main characteristics of these future machines are nevertheless communicated by their manufacturers upstream. We then have recourse to the simulation of the target machines from machines with different material characteristics, called “Benchmark machines”.

• la fréquence du processeur CPU en GHz,
• la bande passante de la mémoire vive ou bande passante mémoire en Go/s,
• la bande passante du système d’interconnexion (réseau rapide, Infiniband) en Go/s.

These simulations are carried out by the teams in charge of application performance, called Benchmark teams, who use models, more or less complex, based essentially on the analysis of "sensitivities" (Comment [JS2]: perhaps it should be explained this term is a kind of partial derivative) of the performance of applications parallel to the different characteristics of a "computing cluster". Among these main characteristics analyzed, we can mention:

• CPU processor frequency in GHz,
• the RAM bandwidth or memory bandwidth in GB/s,
• the bandwidth of the interconnect system (fast network, Infiniband) in GB/s.

The frequency, generally expressed in GHz, designates the number of operations carried out in one second by the CPU processor (Central processing unit in English). For example, a 3 GHz processor performs 3 billion operations per second.

Memory bandwidth, generally expressed in GB/s, designates the number of bytes that can pass between the CPU processor and its memory.

The bandwidth of the interconnection system, also expressed in GB/s, indicates the number of bytes that a connection network can transfer between the different computing nodes.

Benchmark engineers use these characteristics as “sensitivity parameters” of the applications studied. These parameters are obtained, for a given application, by running it under different hardware conditions.

For example, the analysis of the sensitivity to the frequency of the processor is simple, it is quite possible to vary the value of the frequency of the processor via system parameters. This modification of the frequency has the advantage of not generating significant nonlinear effects on the other sensitivity parameters. These nonlinear aspects are very important during performance studies as we will see below.

On the other hand, it is not easy to vary the memory bandwidth or the bandwidth of the interconnection network. Indeed, to vary the memory bandwidth it is generally necessary to modify the hardware configuration of the computing cluster. This implies frequent stops of the machine and a consequent intervention time (changing the memory modules, for example). Similarly, modifying the bandwidth of the interconnection system requires at least reconfiguring all the network switches, this requires restarting a significant portion of the machine. Thus, for practical and time reasons, this type of modification is not feasible. In addition, the calculation machines are shared between several users on different projects, which makes this kind of intervention even less possible.

As a result, it is necessary to resort to the simulation of the characteristics of the computing machine rather than the real modification (at the hardware level) of these.

5) Known solutions

To analyze the bandwidth sensitivity of the interconnect system, benchmark engineers use several indirect methods:

- Profiling and analysis of the application, in particular by using the MPI (Message Passing Interface) communication layer. This method allows you to know the time spent in the “communication” part of the application. However, this time is not only a function of the bandwidth of the interconnection system: this time also takes into account the synchronization waits between the different computing nodes as well as the latencies of the communications, as illustrated in figure 2.

Thus, it is quite possible to measure 30% of the time of an application in the "MPI" part but that this time is in no way exclusively sensitive to the bandwidth of the interconnection system (in other words this time will not vary by doubling the bandwidth of the interconnection network).

• la diminution de la proportion des communications intra-nœud, c’est-à-dire, celles qui s’opèrent au sein d’un même nœud et s’effectuant ainsi directement dans la mémoire du nœud (raison pour laquelle ces communications sont plus rapides que les communications inter-nœud).
• l’augmentation des latences : en effet, il est possible qu’en utilisant plus de nœuds on peut être amené à traverser plus de switchs pour pouvoir communiquer entre les différents processus de l’application. Sachant que plus les messages échangés traversent des switch plus leur latence augmente (traverser un switch augmente la latence des communications).
• La diminution de la bande passante. Sur certaines topologies réseau on peut observer une diminution de la bande passante. En effet, la bande passante entre les switchs peut-être faible que la bande passante au sein d’un switch. Ceci est fortement dépendant du nombre de câbles placés entre les switchs, on parle alors du « facteur de pruning ». Il est courant d’avoir des facteurs de prunings qui peuvent diviser par 2 ou 3 la bande passante lorsque l’on communique entre deux nœuds sur des switchs différents.

- The placement of applications during their execution. One technique is to use twice the number of nodes than normal while using only half the available compute cores. It is nevertheless ensured that, for each computing node, only the cores of a single processor are used (there are generally two processors per node). This makes it possible to double the bandwidth of the interconnection network available per process. Indeed, in this case, we use half as many processes per computing node and therefore per network card. Memory bandwidth and computing capacity are kept constant. However, this modification also impacts the route of the messages exchanged. Indeed, doubling the number of nodes leads to:

• the decrease in the proportion of intra-node communications, that is to say, those which take place within the same node and thus taking place directly in the memory of the node (the reason why these communications are more faster than inter-node communications).
• the increase in latencies: indeed, it is possible that by using more nodes it may be necessary to cross more switches in order to be able to communicate between the different processes of the application. Knowing that the more the messages exchanged cross switches, the more their latency increases (crossing a switch increases the latency of communications).
• The decrease in bandwidth. On some network topologies, a decrease in bandwidth can be observed. Indeed, the bandwidth between switches may be lower than the bandwidth within a switch. This is highly dependent on the number of cables placed between the switches, we then speak of the “pruning factor”. It is common to have pruning factors that can divide the bandwidth by 2 or 3 when communicating between two nodes on different switches.

These different methods for simulating the bandwidth of an interconnection network generate transverse effects that are difficult to apprehend by the usual projection models.

6) Presentation of the invention

The solution proposed in this document consists in simulating, all other things being equal, a lower bandwidth of the interconnection system by increasing the size of the messages transmitted by the MPI communication layer. Indeed, we want to make the modification of the MPI layer simulate lower bandwidth while maintaining both latency and the characteristics intrinsic to the communication scheme of the application (the way the application is synchronized ). For this we propose to artificially increase the size of the messages transmitted by the MPI layer, this modification of the size of the messages must be carried out in a completely transparent way for the application according to the following methodology:

- When process X of the parallel application wants to transfer 1000 bytes to process Y which expects 1000 bytes from process X in return,

- the MPI layer modified by our solution will then transmit 2000 bytes (Comment [TN3]: Wouldn't it be better to halve the size of the messages to simulate a double bandwidth. This makes it possible to replace a scatter at 128 nodes by a normal 'modified' to 64 nodes) to process Y, but will only show 1000 bytes to the application, everything will look like the bandwidth of the interconnect network has been halved.

• Sous forme d’un « wrapper MPI », c’est une librairie informatique qui encapsule le service MPI en exposant la même interface mais dans laquelle on peut intercepter les appels des primitives MPI depuis l’application et en modifier leur fonctionnement. Cependant l’implémentation à cet endroit risquerait de nécessiter des copies mémoire dans des tampons intermédiaires, ce qui pourrait fausser les résultats. Cependant, il est possible de corriger le résultat en mesurant le temps passé dans ces copies.
• Sous forme d’un composant d’une implémentation MPI. MPI est un standard dans lequel il existe plusieurs implémentations (IntelMPI, OpenMPI, MPICH, etc…). Chaque implémentation traduit les primitives du standard MPI en des appels à des librairies d’interfaçage qui vont dialoguer avec les pilotes du système d’interconnexion (sous OpenMPI on parle de BTL ou Byte Transfer Layer, sous intelMPI on peut citer DAPL). Ces librairies d’interfaçage sont, de notre point de vue, un meilleur endroit pour implémenter cette modification, car l’implémentation pourrait se faire sans copies intermédiaires.

It is possible to implement this invention at different levels:

• In the form of an “MPI wrapper”, it is a computer library that encapsulates the MPI service by exposing the same interface but in which one can intercept calls to MPI primitives from the application and modify their operation. However, implementing it there would risk requiring memory copies in intermediate buffers, which could skew the results. However, it is possible to correct the result by measuring the time spent in these copies.
• As a component of an MPI implementation. MPI is a standard in which there are several implementations (IntelMPI, OpenMPI, MPICH, etc…). Each implementation translates the primitives of the MPI standard into calls to interfacing libraries which will dialogue with the pilots of the interconnection system (under OpenMPI we speak of BTL or Byte Transfer Layer, under intelMPI we can cite DAPL). These interfacing libraries are, from our point of view, a better place to implement this modification, because the implementation could be done without intermediate copies.

Claims (1)

.