RU2649748C2 - System and method for interpreting and analyzing dynamic characteristics of the current state of tasks performed - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: invention relates to means for analyzing dynamic characteristics of parallel programs and supercomputers. System contains a set of computational nodes of a supercomputer, each of which is equipped with a monitoring system (CM), a task flow control system (SOUP), information processing server, including a module for data aggregation from monitoring systems of each computing node, a data analysis module from the task flow control system of each computing node, as well as a data storage unit, a tool for visualizing the results of processing. At the same time, SM means include sensors of system monitoring that provide information on the status and degree of use of available resources from each of the available monitoring systems, MEMS facilities are designed to obtain information about the status of tasks, their distribution by nodes and the nature of resource use at compute nodes. Method describes system operation.

EFFECT: technical result consists in increasing efficiency of supercomputer by analyzing current state of problem being solved.

11 cl, 7 dwg, 1 tbl, 1 ex

Description

Technical field

The inventive group of inventions relates to the study of the behavior of programs (tasks) during execution on supercomputer systems and is intended to implement a comprehensive analysis of the dynamic characteristics of parallel programs and supercomputers. During the analysis, the characteristics of the task are taken into account, starting from the moment of queuing and ending with its completion. This allows you to get complete information about both the task itself and the entire set of tasks performed on the supercomputer.

Along with the growing scale of computing systems and the tasks they solve, the complexity of writing effective programs also grows. The properties of the hardware and software of a supercomputer, the properties of the executable program itself, the mutual influence of the executable programs on each other - all this must be taken into account if you want to achieve high efficiency. The main task of any supercomputer installation is to satisfy the needs of users who solve problems that are faced with real problems from various application areas. Therefore, the problem of increasing the efficiency of each individual application is extremely relevant. Its successful solution for a separate application favorably affects the solution not only of the real scientific problem that is behind it, but also on the work of the entire system as a whole.

State of the art

A significant number of software tools available today that are aimed at detecting a specific location of an error in the user program, localizing the part of the program that is inefficient and needs to be optimized, use the tracing method. Most often, a set of recorded events, the granularity of their collection, and the data sources themselves are determined. In addition to collecting information about the occurrence of the events themselves, it is common practice to use the trace file to analyze the final sequence of operations and / or events during application execution.

The approaches based on tracing are quite developed and widespread. The following software tools can serve as the most characteristic representatives of their approaches.

The well-known Scalasca system (http://www.scalasca.org) supports the performance optimization of parallel applications based on measurements of 19 characteristics during program execution and related analysis. This analysis allows you to identify potentially bottlenecks in the application related to communication and process synchronization. The system also provides directions for further in-depth study of problem areas. In the system, you can choose one of two analysis modes: a performance study at the level of function calls based on the total execution times (profiling) and a study of the behavior of the task based on event tracking. The system is available for download under the New BSD Open Source License.

The Score-P system (http://www.vi-hps.org/projects/score-p) is a relatively new system for collecting and analyzing data on the work of programs operating at the application level. The system is available with open source code and is based on instrumentation of program code.

Well-known ThreadSpotter technology (http://www.roguewave.com/products/threadspotter), focused on performance optimization, was developed by Rogue Wave Corporation and is a natural continuation of research projects at Uppsala University, Sweden. In normal mode, heterogeneous information is collected about the behavior of the program in the form of a so-called “fingerprint”. Based on this information, the performance of caches of any size with any line size and with any push strategy can be evaluated by ThreadSpotter'oM without reference to the target system. ThreadSpotter also allows you to detect errors and places where performance drops in applications in the context of certain patterns of data access. Errors of this kind are grouped into 4 groups of problems: problems of littering caches, problems of latency, problems of throughput, and problems of interaction of flows.

The prior art CPS, which can also be used in the implementation of the claimed invention:

SLURM (Simple Linux Utility for Resource Management) is a highly scalable fail-safe cluster manager and task scheduler for computing nodes of large clusters. SLURM maintains a queue of pending jobs and manages the total load of resources during the execution of computational tasks. SLURM also manages available compute nodes. Finally, in addition to monitoring parallel tasks, until their completion, SLURM distributes the load among the selected nodes (https://computing.llnl.gov/liniix/slurm/).

CLEO is a software package that is part of the ParCon system that solves the problems of efficiently managing resources of computing clusters, as well as analyzing the effectiveness of clusters and parallel programs (http://parcon.parallel.ru/cleo.html). This system is oriented to work with parallel applications and supports many parallel environments.

Each of these systems maintains its own record of all the tasks and events known to it related to the progress of the application.

Thus, if information about the flow is required, the main source of data about the structure of the task flow itself is the task flow control system (CPS) itself, also called the Resource Manager.

Also, when implementing the approach described in the claimed invention, data from monitoring systems from each computing node are used. As such monitoring systems, the well-known ClustrX Watch, Ganglia can be used.

ClustrX Watch is a distributed cluster parameter monitoring system designed to organize the collection, recording and processing of data from a large number of sensors of all cluster subsystems and capable of functioning in a fail-safe mode. ClustrX Watch is a development of T-Platforms OJSC, Moscow, and is included in the cluster management system - ClusrtX.

Some of these tools allow you to collect basic information about the status of the computer. But there are many specialized tools that more fully implements the capabilities of system monitoring.

One of the best known systems monitoring data collection tools is the publicly available Ganglia system (http://ganglia.sourceforge.net). It allows you to collect information about the processor load, network load, memory usage, many other resources, but it does not allow the analysis of the collected data in relation to specific running applications. What is important, the system does not provide high time detail for large-scale systems under study.

Supercomputer complexes that are designed to solve problems from different fields of science are characterized by heterogeneity of the tasks being solved: various requirements for the amount of required resources, the possibility of using computational accelerators, the availability of local disks, and much more.

Thus, using this information, it is possible to bind system monitoring data that reflects the state of the computer directly with an application that uses a known set of nodes or sections of a computer system.

The combination of data availability from CPS, describing the entire structure of the computing system, with the rich capabilities of system monitoring allows us to develop comprehensive methods for analyzing the effectiveness of supercomputer applications and systems - from the level of an individual application to the level of the partition or the system as a whole.

Disclosure of invention

The task of the claimed group of inventions is to provide a qualitative analysis of any task from the entire stream of tasks performed by a supercomputer and a quantitative analysis of the average use of computing resources by tasks. The decrease in the operational efficiency of each individual supercomputer application, of each task leads to an increase in the time spent, which leads to a decrease in the overall system performance. Thus, the claimed technical solution is aimed at improving the efficiency of the supercomputer by creating the ability to quickly respond to the current state of the problem.

The technical result achieved by using the claimed invention is to enable the user to evaluate information about the characteristics and dynamic features of the performance of a particular task, solved by a supercomputer, its current state.

The problem is solved in that the claimed system of interpretation and analysis of the dynamic properties of problems solved by a supercomputer includes:

a set of computing nodes of a supercomputer, each of which is equipped with the means of a monitoring system (SM) and a task flow control system (CPS),

an information processing server including a data aggregation module from a monitoring system of each computing node, a data analysis module from a task flow control system of each computing node, and a data storage unit,

means for visualizing the processing results, while

SM tools include system monitoring sensors, which provide information on the status and degree of use of available resources from each of the available monitoring systems,

CPS tools are designed to obtain information about the status of tasks, their distribution among nodes and the nature of the use of resources at computing nodes.

The problem is also solved by the fact that the claimed method of interpretation and analysis of the dynamic properties of tasks solved by a supercomputer, including the following steps:

- collecting data from computing nodes from sensors of system monitoring of the computing node of a supercomputer for each individual task during its implementation and placing the collected data into an aggregation module;

- collecting data from the task flow control system for each individual task and putting the collected data into the analysis module;

- processing the collected data by the information processing server and linking them according to the same tasks being performed;

- the formation of the server information processing report on the task at the request of the user, including the results of processing the collected data for each task;

- visualization of the report.

When processing the data collected by the aggregation and analysis modules, the processing results are saved to the database of the information processing server, which provides storage of task data, storage of dynamic characteristics, storage of integral characteristics. The aggregation module provides alignment of incoming system monitoring data, converting data to uniform time intervals, thinning, filtering monitoring data, generating dynamic characteristics, and storing the obtained dynamic characteristics in the database of the information processing server. The analysis module provides verification of the correctness of the incoming data from the control system, processing the saved system monitoring data, calculating the integral characteristics, storing the processed data in the database of the information processing server, and generating visualization templates for the processing results. System monitoring data includes the flow of information from individual system monitoring sensors indicating the time and / or place of reading the value and / or identification of the source from each of the available monitoring systems. The data of the CPS includes information about each launch, at least the following: the time the task started and / or queued, and / or the time completed, and / or the waiting time, and / or the counting time, and / or list allocated computing nodes, and / or the number of allocated computing cores, and / or the amount of processor hours spent, and / or the launch line, and / or section of the computing system, and / or the status of the task. A report as a result of information processing is a set of textual and / or tabular and / or graphical information that reflects general information, information about the dynamic and integral characteristics of the analyzed problem. As general information, the report includes the start time of the task and / or putting it in the execution queue and / or the completion time, and / or the wait time, and / or the counting time, and / or the list of allocated computing nodes, and / or the number allocated computing cores, and / or the amount of processor hours spent, and / or a launch line, and / or a section of a computing system, and / or task execution status. As integral characteristics, the report includes the minimum, maximum, average (or median) values of the dynamic characteristics during the execution of the task indicating the excess of threshold values. As dynamic characteristics, the report includes time series that reflect the values of dynamic values from the monitoring system, for example CPU_user, LoadAverage, the number of floating-point operations, network exchange rate, I / O utilization rate, memory exchange rate and cache usage characteristics.

The claimed group of inventions is illustrated by the following drawings.

In FIG. 1 schematically presents the relationship of nodes and software modules that are part of the inventive system.

In FIG. 2 shows an example of a report submitted by the system.

In FIG. Figure 3-7 shows an example of graphs obtained as reports on individual tasks, as well as interpretation of graphical information by the user.

The positions in the drawings indicate:

1 - computing node;

2 - monitoring system;

3 - task flow control system;

4 - server processing and storage of information;

5 - analysis module;

6 - aggregation module;

7 - database;

8 - report;

9 - visualization tool.

The inventive system is a technical solution that provides the ability to obtain, analyze and interpret data characterizing the quality of the task on a supercomputer.

The system includes both hardware for its implementation and a software package that provides the ability to implement user requests.

The hardware of the system includes many computing nodes 1 constituting a supercomputer computing system. In addition, the inventive system includes a server for processing and storing information 4, as well as a means of visualizing the processed information 9, which can be used, for example, a PC / laptop / tablet / phone with Internet access and a web browser.

The software package of the claimed invention is implemented as follows.

Each computing node 1 is equipped with CPS 3 tools that provide data on the status of tasks, their distribution among nodes and the nature of the use of various kinds of resources on nodes, and SM 2 tools, which provide data on the nature of various kinds of resources used on nodes in time. Aggregation module 6, which provides data processing from the system monitoring sensors of each computing node, analysis module 5, which provides processing of the CPSS data and processing of the stored system monitoring data, as well as a database 7, in which task data is stored, operate on the processing and storage server, and calculated dynamic and integral characteristics of the task being performed. In addition, to enable the user to receive a report on task 8, visualization tools 9 are equipped with web browsers that support JavaScript, such as Chrome, FireFox, Safari, IE, etc.

Speaking about the monitoring system on the computing nodes of a supercomputer installation, it is understood that in every node of the computing system, in addition to the main computing means, there are system monitoring sensors that provide information about many characteristics of the state of the software and hardware environment on the node. For each characteristic, the data is a sequence of pairs of values of the characteristic and the time of its measurement (Vi, Ti).

The inventive method is implemented as follows.

The sensors of the monitoring system from the nodes of the computing system constantly collect data on the status and degree of use of available resources (processor, memory, network, etc.). Sensors record the time, place of reading the value and provide identification of the source from each of the available monitoring systems.

Each sensor has a unique identifier, the ability to get its value from the operating system, environment variables, or from the available hardware interfaces through an agent program. Examples of such sensors are shown in table 1.

The monitoring system periodically receives values from all sensors. Each individual monitoring system has its own set of available sensors, which significantly depends on both the features of the equipment and the settings of the software environment.

Often, in practice, the number of sensors removed from the processor is very limited and you have to choose the most important ones from them. On the other hand, an analysis based on the selected characteristics should give as comprehensive a picture of the investigated stream of tasks as possible.

CPS provides the aggregation module with task data: queuing, launch, completion times, status, distribution by nodes, launch string, etc.

The aggregation module, based on the specifics of the research, converts the incoming system monitoring data to single time intervals (usually 5 minutes) and filters it, thereby obtaining dynamic characteristics from the monitoring data, and saves them in one table. The dynamic characteristics consist for each interval and each observed parameter of three values: the average for the interval, the minimum and maximum values for the interval.

The dynamic characteristics of user application launches are available to system administrators in full, and for ordinary users access is granted only to launches of their own applications. Of greatest interest is the share of user processes in the total processor load - CPU user time (the time spent on user programs), which most reflects the processor load by the application. For a more detailed study of application behavior, you can use other sensors, however, for general research, including the determination of typical profiles for using supercomputer systems that require a comprehensive assessment as a priority, it suffices to limit the inclusion of CPU User in the list of key characteristics.

Among the most commonly used sensors, sensors that record the processor load can also be distinguished; number of floating point operations; the number of processes ready to take control (Load Average); Intensite exchange rate I / O rate The number of misses when accessing the cache. Of course, such a list can both expand with the addition of new sensors to the review, and be improved in terms of the frequency of obtaining the studied characteristics.

The analysis module, receiving data from the CPS about the change in the status of the task, checks the correctness of the received data and saves them to the database (DB).

If the analysis module receives data on the completion of the task, it selects the dynamic characteristics by time and the nodes of the task and builds the integral characteristics (average over the dynamic characteristics), which it then stores in the database. At the same time, belonging to classes by the level of average resource use (for example, exceeding the threshold), and other similar processing can be attributed to integral characteristics.

The integral characteristics of applications are averaged (or medians) data of the corresponding dynamic characteristics for a given task, as well as data on the resources allocated and spent by the task: partition, time, number of nodes and cores.

The data of different data streams relative to a specific task are connected through the time interval of the task and through the identifier of the computing nodes on which the task was performed. Such data is always available both for system monitoring data streams and from the task flow control system for each task.

Data from the database can be extracted for analysis using the analysis module. As a database for data storage, for example, Cassandra and MongoDB databases are used.

When a request for a list of tasks is received from the user interface from a web browser (following the link), the analysis module selects from the database a list of tasks, their integral characteristics, labels (tags) of belonging to certain classes (if any). It is possible to filter and refine the query through the user interface (prepared links or refine the SQL query manually). The selection result is inserted into the “task list” visualization template with color indication of exceeding certain thresholds by integral characteristics and transmitted via http to the client side, where it is visualized by a web browser. In the list there is an opportunity to follow the link to the report of an individual task.

When a request for a specific task is received from the user interface from a web browser (following the link), the analysis module selects from the database data about this task from the list of tasks, its integral characteristics, labels (tags) of belonging to certain classes (if any ) established for this task. The query result is inserted into the visualization template and transmitted to the client side, where it is visualized by a web browser.

One of the tasks of the analysis module is to perform data processing. Moreover, the analysis can be initiated from the most diverse parts of the system. For example, an analysis request may come from a visualization tool through a request processing center (CSP) as a result of a user working with the system through a web browser. In this case, the main goal of the request may be to analyze the dynamics of the behavior of the parallel program. Data analysis can be initiated by system processes, for example, by a timer once a day to build a daily report on the operation of a supercomputer. Or data may be requested by external integration and visualization systems.

The request indicates the characteristics of the completed task, as well as a visualization template for generating the report. The address of this report is stored in a separate file with the task report, and the user can subsequently open it in a browser.

The queries themselves are stored in text files called “query templates”. Upon completion of the report, the user can view it in a browser. The basis of the report is information about the monitoring data of the selected application. You can include any number of graphs and charts in the report that reflect various parameters of the application - the processor load.

A report, in particular, may contain the following blocks:

1. general data about the task: id, owner, list of nodes, status, launch section, time of enrolling, starting and ending, launch line, processor hours, number of allocated cores, etc .;

2. integral characteristics of the task with color indication of exceeding certain thresholds;

3. a list of labels of belonging to classes (tags, tags), if the corresponding belonging of the task to the analysis module was determined when constructing the integral characteristics;

4. graphs reflecting the behavior of dynamic characteristics based on a selection from the database by nodes and task execution time.

Concrete example

Below is an example of graphs received as reports on individual tasks, as well as interpretation of graphical information by the user.

Program run time: 9 hours 38 minutes.

The number of involved cores: 112.

Used partition with disks: no

On the graph (Fig. 3) of the processor load, periodicity is observed, which indicates the iterative structure of the algorithm. A feature of this graph is the big difference between the maximum and minimum processor load. The maximum load almost does not differ from 100%, while the minimum load is almost 0% almost all the time.

The schedule of cache misses of the first level (Fig. 4) correlates with the schedule of processor loading. A feature of the graph is that at each iteration, a surge in the number of cache misses occurs at the beginning of the iteration. This correlates with bursts of minimal CPU utilization.

The graph of the number of cache misses L2 (Fig. 5) repeats the graph of misses in the cache of the first level. However, the number of misses is lower.

The Ethernet activity graph (Fig. 6) indicates bursts of activity with equal periods. The intensity of network use at these moments reaches 50 MB / s. This is quite a lot of activity, but such a network load occurs at equal and fairly long periods of time. Therefore, the average network load is only 0.13 MB / s.

On the graph of the data transfer rate by InfiniBand (Fig. 7), a correlation with the cache miss graph is visible. The overall high network load of 104.11 MB / s suggests that communications between processes do not contain a very large amount of data. Processes exchange data at the beginning of each iteration.

This profile shows the dependence of increasing the number of cache misses of different levels on the transfer of new data from a file over an Ethernet network. The profile reflects the dependence of the activity of data transmission over the network, the activity of the InfiniBand network and the number of cache misses. This dependence reflects the iterative structure of the algorithm, in which data is exchanged with files at each iteration, and therefore the number of cache misses increases. Waiting for new data reduces processor load, as can be seen on the graph of processor usage. A distinctive feature of this graph is the wide variation between the maximum and minimum values of the processor load and cache misses. This shows the imbalance of the task: some processes are busy computing, while the other is idle. This reflects an average processor load of 55.7%. Because, judging by the readings of the sensors, some of the processors are loaded with a level close to 100%, while the minimum load of other processors fluctuates around 0. This suggests that the calculations are distributed unevenly.

The proposed approach to analysis allows one to efficiently and technologically justify a qualitative assessment of the properties of a real task flow, on the basis of which one can judge the utilization of supercomputer resources, identify problem areas of the architecture and outline possible directions for its optimization.

Claims (22)

1. The system of interpretation and analysis of the dynamic properties of problems solved by a supercomputer, including: a set of computing nodes of a supercomputer, each of which is equipped with the means of a monitoring system (SM) and a task flow control system (CPS), an information processing server including a module for aggregating data from monitoring systems of each computing node, a data analysis module from a task flow control system of each computing node, and also a data storage unit, means for visualizing processing results, wherein SM tools include system monitoring sensors, which provide information on the status and degree of use of available resources from each of the available monitoring systems, CPS tools provide the ability to obtain information about the status of tasks, their distribution among nodes and the nature of the use of resources at computing nodes. 2. A method for interpreting and analyzing the dynamic properties of problems solved by a supercomputer, comprising the following steps: - collecting data from computing nodes from sensors of system monitoring of the computing node of a supercomputer for each individual task during its implementation and placing the collected data into an aggregation module; - collecting data from the task flow control system for each individual task and putting the collected data into the analysis module; - processing the collected data in the information processing server, linking them to the same tasks to be performed; - formation by the server of the information processing of the report on the task at the request of the user, including the results of processing the data collected for each task; - visualization of the report. 3. The method according to p. 2, characterized in that when processing the data collected by the aggregation and analysis modules, the processing results are stored in the database of the information processing server, providing storage of task data, storage of dynamic characteristics, storage of integral characteristics. 4. The method according to claim 2, characterized in that the aggregation module provides alignment of the incoming system monitoring data, converting the data to uniform time intervals, thinning, filtering the monitoring data, generating dynamic characteristics, storing the obtained dynamic characteristics in the database of the information processing server. 5. The method according to claim 2, characterized in that the analysis module verifies the correctness of the incoming data from the control system, processes the stored system monitoring data, calculates the integral characteristics, stores the processed data in the database of the information processing server, generates visualization templates for the processing results. 6. The method according to p. 2, characterized in that the system monitoring data includes the flow of information from individual system monitoring sensors indicating the time and / or location of the value and / or source identification from each of the available monitoring systems. 7. The method according to p. 2, characterized in that the data CPS includes information about each start, at least the following: the time the task started and / or put it in the execution queue, and / or the time it ended, and / or the time expectations, and / or counting time, and / or a list of allocated computing nodes, and / or the number of allocated computing cores, and / or the amount of processor hours spent, and / or the launch line, and / or section of the computing system, and / or status complete the task. 8. The method according to p. 2, characterized in that the report as a result of information processing is a set of textual and / or tabular and / or graphical information that reflects general information, information about the dynamic and integral characteristics of the analyzed problem. 9. The method according to p. 8, characterized in that the general information in the report includes the start time of the task and / or putting it in the execution queue, and / or the completion time of the execution, and / or the waiting time, and / or the counting time , and / or a list of allocated computing nodes, and / or the number of allocated computing cores, and / or the amount of processor hours spent, and / or a launch line, and / or a section of a computing system, and / or task execution status. 10. The method according to p. 8, characterized in that the integral characteristics of the report include the minimum, maximum, average (or median) values of the dynamic characteristics during the execution of the task indicating the excess of threshold values. 11. The method according to claim 8, characterized in that the dynamic characteristics of the report include time series that reflect the values of the dynamic values from the monitoring system, for example CPU_user, LoadAverage, the number of operations with a floating point, the intensity of network exchange, the intensity of I / O , memory exchange rate and cache usage characteristics.

Images