RU2591020C1 - Method of monitoring user program progress, running on computational nodes of computer system - Google Patents

Abstract

FIELD: computer engineering.

SUBSTANCE: invention relates to monitoring progress of program executed on computer system, computer cluster. Method of monitoring user program progress running on computational nodes of a computer system, consists in that operating in each computing unit of computer system software control system tasks reads information from streams of standard issue and diagnosis of running user program and records in distributed software buffer in random-access memory of several service computers, from which software system for controlling tasks, operating on control service server computer system, reads and writes into files on a separate information storage device.

EFFECT: efficient use of user program, which provides for timely detection of cause of failure of execution of user program.

1 cl, 1 dwg

Description

The invention relates to the field of computer technology, in particular to providing control over the progress of a user program running on a computer system or computer cluster.

Modeling of complex physical, chemical or other processes is performed by executing a parallel program on a high-performance computing system (AC) or computing cluster. The executable file (binary file) of the parallel program is launched on the aircraft computers (computational elements, computational nodes) by the program component of the job management system and computing resources (hereinafter simply a job management system). When the program is executed, malfunctions / failures of the hardware or software components of the aircraft occur; therefore, the user must control the progress of the program according to the standard output and diagnostics.

Known methods for monitoring the progress of the user program on the sun, implemented in SLURM (simple linux utility for resource management) job management systems (https://www.computing.llnl.gov/linux/slurm/download.html) and Torque, ( http://www.adaptivecomputing.com/support/download-center/torque-download), which consists in the fact that the software component of the job management system is first installed in each computing node, by means of which information is saved in a certain order on the progress of the program in each computing node (standard output information and diagnosis tics) on which the user program is executed, then, using the job management system, information about the program’s execution is recorded in standard output and diagnostic files located in the storage device of the general-purpose file system of the computing system (network file system or parallel file system in which executable files, files with intermediate data, source codes of programs, etc.).

In the general case, before starting the program on the aircraft, the user places its files in the general-purpose file system, which is connected to each computing node of the aircraft. Then the user generates a text file of the task, which indicates the attributes typical for the task: the time necessary for the program to run, the name of the program executable file that must be run on the computational nodes, the number of computational nodes, as well as the names of the standard output and diagnostic files. After launching the user program executable file on the computational nodes on the computing nodes, the job management system must write the information that the running program writes to the standard output and diagnostics. This information reflects the progress of the program.

Since each executable file of the program running on the aircraft has its own standard output and diagnostics streams, the task of the task management system is to read information from all the standard output and diagnostics streams and write it in its entirety to the standard output and diagnostics files in the same order in which they were read out. Using the utility utility, the user transfers the job file to the job management system, which processes the file and assigns a unique identifier to the job. The job identifier is used by both the user and the job management system to monitor and manage the job. The task management system allocates for the user program the number of computing nodes specified in the task and runs the program executable on them.

In SLURM and Torque, simultaneously with the start of the user program executable file, the first computing node allocated for the user program launches the software component of the job management system, which processes (collects, buffers and records) standard output and diagnostic information. Since the process of information processing is carried out simultaneously with the execution of the program, it interferes with the calculations, since they use the hardware resources of the computing node, which must belong entirely to the program. In addition, the failure of software components on this computing node leads to the loss of raw information.

To increase the fault tolerance of the standard output and diagnostic information processing process, Torque implements a method where the job management system writes information to temporary files on one computing node. The job management system places temporary files on the information storage device, which is connected only to this computing node. However, this method is inconvenient for the user, since the files become available for information analysis only after the program is completed. The method in Torque increases the resistance to file system failures, but does not allow the user to control the progress of the program during its execution. In addition, in the case of computing nodes that load the operating system over the network, this method is unacceptable due to the limited size of the electronic disk located in the RAM of the computing node.

Thus, the methods for monitoring the progress of the user program provided by systems such as SLURM, Torque are closest to the claimed method in technical essence, and the method for monitoring the progress of the user program implemented in SLURM is selected as the closest analogue.

The disadvantages of the method are:

- impact on the user program:

- the use of processor cores designed to execute a user program;

- the consumption of RAM of the computing node for buffering information;

- Dependence on failures of the general file system;

- the inability to save the latest diagnostic messages, with which you can find the cause of the failure of the user program on the computing nodes of the aircraft;

- the inability to determine the failure / failure of the user program.

The technical results to which the claimed invention is directed are to increase the efficiency of the user program, the fault tolerance (fault tolerance) of the process of monitoring the progress of the user program, the expansion of functionality.

This technical result is achieved in that in a method for monitoring the execution of a user program executed on computing nodes of a computing system, which consists in first installing the software component of the job management system into each computing node, by means of which information is saved in a certain order on the progress of the program in each computing node allocated to execute the user program, then using the installed software component inform The progress report of the program is written to the standard output and diagnostic files located in the storage device of the general-purpose file system of the computing system, the recorded information is processed, it is new that the software component of the job management system is additionally installed on the service server, on which the received user's task is processed , assign an identifier to it and send the command to launch the executable file of the program to the corresponding computing node, simultaneously but with this, the software component “control and generation of files of standard output and diagnostics” is launched on the service server, information about the progress of the program on the computing nodes is stored in a distributed software buffer that provides temporary storage of information; then this information is recorded in temporary files of standard output and diagnostics of an additional information storage device with a name for each file and links are created to temporary files created on the storage device of the general-purpose file system of the computing system to monitor the progress of the user program; after the user program is executed, delete links to temporary files and to their place save temporary files of standard output and diagnostics, while a certain part of the information on the progress of the program is stored in the database of the job management system on the service server.

The processing and intermediate storage of information about the progress of the user program outside the general-purpose file system and outside the computing nodes allows the running program to exclusively use the resources of the computing nodes, as well as reducing the dependence of the processing process on the failure / failure of the general-purpose file system storage device. In addition, it is possible at the same time as the user program is running to analyze and monitor the progress of the user program, find the cause of the program failure / refusal in time, and stop the program due to incorrect intermediate results or its freezing. If the program freezes, the described method allows you to free computing resources to perform other tasks (programs) of users. Storage after completion of the user program of a certain part of the information about the course of its execution in the database of the job management system guarantees the availability of information about the last stage of the program execution, including information about the failures of system programs, the completion of the user program when determining its freezing even if the standard output files are deleted and diagnostics. Together, all this increases the efficiency of the user program, the efficiency of the use of computing resources of the computing system, the fault tolerance / fault tolerance of the process of monitoring the progress of the user program, expands the functionality of the job management system as a whole.

The figure shows a structural diagram of a computing system that implements the inventive method for monitoring the progress of a user program running on its computing nodes (hereinafter, a method for monitoring the progress of a user program).

The computing system comprises a block 1 of computing nodes, a distributed software buffer 2, a service server 3, an additional information storage device 4, a general-purpose file system storage device 5 of the computing system.

The group of inputs and outputs of block 1 of the computing nodes is connected to the group of inputs and outputs of the device 5 for storage of a general-purpose file system of a computing system via a communication network, for example, Ethernet. The group of outputs of block 1 of the computing nodes is connected to the first group of inputs of the distributed software buffer 2. The second group of inputs of the distributed software buffer 2 is connected to the first group of outputs of the service server 3. The second and third groups of outputs of the service server 3 are connected respectively to the first groups of inputs of the file storage device 5 general-purpose computing system and additional information storage device 4. The group of outputs of the distributed software buffer 2 is connected to the second group of inputs of the additional device 4 for storing information. The fourth group of outputs of the service server 3 is connected to the group of inputs of block 1 of the computing nodes. The group of outputs of the additional information storage device 4 is connected to the second group of inputs of the general purpose file system storage device 5 of the computing system. The group of inputs of the service server 3 is a group of inputs of user job files.

Block 1 computing nodes contains the computing nodes WU ₁ , ..., WU _m , where m = 1, 2, ..., interconnected, for example, according to the topology of the "star". Computing nodes WU ₁ , ..., WU _m are designed to execute user programs. In each computing node of VU ₁ (VU ₂ , ..., VU _m ), specialized software is installed and runs, including the software component of the job management system.

Distributed software buffer 2 can be implemented on separate office computers B1, ..., Bk, where k = 1, 2, .... Distributed software buffer 2 is intended for temporary storage of information on the progress of user programs. Information buffering is provided by a specialized program distributed over the service computers B1, ..., Bk, which uses the computer random access memory for temporary storage of information.

The software for the job management system is installed and runs in the service server 3, including the program “control and generation of files of standard output and diagnostics”. This program monitors the freeze / crash of the user program and generates temporary files of standard output and diagnostics. Also in the service server 3 there is a database (not shown in the figure) of the job management system. The job management system operating on the service server 3 generates and sends to WU ₁ (WU ₂ , ..., WU _m ) commands to launch the user program executable file on them (the file name is pre-specified by the user in the task) and also reads from the distributed program buffer 2 and writes to temporary files located in the additional device 4 for storing information, information about the progress of the user program (information of standard output and diagnostics of programs).

An additional device 4 for storing information in temporary files of standard output and diagnostics about user programs running on aircraft is designed to store files of standard output and diagnostics for the duration of execution of user programs on aircraft. The device can be made in the form of a hard disk or disk array.

The general purpose file system storage device 5 of the computing system is intended for storing executable files of user programs, files with their input data, files with program source texts, etc., in the general case, it is several disk arrays. The executable file is launched on VU ₁ (VU ₂ , ..., VU _m ) of the user program by reading it from device 5 and transferring it via the communication network to VU ₁ (VU ₂ , ..., VU _m ).

A method for monitoring the progress of a user program executing on computing nodes of a computing system is implemented as follows.

To run the program on WU ₁ , ..., WU _{m of the} computing system, the user pre-generates a task file, which indicates the time that the program should spend on calculating, the name of the program executable file that the job management system should run on the computing nodes of WU ₁ , ..., WU _m , as well as the names of the files of standard output and diagnostics. Next, the user using the appropriate utility transfers the job file to the control program of the job management system, which operates on the service server 3.

The control program of the job management system on the service server 3 processes the attributes specified in the job, assigns an identifier to the job and sends to the software components of the job management system on the computing nodes, for example, ВУn, ..., БУm, where n <m, the service messages “start executable user program file. " At the same time, for this task, an instance of the program “monitoring and generating standard output and diagnostics files” is launched on the service server 3.

On the additional information storage device 4, through this program, temporary output and diagnostic files with the corresponding names are created, and on the general purpose file system storage device 5 of the computing system, links to the created temporary files are formed with the names indicated by the user in the task. Using these links, the user can control the course (steps) of the execution of his program, as if the files would actually be located in the storage device 5 of the general-purpose file system of a computing system. Having completed the above steps, the program “control and generation of standard output and diagnostics files” begins to cycle through the distributed program buffer 2, searching for information by the job identifier.

After starting and during execution on WU ₁ (WU ₂ , ..., WU _m ), the user program writes information to the standard output or standard diagnostics, which the software component of the job management system on WU ₁ (WU ₂ , ..., WU _m ) reads and writes to a distributed program buffer 2. To facilitate the search / reading of information for the program “control and generation of standard output and diagnostic files”, the information is accompanied by a job identifier and separate labels (flags) that mark the information, relative yaschayasya to standard output or diagnosis.

Having found the information with the required job identifier, the program “control and generation of files of standard output and diagnostics” reads it from the distributed program buffer 2 and processes it: standard output and diagnostics are recorded in separate temporary files in the additional information storage device 4. If the user is not sure that the program can successfully complete, then he directs the task file to instruct the job management system to control the program freezes.

 The way to control the program freezing is that if for some time period specified by the user there is no new information from the standard output of the program, then it is considered to be hung. In this case, the job management system completes the program on the computing nodes. After completion of the program, the job management system writes a certain last piece of information about the progress of the user program from temporary files of standard output and diagnostics to its own database, which is also located on the service server 3. Then links to temporary files of standard delivery and diagnosis of the task on storage device 5 general-purpose file system computing systems are deleted. In their place are recorded temporary files from an additional device 4 for storing information. Even if, during the execution of the program, the user intentionally erased the links to temporary files of standard output and diagnostics of the task, the task management system continues to record information in temporary files on the additional information storage device 4. After completing each task, a certain last piece of information from temporary files of standard output and diagnostics is stored in a database located on the service server 3. If the user deletes files transferred to device 5, then the information recorded in the database will make it possible to understand the reason, for example, emergency program completion.

Claims (1)

A method for monitoring the progress of a user program executing on computing nodes of a computing system, which consists in first installing the software component of the job control system into each computing node, by which information is saved in a specific order on the progress of the program in each computing node dedicated to execution user program, then using the installed software component information about the progress of the program is recorded in The standard output and diagnostic files located in the storage device of the general-purpose file system of the computing system process the recorded information, characterized in that the software component of the job management system is additionally installed on the service server, on which the received user's job is processed, assigned an identifier and transferred to the corresponding computing node, the command to run the executable file of the program; at the same time, the programs are launched on the service server hydrochloric component "control and the formation of a standard file issue and diagnostics", information on the implementation of the program on the compute nodes is stored in the distributed software buffer provides temporary storage of information; then this information is recorded in temporary files of standard output and diagnostics of an additional information storage device with a name for each file and links are created to temporary files created on the storage device of the general-purpose file system of the computing system to monitor the progress of the user program; after the user program is executed, delete links to temporary files and to their place save temporary files of standard output and diagnostics, while a certain part of the information on the progress of the program is stored in the database of the job management system on the service server.

Images