CN107038066B

CN107038066B - Job computing system based on Web

Info

Publication number: CN107038066B
Application number: CN201710321776.7A
Authority: CN
Inventors: 郭晓新; 魏晓辉
Original assignee: Jilin University
Current assignee: Jilin University
Priority date: 2017-05-09
Filing date: 2017-05-09
Publication date: 2020-06-16
Anticipated expiration: 2037-05-09
Also published as: CN107038066A

Abstract

The invention discloses a Web-based job computing system, which comprises: a receiving module for receiving jobs and calculating workflows required for the jobs through a network; the script generation module is used for generating a job script according to the job and the workflow; the execution module is used for executing the script through the resource manager to obtain a calculation result file of the operation; the recording module is used for recording the calculation result file; according to the invention, the script generation module generates the job script according to the job and the workflow, so that the obstacles of compiling scripts and developing tools by a user can be eliminated, and the user can more conveniently perform high-performance calculation on the job through the computer cluster managed by the resource manager; the calculation result file is recorded through the recording module, so that a user can conveniently download the calculation result file through a network; moreover, the system can integrate the computer clusters managed by a plurality of resource managers, thereby improving the effect of high-performance calculation of the operation and improving the user experience.

Description

Job computing system based on Web

Technical Field

The invention relates to the field of high-performance computing, in particular to a Web-based job computing system.

Background

With the development of modern society science and technology, high-performance calculation for jobs composed of a large amount of data is more and more emphasized by people. Computing pipelines or workflows has become an important tool for analyzing large amounts of data in many scientific fields today. The computational complexity of these workflows varies significantly, but typically requires days of computational time and a large amount of computational power. To speed up the execution of these jobs, the application of parallel algorithms and high performance computing clusters has become increasingly popular.

Computer clusters provide high performance by aggregating resources from multiple individual computers. A resource manager is a software system required to manage the submission and scheduling of jobs on these clusters and the allocation of resources (e.g., memory and processing cores) for individual jobs. With the popularity of computational modeling and big data analysis in various fields, more and more researchers are demanding the use of high performance computing resources.

In the prior art, a researcher needs to compile a workflow required by a job and a calculation job into a script that can be executed by a resource manager through the researcher's own high-performance calculation of the job, so that the researcher has certain obstacles to compiling the script and developing tools. And because a system specially used for performing high-performance calculation on the jobs does not exist, the computer clusters of the resource manager cannot be integrated, so that the computing capacity of the computer clusters on the jobs is insufficient. Therefore, how to provide a system for performing high-performance calculation on jobs to make it easier for users in professional fields to use high-performance calculation is a problem that needs to be solved urgently nowadays.

Disclosure of Invention

The invention aims to provide a Web-based job computing system, which is used for eliminating the barriers of writing scripts and developing tools by a user, so that the user can more easily perform high-performance computation on jobs, and the user experience is improved.

To solve the above technical problem, the present invention provides a Web-based job computing system, including:

the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving a job and calculating a workflow required by the job through a network; wherein the workflow is comprised of files;

the script generation module is used for generating a job script according to the job and the workflow;

the execution module is used for executing the script through a resource manager to obtain a calculation result file of the operation;

and the recording module is used for recording the calculation result file.

Optionally, the receiving module includes:

and the receiving unit is used for receiving the jobs and the workflows sent by the preset Web interface through a network.

Optionally, the script generating module includes:

the first generation unit is used for generating a job script according to the job, the workflow and a preset request resource;

a second generation unit, configured to generate a job script according to the job, the workflow, and the request resource sent by the Web interface; the request resource comprises a memory required for calculating the operation, the number of cores and suspension time distributed to the operation.

Optionally, the system further comprises:

the access module is used for accessing the resource manager according to a preset time interval and acquiring data corresponding to the script execution state;

and the analysis module is used for analyzing the data, acquiring the state information of the operation corresponding to the data and storing the state information to the recording module.

Optionally, the recording module includes:

and the display unit is used for sending the state information of the data corresponding to the display request to the Web interface according to the display request sent by the Web interface so as to display the state information to a user through the Web interface.

Optionally, the execution module further includes:

the management unit is used for executing corresponding operation on the job according to the management request sent by the Web interface; wherein the management request includes at least one of a delete request, a cancel request, a suspend request, and a change request.

Optionally, the management unit includes:

a deleting subunit, configured to delete, according to the deletion request, the script that is not executed by the resource manager;

a cancellation subunit, configured to cancel, according to the cancellation request, the script that is being executed by the resource manager;

a pause subunit, configured to pause the script that is not executed or is being executed by the resource manager according to the pause request;

and the change subunit is used for correspondingly changing the suspended script according to the change request.

Optionally, the system further comprises:

the verification module is used for judging whether the user login information is the login information of a preset user according to the user login information received by the receiving module through the network; and if so, receiving the operation and the workflow of the user corresponding to the user login information through the receiving module.

Optionally, the system further comprises:

the display module is used for acquiring summary information of the current state of the cluster corresponding to the resource manager; wherein the summary information includes at least one of a number of online nodes, a number of offline nodes, a proportion of running processors, a number of running jobs, a number of jobs waiting to run, and an amount of available disk space in the cluster.

Optionally, the system further comprises:

and the sharing module is used for saving the workflow and/or a preset file in the workflow.

The invention provides a Web-based job computing system, which comprises: the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving a job and calculating a workflow required by the job through a network; wherein the workflow is comprised of files; the script generation module is used for generating a job script according to the job and the workflow; the execution module is used for executing the script through a resource manager to obtain a calculation result file of the operation; the recording module is used for recording the calculation result file;

therefore, the method and the device can eliminate the barriers of compiling scripts and developing tools by the user by generating the operation scripts according to the operation and the workflow through the script generation module, so that the user can more conveniently perform high-performance calculation on the operation through the computer cluster managed by the resource manager; the calculation result file is recorded through the recording module, so that a user can conveniently download the calculation result file at any place through a network; moreover, the system can integrate the computer clusters managed by a plurality of resource managers, thereby improving the effect of high-performance calculation of the operation and improving the user experience.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

FIG. 1 is a block diagram of a Web-based job computing system provided by an embodiment of the present invention;

FIG. 2 is a block diagram of another Web-based job computing system provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of a software architecture of another Web-based job computing system provided by an embodiment of the present invention;

fig. 4 is a schematic diagram of a system architecture of another Web-based job computing system according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Referring to fig. 1, fig. 1 is a block diagram of a Web-based job computing system according to an embodiment of the present invention. The system may include:

a receiving module 100 for receiving a job and calculating a workflow required for the job through a network; wherein the workflow is composed of files.

And the script generation module 200 is used for generating a job script according to the job and the workflow.

And the execution module 300 is configured to execute the script through the resource manager to obtain a calculation result file of the job.

And the recording module 400 is used for recording the calculation result file.

For the mode that the receiving module 100 uploads or sets the jobs and the workflows through the network, that is, the positions of the jobs and the workflows uploaded or set by the user, the jobs and the workflows uploaded or set by the user in the Web can be acquired by the Web server where the system is located through the Web interface; the Web server may also obtain the user through another interface to upload or set the job and the workflow in another corresponding location, for example, the Web server receives the user's upload or set job and workflow in an application program through a network. The present embodiment is not limited to this.

It is understood that, in the process of generating the job script by the script generation module 200 according to the job and the workflow, the required resources such as the memory required for calculating the job, the kernel number and the suspension time allocated to the job need to be set. The setting mode of the request resource can be that a preset request resource is directly used, namely the default request resource of the system or the request resource obtained after the operation is analyzed; it is also possible to request resources set by the user received through the receiving module 100, such as jobs, workflows, and request resources directly uploaded or set by the user in the Web interface. The present embodiment does not set any limit to this.

It should be noted that, the execution module 300 in the system provided in this embodiment may execute the script generated by the script generation module 200 through the resource manager, and obtain the calculation result file of the job; the script received by the receiving module 100 may be directly executed by the resource manager to obtain the calculation result file of the job. As long as the script can be executed by the resource manager to obtain the calculation result file of the job, the obtaining method and process of the script are not limited in any way.

Preferably, the system provided by this embodiment may further include an access module and a parsing module, so as to monitor the execution state of the script, and a recording module that stores the execution state of the script, that is, the computation state of the job, for the convenience of viewing by the user. For example, a user may send a presentation request to the recording module via the Web interface to view the computing state of a task in the Web interface.

Preferably, the system provided by this embodiment may further include a management unit, so that the user may manage the job conveniently. Such as a user can delete a job through a Web interface, cancel a running job, pause and retain a job before or after starting running, and then the user can request to change the job in the future.

Preferably, the system provided by this embodiment may further include an authentication module, so that the user needs to log in a preset Web page or APP first, and then perform high-performance calculation on the job, thereby improving the security of the system.

Preferably, the system provided in this embodiment may further include a display module, so as to facilitate a user to view summary information of the current state of the computer cluster managed by the resource manager, and further facilitate the user to set the requested resource.

Preferably, the system provided by this embodiment may further share a module to store a workflow or a file in the workflow, so that the system can be conveniently used by itself or other users, and thus, the collaboration between researchers and a research community is improved.

Specifically, for the specific setting of the system provided in this embodiment, the system may include only 4 modules provided in this embodiment, may also include one or more modules in the above preferred solution, and may further include other modules to achieve a better effect. As long as the job can be calculated by the resource manager according to the received job and workflow, the purpose of calculating the high performance of the job is achieved, which is not limited in this embodiment. The specific setting position and setting manner of each module in the system provided in this embodiment may be set by a designer according to a practical scenario and a user requirement, for example, the script generating module 200 and the executing module 300 in this embodiment may be set in a database of a Web server as an embodiment of a computing operation function in the system provided in this embodiment. This embodiment also does not impose any limitation.

In this embodiment, the script generation module 200 generates the job script according to the job and the workflow, which can eliminate the barriers of the user to compile scripts and develop tools, so that the user can more conveniently perform high-performance calculation on the job through the computer cluster managed by the resource manager; the calculation result file is recorded through the recording module 400, so that a user can conveniently download the calculation result file at any place through a network; moreover, the system can integrate the computer clusters managed by a plurality of resource managers, thereby improving the effect of high-performance calculation of the operation and improving the user experience.

Referring to fig. 2, 3 and 4, fig. 2 is a block diagram of another Web-based job computing system according to an embodiment of the present invention; FIG. 3 is a schematic diagram of a software architecture of another Web-based job computing system provided by an embodiment of the present invention; fig. 4 is a schematic diagram of a system architecture of another Web-based job computing system according to an embodiment of the present invention. The system may include:

a receiving unit 110 configured to receive a preset Web interface transmission job and a workflow required for calculating the job through a network; wherein the workflow is composed of files.

And the recording module 400 is used for recording the calculation result file.

The verification module 500 is configured to determine whether the user login information is login information of a preset user according to the user login information received by the receiving unit 110 through the network; if yes, the receiving unit 110 receives the job and workflow of the user corresponding to the user login information.

The access module 600 is configured to access the resource manager at preset time intervals, and acquire data corresponding to a script execution state;

the parsing module 700 is configured to parse the data, obtain status information of a job corresponding to the data, and store the status information in the recording module 400.

As shown in fig. 3, the software architecture of the system provided in this embodiment may be developed by using a related architecture for a high-performance computing system that provides jobs for a user through a Web interface, which is not limited in this embodiment.

As shown in fig. 3, the software architecture of the present system may adopt a three-layer architecture: the system is composed of a presentation layer, an application layer and a data storage layer. The presentation layer is responsible for receiving and passing user input to the application layer, i.e., providing a Web interface for the user to enter jobs and workflows, which is not limited in this embodiment. Presentation options corresponding to the access module 600 and the parsing module 700 may also be provided, which may allow a user to monitor and manage jobs on the mobile device.

The application layer is composed of two modules (or applications), namely a user module and a job module in fig. 1, responsible for performing tasks according to input received from the presentation layer. The user module, i.e., the verification module 500, is responsible for user management and authentication, and performs security functions, as well as providing collaboration functions built into the job computing system. The job modules may include script generation module 200 and execution module 300 and modules related to job computation responsible for providing workflow creation, management, submission, management and monitoring of job and job history, and resource manager plug-in management, interaction and configuration resource manager settings related functions.

In addition to the two modules, the application layer also includes a background service and an authentication server. The purpose of the background services, namely the access module 600 and the parsing module 700, is to continuously poll the underlying resource manager to ensure that the job records remain up-to-date in the logging module 400 (database). The authentication server is used for the verification module 500 to verify the user identity through the Linux identity verification system, so that the user has the same authority when logging in the service through the SSH, and if the system can verify the user identity by itself, the authentication server does not need to be set.

The data store layer (database) is responsible for storing jobs, workflows, and user details, as well as resource manager configuration data and settings.

As shown in fig. 4, the present system is installed and configured as a separate component on top of an existing high-performance computing cluster to form a two-tiered system architecture. I.e., existing high-performance computing software can remain unchanged and need not be modified or adjusted in any way. In order to make a job computing system compatible with as many resource managers as possible, a custom plug-in architecture is designed. With this architecture, adding support to other resource managers requires only writing a plug-in for the resource manager. These plug-ins follow three rules:

first, the plug-in must inherit from the base explorer class. This class provides some of the important functions required by the job computing system, but also requires plug-ins to cover some of the unrealized functions.

Second, a number of predefined objects are created as the output of the basic functions that the plug-in must cover. The plug-in must populate these objects with data from the resource manager and return them to the job computing system. By doing so, the job computing system receives and knows how to process the object or list of objects. The underlying explorer object content may vary, so the objects contain metadata describing what type of data they hold in order for the job computing system to adapt its interface to the data.

Third, the plug-in must be copied and pasted into a specific folder that the job computing system knows is to find. To use a plug-in, the plug-in name must be specified in the job computing system configuration file.

With this architecture, it will be possible to quickly increase the number of resource managers supported.

Specifically, the system can provide job calculation functions for three types of users, namely developers, administrators and researchers. These functions include the following:

1. job calculation and management. The job computing system may allow a user to submit new jobs to the cluster at runtime, monitor and manage jobs, and obtain the results of the jobs after the jobs are completed. It is implemented by interfacing with the underlying resource manager and workflow management services.

The job computing system allows a user to upload or create a script to be submitted to the cluster and then request resources, including required memory, kernel count, and hang time to be assigned to the job. Based on these inputs, the job computing system generates a job script via script generation module 200 and submits it to the resource manager for execution. The job computing system can run any program or script that can be executed from the command line, monitoring the job until it is complete. Depending on the resource manager plug-in, including the resources used, the input and output streams of the job, and the job catalog of the job. The job exit status combined with the output stream is used to determine whether the execution was successful. Upon completion, the user may access the results of the job from the output and error stream, or download the calculation result file from the recording module 400. All results are accessible through the "job history" tab of the Web interface.

Real-time monitoring of jobs is accomplished through the interaction of the access module 600 with the explorer plug-in. The access module 600 is continuously polled by the background service to update job details. The returned data is parsed by the parsing module 700 and stored in the job computing system database (logging module 400) to maintain a permanent record of all jobs.

In addition to monitoring the status of jobs, job computing systems provide job management functions for users. Such as the user can delete a job from his job history, cancel a running job, pause or retain a job before or after starting running, and the user can request to change jobs in the future. Since change requests may include requests for other resources, they require approval by the administrator user. If a non-administrator user requests a change to one of its tasks, the request will be forwarded to the administrator user for grant or denial.

2. And (6) workflow management. In addition to interfacing with the underlying resource manager, the job computing system also provides functionality that allows users to build and execute complex computing pipelines or workflows. A workflow is made up of a set of modules, each representing a tool that has been added to a job computing system.

The tool may be any command line utility that has been installed on the cluster, or may be a custom script or executable file uploaded by the user. For each tool, the user provides detailed information to the job computing system, including commands for running the tool or script from the terminal, parameters that the commands can take, resources that should be allocated to the tool by the resource manager, and expected outputs that the tool will generate. All these details are entered into the system through the Web interface and then stored in the database backend. In addition, scripts and executables uploaded by the user are automatically stored in a directory hierarchy managed by the job computing system.

The job computing system provides a workflow creation interface in which various tools can be arranged into a complex workflow. Tools are added to the workflow, can be rearranged, and dependencies can be created between them by drag and drop.

A job computing system allows some modules in a workflow to run in parallel while other modules run in sequence. Workflows and tools are selected and executed from the respective "workflow" and "tools" tabs of the job computing system interface. To run a tool or workflow, the job computing system provides an automatically generated Web interface. This interface allows the user to specify the value of each parameter for the created tool.

3. A display panel. The job computing system provides a display panel containing detailed status information. The panel provides the user with summary information describing the current state of the cluster, including how many nodes are online/offline, the proportion of processors used throughout the cluster, the number of jobs currently running or waiting to run, and the amount of disk space still available.

The user may check the status of each node in the cluster and the current queue of jobs submitted to the cluster through the panel. The specific information displayed by the display panel will depend on the explorer plug-in being used. If the user has the appropriate permissions, the user can cancel the job directly from the queue.

4. Access control and collaboration. Once a user has created a tool or workflow, it can be shared with other users in the system. The creator of the tool or workflow may assign administrator privileges to certain users. Except for removing the creator with the administrator identity, the administrator owns all the rights the creator owns. The administrator may assign rights to other users. These permissions include the ability to execute a tool or workflow, export and download a tool or workflow to be imported into another instance of the job computing system, edit tools, and the like. Tools and workflows can also be exposed so that all users have the right to run them.

5. And (5) cluster configuration. In addition to interfacing with the underlying resource manager to provide job management functionality, the job computing system also provides cluster configuration functionality. This allows an administrator to set up and manage queues, configure servers, set up and add compute nodes. The functionality provided by this page will depend on the explorer plug-in being used.

As for the specific arrangement and number of each module in the system provided in this embodiment, other modules may be correspondingly arranged or added through the description of the above functions. The present embodiment is not limited to this.

In the embodiment of the invention, through the verification module 500, the user needs to log in the preset Web page and then perform high-performance calculation on the operation, so that the safety of the system is improved; the access module 600 and the analysis module 700 store the state information of the job in the recording module 400, so that the user can conveniently check the calculation state of the job, and the user experience is improved; through the preset Web interface, a page which can be used by a user is provided, the interaction between the system and the user is facilitated, and the user experience is improved.

The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the system disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the system part for description.

Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality using different systems for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

The steps of a system or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

The Web-based job computing system provided by the present invention has been described in detail above. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the system and its core concepts of the present invention. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.

Claims

1. A Web-based job computing system, for high-performance computing, comprising:

the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving a job and calculating a workflow required by the job through a network; wherein the workflow is comprised of a set of modular tools including command line utilities, scripts or executable files;

the script generation module is used for generating a job script supporting serial or parallel running of a multi-module tool with a dependency relationship according to the job and the workflow;

the execution module is used for executing the job script through a resource manager based on a plug-in architecture to obtain a calculation result file of the job;

the recording module is used for recording the calculation result file;

the plug-in architecture is an extended interface architecture designed for enabling the high-performance computing-oriented job computing system to be compatible with various resource managers, so that the job computing system is compatible with the various resource managers; using the plug-in architecture, adding support to other resource managers only needs to follow plug-in interface rules to customize a specific plug-in interface for the resource managers; the rules followed by the plug-in include:

firstly, a plug-in inherits from a basic resource manager class; the classes provide some important functionality needed by the job computing system, but also require plug-ins to cover some unrealized functions;

secondly, taking a plurality of predefined objects as the output of the basic function covered by the plug-in; the plug-in populates these objects with data from the resource manager and returns them to the job computing system; through the plug-in, the job computing system receives and knows how to process an object or a list of objects; the underlying explorer object contents are allowed to differ, so the objects contain metadata describing what type of data they hold, so that the job computing system adapts its interface to the data;

thirdly, the plug-in is placed in a specific folder which is known to be searched by the operation computing system, and the name of the plug-in is specified in the configuration file of the operation computing system;

using the plug-in architecture will enable a fast increase in the number of resource managers supported.

2. The Web-based job computing system according to claim 1, wherein the receiving module comprises:

3. The Web-based job computing system according to claim 2, wherein the script generation module comprises:

a second generation unit, configured to generate a job script according to the job, the workflow, and the request resource sent by the Web interface; the request resource sent by the Web interface comprises a memory and a kernel number required by the calculation of the operation and the suspension time distributed to the operation.

4. The Web-based job computing system according to claim 3, further comprising:

the access module is used for accessing the resource manager according to a preset time interval and acquiring data corresponding to the execution state of the job script;

and the analysis module is used for analyzing the data corresponding to the execution state of the job script, acquiring the state information of the job corresponding to the data corresponding to the execution state of the job script, and storing the state information to the recording module.

5. The Web-based job computing system according to claim 4, wherein the recording module comprises:

and the display unit is used for sending the state information corresponding to the display request to the Web interface according to the display request sent by the Web interface so as to display the state information to a user through the Web interface.

6. The Web-based job computing system according to claim 5, wherein the execution module further comprises:

7. The Web-based job computing system according to claim 6, wherein the management unit includes:

a deleting subunit, configured to delete, according to the deletion request, the job script that is not executed by the resource manager;

a cancellation subunit, configured to cancel, according to the cancellation request, the job script that is being executed by the resource manager;

a suspending subunit, configured to suspend, according to the suspension request, the job script that is not executed or is being executed by the resource manager;

and the change subunit is used for correspondingly changing the suspended job script according to the change request.

8. The Web-based job computing system according to any one of claims 1 to 7, further comprising:

9. The Web-based job computing system according to claim 8, further comprising:

10. The Web-based job computing system according to claim 9, further comprising: