CN108536528A - Using the extensive network job scheduling method of perception - Google Patents

Abstract

The invention discloses a kind of extensive network job scheduling methods of application perception.This method is mainly made of the following steps：The first step, user submit interactive interface to submit operation by grid work.Second step, system predict Activity Calculation amount according to operational feature information.Third walks, the large-scale distributed grid node current state information of system queries.4th step, job scheduler accordingly can calculate nodes according to the lookup of operation application demand.5th step, distribution operation to corresponding calculate node.6th step judges in job queue whether all operations dispatch and finishes, do not dispatch and finish, and recycles and executes the 4th step and the 5th step, otherwise waits for next dispatching point and execute this method again.Using this method, grid system resource utilization rate can be effectively improved and meet user demand by being compared compared with the existing job scheduling method based on first-come-first-served policy.

Description

Application-aware Large-Scale Grid Job Scheduling Method

technical field

The invention belongs to the technical field of computer software and large-scale parallel distributed processing system resource management and task scheduling, and relates to an application-aware large-scale grid job scheduling method.

Background technique

The China National Grid is a new-generation information infrastructure test bed that aggregates high-performance computing and transaction processing capabilities. Through resource sharing, collaborative work and service mechanisms, the grid can effectively support applications such as scientific research, resource environment, advanced manufacturing and information services. It is mainly based on the existing high-performance computing environment foundation, focusing on research on key technologies for application service optimization in high-performance computing environments, further improving the resource construction mechanism, and establishing practical high-performance computing applications with new operating mechanisms and rich application resources Serve the environment and application community, reduce the cost of high-performance computing applications, and comprehensively improve the service level of high-performance computing applications.

At present, the China National Grid has aggregated the main nodes in the north, the main nodes in the south, the national supercomputing center and ordinary nodes, forming a large-scale distributed computing grid. Among them, the National Supercomputing Wuxi Center has the world's first supercomputer with a peak computing performance exceeding one billion floating point operations per second - "Sunway Taihu Light". The computing system fully adopts the domestic "Sunway 26010" The many-core processor is also my country's first world-ranked supercomputer built entirely with domestically produced processors. The National Supercomputing Guangzhou Center is equipped with the Tianhe-2 system. The peak computing speed of the first phase is 549 petaflops, the continuous computing speed is 339 petaflops, and the energy efficiency ratio is 1.9 billion double-precision floating-point operations per watt.

The National Supercomputing Tianjin Center is equipped with the "Tianhe-1" high-efficiency computer system with a peak performance of 4,700 trillion operations per second, which ranked first in the world in the 2010 HPC TOP500 ranking. In addition, it is equipped with Tianhe-Tianteng (TH-1) system with exascale computing performance; Tianhe-Tianxiang system with 128 Intel-EX5675 CPUs; and Tianhe-Tianchi system with 96 CPUs. These supercomputing centers all have quite astonishing computing and storage capabilities, and ordinary nodes can perform large-scale computing, such as the Inspur "Tianshuo" high-performance computer system deployed by Tsinghua University, the system's general-purpose processor has a computing capability of 1.04 million per second The calculation capability of the GPU acceleration component is 64 trillion times per second; the Shenzhen Institute of Advanced Technology of the Chinese Academy of Sciences adopts the Sugon 5000A cluster system, which has a general computing capacity of 10 trillion times per second and a special computing capacity of 200 trillion times per second. The storage capacity is 500TB, the internal data exchange capacity is up to 2GB/s, and the resource availability rate of the entire system exceeds 90%. The Gansu Provincial Computing Center is equipped with a high-performance scientific and engineering computing cluster with a peak computing value of 40 trillion times per second, and a network distribution capacity of 35TB storage system. The University of Hong Kong has multiple clusters including two organizations, the Department of Computer Science and the Computer Center of the University of Hong Kong, with a peak general computing capacity of 23.45 trillion operations per second and a dedicated computing capacity of 7.7 trillion operations per second.

However, the core of how to make full and effective use of these resources, improve the efficiency of user job completion and reduce user costs is user job scheduling. The main goal of job scheduling is to determine the job allocation plan and job execution order of user jobs according to the scheduling strategy to meet user and system requirements under the constraints of user time limit and cost. At present, the job scheduling on China's national grid only adopts a simple first-come, first-served strategy, which has relatively low efficiency, which has affected the effective application of the national grid. To solve this problem, the present invention attempts to make a comprehensive balance decision on the applications supported by computing nodes, user applications, and the status quo of computing node resources, in order to achieve efficient scheduling of system operations, thereby providing effective information for national grids and other large-scale distributed systems. Technical Support.

Contents of the invention

The invention proposes an application-aware large-scale grid job scheduling method aiming at the problem of inefficiency caused by resource heterogeneity, regional distribution and user application diversity in the national grid and other large-scale distributed systems. In order to solve the above problems, the technical solution adopted in the present invention is:

An application-aware large-scale grid job scheduling method, characterized in that it includes the following steps:

Step 1: The user submits a job to the grid system, and the grid system stores the job information in the corresponding database grid job table, and then inserts the job into the job queue based on the multi-thread sharing mechanism according to the processing situation. It includes a ready queue, a running queue and a result feedback queue. The ready queue is used to queue jobs waiting to be processed, the running queue is used to queue jobs scheduled to run on a certain computing node, and the result feedback queue It is used to queue the finished jobs and save the results returned from the nodes in the queue;

Step 2: Predict the running time of the job submitted by the user according to the pre-established user job running time model;

Step 3: Query the current status information of large-scale distributed grid nodes in real time, and store them in the database grid node resource information table;

Step 4: Find the grid nodes that can be used for job calculation according to the application requirements of the job to calculate the node;

Step 5: On the computational node set found in step 4, find the nodes that meet the running time limit of the user's job as the schedulable node set, and then select the node with the lowest resource utilization rate from the schedulable node set as the grid node for executing the job;

Step 6: assign jobs to the grid nodes obtained in step 5;

Step 7: Determine whether all jobs in the job queue have been scheduled. If the job is scheduled, wait for the next scheduling point; otherwise, take the job from the job queue and return to step 4 for cyclic execution.

In the above-described application-aware large-scale grid job scheduling method, in the step 1, the job information includes user ID, job ID, application software requirements, version number, license, number of nodes, number of CPUs, Number of cores, running time, job data volume, and expected completion time.

In the above-mentioned application-aware large-scale grid job scheduling method, in the step 2, the user job running time model is based on the historical database of system historical jobs running on each computing node, described as: <Job _i ,Time _i,j >; where the application characteristics of Job _i include data volume Jd(Job _i ), job scale Js(Job _i ); and calculate the application characteristics of the job to be scheduled Job and the historical job Job _i by the following formula Similarity ρ _i

ρ _i ＝|Jd(job _i )-Jd(job)|+|Js(job _i )-Js(job)|

Take the historical job running time Time _i,j with the smallest similarity ρ _i as the predicted running time of the job on each grid computing node.

In the above-mentioned application-aware large-scale grid job scheduling method, in the step 3, the current status information of the large-scale distributed grid nodes includes online running job volume, resource utilization rate, node support Hardware and software information.

In the described method for large-scale grid job scheduling with application awareness, the step 5 includes the following process:

Step 5.1: According to the real-time node resource information, initialize the data structure job_node_starttime of the computable node set job start time;

Step 5.2: Calculate the deviation value θ between the running time submitted by the user's job history and the actual running time, as shown in the following formula:

in, The running time and actual running time of the i-th job submitted by the user are respectively, and m is the number of jobs submitted by the user in the grid system;

Step 5.3: If the deviation value θ<0.2, the job running time used in the job scheduling decision is the running time in the job information submitted by the user, otherwise, the job running time predicted by the system in step 2 is used as the basis for the scheduling decision;

Step 5.4: Using the job running time obtained in step 5.3 and the start execution time of the job on the node, sequentially calculate the job completion time job_node_endtime of each node in the computable node set;

Step 5.5: According to the completion time of the job on each computable node, judge whether it meets the job running time limit, if so, put this node into the schedulable node set;

Step 5.6: Select the node with the lowest resource utilization rate from the schedulable node set, and schedule the job to this grid node.

In the above-mentioned application-aware large-scale grid job scheduling method, in step 6, assigning jobs to corresponding computing nodes includes the system submitting job computing requests to computing nodes, job data transmission, job execution status query and system Scheduling information is fed back to the user.

The technical effect of the present invention is that, compared with the existing job scheduling method based on the first-come-first-serve policy, the method provided by the present invention can effectively improve the resource utilization rate of the grid system and meet user needs.

The present invention will be further described below in conjunction with accompanying drawing.

Description of drawings

Fig. 1 is a flow chart of an application-aware large-scale grid job scheduling method provided by the present invention;

Fig. 2 is a large-scale grid global scheduler architecture diagram provided by the implementation of the present invention;

Fig. 3 is a user operation interface diagram provided by the implementation of the present invention;

Figure 4 is a diagram of job state transitions.

Detailed ways

The present invention proposes an application-aware large-scale grid job scheduling method, the flowchart of which is shown in FIG. 1 . This method is based on the large-scale China National Grid, starting from the user's job application characteristics, and proposes a large-scale distributed job scheduling method that meets the job deadline and application requirements, which can effectively improve the job scheduling efficiency of the China National Grid.

The present invention realizes through following technical scheme:

This embodiment is oriented to the large-scale distributed computing system China National Grid, and schedules jobs to each computing node in the global job scheduler, in order to achieve the integration of idle resources of large-scale computing nodes, make full use of and improve the service quality of China National Grid . The development of this embodiment is based on the Linux platform, using the MySql database. Its architecture is shown in Figure 2. The global job scheduler is mainly composed of scheduling decision-making module, job assignment module, information collection module and communication module. Among them, the scheduling decision-making module integrates scheduling algorithms to make decisions on the scheduling of jobs and corresponding computing nodes; the job execution module automatically generates instructions to submit the current job to the computing center according to the computing nodes selected by the scheduling decision-making module; the information collection module is responsible for collecting information from each computing node Resource status (such as the number of idle servers, number of CPU cores, memory usage, etc.), and can also collect job completion status information, such as the job is running, the estimated completion time, etc., and can feedback this information to the user in real time; the communication module is responsible for the user The main program communicates with each computing node, such as sending the instructions generated by the job execution module to the corresponding computing node, so that the job is submitted and executed on the node.

The user first submits the job through the job interaction interface shown in Figure 3 of the grid system. The main job information includes user ID, job ID, application software requirements, version number, license, number of nodes, number of CPUs, number of cores, running time, job data volume, and expected completion time. When the user clicks the job submission button, this embodiment first stores the user job information in the MySql database grid job table. The main attributes of this table are: jobId-job number, jobName-job name, software_type application software requirements, software_system Software environment, resource_type resource type, resource_value—the resource value under the corresponding type, cpu—the cpu required by the user, runtime—the estimated running time of the job by the user, deadline—the user wants to complete the job before this time, budget—the user budget, priority - job priority, memory - memory required by the job, disk - disk size required by the job. Then, insert this job into the job queue based on the multi-thread sharing mechanism. Job queue is a technology used by this patent to manage jobs submitted by users. When a job is scheduled to run on a certain computing node, the job will be transferred from the "ready queue" to the "running queue"; when the job finishes running, the job If necessary, transfer from the running queue to the result feedback queue, and save the result returned from the node in the queue. Multiple users may submit jobs at the same time, that is, operate on the same queue, so the mutual exclusion of queue operations must be guaranteed. At the same time, a certain amount of concurrency must be maintained. Its operation status transition diagram is shown in Figure 4.

The second step of this embodiment predicts the running time of the job submitted by the user. In this embodiment, a historical database of user jobs running on each computing node is established. Mainly collect job parameters, such as user number/user name, job number, number of CPUs used by the job, job queue, job working directory, job command, job submission time, job start time, job end time, and job exit reason; Grid computing node parameters, such as cluster, host name, time: when to collect data, CPU occupancy rate, CPU microarchitecture data, CPU floating-point computing capability, CPU instruction execution speed, last-level cache hit rate, memory occupancy rate , memory read and write bandwidth, Infiniband network usage, Ethernet usage, and disk/NFS usage. In this embodiment, the running time distribution of all jobs and jobs by users and applications is counted, and it is found that a large number of jobs are short-term jobs. At the same time, the statistical results of a sufficient number of jobs show that the distribution of job duration is a power-law distribution. Secondly, the time for users to submit jobs has the nature of clustering, and there is a large gap between the running time of two jobs with a long distance. Finally, this patent studies user behavior habits, combines application characteristics and user job characteristics, and establishes a user-specific application running time model, thereby using user job information and this model to obtain a predicted value of job running time. The user job running time model referred to in this embodiment is based on the historical database of system historical jobs running on each computing node, described as: <Job _i ,Time _i,j >; wherein the application characteristics of Job _i include the data volume Jd(Job _i ), the job scale Js(Job _i ); and calculate the similarity ρ _i between the job to be scheduled and the historical job _i in terms of application characteristics by the following formula

ρ _i ＝|Jd(job _i )-Jd(job)|+|Js(job _i )-Js(job)|

Take the historical job running time Time _i,j with the smallest similarity ρ _i as the predicted running time of the job on each grid computing node.

The third step of this embodiment is to query the current state information of large-scale distributed grid nodes in real time, such as online operation workload, resource utilization rate, hardware and software information supported by nodes, etc., and store them in the database grid node resource information surface. The main fields of this table are node_Id—node ID, node_name—node name, resource_type—resource type (indicating the type of resource owned by the corresponding node), resource_software (indicating the application software that the user can use), wait_job_num—the number of jobs currently waiting to run, waittime - How long does the user need to wait to execute, cpu - the current number of idle CPU cores, memory - the current available memory status, disk - the current remaining disk capacity, max_runtime - the maximum time allowed for user jobs to run, predict_runtime - prediction The job running time, net_delay-network delay (network delay needs to be considered when cross-center scheduling), cpu_usage-cpu usage, memory_usage-memory usage. The implementation of this part adopts the C/S model. The client is responsible for collecting the current status information of each grid node in real time, and the server is responsible for receiving the information and storing it in the node resource information table of the Mysql database. Its communication adopts the socket socket under the Linux platform, uses the select function to monitor the socket, and establishes a connection to realize the communication. The communication protocol mainly adopts the connection-oriented TCP protocol, which can ensure the security of information. Multi-thread multiplexing is mainly used for concurrent access.

The fourth step is to find the corresponding grid computing nodes according to the application requirements of the job. The software_type and software_system in the grid job table are the application software and supporting software environment requirements required for job running. In this embodiment, these information will be used to search for a set of node resources that meet the requirements in the node resources, and the following data structures and queue operation functions are used in its implementation. For the current job information that needs to be scheduled, select the queue that meets the application requirements from the current grid node queue, and store the queue name and grid node in an array. The key function is intsw_info_read(void*job_info, void*sw_match[]), where the function parameter is a job pointer and a pointer array of structure sw_match_info type. This array stores the sw_match_info structure pointers of grid queues that meet software requirements. The return value is the number of grid queues that meet the software requirements. In this function, the product of constants GN_NUM, QUEUE_NUM, SOFT_NUM (the number of grid nodes, the number of queues, and the number of software) is given to generate the number of candidate grid queues. After the function call is completed, the grid queue grid ID and queue name information matching the software will be obtained. Secondly, according to the currently obtained sw_mathch array, put the queue with the array grid ID and queue name into the linked list. The key function is void queue_info_read(List*l, void*sw_match[]). Among them, List*l in the function parameter is a queue chain list for storing conforming software matching. Then, according to the current job information, if the queue information linked list for storing software matching meets the hardware requirements, put the queue information into the hardware matching queue linked list, and delete the queue directly from the software matching linked list if not. In this way, the node queue linked list after hardware screening is obtained. The key function is intqueShift(List*sw_list,List*hw_list,void*data), the second parameter finally gets the linked list pointer of the hardware matching grid queue, and the return value is the number of grid queues after hardware matching. This implements job computable node set lookups.

The fifth step is that the job scheduler implements job scheduling that meets the user's job running time limit on the basis of the computable node set. In this embodiment, the real-time information of node resources is used to establish the data structure job_node_starttime, which can calculate the start time of the node set job, and then calculate the deviation value θ between the running time submitted by the user's job history and the actual running time. The specific calculation process is as follows: first, based on the real-time node resource information, initialize the data structure job_node_starttime of the computable node set job start time; then calculate the deviation value θ between the running time submitted by the user’s job history and the actual running time, as shown in the following formula:

in, are the submitted running time and actual running time of the i-th job submitted by the user, m is the number of jobs submitted by the user in the grid system; in this embodiment, when the deviation value θ<0.2 of a certain node, the job scheduling decision The time used is the running time in the job information submitted by the user, otherwise the job running time predicted in the second step of this patent is used as the scheduling basis. If the user is submitting a job for the first time, the predicted time of this patent is used as the job scheduling time. Based on the scheduling time calculation method and the start execution time of the job at the node, this patent calculates the job completion time job_node_endtime of each node in the computable node set sequentially. Then, according to the completion time of the job on each computable node, it is judged whether it meets the job running time limit, and if so, put this node into the schedulable node set. Finally, this patent selects the node with the lowest resource utilization rate from the set of schedulable nodes, and schedules jobs to this grid node.

The sixth step is that the system is responsible for assigning jobs to corresponding computing nodes, including job computing requests, job data transmission, job execution status query, and scheduling information feedback. Then, it is judged whether all the jobs in the job queue have been scheduled, and if the scheduling is completed, the next scheduling point is waited for. Otherwise, after completion, take the job from the job queue, and then repeat the fourth, fifth and sixth steps to realize job scheduling.

Claims (6)

1. An application-aware large-scale grid job scheduling method is characterized by comprising the following steps:

step 1: a user submits a job to a grid system, the grid system stores job information into a corresponding database grid job table, and then the job is inserted into a job queue based on a multithreading sharing mechanism according to a processing condition, wherein the job queue comprises a ready queue, a running queue and a result feedback queue, the ready queue is used for arranging and scheduling jobs to be processed, the running queue is used for arranging and scheduling jobs to be run at a certain computing node, and the result feedback queue is used for arranging jobs after running and storing results returned from the nodes in the queue;

step 2: according to a pre-established user operation running time model, carrying out running time prediction on the operation submitted by a user;

and step 3: inquiring the current state information of the large-scale distributed grid nodes in real time, and storing the current state information into a grid node resource information table of a database;

and 4, step 4: searching a grid node which can be used for operation calculation according to the application requirement of the operation, namely a calculation node;

and 5: searching nodes meeting the operation time limit of the user operation on the calculable node set found in the step 4 as a schedulable node set, and then selecting the node with the lowest resource utilization rate from the schedulable node set as a grid node for executing the operation;

step 6: distributing the operation to the grid nodes obtained in the step 5;

and 7: judging whether all the jobs in the job queue are scheduled or not, and waiting for the next scheduling point if the scheduling is finished; otherwise, taking the job from the job queue and returning to the step 4 for circular execution.

2. The method according to claim 1, wherein in step 1, the job information includes user ID, job ID, application software requirement, version number, License, number of nodes, number of CPUs, number of many cores, running time, job data amount, and expected completion time.

3. The application-aware large-scale grid job scheduling method according to claim 1, wherein in step 2, the user job running time model is a historical database based on system historical jobs running at each computing node, and is described as:<Job_i,Time_i,j>(ii) a Wherein Job_iThe application characteristics of (1) include the data volume Jd (Job)_i) Job size Js (Job)_i) (ii) a And pass throughThe Job Job to be scheduled and the historical Job Job are calculated by the following formula_iProximity p on application features_i

ρ_i＝|Jd(job_i)-Jd(job)|+|Js(job_i)-Js(job)|

Taking the proximity rho_iMinimum historical job run Time_i,jAs the predicted run time of the job at each grid compute node.

4. The method according to claim 1, wherein in step 3, the current state information of the large-scale distributed grid nodes includes on-line operation workload, resource utilization rate, and node-supportable hardware and software information.

5. The application-aware large-scale grid job scheduling method according to claim 1, wherein the step 5 comprises the following steps:

step 5.1: initializing a data structure jobnode _ start time capable of calculating the node set operation start time according to the real-time node resource information;

step 5.2: calculating the deviation value theta of the user job history submission running time and the actual running time as shown in the following formula:

wherein,submitting the operation time and the actual operation time for the ith job submitted by the user respectively, wherein m is the number of jobs submitted by the user in the grid system;

step 5.3: if the deviation value theta is less than 0.2, the operation running time adopted by the operation scheduling decision is the running time in the operation information submitted by the user, otherwise, the operation running time predicted by the system in the step 2 is adopted as the basis of the scheduling decision;

step 5.4: sequentially calculating job completion time jobnode end time of each node in the calculable node set by using the job running time obtained in the step 5.3 and the start execution time of the job on the node;

step 5.5: judging whether the operation time limit is met or not according to the completion time of the operation on each calculable node, if so, putting the node into a schedulable node set;

step 5.6: and selecting the node with the lowest resource utilization rate from the schedulable node set, and scheduling the job to the grid node.

6. The method according to claim 1, wherein the step 6 of distributing the job to the corresponding computing node comprises the step of the system sending job calculation request, job data transmission, job execution status query and system scheduling information to the computing node for feedback to the user.