CN112799807A - Cluster task pre-scheduling distribution method, device, medium and computer equipment - Google Patents

Cluster task pre-scheduling distribution method, device, medium and computer equipment Download PDF

Info

Publication number
CN112799807A
CN112799807A CN202110099364.XA CN202110099364A CN112799807A CN 112799807 A CN112799807 A CN 112799807A CN 202110099364 A CN202110099364 A CN 202110099364A CN 112799807 A CN112799807 A CN 112799807A
Authority
CN
China
Prior art keywords
task
cluster
queue
memory
dynamic parameter
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110099364.XA
Other languages
Chinese (zh)
Other versions
CN112799807B (en
Inventor
邓天全
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan Huada Gene Technology Service Co ltd
Original Assignee
Wuhan Huada Gene Technology Service Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan Huada Gene Technology Service Co ltd filed Critical Wuhan Huada Gene Technology Service Co ltd
Priority to CN202110099364.XA priority Critical patent/CN112799807B/en
Publication of CN112799807A publication Critical patent/CN112799807A/en
Application granted granted Critical
Publication of CN112799807B publication Critical patent/CN112799807B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/901—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/903—Querying
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/48—Indexing scheme relating to G06F9/48
    • G06F2209/484—Precedence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/50—Indexing scheme relating to G06F9/50
    • G06F2209/5021—Priority

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a method, a device, a medium and computer equipment for pre-dispatching and distributing cluster tasks, wherein the method comprises the following steps: establishing a cluster queue database table and configuring dynamic parameter tasks, wherein the dynamic parameter tasks comprise queues; inquiring the resource use condition of the queue in a cluster queue database table according to the dynamic parameter task, and acquiring a usable resource table according to the resource use condition of the queue in the cluster queue database table; sequencing the available resource table, and determining task parameters of the dynamic parameter task according to a sequencing result; and submitting the tasks carrying the task parameters to a cluster scheduling system so that the cluster scheduling system allocates and executes the tasks carrying the task parameters. Therefore, waste caused by resource use can be effectively reduced, the resource utilization rate is improved, the task execution speed can be effectively accelerated, resources can be released more quickly, and the universality is high.

Description

Cluster task pre-scheduling distribution method, device, medium and computer equipment
Technical Field
The invention relates to the technical field of cluster application, in particular to a cluster task pre-scheduling distribution method, a device, a medium and computer equipment.
Background
When performing bioinformatic analysis, it is often necessary to perform delivery tasks on a high-performance cluster of servers. The server cluster is generally composed of a login node and a computing node, wherein the login node is used for executing simple Linux commands, cannot execute tasks which are large in resource consumption, and the tasks which are large in resource consumption are delivered to the computing node by the login node.
In the related art, after parameters such as a memory, a thread number, a queue and the like of a task to be delivered are determined, the task is directly submitted to a task scheduling system, the task scheduling system allocates the task, and the task is executed at a computing node meeting conditions. When the technology is adopted, tasks are often delivered through software default threads and memory parameters, but under the condition that the cluster resource idle rate is high, resources required by each task are set by using the default parameters, so that the resource use is wasted, the task execution speed is influenced, the resources cannot be released quickly, and the resource utilization rate is low.
Disclosure of Invention
The present invention is directed to solving, at least to some extent, one of the technical problems in the related art. Therefore, a first objective of the present invention is to provide a method for pre-scheduling and allocating cluster tasks, which can effectively reduce the waste caused by resource usage, improve the resource utilization rate, and simultaneously can effectively accelerate the task execution speed, so that the resources are released faster, and the universality is high.
A second object of the invention is to propose a computer-readable storage medium.
A third object of the invention is to propose a computer device.
The fourth purpose of the present invention is to provide a cluster task pre-scheduling allocation apparatus.
In order to achieve the above object, an embodiment of a first aspect of the present invention provides a method for pre-scheduling and allocating cluster tasks, including: establishing a cluster queue database table and configuring dynamic parameter tasks, wherein the dynamic parameter tasks comprise queues; inquiring the resource use condition of the queue in a cluster queue database table according to the dynamic parameter task, and acquiring a usable resource table according to the resource use condition of the queue in the cluster queue database table; sequencing the available resource table, and determining task parameters of the dynamic parameter task according to a sequencing result; and submitting the tasks carrying the task parameters to a cluster scheduling system so that the cluster scheduling system allocates and executes the tasks carrying the task parameters.
According to the cluster task pre-scheduling allocation method provided by the embodiment of the invention, a cluster queue database table is established at first, and dynamic parameter tasks are configured, wherein the dynamic parameter tasks comprise queues, the resource use condition of the queues in the cluster queue database table is inquired according to the dynamic parameter tasks, a usable resource table is obtained according to the resource use condition of the queues in the cluster queue database table, the usable resource table is sequenced, the task parameters of the dynamic parameter tasks are determined according to the sequencing result, and the tasks carrying the task parameters are submitted to a cluster scheduling system, so that the cluster scheduling system allocates and executes the tasks carrying the task parameters. Therefore, waste caused by resource use can be effectively reduced, the resource utilization rate is improved, the task execution speed can be effectively accelerated, resources can be released more quickly, and the universality is high.
According to one embodiment of the present invention, the cluster queue database table includes node names, memory amounts, thread numbers, and queue information, wherein the node names are not repeatable as primary key contents.
According to an embodiment of the present invention, the dynamic parameter task further includes a task thread for setting a thread interval and a task memory for setting a memory interval, where each thread interval corresponds to one memory interval.
According to one embodiment of the invention, the resource usage of the dynamic parameter task query queue in the cluster queue database table comprises the following steps: and acquiring the memory use condition and thread number use condition of each node corresponding to the queue in the cluster queue database table according to the queue.
According to an embodiment of the present invention, acquiring a usable resource table according to the resource usage of a queue in a cluster queue database table includes: determining the memory size of each node of the currently remained deliverable tasks according to the memory use condition of each node corresponding to the queue in the cluster queue database table, determining the currently usable thread number of each node according to the thread number use condition of each node corresponding to the queue in the cluster queue database table, and merging the memory size of the currently remained deliverable tasks of each node and the currently usable thread number of each node to obtain a usable resource table.
According to one embodiment of the invention, sorting the usable resource table comprises: and sequencing the number of threads which can be currently used by each node in the usable resource table and the size of the memory of the currently remaining deliverable tasks of each node according to the reverse order of the columns to obtain a sequencing result.
According to one embodiment of the invention, the task parameters of the dynamic parameter task are determined according to the sequencing result, and the method comprises the following steps: determining a most idle node according to the sequencing result, and judging whether the currently usable thread number of the most idle node and the memory size of the currently remaining deliverable task simultaneously meet the upper limit of a thread interval and the upper limit of a memory interval in the dynamic parameter task; if yes, the thread number and the memory of the dynamic parameter task are both used for acquiring the upper limit of the thread interval and the upper limit of the memory interval; and if not, the thread number of the dynamic parameter task and the memory are used for acquiring the lower limit of the thread interval and the lower limit of the memory interval.
In order to achieve the above object, a second embodiment of the present invention provides a computer-readable storage medium, on which a cluster task pre-scheduling assignment program is stored, where the cluster task pre-scheduling assignment program, when executed by a processor, implements the above cluster task pre-scheduling assignment method.
According to the computer-readable storage medium of the embodiment of the invention, the cluster task pre-scheduling allocation method is realized when the cluster task pre-scheduling allocation program is executed by the processor, so that the waste caused by resource use can be effectively reduced, the resource utilization rate is improved, the task execution speed can be effectively increased, the resource can be released more quickly, and the universality is high.
In order to achieve the above object, an embodiment of a third aspect of the present invention provides a computer device, which includes a memory, a processor, and a cluster task pre-scheduling assignment program stored on the memory and operable on the processor, where when the processor executes the cluster task pre-scheduling assignment program, the cluster task pre-scheduling assignment method is implemented.
According to the computer equipment provided by the embodiment of the invention, when the processor executes the cluster task pre-dispatching distribution program, the cluster task pre-dispatching distribution method is realized, the waste caused by resource use can be effectively reduced, the resource utilization rate is improved, the task execution speed can be effectively accelerated, the resource is released more quickly, and the universality is high.
In order to achieve the above object, a fourth aspect of the present invention provides a device for pre-scheduling and allocating cluster tasks, including: the creating module is used for creating a cluster queue database table; the configuration module is used for configuring a dynamic parameter task, wherein the dynamic parameter task comprises a queue; the query module is used for querying the resource use condition of the queue in the cluster queue database table according to the dynamic parameter task; the determining module is used for acquiring a usable resource table according to the resource use condition of the queue in the cluster queue database table, sequencing the usable resource table and determining the task parameters of the dynamic parameter task according to the sequencing result; and the sending module is used for submitting the tasks carrying the task parameters to the cluster scheduling system so as to be convenient for the cluster scheduling system to distribute and execute the tasks carrying the task parameters.
According to the cluster task preposed scheduling and distributing device provided by the embodiment of the invention, a cluster queue database table is established through a creation module, and a dynamic parameter task is configured through a configuration module, wherein the dynamic parameter task comprises a queue, the resource use condition of the queue in the cluster queue database table is inquired through an inquiry module according to the dynamic parameter task, a usable resource table is obtained through a determination module according to the resource use condition of the queue in the cluster queue database table, the usable resource table is sequenced, the task parameter of the dynamic parameter task is determined according to the sequencing result, and the task carrying the task parameter is submitted to a cluster scheduling system through a sending module, so that the cluster scheduling system can distribute and execute the task carrying the task parameter. Therefore, waste caused by resource use can be effectively reduced, the resource utilization rate is improved, the task execution speed can be effectively accelerated, resources can be released more quickly, and the universality is high.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
FIG. 1 is a flowchart of a cluster task pre-scheduling assignment method according to an embodiment of the present invention;
FIG. 2 is a block diagram of a computer device according to one embodiment of the present invention;
fig. 3 is a block diagram of a cluster task prescheduling and allocating apparatus according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.
The following describes a cluster task pre-scheduling allocation method, apparatus, medium, and computer device provided in an embodiment of the present invention with reference to the accompanying drawings.
Fig. 1 is a flowchart of a cluster task pre-scheduling allocation method according to an embodiment of the present invention, and referring to fig. 1, the cluster task pre-scheduling allocation method may include the following steps:
s101: establishing a cluster queue database table, and configuring dynamic parameter tasks, wherein the dynamic parameter tasks comprise queues.
In one embodiment, the cluster queue database table includes node names, memory amounts, thread numbers, and queue information, wherein the node names are not repeatable as primary key contents.
Specifically, a cluster Queue database table may be established with reference to table 1, a table Name of the cluster Queue database table may be set to resource. The cluster in this example has 7 nodes: computer-m-1, computer-m-2, computer-m-3, cngb-computer-m 19-2, cngb-computer-m 19-4, cngb-computer-m 19-6, cngb-computer-f 22-3, and 7 nodes are divided into two queues of bc.q and bc _ rd.q.
TABLE 1
Name Memory Threads Queue
compute-m-1 504G 112 bc.q
compute-m-2 157.3G 48 bc.q
compute-m-3 31.5G 24 bc.q
cngb-compute-m19-2 504G 112 bc_rd.q
cngb-compute-m19-4 504G 112 bc_rd.q
cngb-compute-m19-6 504G 112 bc_rd.q
cngb-compute-f22-3 504G 112 bc_rd.q
In one embodiment, the dynamic parameter task further includes a task thread for setting a thread interval and a task memory for setting a memory interval, where each thread interval corresponds to one memory interval.
Specifically, when a dynamic parameter task is configured, three important parameters, namely a task thread, a task memory and a queue, can be set. In specific implementation, as shown in table 2, the task thread is set as a thread interval, and the thread interval corresponds to a memory interval, where the memory interval is a task memory, both of which have an upper limit value and a lower limit value, and the queue is a queue where a user has task operation permission in the cluster.
TABLE 2
Task Thread interval Memory interval Queue
job.sh [4,16] [2,8]G bc_rd.q
S102: and inquiring the resource use condition of the queue in the cluster queue database table according to the dynamic parameter task, and acquiring a usable resource table according to the resource use condition of the queue in the cluster queue database table.
In one embodiment, the query of resource usage of a queue in a cluster queue database table according to dynamic parameters includes: and acquiring the memory use condition and thread number use condition of each node corresponding to the queue in the cluster queue database table according to the queue.
In one embodiment, obtaining the usable resource table according to the resource usage of the queue in the cluster queue database table comprises: determining the memory size of each node of the currently remained deliverable tasks according to the memory use condition of each node corresponding to the queue in the cluster queue database table, determining the currently usable thread number of each node according to the thread number use condition of each node corresponding to the queue in the cluster queue database table, and merging the memory size of the currently remained deliverable tasks of each node and the currently usable thread number of each node to obtain a usable resource table.
Specifically, the resource usage of each node in the cluster queue database table established in step S101 by the queue may be queried according to the dynamic parameter task configured in step S101 to obtain a usable resource table. For example, the memory size of a deliverable task of each node in a Queue bc _ rd.q can be obtained by using a command "qstat-F vf-q bc _ rd.q", wherein qstat represents a query instruction, -F represents information for listing a specified task, vf represents an available memory resource, q represents a Queue to be obtained, and then the number of usable threads of each node in the Queue can be obtained by using a command "qhost-h Name", wherein qhost represents a node query instruction, -h represents the number of usable threads of the node, and Name represents a node Name. In a specific implementation, for the cluster shown in table 1, the queue bc _ rd.q corresponding to the usable resource table is obtained through the above two steps, as shown in table 3, where Free _ Mem is a usable memory size, and NCPU is a usable thread number.
TABLE 3
Serial number Name Memory Threads Queue Free_Mem NCPU
1 cngb-compute-m19-2 504G 112 bc_rd.q 114.000G 28
2 cngb-compute-m19-4 504G 112 bc_rd.q 416.747G 44
3 cngb-compute-m19-6 504G 112 bc_rd.q 406.747G 53
4 cngb-compute-f22-3 504G 112 bc_rd.q 46.089G 26
S103: and sequencing the usable resource table, and determining task parameters of the dynamic parameter task according to a sequencing result.
In one embodiment, sorting the table of usable resources includes: and sequencing the number of threads which can be currently used by each node in the usable resource table and the size of the memory of the currently remaining deliverable tasks of each node according to the reverse order of the columns to obtain a sequencing result.
In one embodiment, determining the task parameters of the dynamic parameter task according to the sorting result includes: determining a most idle node according to the sequencing result, and judging whether the currently usable thread number of the most idle node and the memory size of the currently remaining deliverable task simultaneously meet the upper limit of a thread interval and the upper limit of a memory interval in the dynamic parameter task; if yes, the thread number and the memory of the dynamic parameter task are both used for acquiring the upper limit of the thread interval and the upper limit of the memory interval; and if not, the thread number of the dynamic parameter task and the memory are used for acquiring the lower limit of the thread interval and the lower limit of the memory interval.
Specifically, the number of threads NCPU available in the available resource table shown in table 3 and the memory size Free _ Mem of the currently remaining deliverable tasks may be arranged in reverse order according to the columns (specifically, the ordering priority of the NCPU may be set to be higher than that of Free _ Mem), so that the available resource table after being ordered by the queue bc _ rd.q may be obtained, as shown in table 4.
TABLE 4
Serial number Name Memory Threads Queue Free_Mem NCPU
1 cngb-compute-m19-6 504G 112 bc_rd.q 406.747G 53
2 cngb-compute-m19-4 504G 112 bc_rd.q 416.747G 44
3 cngb-compute-m19-2 504G 112 bc_rd.q 114.000G 28
4 cngb-compute-f22-3 504G 112 bc_rd.q 46.089G 26
Because the currently available thread number NCPU of each node is higher than the upper limit (16) of the thread interval in the table 2, and the memory size Free _ Mem of the currently remaining deliverable tasks of each node is higher than the upper limit (8G) of the memory interval in the table 2, the task thread of the dynamic parameter task is taken as the upper limit (16) of the thread interval, and the upper limit (8G) of the memory interval is accessed in the task of the dynamic parameter task. On the contrary, if the currently available thread number NCPU and the memory size Free _ Mem of the currently remaining deliverable task cannot simultaneously satisfy the condition that the upper limit (16) of the thread interval and the upper limit (8G) of the memory interval in the dynamic parameter task are higher than those in the dynamic parameter task, the task thread and the task memory of the dynamic parameter task are respectively taken as the lower limit (4) of the thread interval and the lower limit (2G) of the memory interval in table 2, and the queue is still bc _ rd.q.
S104: and submitting the tasks carrying the task parameters to a cluster scheduling system so that the cluster scheduling system allocates and executes the tasks carrying the task parameters.
Specifically, the dynamic parameter task (including task thread, task memory and queue) determined in step S103 may be submitted to a cluster scheduling system such as SGE (sun Grid engine) by a command "qsub-cwd-l vf ═ 8G, p ═ 16-q bc _ rd.q job.sh", where-cwd denotes that the task is executed in the current directory, and at the same time, the running log of the SGE is also output in the current directory; -l represents the resources required for the application; vf is 8G, which indicates that the size of the memory expected to be used by the setting task is 8G; p-16 indicates that the number of threads required for setting is 16; -q represents the specified task to be posted to queue bc _ rd.q; sh represents the task that needs to be delivered.
The technical scheme and the effect of the application are further shown by specific examples and comparative examples. The cluster resource used by the case is the bc _ rd.q queue of table 5, the 28.9Gb two-generation sequences (NGS _1.fq.gz and NGS _2.fq.gz) are aligned by SOAP software (version 2.21) using different threads and memory to the already banked human reference genome sequence hg19, and the completed banked command "SOAP-2.21/2 bw-builder hg 19.fa" has been executed before the following examples and comparative examples are executed.
TABLE 5
Name Memory Threads Queue
cngb-compute-m19-2 504G 112 bc_rd.q
cngb-compute-m19-4 504G 112 bc_rd.q
cngb-compute-m19-6 504G 112 bc_rd.q
cngb-compute-f22-3 504G 112 bc_rd.q
Comparative example:
in a state where resources are idle relatively high (the resource usage before executing a task is shown in table 3), when the task is not executed by using the technical scheme of the present application, the comparison command is "soap-2.21/soap-a NGS _1.fq.gz-b NGS _2.fq.gz-D hg19.fa.index-p 4-o pe.sop-2 se.sop", the command "qsub-cwd-l vf ═ 4g, p ═ 4-q bc _ rd.q job.sh" is used to deliver the task, and the comparison time is 10 hours and 4 minutes.
The first embodiment is as follows:
the technical scheme of the application is used for executing the task in the state of higher resource vacancy, and the implementation steps are as follows:
s101: and establishing a cluster queue database table and configuring a dynamic parameter task.
Firstly, establishing a cluster queue database table as shown in table 5, and naming the table as resource.txt; then, the configure dynamic parameters task: as shown in table 6, the upper and lower limits of the pre-estimated set task thread are 4 and 16, the corresponding upper and lower memory limits are 4G and 16G, and the task running queue is "bc _ rd.q". Meanwhile, the comparison command for configuring the job.sh is "soap-2.21/soap-a NGS _1.fq.gz-b NGS _2.fq.gz-D hg19.fa.index-p $ thread-o pe.sop-2 se.sop", wherein it should be noted that $ thread is a thread parameter, and the value is determined in the subsequent step S103 according to the existing resource idle condition.
TABLE 6
Task Thread interval Memory interval Queue
job.sh [4,16] [4,16]G bc_rd.q
S102: and inquiring the resource use condition of the queue in the cluster queue database table according to the dynamic parameter task, and acquiring a usable resource table according to the resource use condition of the queue in the cluster queue database table.
The memory size deliverable by each node in the queue is first obtained using the command "qstat-F vf-q bc _ rd. q", and then the number of threads usable by each node in table 5 is obtained using the command "qhost-h Name", where names are cngb-computer-m 19-2, cngb-computer-m 19-4, cngb-computer-m 19-6, and cngb-computer-F22-3, respectively. Thus, a corresponding usable resource table bc _ rd.q is obtained, as shown in table 3, where Free _ Mem is the deliverable memory size and NCPU is the number of deliverable threads.
S103: and sequencing the usable resource table, and determining task parameters of the dynamic parameter task according to a sequencing result.
And (3) sequencing the NCPU in the column in the table 3 in a reverse order to obtain a sequenced usable resource table, wherein as shown in the table 4, the upper limit of the resource requirement is met by the number of usable threads NCPU and the size of usable memory Free _ Mem in the table, the thread number parameter $ thread of the task job.sh is assigned as an upper limit 16, and the memory is set as an upper limit 16G.
S104: and submitting the tasks carrying the task parameters to a cluster scheduling system so that the cluster scheduling system allocates and executes the tasks carrying the task parameters.
The task is submitted to the cluster scheduling system using the command "qsub-cwd-l vf ═ 16g, p ═ 16-q bc _ rd.q job.sh", and the cluster scheduling system allocates and executes the task.
In the first example, the time required for the alignment task was 3 hours and 51 minutes.
Example two:
the technical scheme of the application is used for executing the task in the state of low resource vacancy, and the implementation steps are as follows:
s101: and establishing a cluster queue database table and configuring a dynamic parameter task.
Firstly, establishing a cluster queue database table as shown in table 5, and naming the table as resource.txt; then, a dynamic parameter task is configured, as shown in table 6, the upper and lower limits of the pre-estimated set task thread are respectively 4 and 16, the corresponding upper and lower memory limits are respectively set to 4G and 16G, and the task running queue is "bc _ rd.q". Meanwhile, the comparison command for configuring the job.sh is "soap-2.21/soap-a NGS _1.fq.gz-b NGS _2.fq.gz-D hg19.fa.index-p $ thread-o pe.sop-2 se.sop", wherein it should be noted that $ thread is a thread parameter, and the value is determined in the subsequent step S103 according to the existing resource idle condition.
S102: and inquiring the resource use condition of the queue in the cluster queue database table according to the dynamic parameter task, and acquiring a usable resource table according to the resource use condition of the queue in the cluster queue database table.
The memory size deliverable by each node in the queue is first obtained using the command "qstat-F vf-q bc _ rd. q", and then the number of threads usable by each node in table 5 is obtained using the command "qhost-h Name", where names are cngb-computer-m 19-2, cngb-computer-m 19-4, cngb-computer-m 19-6, and cngb-computer-F22-3, respectively. Thus, a corresponding usable resource table bc _ rd.q is obtained, as shown in table 7, where Free _ Mem is the deliverable memory size and NCPU is the number of deliverable threads.
TABLE 7
Figure BDA0002915475460000081
Figure BDA0002915475460000091
S103: and sequencing the usable resource table, and determining task parameters of the dynamic parameter task according to a sequencing result.
And (3) sequencing the NCPU in the column in the table 7 in a reverse order to obtain a sequenced usable resource table, wherein as shown in the table 8, because the NCPU with the number of usable threads and the Free _ Mem with the size of the usable memory in the table do not meet the upper limit of the resource requirement, the thread number parameter $ thread of the job.
TABLE 8
Serial number Name Memory Threads Queue Free_Mem NCPU
1 cngb-compute-m19-2 504G 112 bc_rd.q 14.000G 6
2 cngb-compute-m19-4 504G 112 bc_rd.q 6.249G 4
3 cngb-compute-m19-6 504G 112 bc_rd.q 26.667G 0
4 cngb-compute-f22-3 504G 112 bc_rd.q 6.068G 0
S104: and submitting the tasks carrying the task parameters to a cluster scheduling system so that the cluster scheduling system allocates and executes the tasks carrying the task parameters.
The commands "qsub-cwd-l vf ═ 4g, p ═ 4-q bc _ rd. q jobs. sh" are used to submit tasks to the cluster scheduling system, which allocates and executes the tasks.
In the second example, the time required for the alignment task was 10 hours and 11 minutes.
The comparison task time consumption of the comparative example, the first example and the second example is shown in table 9, wherein the comparative example does not use more resources for task operation when the resource vacancy is higher, the task operation time is 2.61 times of that of the first example, and the resource use is not reasonable enough. In the first embodiment, after the task runs for 3 hours and 51 minutes, the memory of 16 threads and 16G is released for other tasks, the running time of 10 hours and 4 minutes is needed for the comparative example, if a large batch of tasks are delivered within the 10 hours and 4 minutes, the running of the tasks needs to be waited for, and the resources can be released for other tasks, and even in the second embodiment, when the resources are in shortage, the normal execution of the tasks is not influenced by using the method. The comparison shows that through the technical scheme of the application, when the resource vacancy is high, the cluster resources can be obviously and effectively used dynamically, so that the resources are released more quickly, the follow-up tasks can be used, the tasks can run more quickly, and the normal execution of the tasks can be ensured when the resource vacancy is low. In addition, the second embodiment and the comparative example adopt the same data and the same task parameters, but the time consumption is 7 minutes more than that of the comparative example, because the task execution speed is influenced by factors such as the saturation of cluster nodes, the read-write speed of a hard disk and the like, and the second embodiment and the comparative example belong to a normal phenomenon.
TABLE 9
Figure BDA0002915475460000092
Figure BDA0002915475460000101
Therefore, the method and the device can dynamically set the resources required by task delivery according to the using condition of the cluster resources, improve the resource utilization rate, have obvious effect under the condition of high resource vacancy, and have the same effect with the related technology under the condition of low resource vacancy, thereby being generally applicable to the preposed scheduling allocation of the cluster tasks. In addition, it is to be understood that the present application is not limited to a specific bioinformatics analysis, which is applicable to various technical fields of application clustering.
In summary, according to the cluster task pre-scheduling allocation method of the embodiment of the present invention, a cluster queue database table is established first, and a dynamic parameter task is configured, where the dynamic parameter task includes a queue, then a resource usage of the queue in the cluster queue database table is queried according to the dynamic parameter task, a usable resource table is obtained according to the resource usage of the queue in the cluster queue database table, the usable resource table is sorted, a task parameter of the dynamic parameter task is determined according to a sorting result, and a task carrying the task parameter is submitted to a cluster scheduling system, so that the cluster scheduling system allocates and executes a task carrying the task parameter. Therefore, waste caused by resource use can be effectively reduced, the resource utilization rate is improved, the task execution speed can be effectively accelerated, resources can be released more quickly, and the universality is high.
One embodiment of the present invention provides a computer-readable storage medium, on which a cluster task pre-scheduling assignment program is stored, where the cluster task pre-scheduling assignment program, when executed by a processor, implements the above-mentioned cluster task pre-scheduling assignment method.
For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
According to the computer-readable storage medium of the embodiment of the invention, the cluster task pre-scheduling allocation method is realized when the cluster task pre-scheduling allocation program is executed by the processor, so that the waste caused by resource use can be effectively reduced, the resource utilization rate is improved, the task execution speed can be effectively increased, the resource can be released more quickly, and the universality is high.
FIG. 2 is a block diagram of a computer device according to one embodiment of the present invention. Referring to fig. 2, the computer device 200 includes a memory 201, a processor 202, and a cluster task pre-scheduling assignment program stored in the memory 201 and executable on the processor 202, and when the processor 202 executes the cluster task pre-scheduling assignment program, the above-mentioned cluster task pre-scheduling assignment method is implemented.
According to the computer equipment provided by the embodiment of the invention, when the processor executes the cluster task pre-dispatching distribution program, the cluster task pre-dispatching distribution method is realized, the waste caused by resource use can be effectively reduced, the resource utilization rate is improved, the task execution speed can be effectively accelerated, the resource is released more quickly, and the universality is high.
Fig. 3 is a block diagram of a cluster task prescheduling and allocating apparatus according to an embodiment of the present invention. Referring to fig. 3, the cluster task pre-scheduling assignment device 300 includes: a creation module 301, a configuration module 302, a query module 303, a determination module 304, and a sending module 305.
The creating module 301 is configured to create a cluster queue database table; the configuration module 302 is configured to configure a dynamic parameter task, where the dynamic parameter task includes a queue; the query module 303 is configured to query resource usage of the queue in the cluster queue database table according to the dynamic parameter task; the determining module 304 is configured to obtain a usable resource table according to a resource usage condition of the queue in the cluster queue database table, sort the usable resource table, and determine a task parameter of the dynamic parameter task according to a sorting result; the sending module 305 is configured to submit the task carrying the task parameter to the cluster scheduling system, so that the cluster scheduling system allocates and executes the task carrying the task parameter.
In one embodiment, the cluster queue database table includes node names, memory amounts, thread numbers, and queue information, wherein the node names are not repeatable as primary key contents.
In one embodiment, the dynamic parameter task further includes a task thread for setting a thread interval and a task memory for setting a memory interval, where each thread interval corresponds to one memory interval.
In an embodiment, when the query module 303 queries the resource usage of the queue in the cluster queue database table according to the dynamic parameter task, the query module is specifically configured to: and acquiring the memory use condition and thread number use condition of each node corresponding to the queue in the cluster queue database table according to the queue.
In an embodiment, when the determining module 304 obtains the available resource table according to the resource usage of the queue in the cluster queue database table, the determining module is specifically configured to: determining the memory size of each node of the currently remained deliverable tasks according to the memory use condition of each node corresponding to the queue in the cluster queue database table, determining the currently usable thread number of each node according to the thread number use condition of each node corresponding to the queue in the cluster queue database table, and merging the memory size of the currently remained deliverable tasks of each node and the currently usable thread number of each node to obtain a usable resource table.
In an embodiment, when the determining module 304 sorts the available resource table, it is specifically configured to: and sequencing the number of threads which can be currently used by each node in the usable resource table and the size of the memory of the currently remaining deliverable tasks of each node according to the reverse order of the columns to obtain a sequencing result.
In an embodiment, when the determining module 304 determines the task parameter of the dynamic parameter task according to the sorting result, it is specifically configured to: determining a most idle node according to the sequencing result, and judging whether the currently usable thread number of the most idle node and the memory size of the currently remaining deliverable task simultaneously meet the upper limit of a thread interval and the upper limit of a memory interval in the dynamic parameter task; if yes, the thread number and the memory of the dynamic parameter task are both used for acquiring the upper limit of the thread interval and the upper limit of the memory interval; and if not, the thread number of the dynamic parameter task and the memory are used for acquiring the lower limit of the thread interval and the lower limit of the memory interval.
It should be noted that, for the description of the apparatus for allocating pre-scheduling of cluster tasks in the present application, please refer to the description of the method for allocating pre-scheduling of cluster tasks in the present application, which is not described herein again in detail.
According to the cluster task preposed scheduling and distributing device provided by the embodiment of the invention, a cluster queue database table is established through a creation module, and a dynamic parameter task is configured through a configuration module, wherein the dynamic parameter task comprises a queue, the resource use condition of the queue in the cluster queue database table is inquired through an inquiry module according to the dynamic parameter task, a usable resource table is obtained through a determination module according to the resource use condition of the queue in the cluster queue database table, the usable resource table is sequenced, the task parameter of the dynamic parameter task is determined according to the sequencing result, and the task carrying the task parameter is submitted to a cluster scheduling system through a sending module, so that the cluster scheduling system can distribute and execute the task carrying the task parameter. Therefore, waste caused by resource use can be effectively reduced, the resource utilization rate is improved, the task execution speed can be effectively accelerated, resources can be released more quickly, and the universality is high.
It should be noted that the logic and/or steps represented in the flowcharts or otherwise described herein, such as an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
In the present invention, unless otherwise expressly stated or limited, the terms "mounted," "connected," "secured," and the like are to be construed broadly and can, for example, be fixedly connected, detachably connected, or integrally formed; can be mechanically or electrically connected; they may be directly connected or indirectly connected through intervening media, or they may be connected internally or in any other suitable relationship, unless expressly stated otherwise. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.

Claims (10)

1. A cluster task pre-scheduling allocation method is characterized by comprising the following steps:
establishing a cluster queue database table and configuring dynamic parameter tasks, wherein the dynamic parameter tasks comprise queues;
inquiring the resource use condition of the queue in the cluster queue database table according to the dynamic parameter task, and acquiring a usable resource table according to the resource use condition of the queue in the cluster queue database table;
sequencing the usable resource table, and determining task parameters of the dynamic parameter task according to a sequencing result;
and submitting the tasks carrying the task parameters to a cluster scheduling system so that the cluster scheduling system distributes and executes the tasks carrying the task parameters.
2. The method according to claim 1, wherein the cluster queue database table comprises node names, memory amounts, thread numbers and queue information, wherein the node names are not repeatable as primary key contents.
3. The method according to claim 2, wherein the dynamic parameter task further comprises a task thread for setting a thread interval and a task memory for setting a memory interval, wherein each thread interval corresponds to one memory interval.
4. The method as claimed in claim 3, wherein querying the resource usage of the queue in the cluster queue database table according to the dynamic parameter task comprises:
and acquiring the memory use condition and thread number use condition of each node corresponding to the queue in the cluster queue database table according to the queue.
5. The method as claimed in claim 4, wherein the obtaining the available resource table according to the resource usage of the queue in the cluster queue database table includes:
determining the memory size of each node of the currently remaining deliverable tasks according to the memory use condition of each node corresponding to the queue in the cluster queue database table, determining the currently usable thread number of each node according to the thread number use condition of each node corresponding to the queue in the cluster queue database table, and merging the memory size of the currently remaining deliverable tasks of each node and the currently usable thread number of each node to obtain the usable resource table.
6. The method of claim 5, wherein sorting the table of usable resources comprises:
and sequencing the number of threads which can be currently used by each node in the usable resource table and the size of the memory of the currently remaining deliverable tasks of each node according to a column in a reverse order to obtain the sequencing result.
7. The method according to any of claims 3 to 6, wherein determining the task parameter of the dynamic parameter task according to the sorting result comprises:
determining a most idle node according to the sorting result, and judging whether the currently usable thread number of the most idle node and the memory size of the currently remaining deliverable task simultaneously meet the upper limit of the thread interval and the upper limit of the memory interval in the dynamic parameter task;
if yes, the thread number and the memory of the dynamic parameter task are both the upper limit of the thread interval and the upper limit of the memory interval;
and if not, taking the thread number and the memory of the dynamic parameter task as the lower limit of the thread interval and the lower limit of the memory interval.
8. A computer-readable storage medium, on which a cluster task pre-scheduling assignment program is stored, which when executed by a processor implements the cluster task pre-scheduling assignment method according to any one of claims 1 to 7.
9. A computer device, comprising a memory, a processor and a cluster task pre-scheduling assignment program stored on the memory and executable on the processor, wherein the processor implements the cluster task pre-scheduling assignment method according to any one of claims 1 to 7 when executing the cluster task pre-scheduling assignment program.
10. A cluster task pre-scheduling assignment apparatus, comprising:
the creating module is used for creating a cluster queue database table;
the configuration module is used for configuring a dynamic parameter task, wherein the dynamic parameter task comprises a queue;
the query module is used for querying the resource use condition of the queue in the cluster queue database table according to the dynamic parameter task;
the determining module is used for acquiring a usable resource table according to the resource use condition of the queue in the cluster queue database table, sequencing the usable resource table and determining the task parameters of the dynamic parameter task according to the sequencing result;
and the sending module is used for submitting the tasks carrying the task parameters to a cluster scheduling system so that the cluster scheduling system can distribute and execute the tasks carrying the task parameters.
CN202110099364.XA 2021-01-25 2021-01-25 Cluster task front-end scheduling distribution method, device, medium and computer equipment Active CN112799807B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110099364.XA CN112799807B (en) 2021-01-25 2021-01-25 Cluster task front-end scheduling distribution method, device, medium and computer equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110099364.XA CN112799807B (en) 2021-01-25 2021-01-25 Cluster task front-end scheduling distribution method, device, medium and computer equipment

Publications (2)

Publication Number Publication Date
CN112799807A true CN112799807A (en) 2021-05-14
CN112799807B CN112799807B (en) 2024-05-07

Family

ID=75811619

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110099364.XA Active CN112799807B (en) 2021-01-25 2021-01-25 Cluster task front-end scheduling distribution method, device, medium and computer equipment

Country Status (1)

Country Link
CN (1) CN112799807B (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105868025A (en) * 2016-03-30 2016-08-17 华中科技大学 System for settling fierce competition of memory resources in big data processing system
CN106933664A (en) * 2017-03-09 2017-07-07 北京搜狐新媒体信息技术有限公司 A kind of resource regulating method and device of Hadoop clusters
CN110213338A (en) * 2019-05-09 2019-09-06 国家计算机网络与信息安全管理中心 A kind of clustering acceleration calculating method and system based on cryptographic calculation
CN111045795A (en) * 2018-10-11 2020-04-21 浙江宇视科技有限公司 Resource scheduling method and device
US20200285508A1 (en) * 2019-03-08 2020-09-10 Beijing Baidu Netcom Science And Technology Co., Ltd. Method and Apparatus for Assigning Computing Task

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105868025A (en) * 2016-03-30 2016-08-17 华中科技大学 System for settling fierce competition of memory resources in big data processing system
CN106933664A (en) * 2017-03-09 2017-07-07 北京搜狐新媒体信息技术有限公司 A kind of resource regulating method and device of Hadoop clusters
CN111045795A (en) * 2018-10-11 2020-04-21 浙江宇视科技有限公司 Resource scheduling method and device
US20200285508A1 (en) * 2019-03-08 2020-09-10 Beijing Baidu Netcom Science And Technology Co., Ltd. Method and Apparatus for Assigning Computing Task
CN110213338A (en) * 2019-05-09 2019-09-06 国家计算机网络与信息安全管理中心 A kind of clustering acceleration calculating method and system based on cryptographic calculation

Also Published As

Publication number Publication date
CN112799807B (en) 2024-05-07

Similar Documents

Publication Publication Date Title
CN109684065B (en) Resource scheduling method, device and system
Majumdar et al. Scheduling in multiprogrammed parallel systems
CN108549583B (en) Big data processing method and device, server and readable storage medium
CN104915407B (en) A kind of resource regulating method based under Hadoop multi-job environment
CN111045795A (en) Resource scheduling method and device
US8627325B2 (en) Scheduling memory usage of a workload
CN110231991B (en) Task allocation method and device, electronic equipment and readable storage medium
CN112699098B (en) Index data migration method, device and equipment
CN104123304A (en) Data-driven parallel sorting system and method
US20140372649A1 (en) Operating System-Managed Interrupt Steering in Multiprocessor Systems
CN112882828B (en) Method for managing and scheduling a processor in a processor-based SLURM operation scheduling system
CN102004664A (en) Scheduling method of embedded real-time operating system of space vehicle
CN105488134A (en) Big data processing method and big data processing device
CN109117189B (en) Data processing method and device and computer equipment
CN108268546A (en) A kind of method and device for optimizing database
CN113010315A (en) Resource allocation method, resource allocation device and computer-readable storage medium
CN110413393B (en) Cluster resource management method and device, computer cluster and readable storage medium
CN113010286A (en) Parallel task scheduling method and device, computer equipment and storage medium
CN108984286A (en) A kind of resource regulating method and system of cloud computing platform
CN117421129B (en) Service execution method and device based on heterogeneous storage cluster and electronic equipment
CN111143210A (en) Test task scheduling method and system
WO2017020798A1 (en) Core load knowledge for elastic load balancing of threads
CN112799807A (en) Cluster task pre-scheduling distribution method, device, medium and computer equipment
CN115629854A (en) Distributed task scheduling method, system, electronic device and storage medium
US20060156309A1 (en) Method for controlling resource utilization and computer system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant