CN112799807B

CN112799807B - Cluster task front-end scheduling distribution method, device, medium and computer equipment

Info

Publication number: CN112799807B
Application number: CN202110099364.XA
Authority: CN
Inventors: 邓天全
Original assignee: Wuhan Huada Gene Technology Service Co ltd
Current assignee: Wuhan Huada Gene Technology Service Co ltd
Priority date: 2021-01-25
Filing date: 2021-01-25
Publication date: 2024-05-07
Anticipated expiration: 2041-01-25
Also published as: CN112799807A

Abstract

The invention discloses a cluster task front-end scheduling distribution method, a device, a medium and computer equipment, wherein the method comprises the following steps: establishing a cluster queue database table and configuring dynamic parameter tasks, wherein the dynamic parameter tasks comprise queues; inquiring the resource use condition of the queue in the cluster queue database table according to the dynamic parameter task, and acquiring a usable resource table according to the resource use condition of the queue in the cluster queue database table; sequencing the usable resource table, and determining task parameters of the dynamic parameter task according to the sequencing result; and submitting the task carrying the task parameters to the cluster scheduling system so that the cluster scheduling system distributes and executes the task carrying the task parameters. Therefore, the waste caused by resource utilization can be effectively reduced, the resource utilization rate is improved, and meanwhile, the task execution speed can be effectively accelerated, so that the resources are released more quickly, and the universality is high.

Description

Cluster task front-end scheduling distribution method, device, medium and computer equipment

Technical Field

The present invention relates to the field of cluster application technologies, and in particular, to a method, an apparatus, a medium, and a computer device for front-end scheduling and allocation of cluster tasks.

Background

In performing bioinformatics analysis, delivery tasks often need to be performed on a high-performance server cluster. The server cluster is typically composed of login nodes and computing nodes, wherein the login nodes are used for executing simple Linux commands, tasks consuming large resources cannot be executed, and tasks consuming large resources are delivered to the computing nodes by the login nodes.

In the related art, after determining parameters such as memory, thread number and queue of a task to be delivered, the task is directly submitted to a task scheduling system, the task scheduling system allocates the task, and the task is executed at a computing node meeting the condition. When the technology is adopted, tasks are usually delivered through threads and memory parameters defaulted by software, but when the cluster resource idling rate is very high, resources required by each task are set by using default parameters, waste is caused to resource use, and the task execution speed is influenced, so that the resources cannot be quickly released, and the resource utilization rate is low.

Disclosure of Invention

The present invention aims to solve at least one of the technical problems in the related art to some extent. Therefore, a first object of the present invention is to provide a method for pre-scheduling and allocating cluster tasks, which can effectively reduce waste caused by resource usage, improve resource utilization, and effectively accelerate task execution speed, so that resources are released more quickly, and the universality is high.

A second object of the present invention is to propose a computer readable storage medium.

A third object of the invention is to propose a computer device.

The fourth purpose of the invention is to provide a cluster task pre-scheduling and distributing device.

In order to achieve the above objective, an embodiment of a first aspect of the present invention provides a method for pre-scheduling and allocating cluster tasks, including: establishing a cluster queue database table and configuring dynamic parameter tasks, wherein the dynamic parameter tasks comprise queues; inquiring the resource use condition of the queue in the cluster queue database table according to the dynamic parameter task, and acquiring a usable resource table according to the resource use condition of the queue in the cluster queue database table; sequencing the usable resource table, and determining task parameters of the dynamic parameter task according to the sequencing result; and submitting the task carrying the task parameters to the cluster scheduling system so that the cluster scheduling system distributes and executes the task carrying the task parameters.

According to the cluster task pre-scheduling distribution method, a cluster queue database table is firstly established, dynamic parameter tasks are configured, wherein the dynamic parameter tasks comprise queues, then the resource use condition of the queues in the cluster queue database table is inquired according to the dynamic parameter tasks, the usable resource table is obtained according to the resource use condition of the queues in the cluster queue database table, the usable resource table is ordered, task parameters of the dynamic parameter tasks are determined according to the ordering result, and the tasks carrying the task parameters are submitted to a cluster scheduling system so that the cluster scheduling system can distribute and execute the tasks carrying the task parameters. Therefore, the waste caused by resource utilization can be effectively reduced, the resource utilization rate is improved, and meanwhile, the task execution speed can be effectively accelerated, so that the resources are released more quickly, and the universality is high.

According to one embodiment of the invention, the cluster queue database table includes node names, memory amounts, thread numbers, and queue information, wherein the node names are not repeatable as primary key contents.

According to one embodiment of the present invention, the dynamic parameter task further includes a task thread for setting a thread interval and a task memory for setting a memory interval, wherein each thread interval corresponds to one memory interval.

According to one embodiment of the present invention, the resource usage of the task query queue in the cluster queue database table according to the dynamic parameters includes: and acquiring the memory use condition and the thread number use condition of each node corresponding to the queue in the cluster queue database table according to the queue.

According to one embodiment of the present invention, obtaining a usable resource table according to a resource usage of a queue in a cluster queue database table includes: determining the memory size of the current residual deliverable tasks of each node according to the memory use condition of each node corresponding to the queue in the cluster queue database table, determining the current usable thread number of each node according to the thread number use condition of each node corresponding to the queue in the cluster queue database table, and combining the memory size of the current residual deliverable tasks of each node and the current usable thread number of each node to obtain a usable resource table.

According to one embodiment of the invention, ordering the table of available resources includes: and sequencing the number of threads currently available to each node in the available resource table and the memory size of the current remaining deliverable tasks of each node in a reverse order according to the columns to obtain a sequencing result.

According to one embodiment of the invention, determining task parameters of the dynamic parameter task according to the sequencing result comprises: determining the most idle node according to the sequencing result, and judging whether the current available thread number of the most idle node and the memory size of the current residual deliverable task simultaneously meet the upper limit of a thread interval and the upper limit of a memory interval in the dynamic parameter task; if yes, the thread number and the memory of the dynamic parameter task are taken as the upper limit of the thread interval and the upper limit of the memory interval; if not, the thread number and the memory of the dynamic parameter task are taken as the lower limit of the thread interval and the lower limit of the memory interval.

To achieve the above object, a second embodiment of the present invention provides a computer readable storage medium having stored thereon a cluster task pre-scheduling assignment program, which when executed by a processor, implements the above-mentioned cluster task pre-scheduling assignment method.

According to the computer readable storage medium, the cluster task pre-scheduling distribution method is realized when the cluster task pre-scheduling distribution program is executed by the processor, so that the waste caused by resource use can be effectively reduced, the resource utilization rate is improved, the task execution speed can be effectively accelerated, the resources are released more quickly, and the universality is high.

In order to achieve the above objective, an embodiment of a third aspect of the present invention provides a computer device, including a memory, a processor, and a cluster task pre-scheduling distribution program stored in the memory and capable of running on the processor, where the processor implements the above method for cluster task pre-scheduling distribution when executing the cluster task pre-scheduling distribution program.

According to the computer equipment provided by the embodiment of the invention, when the processor executes the cluster task pre-scheduling distribution program, the cluster task pre-scheduling distribution method is realized, so that the waste caused by resource use can be effectively reduced, the resource utilization rate is improved, the task execution speed can be effectively accelerated, the resources are released more quickly, and the universality is high.

To achieve the above object, a fourth embodiment of the present invention provides a cluster task pre-scheduling and distributing device, including: the creating module is used for creating a cluster queue database table; the configuration module is used for configuring dynamic parameter tasks, wherein the dynamic parameter tasks comprise queues; the query module is used for querying the resource use condition of the queue in the cluster queue database table according to the dynamic parameter task; the determining module is used for acquiring a usable resource table according to the resource use condition of the queue in the cluster queue database table, sequencing the usable resource table and determining task parameters of the dynamic parameter task according to the sequencing result; and the sending module is used for submitting the task carrying the task parameter to the cluster scheduling system so that the cluster scheduling system distributes and executes the task carrying the task parameter.

According to the cluster task prepositive scheduling distribution device, a cluster queue database table is established through a creation module, dynamic parameter tasks are configured through a configuration module, wherein the dynamic parameter tasks comprise queues, then the resource use condition of the queues in the cluster queue database table is inquired through an inquiry module according to the dynamic parameter tasks, the usable resource table is acquired through a determination module according to the resource use condition of the queues in the cluster queue database table, the usable resource table is ordered, task parameters of the dynamic parameter tasks are determined according to the ordering result, and the task carrying the task parameters are submitted to a cluster scheduling system through a sending module, so that the cluster scheduling system distributes and executes the task carrying the task parameters. Therefore, the waste caused by resource utilization can be effectively reduced, the resource utilization rate is improved, and meanwhile, the task execution speed can be effectively accelerated, so that the resources are released more quickly, and the universality is high.

Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

Drawings

FIG. 1 is a flow chart of a cluster task pre-scheduling assignment method according to one embodiment of the invention;

FIG. 2 is a block diagram of a computer device according to one embodiment of the invention;

fig. 3 is a block diagram of a cluster task pre-scheduling assignment apparatus according to an embodiment of the present invention.

Detailed Description

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative and intended to explain the present invention and should not be construed as limiting the invention.

The method, the device, the medium and the computer equipment for distributing the front-end scheduling of the cluster task provided by the embodiment of the invention are described below with reference to the accompanying drawings.

Fig. 1 is a flowchart of a cluster task pre-scheduling assignment method according to an embodiment of the present invention, and referring to fig. 1, the cluster task pre-scheduling assignment method may include the steps of:

s101: and establishing a cluster queue database table and configuring dynamic parameter tasks, wherein the dynamic parameter tasks comprise queues.

In one embodiment, the cluster queue database table includes node names, memory amounts, thread numbers, and queue information, where the node names are not repeatable as primary key content.

Specifically, a cluster Queue database table may be established with reference to table 1, whose table Name may be set to resource. The cluster in this example has 7 nodes ：compute-m-1、compute-m-2、compute-m-3、cngb-compute-m19-2、cngb-compute-m19-4、cngb-compute-m19-6、cngb-compute-f22-3, and the 7 nodes are split into two queues bc.q and bc_rd.q.

TABLE 1

Name	Memory	Threads	Queue
				compute-m-1	504G	112	bc.q
compute-m-2	157.3G	48	bc.q
				compute-m-3	31.5G	24	bc.q
cngb-compute-m19-2	504G	112	bc_rd.q
				cngb-compute-m19-4	504G	112	bc_rd.q
cngb-compute-m19-6	504G	112	bc_rd.q
				cngb-compute-f22-3	504G	112	bc_rd.q

In one embodiment, the dynamic parameter task further includes a task thread for setting thread intervals and task memory for setting memory intervals, wherein each thread interval corresponds to one memory interval.

Specifically, when a dynamic parameter task is configured, three important parameters of a task thread, a task memory and a queue can be set. In a specific implementation, as shown in table 2, the task thread is set as a thread interval, and at the same time, the thread interval corresponds to a memory interval, where the memory interval is a task memory, and both the memory interval and the memory interval have an upper limit value and a lower limit value, and the queue is a queue where a user has task operation permission in the cluster.

TABLE 2

Tasks	Thread interval	Memory interval	Queues
				job.sh	[4,16]	[2,8]G	bc_rd.q

S102: and inquiring the resource use condition of the queue in the cluster queue database table according to the dynamic parameter task, and acquiring a usable resource table according to the resource use condition of the queue in the cluster queue database table.

In one embodiment, querying resource usage of a queue in a cluster queue database table according to a dynamic parameter task includes: and acquiring the memory use condition and the thread number use condition of each node corresponding to the queue in the cluster queue database table according to the queue.

In one embodiment, obtaining the usable resource table according to the resource usage of the queue in the cluster queue database table includes: determining the memory size of the current residual deliverable tasks of each node according to the memory use condition of each node corresponding to the queue in the cluster queue database table, determining the current usable thread number of each node according to the thread number use condition of each node corresponding to the queue in the cluster queue database table, and combining the memory size of the current residual deliverable tasks of each node and the current usable thread number of each node to obtain a usable resource table.

Specifically, the resource usage of each node in the cluster queue database table established in step S101 may be queried according to the dynamic parameter task configured in step S101 to obtain the usable resource table. For example, the command "qstat-fvf-qbc_rd.q" may be used to query the memory size of the deliverable tasks of each node in the get Queue bc_rd.q, where qstat represents a query instruction, -F represents information listing the specified tasks, -vf represents available memory resources, -q represents the Queue that needs to be obtained, and then the command "qhost-hname" may be used to query the number of threads available to each node in the Queue, where qhost represents a node query instruction, -h represents the number of threads available to a node, and Name represents the node Name. In a specific implementation, for the cluster shown in table 1, the queue bc_rd.q is obtained through the above two steps, and the table of available resources is shown in table 3, where free_mem is the available memory size, and NCPU is the available thread number.

TABLE 3 Table 3

Sequence number

Name

Memory

Threads

Queue

Free_Mem

NCPU

1

cngb-compute-m19-2

504G

112

bc_rd.q

114.000G

28

2

cngb-compute-m19-4

504G

112

bc_rd.q

416.747G

44

3

cngb-compute-m19-6

504G

112

bc_rd.q

406.747G

53

4

cngb-compute-f22-3

504G

112

bc_rd.q

46.089G

26

S103: and sequencing the usable resource table, and determining task parameters of the dynamic parameter task according to the sequencing result.

In one embodiment, ordering the table of available resources includes: and sequencing the number of threads currently available to each node in the available resource table and the memory size of the current remaining deliverable tasks of each node in a reverse order according to the columns to obtain a sequencing result.

In one embodiment, determining task parameters of the dynamic parameter task based on the ordering result includes: determining the most idle node according to the sequencing result, and judging whether the current available thread number of the most idle node and the memory size of the current residual deliverable task simultaneously meet the upper limit of a thread interval and the upper limit of a memory interval in the dynamic parameter task; if yes, the thread number and the memory of the dynamic parameter task are taken as the upper limit of the thread interval and the upper limit of the memory interval; if not, the thread number and the memory of the dynamic parameter task are taken as the lower limit of the thread interval and the lower limit of the memory interval.

Specifically, the number of threads NCPU available in the available resource table shown in table 3 and the memory size free_mem of the current remaining deliverable tasks may be arranged in reverse order according to the columns (specifically, the NCPU may be set to have a higher priority than free_mem), so as to obtain the available resource table after the queue bc_rd.q is ordered, as shown in table 4.

TABLE 4 Table 4

Sequence number

Name

Memory

Threads

Queue

Free_Mem

NCPU

1

cngb-compute-m19-6

504G

112

bc_rd.q

406.747G

53

2

cngb-compute-m19-4

504G

112

bc_rd.q

416.747G

44

3

cngb-compute-m19-2

504G

112

bc_rd.q

114.000G

28

4

cngb-compute-f22-3

504G

112

bc_rd.q

46.089G

26

Since the number of currently available threads NCPU of each node is higher than the upper limit (16) of the thread interval in table 2, and the memory size free_mem of the currently remaining deliverable task of each node is higher than the upper limit (8G) of the memory interval in table 2, the task thread of the dynamic parameter task is taken as the upper limit (16) of the thread interval, and the upper limit (8G) of the access memory interval in the task of the dynamic parameter task is taken. Otherwise, if the current available thread number NCPU and the memory size free_mem of the current remaining deliverable task cannot simultaneously satisfy the upper limit (16) of the thread interval and the upper limit (8G) of the memory interval in the dynamic parameter task, the task thread and the task memory of the dynamic parameter task are respectively taken as the lower limit (4) of the thread interval and the lower limit (2G) of the memory interval in table 2, and the queue is still bc_rd.q.

S104: and submitting the task carrying the task parameters to the cluster scheduling system so that the cluster scheduling system distributes and executes the task carrying the task parameters.

Specifically, the dynamic parameter task (including task thread, task memory and queue) determined in step S103 may be submitted to the cluster scheduling system such as SGE (Sun GRID ENGINE) through commands "qsub-cwd-lvf=8g, p=16-qbc_rd.qjob.sh", where-cwd indicates that the task is executed under the current directory, and the running log of the SGE is also output in the current directory; -l represents the resources required for the application; vf=8g represents that the memory size that the task is expected to use is set to 8G; p=16 represents the number of threads required for setting to 16; q represents the task specified to be posted to the queue bc_rd.q; job.sh represents the task that needs to be delivered.

The technical scheme and effect of the present application will be further embodied by specific examples and comparative examples. The cluster resource used for the case is the bc_rd.q queue of Table 5, the 28.9Gb second generation sequences (NGS_1. Fq.gz and NGS_2. Fq.gz) were aligned by SOAP software (version 2.21) to the already-built human reference genome sequence hg19 using different threads and memory, and the completed library building command "SOAP-2.21/2bwt-builder hg19.Fa" was executed before the following examples and comparative examples were performed.

TABLE 5

Name	Memory	Threads	Queue
				cngb-compute-m19-2	504G	112	bc_rd.q
cngb-compute-m19-4	504G	112	bc_rd.q
				cngb-compute-m19-6	504G	112	bc_rd.q
cngb-compute-f22-3	504G	112	bc_rd.q

Comparative example:

In a state of higher resource idling (the resource use condition before task execution is shown in table 3), when the technical scheme of the application is not adopted to execute the task, the comparison command is 'soap-2.21/soap-a NGS_1.Fq.gz-b NGS_2.Fq.gz-D hg19. Index-p 4-o pe.sop-2 se.sop', and the comparison time is 10 hours and 4 minutes when the command is used for delivering the task 'qsub-cwd-lvf=4g and p=4-qbc_rd.qjob.sh'.

Embodiment one:

the technical scheme of the application is used for executing the task in the state of higher resource empty, and the implementation steps are as follows:

s101: and establishing a cluster queue database table and configuring dynamic parameter tasks.

Firstly, establishing a cluster queue database table, as shown in table 5, and naming the table as resource. Then, the dynamic parameter task is configured: as shown in table 6, the upper and lower limits of the task thread are set to 4 and 16, respectively, and the upper and lower limits of the corresponding memory are set to 4G and 16G, respectively, and the task running queue is "bc_rd.q". Meanwhile, a job.sh comparison command is configured as "soap-2.21/soap-a ngs_1.Fq.gz-b ngs_2.Fq.gz-D hg19.Fa. Index-p$thread-o pe.sop-2se. Sop", wherein it is to be noted that $thread is a thread parameter, and the value is determined in a subsequent step S103 according to the existing resource idling condition.

TABLE 6

Tasks	Thread interval	Memory interval	Queues
				job.sh	[4,16]	[4,16]G	bc_rd.q

The amount of memory that each node in the queue can deliver is first obtained using the command "qstat-fvf-qbc_rd.q", and then the number of threads that each node in table 5 can use is obtained using the command "qhost-hname", where Name is cngb-computer-m 19-2, cngb-computer-m 19-4, cngb-computer-m 19-6, and cngb-computer-F22-3, respectively. Thus, a table of available resources for bc_rd.q is obtained, as shown in Table 3, where free_Mem is the deliverable memory size and NCPU is the number of deliverable threads.

The columns NCPU in Table 3 are ordered in reverse order to obtain an ordered usable resource table, and as shown in Table 4, the upper limit of resource demand is met by the usable thread number NCPU and the usable memory size free_Mem in the table, the thread number parameter $thread of the task job.sh is assigned as the upper limit 16, and the memory is set as the upper limit 16G.

The command "qsub-cwd-lvf=16 g, p=16-qbc_rd.qjob.sh" is used to submit the task to the cluster scheduling system, which allocates and executes the task.

In the first embodiment, the alignment task takes 3 hours and 51 minutes.

Embodiment two:

The technical scheme of the application is used for executing the task in the state of low resource empty, and the implementation steps are as follows:

Firstly, establishing a cluster queue database table, as shown in table 5, and naming the table as resource. Then, the dynamic parameter task is configured, as shown in table 6, the upper limit and the lower limit of the task thread are set to be 4G and 16G respectively, the upper limit and the lower limit of the corresponding memory are set to be 4G and 16G respectively, and the task running queue is "bc_rd.q". Meanwhile, a job.sh comparison command is configured as "soap-2.21/soap-a ngs_1.Fq.gz-b ngs_2.Fq.gz-D hg19.Fa. Index-p$thread-o pe.sop-2se. Sop", wherein it is to be noted that $thread is a thread parameter, and the value is determined in a subsequent step S103 according to the existing resource idling condition.

The amount of memory that each node in the queue can deliver is first obtained using the command "qstat-fvf-qbc_rd.q", and then the number of threads that each node in table 5 can use is obtained using the command "qhost-hname", where Name is cngb-computer-m 19-2, cngb-computer-m 19-4, cngb-computer-m 19-6, and cngb-computer-F22-3, respectively. Thus, a table of available resources for bc_rd.q is obtained, as shown in Table 7, where free_Mem is the deliverable memory size and NCPU is the number of deliverable threads.

TABLE 7

The columns NCPU in Table 7 are ordered in reverse order to obtain an ordered usable resource table, and as shown in Table 8, the usable thread number NCPU and the usable memory size free_Mem in the table do not meet the upper limit of resource requirements, the thread number parameter $thread of the task job.sh is assigned as the lower limit 4, and the memory is set as the lower limit 4G.

TABLE 8

Sequence number

Name

Memory

Threads

Queue

Free_Mem

NCPU

1

cngb-compute-m19-2

504G

112

bc_rd.q

14.000G

6

2

cngb-compute-m19-4

504G

112

bc_rd.q

6.249G

4

3

cngb-compute-m19-6

504G

112

bc_rd.q

26.667G

0

4

cngb-compute-f22-3

504G

112

bc_rd.q

6.068G

0

The command "qsub-cwd-lvf=4g, p=4-qbc_rd.qjob.sh" is used to submit the task to the cluster scheduling system, which allocates and executes the task.

In the second embodiment, the alignment task takes 10 hours and 11 minutes.

The time consumption of the comparison tasks of the comparison example, the first embodiment and the second embodiment is shown in table 9, wherein when the resource is empty and high, the comparison example does not use more resources to perform task operation, the task operation time is 2.61 times that of the first embodiment, and the resource use is unreasonable. In the embodiment, after the 3-hour 51 minute task is run, the memory of 16 threads and 16G is released for other tasks, the comparative example needs 10 hours and 4 minutes of running time, if a large number of tasks are delivered within the 10 hours and 4 minutes, the tasks need to be waited for to run, and the resources can be released for other tasks, even in the embodiment II, when the resources are tense, the normal execution of the tasks is not influenced by the application. Compared with the prior art, the method and the device have the advantages that when the resource blank is high, the cluster resource can be obviously, effectively and reasonably used dynamically, so that the resource is released faster for the subsequent task, the task can be operated faster, and when the resource blank is low, the normal execution of the task can be ensured. In addition, the second embodiment and the comparative example adopt the same data and the same task parameters, but the time is 7 minutes more than that of the comparative example, because factors such as the saturation of the cluster node and the read-write speed of the hard disk affect the task execution speed, and the second embodiment belongs to a normal phenomenon.

TABLE 9

Therefore, the application can dynamically set the resources required by task delivery according to the use condition of the cluster resources, improves the utilization rate of the resources, has obvious effect under the condition of higher resource vacancy and has equivalent effect to the related technology under the condition of lower resource vacancy, thereby being widely applicable to the front-end dispatch distribution of the cluster tasks. In addition, it is to be understood that the application is not limited to a specific bioinformatic analysis, which is applicable to various technical fields of application clusters.

In summary, according to the method for pre-scheduling and allocating cluster tasks in the embodiment of the present invention, a cluster queue database table is first established, and dynamic parameter tasks are configured, where the dynamic parameter tasks include queues, then the resource usage situation of the queues in the cluster queue database table is queried according to the dynamic parameter tasks, and the available resource table is obtained according to the resource usage situation of the queues in the cluster queue database table, and the available resource table is ordered, and task parameters of the dynamic parameter tasks are determined according to the ordering result, and the tasks carrying the task parameters are submitted to a cluster scheduling system, so that the cluster scheduling system allocates and executes the tasks carrying the task parameters. Therefore, the waste caused by resource utilization can be effectively reduced, the resource utilization rate is improved, and meanwhile, the task execution speed can be effectively accelerated, so that the resources are released more quickly, and the universality is high.

An embodiment of the present invention provides a computer readable storage medium, on which a cluster task pre-scheduling assignment program is stored, where the cluster task pre-scheduling assignment program implements the above-mentioned cluster task pre-scheduling assignment method when executed by a processor.

For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

Fig. 2 is a block diagram of a computer device according to one embodiment of the invention. Referring to fig. 2, the computer device 200 includes a memory 201, a processor 202, and a cluster task pre-scheduling assignment program stored in the memory 201 and capable of running on the processor 202, where the processor 202 implements the cluster task pre-scheduling assignment method when executing the cluster task pre-scheduling assignment program.

Fig. 3 is a block diagram of a cluster task pre-scheduling assignment apparatus according to an embodiment of the present invention. Referring to fig. 3, the cluster task pre-scheduling assignment apparatus 300 includes: a creation module 301, a configuration module 302, a query module 303, a determination module 304 and a transmission module 305.

The creating module 301 is configured to create a cluster queue database table; the configuration module 302 is configured to configure a dynamic parameter task, where the dynamic parameter task includes a queue; the query module 303 is configured to query a resource usage condition of the queue in the cluster queue database table according to the dynamic parameter task; the determining module 304 is configured to obtain a usable resource table according to a resource usage condition of the queue in the cluster queue database table, order the usable resource table, and determine task parameters of the dynamic parameter task according to an ordering result; the sending module 305 is configured to submit the task carrying the task parameter to the cluster scheduling system, so that the cluster scheduling system allocates and executes the task carrying the task parameter.

In one embodiment, the query module 303 is specifically configured to, when the query module queries the resource usage of the queue in the cluster queue database table according to the dynamic parameter task: and acquiring the memory use condition and the thread number use condition of each node corresponding to the queue in the cluster queue database table according to the queue.

In one embodiment, the determining module 304 is specifically configured to, when obtaining the available resource table according to the resource usage of the queue in the cluster queue database table: determining the memory size of the current residual deliverable tasks of each node according to the memory use condition of each node corresponding to the queue in the cluster queue database table, determining the current usable thread number of each node according to the thread number use condition of each node corresponding to the queue in the cluster queue database table, and combining the memory size of the current residual deliverable tasks of each node and the current usable thread number of each node to obtain a usable resource table.

In one embodiment, the determining module 304 is specifically configured to, when ordering the available resource table: and sequencing the number of threads currently available to each node in the available resource table and the memory size of the current remaining deliverable tasks of each node in a reverse order according to the columns to obtain a sequencing result.

In one embodiment, when the determining module 304 determines the task parameters of the dynamic parameter task according to the sorting result, the determining module is specifically configured to: determining the most idle node according to the sequencing result, and judging whether the current available thread number of the most idle node and the memory size of the current residual deliverable task simultaneously meet the upper limit of a thread interval and the upper limit of a memory interval in the dynamic parameter task; if yes, the thread number and the memory of the dynamic parameter task are taken as the upper limit of the thread interval and the upper limit of the memory interval; if not, the thread number and the memory of the dynamic parameter task are taken as the lower limit of the thread interval and the lower limit of the memory interval.

It should be noted that, for the description of the apparatus for pre-scheduling and distributing cluster tasks in the present application, please refer to the description of the method for pre-scheduling and distributing cluster tasks in the present application, and detailed descriptions thereof are omitted herein.

It should be noted that the logic and/or steps represented in the flowcharts or otherwise described herein, for example, may be considered as a ordered listing of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.

It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present invention, the meaning of "plurality" means at least two, for example, two, three, etc., unless specifically defined otherwise.

In the present invention, unless explicitly specified and limited otherwise, the terms "mounted," "connected," "secured," and the like are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally formed; can be mechanically or electrically connected; either directly or indirectly, through intermediaries, or both, may be in communication with each other or in interaction with each other, unless expressly defined otherwise. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances.

While embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the invention, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the invention.

Claims

1. The cluster task pre-scheduling and distributing method is characterized by comprising the following steps of:

Establishing a cluster queue database table and configuring dynamic parameter tasks, wherein the dynamic parameter tasks comprise queues;

Inquiring the resource use condition of the queue in the cluster queue database table according to the dynamic parameter task, and acquiring a usable resource table according to the resource use condition of the queue in the cluster queue database table;

sorting the usable resource table, and determining task parameters of the dynamic parameter task according to a sorting result;

submitting the task carrying the task parameter to a cluster scheduling system so that the cluster scheduling system distributes and executes the task carrying the task parameter;

The cluster queue database table comprises node names, memory quantity, thread numbers and queue information, wherein the node names are unrepeatable as main key contents;

The dynamic parameter task further comprises a task thread for setting thread intervals and a task memory for setting memory intervals, wherein each thread interval corresponds to one memory interval.

2. The method for pre-scheduling and distributing cluster tasks according to claim 1, wherein querying the resource usage of the queue in the cluster queue database table according to the dynamic parameter task comprises:

And acquiring the memory use condition and the thread number use condition of each node corresponding to the queue in the cluster queue database table according to the queue.

3. The method for pre-scheduling and distributing cluster tasks according to claim 2, wherein obtaining a usable resource table according to the resource usage of the queue in the cluster queue database table comprises:

Determining the memory size of the current residual deliverable tasks of each node according to the memory use condition of each node corresponding to the queue in the cluster queue database table, determining the current usable thread number of each node according to the thread number use condition of each node corresponding to the queue in the cluster queue database table, and merging the memory size of the current residual deliverable tasks of each node and the current usable thread number of each node to obtain the usable resource table.

4. The method of assigning a cluster task pre-schedule of claim 3, wherein ordering the list of available resources comprises:

And sequencing the number of threads currently available to each node in the available resource table and the memory size of the current remaining deliverable tasks of each node in a reverse order according to the columns to obtain the sequencing result.

5. The method for distributing the front-end scheduling of the cluster tasks according to any one of claims 1 to 4, wherein determining the task parameters of the dynamic parameter task according to the sequencing result comprises:

Determining the most idle node according to the sequencing result, and judging whether the number of threads which can be used currently and the memory size of the current rest deliverable tasks of the most idle node simultaneously meet the upper limit of the thread interval and the upper limit of the memory interval in the dynamic parameter task;

if yes, the thread number and the memory of the dynamic parameter task are taken as the upper limit of the thread interval and the upper limit of the memory interval;

if not, the thread number and the memory of the dynamic parameter task are taken as the lower limit of the thread interval and the lower limit of the memory interval.

6. A computer readable storage medium, having stored thereon a clustered task front-end scheduling assignment program, which when executed by a processor implements a clustered task front-end scheduling assignment method as claimed in any one of claims 1-5.

7. A computer device comprising a memory, a processor and a cluster task pre-scheduling assignment program stored on the memory and executable on the processor, the processor implementing the cluster task pre-scheduling assignment method according to any of claims 1-5 when executing the cluster task pre-scheduling assignment program.

8. A cluster task pre-scheduling and distributing device, comprising:

the creating module is used for creating a cluster queue database table;

The configuration module is used for configuring dynamic parameter tasks, wherein the dynamic parameter tasks comprise queues;

The query module is used for querying the resource use condition of the queue in the cluster queue database table according to the dynamic parameter task;

The determining module is used for acquiring a usable resource table according to the resource use condition of the queue in the cluster queue database table, sequencing the usable resource table and determining task parameters of the dynamic parameter task according to the sequencing result;

The sending module is used for submitting the task carrying the task parameter to the cluster scheduling system so that the cluster scheduling system distributes and executes the task carrying the task parameter;