CN111258757A - Automatic task arranging method and device, computer equipment and storage medium - Google Patents
Automatic task arranging method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN111258757A CN111258757A CN202010026629.9A CN202010026629A CN111258757A CN 111258757 A CN111258757 A CN 111258757A CN 202010026629 A CN202010026629 A CN 202010026629A CN 111258757 A CN111258757 A CN 111258757A
- Authority
- CN
- China
- Prior art keywords
- big data
- resource
- data cluster
- target task
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a method, a device, computer equipment and a storage medium for automatically arranging tasks, wherein the method comprises the following steps: if receiving a submission instruction of a plurality of tasks, acquiring the current idle resource condition of the big data cluster; acquiring one of the tasks as a target task, and determining the resource demand for executing the target task; judging whether the current idle resource condition of the big data cluster meets the resource demand required by executing the target task or not; and if the current idle resource condition of the big data cluster can not meet the resource demand required by executing the target task, selecting another task from the tasks as the target task and returning to the step of determining the resource demand for executing the target task. The invention can reasonably arrange the tasks, ensure the smooth execution of the tasks under the condition of limited resources and improve the efficiency of task execution.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for automatically arranging tasks, a computer device, and a storage medium.
Background
In the existing big data system, the core function of the task scheduling system is to submit tasks to the big data cluster for operation, and the tasks submitted by users temporarily or regularly are submitted to the big data cluster for operation, but the big data cluster has certain resource limitation, if the submission of the tasks is not reasonably arranged, a large number of tasks compete for a small amount of operation resources, and finally all tasks run very slowly, so that the expected requirements are difficult to meet.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for automatically arranging tasks, a computer device, and a storage medium, which can reasonably arrange tasks, ensure that the tasks are smoothly executed under the condition of limited resources, and improve the efficiency of executing the tasks.
In one aspect, an embodiment of the present invention provides an automatic task scheduling method, where the method includes:
if receiving a submission instruction of a plurality of tasks, acquiring the current idle resource condition of the big data cluster, wherein the current idle resource condition of the big data cluster comprises the sum of current idle resources of all computer equipment forming the big data cluster;
acquiring one of the tasks as a target task, and determining the resource demand for executing the target task;
judging whether the current idle resource condition of the big data cluster meets the resource demand required by executing the target task or not;
and if the current idle resource condition of the big data cluster can not meet the resource demand required by executing the target task, selecting another task from the tasks as the target task and returning to the step of determining the resource demand for executing the target task.
In another aspect, an embodiment of the present invention provides an automatic task scheduling apparatus, where the apparatus includes:
the system comprises a first obtaining unit and a second obtaining unit, wherein the first obtaining unit is used for obtaining the current idle resource condition of a big data cluster if receiving a submitting instruction of a plurality of tasks, and the current idle resource condition of the big data cluster comprises the sum of the current idle resources of all computer equipment forming the big data cluster;
the determining unit is used for acquiring one of the tasks as a target task and determining the resource demand for executing the target task;
the judging unit is used for judging whether the current idle resource condition of the big data cluster meets the resource demand required by executing the target task;
and the selecting unit is used for selecting another task from the plurality of tasks as the target task and returning to the step of determining the resource demand for executing the target task if the current idle resource condition of the big data cluster cannot meet the resource demand required for executing the target task.
In still another aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the task automatic arranging method as described above when executing the computer program.
In still another aspect, the present invention further provides a computer-readable storage medium, where one or more computer programs are stored, and the one or more computer programs are executable by one or more processors to implement the task automatic orchestration method described above.
The embodiment of the invention provides a method and a device for automatically arranging tasks, computer equipment and a storage medium, wherein the method comprises the following steps: if receiving a submission instruction of a plurality of tasks, acquiring the current idle resource condition of the big data cluster, wherein the current idle resource condition of the big data cluster comprises the sum of current idle resources of all computer equipment forming the big data cluster; acquiring one of the tasks as a target task, and determining the resource demand for executing the target task; judging whether the current idle resource condition of the big data cluster meets the resource demand required by executing the target task or not; and if the current idle resource condition of the big data cluster can not meet the resource demand required by executing the target task, selecting another task from the tasks as the target task and returning to the step of determining the resource demand for executing the target task. The invention can reasonably arrange the tasks, ensure the smooth execution of the tasks under the condition of limited resources and improve the efficiency of task execution.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
Fig. 1 is a schematic view of an application scenario of an automatic task orchestration method according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a task automatic scheduling method according to an embodiment of the present invention;
FIG. 3 is another schematic flow chart of a task automatic scheduling method according to an embodiment of the present invention;
FIG. 4 is another schematic flow chart of a task automatic scheduling method according to an embodiment of the present invention;
FIG. 5 is another schematic flow chart of a task automatic scheduling method according to an embodiment of the present invention;
FIG. 6 is a schematic block diagram of an automatic task orchestration device according to an embodiment of the present invention;
FIG. 7 is another schematic block diagram of an automatic task orchestration device according to embodiments of the present invention;
FIG. 8 is another schematic block diagram of an automatic task orchestration device according to embodiments of the present invention;
FIG. 9 is another schematic block diagram of an automatic task orchestration device according to embodiments of the present invention;
fig. 10 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Referring to fig. 1 and fig. 2, fig. 1 is a schematic view of an application scenario of an automatic task scheduling method according to an embodiment of the present invention, and fig. 2 is a schematic view of a flowchart of the automatic task scheduling method according to the embodiment of the present invention. The task automatic arrangement method is applied to a big data cluster 10, as shown in fig. 1, wherein the big data cluster 10 is composed of a plurality of computer devices. As an application, as shown in fig. 1, the task automatic arrangement method is applied to each computer device in a big data cluster 10, where the big data cluster 10 is responsible for acquiring a task from a client 20 and determining whether a current idle resource condition of the big data cluster meets a resource demand required for executing the task; if the current idle resource condition of the big data cluster cannot meet the resource demand required for executing the task, one of the tasks to be executed in the big data cluster is selected to execute, wherein the current idle resource condition of the big data cluster refers to the sum of the current idle resources of all the computer devices forming the big data cluster, and the client 20 can be a desktop computer, a notebook computer, a smart phone or the like.
Referring to fig. 2, fig. 2 is a schematic flow chart of a task automatic scheduling method according to an embodiment of the present invention. As shown in fig. 2, the method includes the following steps S101 to S104.
S101, if receiving a submitting instruction of a plurality of tasks, acquiring the current idle resource condition of the big data cluster, wherein the current idle resource condition of the big data cluster comprises the sum of the current idle resources of all computer equipment forming the big data cluster.
In the embodiment of the present invention, the task submission instruction may be generated by a user clicking a task submission icon on a client connected to the big data cluster. And the task submitting instruction is used for sending a plurality of tasks uploaded by the user through the client to the big data cluster and queuing in the big data cluster for execution. The current idle resource condition of the big data cluster refers to the sum of current idle resources of all computer devices forming the big data cluster, the current idle resource condition of the big data cluster includes the memory idle resource condition and the CPU idle resource condition of the big data cluster, and the acquiring the current idle resource condition of the big data cluster includes: and acquiring the memory idle resource condition and the CPU idle resource condition of the big data cluster. Specifically, in this embodiment, the current idle resource condition includes a memory idle resource condition and a CPU idle resource condition of the big data cluster, for example, the current occupied memory resource condition of the big data cluster is 100g, the current occupied CPU resource condition is 50 cores, and the memory resource and the CPU resource of the big data cluster are 150g and 80 cores respectively when they are not used, so that the current idle resource condition of the big data cluster is 50g and the CPU idle resource condition is 30 cores respectively.
S102, one of the tasks is obtained to serve as a target task, and the resource demand for executing the target task is determined.
In the embodiment of the present invention, as shown in fig. 3, the step of determining a resource demand for executing one of the submitted tasks specifically includes the following steps S201 to S202: s201, acquiring the data volume of one of a plurality of tasks needing to be executed; s202, determining the resource demand for executing the target task according to the data volume of the task. Specifically, the resource demand of the target task refers to a resource condition in a big data cluster that the target task needs to occupy when executing, and the demand of the target task may be a memory resource amount and a CPU resource amount that occupy the big data cluster, and more specifically, the resource demand of the task may be determined by a data amount of the executed task, for example, the data amount of the task a is 100g, if the task with the data amount needs to be processed, according to a preset setting of the big data cluster, fixed resources (for example, 10g of memory resources and 20-core CPU resources) are allocated to every 128M data amount in the task a for processing, and then approximately the total resource demand is 80g of the memory resource that needs to occupy the big data cluster and 160 cores of the CPU resource; it should be noted that, in a large data cluster, a task whose data amount exceeds a preset threshold cannot be processed at one time, and the processing flow of the task is to execute the task in batches, so that a large number of tasks processed in batches occupy resources of the large data cluster for a long time, and other tasks are submitted and cannot be allocated sufficient resources to run.
S103, judging whether the current idle resource condition of the big data cluster meets the resource demand required by executing the target task.
In this embodiment of the present invention, the current idle resource condition of the big data cluster includes a memory idle resource condition and a CPU idle resource condition of the big data cluster, and the resource demand required for executing the task includes a memory resource amount and a CPU resource amount required for executing the task, as shown in fig. 4, the step of determining whether the current idle resource condition of the big data cluster satisfies the resource demand required for executing the target task specifically includes the following steps S301 to S302: s301, judging whether the memory idle resource condition of the big data cluster is larger than the memory resource amount required for executing the target task and whether the CPU idle resource condition of the big data cluster is larger than the CPU resource amount required for executing the target task; s302, if the memory free resource condition of the big data cluster is larger than the memory resource amount required for executing the target task and the CPU free resource condition of the big data cluster is larger than the CPU resource amount required for executing the target task, judging that the current free resource condition of the big data cluster meets the resource demand amount required for executing the target task, otherwise, judging that the current free resource condition of the big data cluster cannot meet the resource demand amount required for executing the target task. Specifically, in this embodiment, the current idle resource condition of the big data cluster specifically refers to a memory idle resource condition and a CPU idle resource condition of the big data cluster, the current memory free resource condition of the big data cluster is equal to the total memory resource of the big data cluster minus the currently occupied memory resource of the big data cluster, the current CPU free resource condition of the big data cluster is equal to the total CPU resource of the big data cluster minus the currently occupied CPU resource of the big data cluster, for example, the total memory resource of a big data cluster is 1000g, the total CPU resource is 2000 cores, the currently occupied memory resource reaches 500g, and the currently occupied CPU resource reaches 300 cores, so that the current memory idle resource condition of the big data cluster is 500 g-1000 g, and the current CPU idle resource condition of the big data cluster is 1700 cores-2000 cores-1700 cores. The resource demand required for executing the task specifically refers to the memory resource amount and the CPU resource amount required for executing the task, for example, the data amount of the task a is 100g, if the task of the data amount is to be processed, fixed resources (for example, 10g of memory resources and 20-core CPU resources) are allocated to every 128M of the data amount in the task a for processing according to the preset of the big data cluster, and then the approximate total resource demand is that 80g of the memory resources of the big data cluster needs to be occupied and 160 cores of the CPU resources need to be occupied; in this embodiment, whether the memory idle resource condition of the big data cluster is greater than the memory resource amount required for executing the target task and whether the CPU idle resource condition of the big data cluster is greater than the CPU resource amount required for executing the target task are determined; and when the condition of the idle memory resources of the big data cluster is greater than the amount of the memory resources required for executing the target task and the condition of the idle CPU resources of the big data cluster is greater than the amount of the CPU resources required for executing the target task, indicating that the condition of the current idle resources of the big data cluster meets the amount of the resource demand required for executing the target task, otherwise, indicating that the condition of the current idle resources of the big data cluster cannot meet the amount of the resource demand required for executing the target task. It should be noted that, when the memory idle resource condition of the big data cluster is greater than the amount of memory resources required for executing the target task and the CPU idle resource condition of the big data cluster is less than the amount of CPU resources required for executing the target task, the current idle resource condition of the big data cluster cannot satisfy the amount of resource demand required for executing the target task, or, when the memory idle resource condition of the big data cluster is less than the amount of memory resources required for executing the target task and the CPU idle resource condition of the big data cluster is greater than the amount of CPU resources required for executing the target task, the current idle resource condition of the big data cluster cannot satisfy the amount of resource demand required for executing the target task, or, when the memory idle resource condition of the big data cluster is less than the amount of memory resources required for executing the target task and the CPU idle resource condition of the big data cluster is less than the amount of CPU resources required for executing the target task, the current idle resource condition of the big data cluster cannot meet the resource demand needed to execute the target task.
And S104, if the current idle resource condition of the big data cluster can not meet the resource demand required by executing the target task, selecting another task from the tasks as the target task and returning to the step of determining the resource demand for executing the target task.
In this embodiment of the present invention, as shown in fig. 5, if the current idle resource condition of the big data cluster cannot satisfy the resource demand required for executing the target task, selecting another task from the multiple tasks as the target task and returning to execute the step of determining the resource demand for executing the target task includes the following steps S401 to S402: s401, if the current idle resource condition of the big data cluster can not meet the resource demand required by executing the target task, calculating the resource demand required by other tasks to be executed during execution; s402, comparing the calculated resource demand required by each other task to be executed during execution with the current idle resource condition of the big data cluster, selecting another task of the other tasks to be executed, of which the resource demand required during execution is less than the current idle resource condition of the big data cluster, as a target task according to the comparison result, and returning to the step of executing the resource demand for determining and executing the target task. Specifically, the resource demand required for executing the task can be met under the current idle resource condition of the big data cluster, that is, the following three conditions are involved: (1) the condition of the memory free resources of the big data cluster is more than the amount of the memory resources needed for executing the task and the condition of the CPU free resources of the big data cluster is less than the amount of the CPU resources needed for executing the task, (2) the condition of the memory free resources of the big data cluster is less than the amount of the memory resources needed for executing the task and the condition of the CPU free resources of the big data cluster is more than the amount of the CPU resources needed for executing the task, (3) the condition of the memory free resources of the big data cluster is less than the amount of the memory resources needed for executing the task and the condition of the CPU free resources of the big data cluster is less than the amount of the CPU resources needed for executing; when the three conditions are met, one of the tasks to be executed in the big data cluster is selected to be executed, when the other tasks to be executed are selected to be executed, the required resource demand of the selected task in the execution process needs to be calculated, the required resource demand of the calculated task in the execution process is compared with the current idle resource condition of the big data cluster, and one of the tasks to be executed, of which the required resource demand is smaller than the current idle resource condition of the big data cluster in the execution process, is selected to be executed according to the comparison result. For example, the data volume of the task a is 100g, if the task with the data volume needs to be processed, fixed resources (for example, 10g of memory resources and 20-core CPU resources) are allocated to the data volume of every 128M in the task a for processing according to the preset of the big data cluster, and then the total resource demand is approximately 80g of the memory resources of the big data cluster and 160 cores of the CPU resources.
Optionally, in this embodiment, after selecting one of the other tasks to be executed in the big data cluster to execute, the execution state of the task is monitored to determine whether the task is completely executed, after the task is submitted to the big data cluster to execute, the execution state of the task is updated all the time, and when the execution state of the task shows that the execution is completed, the big data cluster acquires the task with a large resource demand that is not completely executed to continue executing, so that the task with less required resources is prevented from being executed too long due to the fact that the task with long time consumption is waited for the task to be executed.
As can be seen from the above, in the embodiment of the present invention, if a submission instruction of a plurality of tasks is received, a current idle resource condition of a big data cluster is obtained; acquiring one of the tasks as a target task, and determining the resource demand for executing the target task; judging whether the current idle resource condition of the big data cluster meets the resource demand required by executing the target task or not; and if the current idle resource condition of the big data cluster can not meet the resource demand required by executing the target task, selecting another task from the tasks as the target task and returning to the step of determining the resource demand for executing the target task. The invention can reasonably arrange the tasks, ensure the smooth execution of the tasks under the condition of limited resources and improve the efficiency of task execution.
Referring to fig. 6, in response to the above-mentioned method for automatically arranging tasks, an embodiment of the present invention further provides an apparatus for automatically arranging tasks, where the apparatus 100 includes: a first acquisition unit 101, a determination unit 102, a judgment unit 103, and a selection unit 104.
The first obtaining unit 101 is configured to obtain a current idle resource condition of a big data cluster if a submission instruction of a plurality of tasks is received, where the current idle resource condition of the big data cluster includes a sum of current idle resources of all computer devices that form the big data cluster; a determining unit 102, configured to acquire one of the multiple tasks as a target task, and determine a resource demand for executing the target task; a judging unit 103, configured to judge whether a current idle resource condition of the big data cluster meets a resource demand required for executing the target task; and a selecting unit 104, configured to select another task from the multiple tasks as a target task and return to executing the step of determining the resource demand for executing the target task if the current idle resource condition of the big data cluster cannot meet the resource demand required for executing the target task.
The current idle resource condition of the big data cluster includes a memory idle resource condition and a CPU idle resource condition of the big data cluster, and the first obtaining unit 101 is specifically configured to: and acquiring the memory idle resource condition and the CPU idle resource condition of the big data cluster.
As shown in fig. 7, the determining unit 102 includes:
a second obtaining unit 102a, configured to obtain a data amount of one of a plurality of tasks that need to be executed; the determining subunit 102b is configured to determine, according to the data size of the task, a resource requirement for executing the target task.
As shown in fig. 8, the current idle resource condition of the big data cluster includes a memory idle resource condition and a CPU idle resource condition of the big data cluster, the resource demand required for executing the task includes a memory resource amount and a CPU resource amount required for executing the task, and the determining unit 103 includes:
a judging subunit 103a, configured to judge whether a memory idle resource condition of the big data cluster is greater than a memory resource amount required for executing the target task and whether a CPU idle resource condition of the big data cluster is greater than a CPU resource amount required for executing the target task; a determining unit 103b, configured to determine that the current idle resource condition of the big data cluster satisfies the resource demand required for executing the target task if the memory idle resource condition of the big data cluster is greater than the memory resource amount required for executing the target task and the CPU idle resource condition of the big data cluster is greater than the CPU resource amount required for executing the target task, otherwise, determine that the current idle resource condition of the big data cluster cannot satisfy the resource demand required for executing the target task.
As shown in fig. 9, the selecting unit 104 includes:
the computing unit 104a is configured to compute a resource demand required by each other task to be executed when executing the target task if the current idle resource condition of the big data cluster cannot meet the resource demand required by executing the target task; and the selecting subunit 104b is configured to compare the calculated resource demand required by each of the other tasks to be executed during execution with the current idle resource condition of the big data cluster, select, according to the comparison result, another task of the other tasks to be executed whose resource demand is smaller than the current idle resource condition of the big data cluster, as a target task, and return to execute the step of determining the resource demand for executing the target task.
As can be seen from the above, in the embodiment of the present invention, if a submission instruction of a plurality of tasks is received, a current idle resource condition of a big data cluster is obtained; acquiring one of the tasks as a target task, and determining the resource demand for executing the target task; judging whether the current idle resource condition of the big data cluster meets the resource demand required by executing the target task or not; and if the current idle resource condition of the big data cluster can not meet the resource demand required by executing the target task, selecting another task from the tasks as the target task and returning to the step of determining the resource demand for executing the target task. The invention can reasonably arrange the tasks, ensure the smooth execution of the tasks under the condition of limited resources and improve the efficiency of task execution.
The task automatic arrangement device corresponds to the task automatic arrangement method one to one, and the specific principle and process are the same as those of the method in the embodiment, which is not described again.
The above-mentioned task automatic orchestration means may be implemented in the form of a computer program that can be run on a computer device as shown in fig. 10.
FIG. 10 is a schematic diagram of a computer device according to the present invention. The device can be a terminal or a server, wherein the terminal can be an electronic device such as a notebook computer, a desktop computer and the like. The server may be an independent server or a server cluster composed of a plurality of servers. Referring to fig. 10, the computer apparatus 500 includes a processor 502, a nonvolatile storage medium 503, an internal memory 504, and a network interface 505, which are connected by a system bus 501. The non-volatile storage medium 503 of the computer device 500 may store, among other things, an operating system 5031 and a computer program 5032, which, when executed, may cause the processor 502 to perform an automatic task orchestration method. The processor 502 of the computer device 500 is used to provide computing and control capabilities that support the overall operation of the computer device 500. The internal memory 504 provides an environment for the execution of a computer program 5032 on the non-volatile storage medium 503, which when executed by the processor causes the processor 502 to perform a method for automatic task orchestration. The network interface 505 of the computer device 500 is used for network communication. Those skilled in the art will appreciate that the illustration in fig. 10 is merely a block diagram of a portion of the structure associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those illustrated, or may combine certain components, or have a different arrangement of components.
Wherein the processor 502 implements the following operations when executing the computer program:
if receiving a submission instruction of a plurality of tasks, acquiring the current idle resource condition of the big data cluster, wherein the current idle resource condition of the big data cluster comprises the sum of current idle resources of all computer equipment forming the big data cluster;
acquiring one of the tasks as a target task, and determining the resource demand for executing the target task;
judging whether the current idle resource condition of the big data cluster meets the resource demand required by executing the target task or not;
and if the current idle resource condition of the big data cluster can not meet the resource demand required by executing the target task, selecting another task from the tasks as the target task and returning to the step of determining the resource demand for executing the target task.
In one embodiment, the obtaining of the current idle resource condition of the big data cluster includes:
and acquiring the memory idle resource condition and the CPU idle resource condition of the big data cluster.
In one embodiment, the obtaining one of the plurality of tasks as a target task and determining a resource requirement for executing the target task includes:
acquiring the data volume of one task in a plurality of tasks needing to be executed;
and determining the resource demand for executing the target task according to the data volume of the task.
In one embodiment, the determining whether the current idle resource condition of the big data cluster meets the resource demand required for executing the target task includes:
judging whether the memory idle resource condition of the big data cluster is larger than the memory resource amount required for executing the target task and whether the CPU idle resource condition of the big data cluster is larger than the CPU resource amount required for executing the target task;
and if the memory free resource condition of the big data cluster is larger than the memory resource quantity required for executing the target task and the CPU free resource condition of the big data cluster is larger than the CPU resource quantity required for executing the target task, judging that the current free resource condition of the big data cluster meets the resource demand quantity required for executing the target task, otherwise, judging that the current free resource condition of the big data cluster cannot meet the resource demand quantity required for executing the target task.
In an embodiment, if the current idle resource condition of the big data cluster cannot satisfy the resource demand required for executing the target task, selecting another task from the multiple tasks as the target task and returning to execute the step of determining the resource demand for executing the target task, includes:
if the current idle resource condition of the big data cluster can not meet the resource demand required by executing the target task, calculating the resource demand required by other tasks to be executed during execution;
and comparing the calculated resource demand required by each other task to be executed during execution with the current idle resource condition of the big data cluster, selecting another task of the other tasks to be executed, of which the resource demand required during execution is less than the current idle resource condition of the big data cluster, as a target task according to the comparison result, and returning to execute the step of determining the resource demand for executing the target task.
Those skilled in the art will appreciate that the embodiment of a computer device illustrated in fig. 10 does not constitute a limitation on the specific construction of the computer device, and that in other embodiments a computer device may include more or fewer components than those illustrated, or some components may be combined, or a different arrangement of components. For example, in some embodiments, the computer device only includes a memory and a processor, and in such embodiments, the structures and functions of the memory and the processor are the same as those of the embodiment shown in fig. 10, and are not described herein again.
The present invention provides a computer readable storage medium storing one or more computer programs, the one or more computer programs being executable by one or more processors to perform the steps of:
if receiving a submission instruction of a plurality of tasks, acquiring the current idle resource condition of the big data cluster, wherein the current idle resource condition of the big data cluster comprises the sum of current idle resources of all computer equipment forming the big data cluster;
acquiring one of the tasks as a target task, and determining the resource demand for executing the target task;
judging whether the current idle resource condition of the big data cluster meets the resource demand required by executing the target task or not;
and if the current idle resource condition of the big data cluster can not meet the resource demand required by executing the target task, selecting another task from the tasks as the target task and returning to the step of determining the resource demand for executing the target task.
In one embodiment, the obtaining of the current idle resource condition of the big data cluster includes:
and acquiring the memory idle resource condition and the CPU idle resource condition of the big data cluster.
In one embodiment, the obtaining one of the plurality of tasks as a target task and determining a resource requirement for executing the target task includes:
acquiring the data volume of one task in a plurality of tasks needing to be executed;
and determining the resource demand for executing the target task according to the data volume of the task.
In one embodiment, the determining whether the current idle resource condition of the big data cluster meets the resource demand required for executing the target task includes:
judging whether the memory idle resource condition of the big data cluster is larger than the memory resource amount required for executing the target task and whether the CPU idle resource condition of the big data cluster is larger than the CPU resource amount required for executing the target task;
and if the memory free resource condition of the big data cluster is larger than the memory resource quantity required for executing the target task and the CPU free resource condition of the big data cluster is larger than the CPU resource quantity required for executing the target task, judging that the current free resource condition of the big data cluster meets the resource demand quantity required for executing the target task, otherwise, judging that the current free resource condition of the big data cluster cannot meet the resource demand quantity required for executing the target task.
In an embodiment, if the current idle resource condition of the big data cluster cannot satisfy the resource demand required for executing the target task, selecting another task from the multiple tasks as the target task and returning to execute the step of determining the resource demand for executing the target task, includes:
if the current idle resource condition of the big data cluster can not meet the resource demand required by executing the target task, calculating the resource demand required by other tasks to be executed during execution;
and comparing the calculated resource demand required by each other task to be executed during execution with the current idle resource condition of the big data cluster, selecting another task of the other tasks to be executed, of which the resource demand required during execution is less than the current idle resource condition of the big data cluster, as a target task according to the comparison result, and returning to execute the step of determining the resource demand for executing the target task.
The foregoing storage medium of the present invention includes: various media that can store program codes, such as a magnetic disk, an optical disk, and a Read-Only Memory (ROM).
The elements of all embodiments of the present invention may be implemented by a general purpose integrated circuit, such as a CPU (central processing Unit), or by an ASIC (Application Specific integrated circuit).
The steps in the automatic task scheduling method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs.
The units in the automatic task scheduling device of the embodiment of the invention can be merged, divided and deleted according to actual needs.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A method for automatically arranging tasks, which is characterized by comprising the following steps:
if receiving a submission instruction of a plurality of tasks, acquiring the current idle resource condition of the big data cluster, wherein the current idle resource condition of the big data cluster comprises the sum of current idle resources of all computer equipment forming the big data cluster;
acquiring one of the tasks as a target task, and determining the resource demand for executing the target task;
judging whether the current idle resource condition of the big data cluster meets the resource demand required by executing the target task or not;
and if the current idle resource condition of the big data cluster can not meet the resource demand required by executing the target task, selecting another task from the tasks as the target task and returning to the step of determining the resource demand for executing the target task.
2. The method of claim 1, wherein the current idle resource condition of the big data cluster comprises a memory idle resource condition and a CPU idle resource condition of the big data cluster, and the obtaining the current idle resource condition of the big data cluster comprises:
and acquiring the memory idle resource condition and the CPU idle resource condition of the big data cluster.
3. The method of claim 1, wherein the obtaining one of the plurality of tasks as a target task and determining a resource requirement to execute the target task comprises:
acquiring the data volume of one task in a plurality of tasks needing to be executed;
and determining the resource demand for executing the target task according to the data volume of the task.
4. The method of claim 1, wherein the current free resource condition of the big data cluster includes a memory free resource condition and a CPU free resource condition of the big data cluster, the resource demand required to execute the task includes a memory resource amount and a CPU resource amount required to execute the task, and the determining whether the current free resource condition of the big data cluster satisfies the resource demand required to execute the target task comprises:
judging whether the memory idle resource condition of the big data cluster is larger than the memory resource amount required for executing the target task and whether the CPU idle resource condition of the big data cluster is larger than the CPU resource amount required for executing the target task;
and if the memory free resource condition of the big data cluster is larger than the memory resource quantity required for executing the target task and the CPU free resource condition of the big data cluster is larger than the CPU resource quantity required for executing the target task, judging that the current free resource condition of the big data cluster meets the resource demand quantity required for executing the target task, otherwise, judging that the current free resource condition of the big data cluster cannot meet the resource demand quantity required for executing the target task.
5. The method of claim 1, wherein said selecting another task from said plurality of tasks as a target task and returning to performing said step of determining the amount of resource required to execute said target task if the current free resources condition of the big data cluster fails to meet the amount of resource required to execute said target task comprises:
if the current idle resource condition of the big data cluster can not meet the resource demand required by executing the target task, calculating the resource demand required by other tasks to be executed during execution;
and comparing the calculated resource demand required by each other task to be executed during execution with the current idle resource condition of the big data cluster, selecting another task of the other tasks to be executed, of which the resource demand required during execution is less than the current idle resource condition of the big data cluster, as a target task according to the comparison result, and returning to execute the step of determining the resource demand for executing the target task.
6. An automatic task orchestration device, the device comprising:
the system comprises a first obtaining unit and a second obtaining unit, wherein the first obtaining unit is used for obtaining the current idle resource condition of a big data cluster if receiving a submitting instruction of a plurality of tasks, and the current idle resource condition of the big data cluster comprises the sum of the current idle resources of all computer equipment forming the big data cluster;
the determining unit is used for acquiring one of the tasks as a target task and determining the resource demand for executing the target task;
the judging unit is used for judging whether the current idle resource condition of the big data cluster meets the resource demand required by executing the target task;
and the selecting unit is used for selecting another task from the plurality of tasks as the target task and returning to the step of determining the resource demand for executing the target task if the current idle resource condition of the big data cluster cannot meet the resource demand required for executing the target task.
7. The apparatus of claim 6, wherein the current idle resource status of the big data cluster includes a memory idle resource status and a CPU idle resource status of the big data cluster, and the first obtaining unit is specifically configured to: and acquiring the memory idle resource condition and the CPU idle resource condition of the big data cluster.
8. The apparatus of claim 6, wherein the determining unit comprises:
the second acquisition unit is used for acquiring the data volume of one of the tasks needing to be executed;
and the determining subunit is used for determining the resource demand for executing the target task according to the data volume of the task.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements a method of automatically orchestrating tasks according to any one of claims 1-5 when executing the computer program.
10. A computer-readable storage medium storing one or more computer programs, the one or more computer programs being executable by one or more processors to implement the method for automatic task orchestration according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010026629.9A CN111258757A (en) | 2020-01-10 | 2020-01-10 | Automatic task arranging method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010026629.9A CN111258757A (en) | 2020-01-10 | 2020-01-10 | Automatic task arranging method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111258757A true CN111258757A (en) | 2020-06-09 |
Family
ID=70950348
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010026629.9A Pending CN111258757A (en) | 2020-01-10 | 2020-01-10 | Automatic task arranging method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111258757A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111722928A (en) * | 2020-06-12 | 2020-09-29 | 北京字节跳动网络技术有限公司 | Resource scheduling method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101916209A (en) * | 2010-08-06 | 2010-12-15 | 华东交通大学 | Cluster task resource allocation method for multi-core processor |
| CN107729126A (en) * | 2016-08-12 | 2018-02-23 | 中国移动通信集团浙江有限公司 | A kind of method for scheduling task and device of container cloud |
| CN108469988A (en) * | 2018-02-28 | 2018-08-31 | 西北大学 | A kind of method for scheduling task based on isomery Hadoop clusters |
-
2020
- 2020-01-10 CN CN202010026629.9A patent/CN111258757A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101916209A (en) * | 2010-08-06 | 2010-12-15 | 华东交通大学 | Cluster task resource allocation method for multi-core processor |
| CN107729126A (en) * | 2016-08-12 | 2018-02-23 | 中国移动通信集团浙江有限公司 | A kind of method for scheduling task and device of container cloud |
| CN108469988A (en) * | 2018-02-28 | 2018-08-31 | 西北大学 | A kind of method for scheduling task based on isomery Hadoop clusters |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111722928A (en) * | 2020-06-12 | 2020-09-29 | 北京字节跳动网络技术有限公司 | Resource scheduling method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9946577B1 (en) | Systems and methods for distributed resource management | |
| US11645121B2 (en) | Systems and methods for distributed resource management | |
| JPWO2007072544A1 (en) | Information processing apparatus, computer, resource allocation method, and resource allocation program | |
| CN109766172B (en) | Asynchronous task scheduling method and device | |
| CN111104227B (en) | Resource control method and device of K8s platform and related components | |
| US20240160474A1 (en) | Multi-core processor task scheduling method, and device and storage medium | |
| CN115454589A (en) | Task scheduling method and device and Kubernetes scheduler | |
| CN107682389B (en) | Method, terminal and computer readable storage medium for executing network request | |
| CN113886069A (en) | Resource allocation method and device, electronic equipment and storage medium | |
| CN114816709A (en) | Task scheduling method, device, server and readable storage medium | |
| CN111258757A (en) | Automatic task arranging method and device, computer equipment and storage medium | |
| CN116360990B (en) | Distributed computing task rationality pre-judging method, system, equipment and storage medium | |
| CN111359205B (en) | Operation method and device of cloud game, computer equipment and storage medium | |
| CN117331668A (en) | Job scheduling method, device, equipment and storage medium | |
| CN114448909B (en) | Network card queue polling method and device based on ovs, computer equipment and medium | |
| CN111143063A (en) | Task resource reservation method and device | |
| CN114911538A (en) | Starting method of running system and computing equipment | |
| CN114490374A (en) | Applied pressure test method, electronic device, storage medium, and program product | |
| CN115129438A (en) | Method and device for task distributed scheduling | |
| US10417040B2 (en) | Job scheduler test program, job scheduler test method, and information processing apparatus | |
| CN111399994B (en) | Request scheduling method, request scheduling device, electronic equipment and storage medium | |
| CN117076138B (en) | System simulation-oriented resource cross-platform integration and scheduling method and device | |
| CN114598705B (en) | Message load balancing method, device, equipment and medium | |
| CN113535187B (en) | Service online method, service updating method and service providing method | |
| CN117591256A (en) | Automatic driving task scheduling method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |