CN113076188A - Scheduling method and device of distributed system - Google Patents
Scheduling method and device of distributed system Download PDFInfo
- Publication number
- CN113076188A CN113076188A CN202010004365.7A CN202010004365A CN113076188A CN 113076188 A CN113076188 A CN 113076188A CN 202010004365 A CN202010004365 A CN 202010004365A CN 113076188 A CN113076188 A CN 113076188A
- Authority
- CN
- China
- Prior art keywords
- task flow
- task
- resource
- lock file
- resource node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000010586 diagram Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 241001522296 Erithacus rubecula Species 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
- G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a dispatching method and a device of a distributed system, wherein the method comprises the following steps: receiving a first task flow for performing an operation on a first resource node of a distributed system; creating a first resource lock file corresponding to the first resource node, wherein the first resource lock file comprises identification information of the first task flow and timeout time of the first task flow; scheduling the first task flow to the first resource node based on the first resource lock file.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a scheduling method and apparatus for a distributed system.
Background
With the rise of cloud computing, distributed systems are widely used. Distributed systems are deployed with a large number of resource nodes that support task flow execution. In order to ensure mutual exclusivity of task streams accessing the same resource when the task streams are scheduled by a distributed system, resource nodes are generally coordinated by using a resource lock file.
In the existing method, if a plurality of task flows need to operate the same resource node, resource lock files with increasing sequence numbers are created for the task flows in the same directory according to the receiving sequence of the task flows, and the task flows execute operations on the resource node according to the sequence of the resource lock files. For example, when the first task flow is executed, the created resource lock file is 1, and the created resource lock file for the second task flow is 2, then the second task flow must execute the operation on the resource node after the resource lock file 1 is deleted. The task flow scheduling method has at least two disadvantages: first, the resource lock file is deleted according to the callback notification of the callback function, but because the callback function notification is not timely, the situation that the resource lock file of the task flow which is executing the operation on the resource node is not deleted and the next task flow already executes the operation on the resource node often occurs. Secondly, if the load of the resource node is too high when the task flow executes the operation on the resource node, the task flow is suspended, so that the task flow cannot be executed, the resource lock file of the task flow cannot be released all the time, and a new task flow cannot be scheduled all the time.
Disclosure of Invention
In view of this, the present invention provides a scheduling method and apparatus for a distributed system, and mainly aims to efficiently and accurately schedule a task flow in the distributed system.
In a first aspect, the present invention provides a scheduling method for a distributed system, where the method includes:
receiving a first task flow for performing an operation on a first resource node of a distributed system;
creating a first resource lock file corresponding to the first resource node, wherein the first resource lock file comprises identification information of the first task flow and timeout time of the first task flow;
scheduling the first task flow to the first resource node based on the first resource lock file.
In a second aspect, the present invention provides a scheduling apparatus for a distributed system, the apparatus comprising:
a first receiving unit, configured to receive a first task flow, where the first task flow is used to perform an operation on a first resource node of a distributed system;
a creating unit, configured to create a first resource lock file corresponding to the first resource node, where the first resource lock file includes identification information of the first task flow and timeout time of the first task flow;
a scheduling unit, configured to schedule the first task flow to the first resource node based on the first resource lock file.
In a third aspect, the present invention provides a computer-readable storage medium, where the computer-readable storage medium includes a stored program, and when the program runs, a device in which the storage medium is located is controlled to execute the scheduling method of the distributed system in the first aspect.
In a fourth aspect, the present invention provides a storage management device, comprising: a memory for storing a program; a processor, coupled to the memory, for executing the program to perform the scheduling method of the distributed system described in the first aspect.
By means of the technical scheme, the scheduling method and the scheduling device of the distributed system provided by the invention firstly create the first resource lock file corresponding to the first resource node when receiving the first task stream for executing the operation on the first resource node of the distributed system. A first task flow is scheduled to a first resource node based on the first resource lock file. The first resource lock file comprises the identification information of the first task flow and the overtime of the first task flow, and therefore the occupation of the task flow on the resource node can be controlled through the identification information of the task flow and the overtime, and the scheme provided by the invention can realize efficient and accurate scheduling of the task flow in the distributed system.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a flowchart illustrating a scheduling method of a distributed system according to an embodiment of the present invention;
fig. 2 is a flowchart illustrating a scheduling method of a distributed system according to another embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating a resource lock control manner of a distributed system according to another embodiment of the present invention;
fig. 4 is a schematic structural diagram illustrating a scheduling apparatus of a distributed system according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram illustrating a scheduling apparatus of a distributed system according to another embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
An embodiment of the present invention provides a scheduling method for a distributed system, and as shown in fig. 1, the method mainly includes:
101. a first task flow is received for performing an operation on a first resource node of a distributed system.
The task flow described in this embodiment generally includes a plurality of task items, a set execution sequence exists among the plurality of task items, and the execution of the plurality of task items has an uninterruptible characteristic, and once a certain task item is interrupted, the entire task flow fails to be executed. The execution of the task flow depends on resources in resource nodes of the distributed system, and once a fault exists when the task flow performs an operation on one resource node of the distributed system and a task project is interrupted, the task flow needs to be scheduled to another resource node in the distributed system again in order to ensure the smooth execution of the task flow. It can be seen that the first task stream exists in at least two types: first, the first task flow is a task flow which does not perform any operation on any resource node of the distributed system; second, the first task flow is an executed but interrupted task flow, which has already executed an operation on at least one resource node in the distributed system, but a failure occurs during the operation, and in order to ensure that the first task flow can be executed and completed, an operation needs to be executed on another resource node in the distributed system.
102. And creating a first resource lock file corresponding to the first resource node, wherein the first resource lock file comprises the identification information of the first task flow and the timeout time of the first task flow.
In practical applications, there are multiple resource nodes in a distributed system. When the task flow is received, the task flow needs to be scheduled to a resource node which needs to execute the operation, and the resource node provides resources for the execution of the task flow. In order to enable the task flow to operate the required resource node, a resource lock file corresponding to the required resource node needs to be created, so that the task flow can preempt the resource node based on the resource lock file.
In this embodiment, the principle of creating the first resource lock file corresponding to the first resource node at least includes the following two principles:
1. in order to ensure the mutual exclusivity of the task flow in executing the operation on the same resource, one resource node is occupied by only one task flow at the same time, and only one resource lock file corresponding to one task flow exists in one resource node at the same time. The meaning of the resource lock file corresponding to only one task flow includes: (1) when the task flow completes the execution operation of the resource node, the resource lock file of the resource node aiming at the task flow is automatically deleted, and when the resource node has the corresponding resource lock file, the resource lock file is effective, and the task flow can be scheduled to the resource node based on the effective resource lock file; (2) when the task flow completes the operation on the resource node, the resource lock file of the resource node for the task flow is identified to be invalid, that is, only one resource lock file in the resource lock files of the resource node is valid at the same time, and the task flow can be scheduled to the resource node based on the valid resource lock file.
2. In order to ensure that a plurality of task flows can execute operations based on the same resource node, a plurality of effective resource lock files exist in the same resource at the same time, and each resource lock file corresponds to one task flow.
In this embodiment, creating the first resource lock file corresponding to the first resource node may face the following situations:
and when the first resource node is occupied by the task flow and the number of the task flows reaches a preset number threshold, waiting for the task flow to release the first resource node, and creating a first resource lock file corresponding to the first resource node when any one task flow releases the first resource node.
Or the first resource node is not occupied by the task flow currently or occupied by the task flow, but the number of the task flows does not reach the preset number threshold, and then the first resource lock file corresponding to the first resource node is directly created.
In this embodiment, the specific type of the resource lock file may be determined based on the service requirement, and may include, but is not limited to, any one of a TXT type, a json type, and an XML type. The resource lock file comprises identification information of the first task flow and timeout time of the first task flow. The identification information of the first task flow is used to identify the first task flow so as to clearly distinguish which task flow currently preempts the first resource node based on the identification information. The timeout time of the first task flow refers to a time limit within which the first task flow can execute operations on the first resource node, and once the time for the first task flow to execute operations on the first resource node reaches the timeout time, it indicates that the first task flow occupies the resource node to the end, and other task flows may preempt the first resource node and create a resource lock file for the first resource node. After the occupation of the resource node is finished, the resource lock file can be deleted or the invalidity of the resource lock file can be identified.
103. Scheduling the first task flow to the first resource node based on the first resource lock file.
In this embodiment, after the first resource lock file corresponding to the first resource node is created, which indicates that the first resource node has been successfully preempted by the first task flow, the first task flow is scheduled to the first resource node according to the identification information of the first task flow in the first resource lock file, so that the first task flow performs an operation on the first resource node.
In this embodiment, in order to ensure that the first resource node can be released in time for other task flows to perform operations, after the first task flow is scheduled to the first resource node, the first task flow needs to be monitored according to the timeout time of the first task flow in the first resource lock file.
Specifically, when the execution time of the first task flow for executing the operation on the first resource node reaches the timeout time, it indicates that the occupation of the first resource node by the first task flow is finished, and in order to enable other task flows to be dispatched to the first resource node, it is necessary to delete the first resource lock file or identify that the first resource lock file is invalid. Meanwhile, the process of the first task flow scheduling to the first resource node can be timely terminated according to the identification information of the first task flow in the first resource lock file, so that the first task flow stops executing the operation on the first resource node.
Specifically, when the execution time of the first task flow for executing the operation on the first resource node does not reach the timeout time, it indicates that the first task flow can still occupy the first resource node, and it is sufficient to maintain the first task flow to execute the operation on the first resource node.
The scheduling method of the distributed system provided by the embodiment of the invention firstly creates the first resource lock file corresponding to the first resource node when receiving the first task stream for executing the operation on the first resource node of the distributed system. A first task flow is scheduled to a first resource node based on the first resource lock file. The first resource lock file comprises identification information of the first task flow and timeout time of the first task flow, and therefore occupation of the first task flow on the first resource node can be controlled through the identification information and the timeout time.
Further, according to the method shown in fig. 1, another embodiment of the present invention further provides a scheduling method of a distributed system, as shown in fig. 2, the method mainly includes:
201. a first task flow is received for performing an operation on a first resource node of a distributed system.
202. And creating a first resource lock file corresponding to the first resource node, wherein the first resource lock file comprises the identification information of the first task flow and the timeout time of the first task flow.
203. Scheduling the first task flow to the first resource node based on the first resource lock file.
In this embodiment, a preset process pool exists in the distributed system, and when the first task flow is scheduled to the first resource node based on the first resource lock file, a process is allocated to the first task flow from the process pool, and the first task flow is scheduled to the first resource node by using the process.
204. Receiving a second task flow, the second task flow being used to perform an operation on the first resource node.
In this embodiment, the task flow has an uninterruptible characteristic, and once a certain task item is interrupted, the whole task flow fails to execute. The task flow is required to be subjected to fault monitoring in the task flow execution process, so that when the task flow is in fault, the task flow is timely scheduled to another resource node in the distributed system again. The new resource node is a substitute of the previous resource node, and the task flow can be executed and completed according to the required resources. It can be seen that the second task stream exists in at least two types: the first type, the second task flow is a task flow which does not execute operations on any resource node of the distributed system; and secondly, the second task flow is an interrupted first task flow which has already performed operation on at least one resource node in the distributed system but has a fault in the operation process, and in order to ensure that the first task flow can be performed completely, the first task flow is dispatched to another resource node in the distributed system to perform operation.
205. And judging whether the identification information of the second task flow is consistent with the identification information of the first task flow or not based on the first resource lock file.
In this embodiment, when a task flow preempts a resource node, in order to enable the task flow to be able to schedule other resource nodes to execute operations in time when the task flow fails, the task flow directly preempts two or more resource nodes when the task flow preempts the resource node. When a resource node fails in the task flow scheduling process, the resource node does not need to be preempted any more, and the task flow can be directly scheduled to the preempted resource node to continue to execute the operation. Therefore, in order to determine whether the first resource lock file of the first resource node is a resource node pre-preempted before the second task flow, it is determined whether the identification information of the second task flow is consistent with the identification information of the first task flow based on the first resource lock file.
It should be noted that the resource nodes preempted by the task flow all create the resource lock file for them. It should be noted that the resource lock files of the preemptive resource nodes include the same timeout time, and may also include different timeout times. For example, the timeout time in the resource lock file corresponding to the resource node serving as disaster recovery is relatively late.
206. And processing the second task flow according to the judgment result.
In this embodiment, the method for processing the second task flow according to the determination result is related to the determination result, and there are at least the following methods:
first, if the identification information of the second task flow is consistent with the identification information of the first task flow, determining whether the first task flow is overtime based on the overtime time of the first task flow; if not, scheduling the second task flow to the first resource node based on the first resource lock file; and if the time is out, terminating the execution of the first task flow.
Specifically, when it is determined that the identification information of the second task flow is consistent with the identification information of the first task flow, it is indicated that the second task flow and the first task flow are the same task flow, and the first resource node is a resource node preset to be preempted by the task flow. At this time, it is only required to determine whether the first task flow is overtime directly according to the overtime time of the first task flow in the first resource lock file.
Specifically, when it is determined that the first task flow is not timed out, it is indicated that the first resource node is still occupied by the first task flow node, and the second task flow is scheduled to the first resource node based on the first resource lock file.
Specifically, if the second task flow is the first task flow after the fault occurs, the task item with the fault occurs in the task flow is determined, and the task item with the fault and the task item after the task item with the fault occur are dispatched to the first resource node according to the identification information of the first task flow in the first resource lock file, so that the task item dispatched to the first resource node within the timeout time can execute the operation on the first resource node, and the whole first task flow is completed.
Illustratively, the first task flow A includes task item 1, task item 2, task item 3, and task item 4. When the first task flow A accesses the resource node1, a fault occurs in the task item 2, and a substitute resource node 'resource node 2' which replaces the resource node1 is determined. If the resource node2 is a resource node which is preempted before and aims at the first task flow A, and the resource lock file is not overtime, the task items 2 to 4 are directly dispatched to the resource node 2.
Specifically, when the first task flow is determined to be timed out, it is indicated that the first task flow occupies the first resource node, and in order to enable other task flows to occupy the first resource node, it is necessary to terminate execution of the first task flow and terminate occupation of the first resource node by the first task flow. And after the execution of the first task flow is terminated, a second resource lock file corresponding to the second resource node is created, wherein the second resource lock file comprises identification information of the second task flow and the timeout time of the second task flow, and then the second task flow is dispatched to the first resource node based on the second resource lock file. Similarly, if the second task flow is the first task flow after the fault occurs, the task item with the fault in the task flow is determined, and the task item with the fault and the task item after the task item with the fault are dispatched to the first resource node according to the identification information of the first task flow in the first resource lock file, so that the task item dispatched to the first resource node within the timeout time can execute the operation on the first resource node, and the whole first task flow is completed.
Secondly, if the identification information of the second task flow is inconsistent with the identification information of the first task flow, determining whether the first task flow is overtime or not based on the overtime time of the first task flow; if not, waiting for the first task flow to release the first resource node, and creating a second resource lock file corresponding to the first resource node when the first task flow releases the first resource node, wherein the second resource lock file comprises identification information of the second task flow and the timeout time of the second task flow; and if the time is out, terminating the execution of the first task flow.
Specifically, if it is determined that the identification information of the second task flow is inconsistent with the identification information of the first task flow, it is determined that the second task flow is the first task flow and is not the same task flow. At this time, it is necessary to determine whether the first task flow is overtime according to the timeout time of the first task flow in the first resource lock file, so that when the first task flow is overtime, execution of the first task flow is terminated, so that the first task flow timely releases the first resource node, and other task flows preempt the first resource node.
Specifically, when it is determined that the first resource lock file is not timed out, it indicates that the first resource node is still occupied by the first task flow, and at this time, the following two operations may be performed: the first method is to wait for a first task flow to release a first resource node, and when the first task flow releases the first resource node, create a second resource lock file corresponding to the first resource node, wherein the second resource lock file comprises identification information of the second task flow and timeout time of the second task flow. A second task flow is then scheduled to the first resource node based on the second resource lock file. And secondly, continuously searching other resource nodes capable of replacing the first resource node, if the other resource nodes are not occupied by other task flows, creating a second resource lock file corresponding to the resource node, and scheduling the second task flow to the resource node based on the second resource lock file. And if other resource nodes are also occupied, the first resource node and the other resource nodes are trained in turn until one of the resource nodes is released by the task flow. And then creating a second resource lock file corresponding to the released resource node, and scheduling a second task flow to the resource node based on the second resource lock file. It should be noted that there are at least the following two cases when the first task flow releases the first resource node: first, the first task flow is overtime, the first task flow is terminated to be executed, and therefore the first task flow releases the first resource node; second, the first task flow is not timed out, but the first task flow is based on executing operations on the first resource node, the first task flow execution is complete, and the first task flow autonomously terminates execution.
Specifically, when the first task flow is determined to be timed out, it is indicated that the first task flow occupies the first resource node, and in order to enable other task flows to occupy the first resource node, execution of the first task flow is terminated to end occupation of the first resource node by the first task flow. And after the execution of the first task flow is terminated, a second resource lock file corresponding to the second resource node is created, wherein the second resource lock file comprises identification information of the second task flow and the timeout time of the second task flow, and then the second task flow is dispatched to the first resource node based on the second resource lock file.
Specifically, the specific process of scheduling the second task flow to the first resource node based on the second resource lock file is related to the specific type of the second task flow:
firstly, if the second task flow is an unexecuted task flow, the second task flow is completely dispatched to the first resource node directly according to the identification information of the second task flow in the second resource lock file, so that the task flow dispatched to the first resource node within the overtime carries out operation on the first resource node, and the whole second task flow is completed.
And secondly, if the second task flow is the task flow after the fault occurs, determining the task item with the fault in the task flow, and scheduling the task item with the fault and the task item behind the task item with the fault to the first resource node according to the identification information of the second task flow in the second resource lock file, so that the task item scheduled to the first resource node within the overtime can execute operation on the first resource node, and the whole second task flow is completed.
In the present embodiment, in both methods described above, it is mentioned that the execution of the first task flow is terminated when the first task flow times out. When the first task flow is terminated to execute, the first resource lock file comprising the identification information of the first task flow and the overtime time is invalid, and in order to reduce the storage space occupied by the first resource lock file, the first resource lock file is deleted based on the identification information of the first task flow when the first task flow is overtime.
Specifically, the following specific process for deleting the first resource lock file is as follows: and when the first resource lock file is deleted, calling a first interface, and deleting the first resource lock file based on the identification information of the first task flow, wherein the first interface is used for forcibly deleting the resource lock file corresponding to the resource node of the distributed system.
Further, taking fig. 3 as an example, a scheduling method of a distributed system is described below, as shown in fig. 3, fig. 3 includes a resource node1 "node 1", a resource node2 "node 2", and a resource node3 "node 3", a master node "master" of the distributed system, and the distributed system has the following three interfaces: an overlay write File interface (Create File Without Overwrite), a first interface (Condition Delete), and a second interface (Session express Time). It should be noted that, all of the three interfaces described above may be API interfaces.
The distributed system schedules the execution of a task flow 'ProcessID 1', and when a resource node is robbed, the distributed system operates and executes the following steps aiming at a node 1:
first, the distributed system receives a task stream "ProcessID 1".
Next, a resource lock File "Normal File" corresponding to the resource node1 "node 1" is created using an Overwrite File interface (Create File Without Overwrite), the content of the resource lock File including identification information "process id 1" and timeout time "TTL" of the task flow "process id 1", and as can be seen from fig. 3, the resource lock File is created by invoking a distributed consistency system such as zookeeper or etcd through the Overwrite File interface. Based on the resource lock file, the task flow "ProcessID 1" is scheduled to resource node1 "node 1". When the task flow "process id 1" is dispatched to the resource node1 "node 1", the second interface (Session exception Time) determines whether the task flow "process id 1" is overtime based on the timeout Time "TTL" round robin of the task flow "process id 1" in the resource lock file, so as to call the first interface (Condition Delete) to Delete the resource lock file at the Time-out.
Further, the task stream "ProcessID 1" includes task item 1, task item 2, task item 3, and task item 4. The task flow "ProcessID 1" fails at task item 2 when it accesses node1 and determines the replacement resource node "node 2" that replaces node 1. It is detected that node2 has been occupied by task stream "ProcessID 3" and that there is identification information and timeout time including task stream "ProcessID 3". If the identification information of the task flow in the resource lock file of node2 does not match the identification information of the task flow "ProcessID 1", it is determined whether the task flow "ProcessID 3" has timed out based on the time-out time of the task flow "ProcessID 3" in the resource lock file. If the task flow 'ProcessID 3' is determined to be overtime, execution of the task flow 'ProcessID 3' is terminated, the first interface is called, and the resource lock file is deleted based on the identification information of 'ProcessID 3' in the resource lock file. When the second interface is called to determine that the resource lock File is really deleted, a Create File Without timeout (Overwrite) interface is called to Create a resource lock File including identification information of the task flow "ProcessID 1" and timeout time for the resource node 2. Then, task item 2 to task item 4 in the task stream "ProcessID 1" are scheduled to node2, so that the task stream "ProcessID 1" is completed.
Similarly, the execution process of the task flow "ProcessID 2" in fig. 3 is substantially the same as the execution process of the task flow "ProcessID 1" described above, and thus will not be described herein again.
Further, according to the foregoing method embodiment, another embodiment of the present invention further provides a scheduling apparatus of a distributed system, as shown in fig. 4, the apparatus includes:
a first receiving unit 31, configured to receive a first task flow, where the first task flow is used to perform an operation on a first resource node of a distributed system;
a creating unit 32, configured to create a first resource lock file corresponding to the first resource node, where the first resource lock file includes identification information of the first task flow and a timeout time of the first task flow;
a scheduling unit 33, configured to schedule the first task flow to the first resource node based on the first resource lock file.
The scheduling apparatus of a distributed system according to the embodiment of the present invention first creates a first resource lock file corresponding to a first resource node when receiving a first task stream that performs an operation on the first resource node of the distributed system. A first task flow is scheduled to a first resource node based on the first resource lock file. The first resource lock file comprises identification information of the first task flow and the overtime of the first task flow, and therefore the occupation of the first task flow on the first resource node can be controlled through the overtime.
Further, as shown in fig. 5, the apparatus further includes:
a second receiving unit 34, configured to receive a second task stream, where the second task stream is used to perform an operation on the first resource node;
a determining unit 35, configured to determine, based on the first resource lock file, whether the identification information of the second task flow is consistent with the identification information of the first task flow;
and the processing unit 36 is configured to process the second task flow according to the determination result.
Further, as shown in fig. 5, the processing unit 36 includes:
a first determining module 361, configured to determine whether the first task flow is timeout based on the timeout time of the first task flow if the identification information of the second task flow is consistent with the identification information of the first task flow; if not, the first scheduling module 362 is triggered; if yes, triggering the termination module 363;
the first scheduling module 362, configured to schedule the second task flow to the first resource node based on the first resource lock file under the trigger of the first determining module 361;
the terminating module 363 is configured to terminate the execution of the first task flow under the trigger of the first determining module 361.
Further, as shown in fig. 5, the processing unit 36 includes:
a second determining module 364, configured to determine whether the first task flow is time-out based on the time-out time of the first task flow if the identification information of the second task flow is inconsistent with the identification information of the first task flow; triggering the preemption module 365; otherwise, trigger the termination module 363;
the preempting module 365 is configured to wait for the first task flow to release the first resource node if the second determining module 364 determines that the first task flow is not overtime, and create a second resource lock file corresponding to the first resource node when the first task flow releases the first resource node, where the second resource lock file includes identification information of the second task flow and overtime of the second task flow;
the terminating module 363 is configured to terminate executing the first task flow if the second determining module 364 determines that the first task flow is time out.
Further, as shown in fig. 5, the apparatus further includes: a deleting unit 37, configured to delete the first resource lock file based on the identification information of the first task flow if the first task flow is overtime.
Further, as shown in fig. 5, the creating unit 32 is further configured to create a second resource lock file corresponding to the second resource node after the terminating module 363 terminates executing the first task flow, where the second resource lock file includes identification information of the second task flow and a timeout time of the second task flow;
the scheduling unit 33 is further configured to schedule the second task flow to the first resource node based on the second resource lock file.
Further, as shown in fig. 5, the scheduling unit 33 includes:
a third determining module 331, configured to determine whether the second task flow is a failed task flow; if so, the second scheduling module 332 is triggered;
the second scheduling module 332 is configured to determine a task item corresponding to the fault in the second task stream, and schedule the task item and task items subsequent to the task item to the first resource node.
Because the contents of information interaction, execution process, and the like between the modules in the device are based on the same concept as the scheduling method embodiment of the distributed system in the embodiment of the present invention, specific contents may refer to the description in the method embodiment of the present invention, and are not described herein again. Based on the description in the above embodiments, the embodiments of the apparatus can be freely combined into a new embodiment according to business needs.
The embodiment of the invention provides a computer-readable storage medium, which includes a stored program, and when the program runs, controls a device where the storage medium is located to execute a scheduling method of a distributed system of a workflow scheduling system described in any one of the above.
An embodiment of the present invention provides a storage management device, where the storage management device includes: a memory for storing a program; a processor, coupled to the memory, for executing the program to perform the scheduling method of the distributed system of the workflow scheduling system according to any one of the above embodiments.
In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
It will be appreciated that the relevant features of the method and apparatus described above are referred to one another. In addition, "first", "second", and the like in the above embodiments are for distinguishing the embodiments, and do not represent merits of the embodiments.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components of the method, apparatus and framework for operation of a deep neural network model in accordance with embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.
Claims (16)
1. A scheduling method of a distributed system comprises the following steps:
receiving a first task flow for performing an operation on a first resource node of a distributed system;
creating a first resource lock file corresponding to the first resource node, wherein the first resource lock file comprises identification information of the first task flow and timeout time of the first task flow;
scheduling the first task flow to the first resource node based on the first resource lock file.
2. The method of claim 1, further comprising:
receiving a second task flow, wherein the second task flow is used for executing operation on the first resource node;
judging whether the identification information of the second task flow is consistent with the identification information of the first task flow or not based on the first resource lock file;
and processing the second task flow according to the judgment result.
3. The method of claim 2, wherein processing the second task stream according to the determination comprises:
if the identification information of the second task flow is consistent with the identification information of the first task flow, determining whether the first task flow is overtime or not based on the overtime time of the first task flow;
if not, scheduling the second task flow to the first resource node based on the first resource lock file;
and if the time is out, terminating the execution of the first task flow.
4. The method of claim 2, wherein processing the second task stream according to the determination comprises:
if the identification information of the second task flow is inconsistent with the identification information of the first task flow, determining whether the first task flow is overtime or not based on the overtime time of the first task flow;
if not, waiting for the first task flow to release the first resource node, and creating a second resource lock file corresponding to the first resource node when the first task flow releases the first resource node, wherein the second resource lock file comprises identification information of the second task flow and the timeout time of the second task flow;
and if the time is out, terminating the execution of the first task flow.
5. The method of claim 3 or 4, further comprising:
and if the first task flow is overtime, deleting the first resource lock file based on the identification information of the first task flow.
6. The method of claim 3 or 4, wherein after terminating execution of the first task flow, further comprising:
creating a second resource lock file corresponding to the second resource node, wherein the second resource lock file comprises identification information of the second task flow and timeout time of the second task flow;
scheduling the second task flow to the first resource node based on the second resource lock file.
7. The method of claim 6, wherein scheduling the second task flow to the first resource node comprises:
determining whether the second task flow is a failed task flow;
if yes, determining a task item corresponding to the fault in the second task flow, and scheduling the task item and task items behind the task item to the first resource node.
8. A scheduling apparatus of a distributed system, comprising:
a first receiving unit, configured to receive a first task flow, where the first task flow is used to perform an operation on a first resource node of a distributed system;
a creating unit, configured to create a first resource lock file corresponding to the first resource node, where the first resource lock file includes identification information of the first task flow and timeout time of the first task flow;
a scheduling unit, configured to schedule the first task flow to the first resource node based on the first resource lock file.
9. The apparatus of claim 8, further comprising:
a second receiving unit, configured to receive a second task stream, where the second task stream is used to perform an operation on the first resource node;
a determining unit, configured to determine whether the identification information of the second task flow is consistent with the identification information of the first task flow based on the first resource lock file;
and the processing unit is used for processing the second task flow according to the judgment result.
10. The apparatus of claim 9, wherein the processing unit comprises:
a first determining module, configured to determine whether the first task flow is overtime based on the overtime time of the first task flow if the identification information of the second task flow is consistent with the identification information of the first task flow; if not, triggering a first scheduling module; if overtime, triggering the termination module;
the first scheduling module is configured to schedule the second task flow to the first resource node based on the first resource lock file under the trigger of the first determining module;
the termination module is configured to terminate execution of the first task flow under the trigger of the first determination module.
11. The apparatus of claim 9, wherein the processing unit comprises:
a second determining module, configured to determine whether the first task flow is overtime based on the overtime time of the first task flow if the identification information of the second task flow is inconsistent with the identification information of the first task flow; triggering a preemption module; otherwise, triggering a termination module;
the preemption module is configured to wait for the first task flow to release the first resource node if the second determination module determines that the first task flow is not overtime, and create a second resource lock file corresponding to the first resource node when the first task flow releases the first resource node, where the second resource lock file includes identification information of the second task flow and overtime of the second task flow;
the terminating module is configured to terminate executing the first task flow if the second determining module determines that the first task flow is overtime.
12. The apparatus of claim 10 or 11, further comprising:
and the deleting unit is used for deleting the first resource lock file based on the identification information of the first task flow if the first task flow is overtime.
13. The apparatus according to claim 10 or 11, wherein the creating unit is further configured to create a second resource lock file corresponding to the second resource node after the terminating module terminates execution of the first task stream, the second resource lock file including identification information of the second task stream and a timeout time of the second task stream;
the scheduling unit is further configured to schedule the second task flow to the first resource node based on the second resource lock file.
14. The apparatus of claim 13, wherein the scheduling unit comprises:
a third determining module, configured to determine whether the second task flow is a failed task flow; if yes, triggering a second scheduling module;
and the second scheduling module is used for determining a task item corresponding to the fault in the second task flow and scheduling the task item and task items after the task item to the first resource node.
15. A computer-readable storage medium, comprising a stored program, wherein when the program runs, a device in which the storage medium is located is controlled to execute the scheduling method of the distributed system according to any one of claims 1 to 7.
16. A storage management device, comprising:
a memory for storing a program;
a processor, coupled to the memory, for executing the program to perform the scheduling method of the distributed system according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010004365.7A CN113076188B (en) | 2020-01-03 | Scheduling method and device of distributed system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010004365.7A CN113076188B (en) | 2020-01-03 | Scheduling method and device of distributed system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113076188A true CN113076188A (en) | 2021-07-06 |
| CN113076188B CN113076188B (en) | 2024-05-14 |
Family
ID=
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106933662A (en) * | 2017-03-03 | 2017-07-07 | 广东神马搜索科技有限公司 | Distributed system and its dispatching method and dispatching device |
| WO2020000944A1 (en) * | 2018-06-25 | 2020-01-02 | 星环信息科技（上海）有限公司 | Preemptive scheduling based resource sharing use method, system and |
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106933662A (en) * | 2017-03-03 | 2017-07-07 | 广东神马搜索科技有限公司 | Distributed system and its dispatching method and dispatching device |
| WO2020000944A1 (en) * | 2018-06-25 | 2020-01-02 | 星环信息科技（上海）有限公司 | Preemptive scheduling based resource sharing use method, system and |
Non-Patent Citations (2)
| Title |
|---|
| TIANXIAO LIU: "DScheduler: Dynamic Network Scheduling Method for MapReduce in Distributed Controllers", IEEE, 19 January 2017 (2017-01-19) * |
| 朱映映;陈阳;明仲;: "云系统中面向海量多媒体数据的动态任务调度算法", 小型微型计算机系统, no. 04, 15 April 2013 (2013-04-15) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106802826B (en) | Service processing method and device based on thread pool | |
| CN107291547B (en) | Task scheduling processing method, device and system | |
| WO2020181813A1 (en) | Task scheduling method based on data processing and related device | |
| CN111338773B (en) | Distributed timing task scheduling method, scheduling system and server cluster | |
| CN111949386A (en) | Task scheduling method, system, computing device and readable storage medium | |
| CN106156939A (en) | Dispatching System based on job stream and application process | |
| WO2023198061A1 (en) | Container scheduling method, electronic device, and storage medium | |
| CN110389843B (en) | Service scheduling method, device, equipment and readable storage medium | |
| CN112000455A (en) | Multithreading task processing method and device and electronic equipment | |
| WO2015131542A1 (en) | Data processing method, device and system | |
| CN110971700A (en) | Method and device for realizing distributed lock | |
| CN114153580A (en) | Cross-multi-cluster work scheduling method and device | |
| CN114168302A (en) | Task scheduling method, device, equipment and storage medium | |
| CN110555074A (en) | method for extracting and processing data at fixed time, terminal and computer readable storage medium | |
| CN113886069A (en) | Resource allocation method and device, electronic equipment and storage medium | |
| CN112181627A (en) | Timing task scheduling method, device and system | |
| CN108037983A (en) | Method for scheduling task and distributed scheduling system in distributed scheduling system | |
| CN113986502A (en) | Thread pool management method and device, computer terminal and storage medium | |
| CN113626173A (en) | Scheduling method, device and storage medium | |
| CN111831408A (en) | Asynchronous task processing method and device, electronic equipment and medium | |
| CN113076188A (en) | Scheduling method and device of distributed system | |
| CN113076188B (en) | Scheduling method and device of distributed system | |
| CN117076096A (en) | Task flow execution method and device, computer readable medium and electronic equipment | |
| CN115964176B (en) | Cloud computing cluster scheduling method, electronic equipment and storage medium | |
| CN116954816A (en) | Container cluster control method, device, equipment and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |