CN102541648A - Method and device for dynamically scheduling batch processing task - Google Patents
Method and device for dynamically scheduling batch processing task Download PDFInfo
- Publication number
- CN102541648A CN102541648A CN2010106191413A CN201010619141A CN102541648A CN 102541648 A CN102541648 A CN 102541648A CN 2010106191413 A CN2010106191413 A CN 2010106191413A CN 201010619141 A CN201010619141 A CN 201010619141A CN 102541648 A CN102541648 A CN 102541648A
- Authority
- CN
- China
- Prior art keywords
- task
- execution
- nodes
- flow
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 32
- 238000012544 monitoring process Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000000725 suspension Substances 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a method for dynamically scheduling batch processing tasks. The tasks comprise a first task and a second task. The method is characterized by comprising the following steps of: a first step of decomposing the first task and the second task into a plurality of task nodes and compositing a plurality of task nodes into a first process corresponding to the first task and a second process corresponding to the second task; and a second step of controlling to execute the first process and the second process by taking the task nodes as units according to information of each node, wherein the information comprises an executing state and an executing condition of each task node in a plurality of task nodes.
Description
Technical Field
The present invention relates to the field of computer technology, and in particular to dynamic scheduling of batch tasks.
Background
A batch task refers to a job that executes a series of programs in a computer system without human intervention. The existing batch processing task scheduling system focuses on the timing scheduling of tasks, but when the execution time of some tasks exceeds the expectation, the parallel execution of a plurality of tasks can be caused, and if the tasks consume too much system resources, the deadlock and even the system crash can be caused. The traditional sequential scheduling can ensure the abundance of available resources for light-weight batch tasks, but can possibly cause the waste of system resources and low working efficiency; the scheduling of large tasks lacks controllability, and scheduling operations such as suspension, restart and continuation are difficult to achieve in time.
Therefore, there is a need for a method and apparatus that can dynamically schedule batch tasks.
Disclosure of Invention
To solve the above problems, the present invention provides a method for dynamically scheduling batch tasks, the tasks including a first task and a second task, the method comprising the steps of:
a first step of decomposing a first task and a second task into a plurality of task nodes, and composing the plurality of task nodes into a first flow corresponding to the first task and a second flow corresponding to the second task;
and a second step of controlling the execution of the first flow and the second flow by taking the task node as a unit according to the information of each task node, and information the execution state and the execution condition of the task node.
Preferably, the method further comprises the step of obtaining system resource information; the second step further includes controlling execution of each task node according to the system resource information.
Preferably, the first step further comprises setting priorities of the first flow and the second flow; the second step further includes controlling execution of the first flow and the second flow according to the priority.
According to another object of the present invention, there is provided an apparatus for dynamically scheduling batch tasks, the tasks including a first task and a second task, characterized in that the apparatus includes a setting module and a control module, wherein,
the setting module is used for decomposing the first task and the second task into a plurality of task nodes and forming the plurality of task nodes into a first flow corresponding to the first task and a second flow corresponding to the second task;
the control module is used for controlling the execution of the first flow and the second flow by taking the task nodes as a unit according to the information of each task node, and the information comprises the execution state and the execution condition of each task node in the plurality of task nodes.
Preferably, the device further comprises a resource monitoring module, configured to obtain system resource information; the control module is further configured to control execution of each task node according to the system resource information.
Preferably, the setting module is further configured to set priorities of the first flow and the second flow; the control module is further configured to control execution of the first flow and the second flow according to the priority.
Preferably, the task node is the smallest unit of program with data integrity.
The invention provides the resource utilization rate of the system through the technical scheme, and can carry out corresponding processing in time when a certain task node fails through analyzing the dependency relationship, thereby improving the fault tolerance of the system. The invention has strong flexibility for scheduling the batch processing tasks based on the dynamic change of system resources and the static resource characteristics of the task nodes. In addition, because the tasks are serialized into the task nodes, the scheduling and the tasks are separated from each other, the tasks are scheduled in a flow form, and the method can be well adapted to the change of services.
Drawings
The various aspects of the present invention will become more apparent to those of ordinary skill in the art after reading the detailed description of the invention in light of the accompanying drawings. It should be understood by those skilled in the art that the drawings are designed solely for purposes of illustrating the subject technology in connection with the detailed description thereof and are not intended as a definition of the limits of the invention. Wherein,
FIG. 1 is a flow diagram of a method of dynamically scheduling batch tasks according to an embodiment of the invention;
FIG. 2 is a diagrammatic view of a method of dynamically scheduling batch tasks in accordance with an embodiment of the present invention;
FIG. 3 is a schematic diagram of an apparatus for dynamically scheduling batch tasks according to an embodiment of the present invention.
Detailed Description
Hereinafter, specific embodiments of the present invention will be described in detail with reference to the accompanying drawings. Descriptions of well-known functions and constructions in the art, which would obscure the subject matter of the present invention, are omitted for clarity and conciseness.
FIG. 1 is a flowchart of a method for dynamically scheduling batch processing tasks according to an embodiment of the present invention, wherein:
in step S1, the batch processing task is serialized into a flow composed of task nodes. As an example, a task may be decomposed into a plurality of node units with data integrity, and then the node units are spliced into a flow for scheduling by configuration, and the generated flow corresponds to the original task. A task node may be the smallest program unit that is a logical unit that can be executed independently.
In step S2, the flow is dynamically scheduled according to the information of the task node. The information of the task node may include an execution state and an execution condition of the task node. The execution state of each independently executable task node may include ready to execute, executing, and execution complete. The execution condition may include corresponding resource consumption data of the task node for describing the resource demand condition of the task node or may also include the execution time of the task node and the dependency relationship between the task node and other task nodes in the process.
Reference is now made to FIG. 2, which is a diagram illustrating a method of dynamically scheduling batch tasks, in accordance with an embodiment of the present invention. As shown, the first and second flows may be composed of a plurality of such task nodes A, B, C, D, E, F in a serial-parallel manner. According to the idea of the invention, the starting, the pause and the continuation of the process are implemented by taking the task nodes as units, and the data integrity of the nodes ensures that the process scheduling can be freely carried out among a plurality of tasks, thereby greatly increasing the flexibility of the calling. For example, in fig. 2, it is known from the dependency relationship between the task node D and other nodes, such as nodes B and C, that the execution condition of the task node D in the first flow is that the task nodes B and C are executed completely, and then the execution condition of the task node D may be set accordingly, for example, the execution time of the task node D may be set after the execution times of the task nodes B and C. Alternatively, the execution of task node D may also be triggered by determining the execution status of task nodes B and C. When the pause operation is needed, the task node to be released can be determined according to the resource consumption data of the task node, and therefore the task node information also provides a precondition for the resource release of the pause operation and the like.
FIG. 3 is a schematic diagram of an apparatus for dynamically scheduling batch tasks according to an embodiment of the present invention. As shown in the figure, the device comprises a setting module, a control module and a resource monitoring module, wherein the setting module is used for decomposing a task into a plurality of task nodes and forming the task nodes into a flow 1-4. And the control module is used for controlling the processes 1-4 by taking the task nodes as units according to the information of each task node. The information of the task node may include an execution state and an execution condition of the task node. The resource monitoring module is used for acquiring system resource information, and then the control module controls the execution of each task node according to the system resource information. For example, system resource information may be obtained through an API of an operating system and provided as a reference condition for flow scheduling. For example, a priority may be set for a flow, and when the available resources of the system exceed the warning line, a flow with a low priority is suspended; when the system resources are abundant, the suspended flow is continued in time. Therefore, the scheduling decision of the process and the pre-judgment of the real-time scheduling of the process can be made according to the resource information. The method solves the resource conflict among tasks while ensuring the efficient running of the program. By way of example, the resource consumption data of a task node is described in the following table 1:
| task node numbering | CPU requirements | IO demand | Memory requirements |
| A | 7 | 3 | 5 |
| B | 2 | 5 | 6 |
| C | 4 | 3 | 3 |
(Table 1)
Assuming that the current resource alert values are all 10, the following scheduling decisions are made:
1) CPU requirement of A and C incapable of coexisting exceeds a warning value
2) The memory index of A and B can not coexist exceeds the warning value
3) The sum of all indexes of the B and C coexisting port is less than 10
For example, if two running processes are going to start task nodes that cannot coexist at the same time, the control module will make scheduling decisions according to the process priorities, and the suboptimal process will be suspended.
Through the above description of the embodiments, those skilled in the art will clearly understand that the present invention may be implemented by combining software and a hardware platform, and of course, may be implemented by hardware entirely. Those skilled in the art will appreciate that various modifications and substitutions can be made to the specific embodiments of the present invention without departing from the spirit and scope of the invention. Such modifications and substitutions are intended to be included within the scope of the present invention as defined by the appended claims.
Claims (7)
1. A method for dynamically scheduling batch tasks, said tasks including a first task and a second task, the method comprising the steps of:
a first step of decomposing the first task and the second task into a plurality of task nodes, and composing the plurality of task nodes into a first flow corresponding to the first task and a second flow corresponding to the second task;
and a second step of controlling the execution of the first flow and the second flow by taking the task nodes as units according to the information of each task node, wherein the information comprises the execution state and the execution condition of the task node.
2. The method of claim 1,
the method also comprises the step of acquiring system resource information;
the second step further comprises controlling the execution of each of the task nodes according to system resource information.
3. The method of claim 1 or 2,
the first step further comprises setting priorities of the first and second flows;
the second step further includes controlling execution of the first and second flows according to the priority.
4. An apparatus for dynamically scheduling batch tasks, said tasks comprising a first task and a second task, characterized in that the apparatus comprises a setup module and a control module, wherein,
the setting module is used for decomposing the first task and the second task into a plurality of task nodes and forming the plurality of task nodes into a first flow corresponding to the first task and a second flow corresponding to the second task;
the control module is used for controlling the execution of the first flow and the second flow by taking the task nodes as units according to the information of each task node, wherein the information comprises the execution state and the execution condition of the task node.
5. The apparatus of claim 4,
the device also comprises a resource monitoring module used for acquiring system resource information;
the control module is further configured to control execution of the each task node according to system resource information.
6. The apparatus of claim 4 or 5,
the setting module is further configured to set priorities of the first and second flows;
the control module is further configured to control execution of the first and second flows according to the priority.
7. The apparatus of claim 5, wherein the task node is a minimal program unit with data integrity.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010106191413A CN102541648A (en) | 2010-12-29 | 2010-12-29 | Method and device for dynamically scheduling batch processing task |
| PCT/CN2011/002208 WO2012088766A1 (en) | 2010-12-29 | 2011-12-29 | Method and device for dynamically scheduling batch processing tasks |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010106191413A CN102541648A (en) | 2010-12-29 | 2010-12-29 | Method and device for dynamically scheduling batch processing task |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102541648A true CN102541648A (en) | 2012-07-04 |
Family
ID=46348604
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2010106191413A Pending CN102541648A (en) | 2010-12-29 | 2010-12-29 | Method and device for dynamically scheduling batch processing task |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN102541648A (en) |
| WO (1) | WO2012088766A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109117359A (en) * | 2017-06-22 | 2019-01-01 | 深圳市中兴微电子技术有限公司 | Serialize test method and device |
| CN112230572A (en) * | 2019-06-30 | 2021-01-15 | 比亚迪股份有限公司 | Integrated control chip, control method thereof, storage medium, and vehicle |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0613085A2 (en) * | 1993-02-26 | 1994-08-31 | Nippondenso Co., Ltd. | Multitask processing unit |
| CN101566957A (en) * | 2008-04-25 | 2009-10-28 | 恩益禧电子股份有限公司 | Information processing system and task execution control method |
| CN101593088A (en) * | 2008-05-28 | 2009-12-02 | 鸿富锦精密工业（深圳）有限公司 | Concentrating type print system and method |
-
2010
- 2010-12-29 CN CN2010106191413A patent/CN102541648A/en active Pending
-
2011
- 2011-12-29 WO PCT/CN2011/002208 patent/WO2012088766A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0613085A2 (en) * | 1993-02-26 | 1994-08-31 | Nippondenso Co., Ltd. | Multitask processing unit |
| CN101566957A (en) * | 2008-04-25 | 2009-10-28 | 恩益禧电子股份有限公司 | Information processing system and task execution control method |
| CN101593088A (en) * | 2008-05-28 | 2009-12-02 | 鸿富锦精密工业（深圳）有限公司 | Concentrating type print system and method |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109117359A (en) * | 2017-06-22 | 2019-01-01 | 深圳市中兴微电子技术有限公司 | Serialize test method and device |
| CN112230572A (en) * | 2019-06-30 | 2021-01-15 | 比亚迪股份有限公司 | Integrated control chip, control method thereof, storage medium, and vehicle |
| CN112230572B (en) * | 2019-06-30 | 2021-09-03 | 比亚迪股份有限公司 | Integrated control chip, control method thereof, storage medium, and vehicle |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2012088766A1 (en) | 2012-07-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108762896B (en) | Hadoop cluster-based task scheduling method and computer equipment | |
| Monot et al. | Multisource software on multicore automotive ECUs—Combining runnable sequencing with task scheduling | |
| WO2017080273A1 (en) | Task management methods and system, and computer storage medium | |
| CN109684060B (en) | Mixed scheduling method for multiple types of time-critical tasks | |
| CN105159769A (en) | Distributed job scheduling method suitable for heterogeneous computational capability cluster | |
| US10467054B2 (en) | Resource management method and system, and computer storage medium | |
| CN104598426A (en) | task scheduling method applied to a heterogeneous multi-core processor system | |
| CN102096603A (en) | Task decomposition control method in MapReduce system and scheduling node equipment | |
| CN104375893A (en) | Partition scheduling method based on two-stage operating system structure | |
| CN105389207A (en) | Adaptive data request processing method and apparatus, task scheduler and adaptive data request processing system | |
| CN102123084A (en) | Resource scheduling method and system in cloud computing operating system | |
| CN114489867A (en) | Algorithm module scheduling method, algorithm module scheduling device and readable storage medium | |
| CN114217966A (en) | Deep learning model dynamic batch processing scheduling method and system based on resource adjustment | |
| CN112506624A (en) | Timing task scheduling system, method and corresponding computer equipment and storage medium | |
| CN106874129B (en) | Method for determining process scheduling sequence of operating system and control method | |
| CN102541648A (en) | Method and device for dynamically scheduling batch processing task | |
| CN112860401A (en) | Task scheduling method and device, electronic equipment and storage medium | |
| CN109656716B (en) | Slurm job scheduling method and system | |
| CN108399096A (en) | Multi-task scheduling time sequence monitoring method and system for battery management system | |
| CN114064397A (en) | Heterogeneous data source synchronization quality monitoring method and device and electronic equipment | |
| CN109634749B (en) | Distributed unified scheduling method and device | |
| CN101615120B (en) | Dispatch analysis method of AADL model | |
| CN113656468B (en) | Task flow triggering method and device based on NIFI | |
| CN110502580A (en) | A kind of method of clustered database system resource control scheduling | |
| CN113254143A (en) | Virtual network function network element arranging and scheduling method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20120704 |