CN104714838A - Task scheduling method and device - Google Patents
Task scheduling method and device Download PDFInfo
- Publication number
- CN104714838A CN104714838A CN201310681601.9A CN201310681601A CN104714838A CN 104714838 A CN104714838 A CN 104714838A CN 201310681601 A CN201310681601 A CN 201310681601A CN 104714838 A CN104714838 A CN 104714838A
- Authority
- CN
- China
- Prior art keywords
- task
- priority
- scheduling
- dependence
- tasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention discloses a task scheduling method and device. The method includes the steps of analyzing task configuration files of a first task to obtain task configuration items, inquiring dependent conditions for completing the first task according to the task configuration items, and scheduling the first task according to the dependent conditions for completing the first task.
Description
Technical field
The present invention relates to technical field of data processing, particularly relate to a kind of method for scheduling task and device.
Background technology
Along with the development of infotech, quantity of information is explosive growth, from mass data, how to excavate valuable information rapidly become and become more and more important, large data system is such data processing and service platform just, as the platform providing data calculation services, usually assume responsibility for hundreds and thousands of even more task, data handling procedure generally involves complicated flow process, once complete data processing task generally includes multiple subtask, how effectively to manage and control well these tasks, allow them cooperate in harmony, have influence on large data system can realize one of its importance factor be worth.
The principal feature of large data system comprises:
(1) data volume is huge: system has the data volume that some TB even go up PB usually;
(2) task substantial amounts: as data service platform, receives the Processing tasks that user submits at any time;
(3) task is usually centered by data processing: the major responsibility of task be data cleaned, transform, analyze, excavation etc., form a data processing stream between each task;
(4) relation between task is set up by data usually.
At present, task scheduling mode generally adopts the principle of control flow check, namely carrys out the execution flow process of control task according to task status, so-called task status, refers to the operation result situation of task, i.e. success or failure.Task scheduling is carried out according to such logic usually: if task execution plan comprises A, B, C tri-tasks, the dependence of three is: C > B > A, namely can C start the running status whether success depending on B, can B start, and depends on that whether the running status of A is successful.
In prior art, what generally adopt is task scheduling mode based on control flow check and task status, which is when number of tasks is less, be easy to realize, but when task amount is very huge time, especially, time the dependence between task is very complicated, will be a very large challenge to the maintenance of task nexus.On the other hand, if in the task chain that A, B, C are formed, if A task produces multiple Output rusults, B will use wherein a certain partial output results, certain time interval is there is between these Output rusults, if the resource that task A relies at moment t1 output B, complete in t2 moment A task, so under this scheduling mechanism, task B Earliest Starting Time is that t2---namely task A all completes, for B, its resource relied on just existed in the t1 moment, then (t2-t1) is task B the idle waiting time.
In sum, the shortcoming of existing scheme is mainly reflected in the following aspects:
1, dispatching system must be safeguarded the relation between task, and what which increase dispatching system realizes difficulty and operating cost;
2, there is the idle waiting time between task, the delay phenomenon of waste in resource and task can be caused.
Summary of the invention
In order to solve task scheduling complexity in large data system and there is the technical matters postponed, the present invention proposes a kind of method for scheduling task and device.
One aspect of the present invention, provides a kind of method for scheduling task, comprising:
Resolve from the task configuration file of first task and obtain task configuration item;
The dependence condition that described first task completes is inquired about according to described task configuration item;
According to first task described in the dependence conditional scheduling that described first task completes.
Another aspect of the present invention, provides a kind of task scheduling apparatus, comprising:
Parsing module, obtains task configuration item for resolving from the task configuration file of first task;
Enquiry module, for inquiring about the dependence condition that described first task completes according to described task configuration item;
Scheduler module, for first task described in the dependence conditional scheduling that completes according to described first task.Method for scheduling task of the present invention and device, by judging the dependence condition of tasks carrying, carrying out operation dispatching to task, the relation between Maintenance free task, reducing the complexity of system, increase system efficiency; The management and running of extensive task group are achieved; At utmost reduce the task idles stand-by period, improve the efficiency of system.
Accompanying drawing explanation
Fig. 1 is the process flow diagram of method for scheduling task embodiment of the present invention;
Fig. 2 is the process flow diagram of another embodiment of method for scheduling task of the present invention;
Fig. 3 is the process flow diagram of a method for scheduling task of the present invention embodiment again;
Fig. 4 is the structural drawing of task scheduling apparatus embodiment of the present invention.
Embodiment
The present invention adopts the thinking of carrying out task scheduling based on data mode.Data mode, namely the existence (whether ready) of data, is considered as data processing unit by task, the data of a data that concern task relies on and output, does not pay close attention to the relation between the state of task and task.Whether the unique basis for estimation whether starting a task is exactly the data that rely on of this task ready.Below in conjunction with accompanying drawing, the present invention is described in detail.
As shown in Figure 1, the invention provides a kind of method for scheduling task embodiment, comprise the following steps:
Step 101, resolves and obtains task configuration item from the task configuration file of first task;
Step 102, according to the dependence condition that task configuration item inquiry first task completes;
Step 103, according to the dependence conditional scheduling first task that first task completes.
In the present embodiment, by judging the dependence condition of tasks carrying, operation dispatching being carried out to task, the relation between Maintenance free task, reducing the complexity of system, increase system efficiency; The management and running of extensive task group are achieved; At utmost reduce the task idles stand-by period, improve the efficiency of system.
In practical operation, in step 101, parsing task configuration item from the task configuration file (JobConfig) of task (Job), and add new task queue (NewTaskList) after being assembled into executable mission script (new task Task).
In step 103, all tasks in poll NewTaskList, move to ready for dependence condition task and prepare in queue (ToRunList) from new task queue, until new task queue is empty.
Such as: current new task is T1, then according to the dependence task title T2 of T1, complete queue from task and inquire about relied on task, Query Result in two kinds of situation:
(1) if the T2 obtained is for empty, represent that T2 does not also perform, then abandon the condition judgment of T1, proceed to next new task condition judgment;
(2) if T2 is not empty, the completion status of T2 is judged:
If completion status is " failure " or " abandoning ", then abandon the condition judgment of T1, and add queue (FinisedList) after the completion status of T1 is set to " abandoning ";
If state is " success ", then judge the dependence condition of T1, if condition meets, then task is added ToRunList, otherwise, proceed to the judgement of next new task.
When the dependence condition that first task completes comprises the second task, according to the node state scheduling first task of the second task, specifically comprise:
When the state of the second task is not for performing, do not perform first task;
When the state of the second task is for completing unsuccessfully, do not perform first task;
When the state of the second task is for completing successfully, judging other dependence conditions that first task completes, when other dependence conditions meet, performing first task.
In the present embodiment, when first task complete the completion status needing to depend on the second task time, the state of the second task is judged, only when the second task completes, just first task is joined preparation queue, prepare to perform first task.Like this, in large data system, the relation between Maintenance free task, starting unique basis for estimation of a task is exactly whether the data that rely on of this task are ready, reduces the complexity of system, increase system efficiency; The management and running of extensive task group are achieved; At utmost reduce the task idles stand-by period, improve the efficiency of system.
Preferably, as shown in Figure 2, the method also comprises:
Step 201, the execution result of monitoring first task;
Step 202, after first task completes, the status indicator arranging first task, for completing, comprises status indicator in task configuration item.
In task family (JobFamily), each task is likely the dependence condition that other tasks complete, and therefore monitors the execution result of each task, when task completes, the status indicator of amendment task, to judge with this task as whether other tasks relying on condition perform.Like this, for the method for above-mentioned task scheduling provides support, reduce the complexity of system, increase system efficiency, the management and running of extensive task group are achieved, at utmost reduce the task idles stand-by period, improve the efficiency of system.
Meanwhile, judge whether all members in the JobFamily of current task place all complete according to status indicator, if all completed, then show that this Job completes, program by result feedback to external program.
Preferably, as shown in Figure 3, the method also comprises:
Step 301, obtains the priority of priority as first task of the task family at first task place, and obtains the job invocation time of first task according to task configuration item;
Step 302, according to dependence condition, the priority of first task and the job invocation time scheduling first task of first task that first task completes.
Wherein, step 302 specifically comprises:
When the dependence condition of first task and other tasks is all satisfied, and the priority of first task higher than other tasks time, priority scheduling first task;
When the dependence condition of first task and other tasks is all satisfied, when the priority of first task is identical with other tasks, and the job invocation time of first task prior to other tasks time, priority scheduling first task;
When the priority of first task is identical with other tasks, the dependence condition of first task meets and the dependence condition of other tasks does not meet time, priority scheduling first task.
As shown in table 1 below, rely on the execution sequence impact on task of condition, task priority and these three conditions of job invocation time as follows:
Table 1
(priority is one, two, three from high to low ...)
As can be seen from Table 1, when the priority of two tasks is identical and dependence condition all meets, submission time task formerly of preferentially executing the task.When priority is different, but the dependence condition of one of them task does not meet, then no matter the job invocation time who formerly, preferential performing dependence condition and having met of task.When the priority of two tasks is different and the condition of dependence is all satisfied, then no matter the job invocation time who formerly, the task that preferential execution priority is higher.
In the present embodiment, except dependence condition, also determine whether the some tasks of priority scheduling according to task priority and job invocation time further.In large data system, the quantity of task is very huge, need to manage the execution sequence of task, tasks carrying order is determined according to these three conditions of dependence condition, task priority and job invocation time, make the dispatching sequence of task more reasonable, tasks carrying efficiency is higher, reduces task waiting time further, improves the efficiency of system.
On the other hand, the present invention also provides a kind of task scheduling apparatus 40, comprising:
Parsing module 41, obtains task configuration item for resolving from the task configuration file of first task;
Enquiry module 42, for inquiring about the dependence condition that described first task completes according to described task configuration item;
Scheduler module 43, for first task described in the dependence conditional scheduling that completes according to described first task.
Preferably, when the dependence condition that described first task completes comprises the second task, described scheduler module 43, for first task according to the node state scheduling of described second task, when the state of described second task is not for performing, does not perform described first task; When the state of described second task is for completing unsuccessfully, do not perform described first task; When the state of described second task is for completing successfully, judging other dependence conditions that described first task completes, when other dependence conditions meet, performing described first task.
Preferably, this device 40 also comprises:
Monitoring module 44, for monitoring the execution result of described first task;
Module 45 is set, for after described first task completes, the status indicator of described first task is set for completing, in described task configuration item, comprises described status indicator.
Preferably, this device 40 also comprises:
Acquisition module 46, for obtaining the priority of priority as described first task of the task family at described first task place according to described task configuration item, and obtains the job invocation time of described first task;
Scheduler module 43, for first task described in the priority of the dependence condition completed according to described first task, described first task and the job invocation time scheduling of described first task.
Preferably, scheduler module 43, all satisfied for the dependence condition when described first task and other tasks, and the priority of described first task higher than other tasks time, first task described in priority scheduling; When the dependence condition of described first task and other tasks is all satisfied, when the priority of described first task is identical with other tasks, and the job invocation time of described first task prior to other tasks time, first task described in priority scheduling; When the priority of described first task is identical with other tasks, the dependence condition of described first task meets and the dependence condition of other tasks does not meet time, first task described in priority scheduling.
The task scheduling apparatus of the present embodiment, by judging the dependence condition of tasks carrying, carrying out operation dispatching to task, the relation between Maintenance free task, reducing the complexity of system, increase system efficiency; The management and running of extensive task group are achieved; At utmost reduce the task idles stand-by period, improve the efficiency of system.
It is noted that above embodiment is only in order to illustrate the present invention and unrestricted, the present invention is also not limited in above-mentioned citing, and all do not depart from technical scheme and the improvement thereof of the spirit and scope of the present invention, and it all should be encompassed in right of the present invention.
Claims (10)
1. a method for scheduling task, is characterized in that, comprising:
Resolve from the task configuration file of first task and obtain task configuration item;
The dependence condition that described first task completes is inquired about according to described task configuration item;
According to first task described in the dependence conditional scheduling that described first task completes.
2. method according to claim 1, is characterized in that, when the dependence condition that described first task completes comprises the second task, first task according to the node state scheduling of described second task, comprising:
When the state of described second task is not for performing, do not perform described first task;
When the state of described second task is for completing unsuccessfully, do not perform described first task;
When the state of described second task is for completing successfully, judging other dependence conditions that described first task completes, when other dependence conditions meet, performing described first task.
3. method according to claim 2, is characterized in that, described method also comprises:
Monitor the execution result of described first task;
After described first task completes, the status indicator of described first task being set for completing, in described task configuration item, comprising described status indicator.
4. method according to claim 2, is characterized in that, described method also comprises:
Obtain the priority of priority as described first task of the task family at described first task place according to described task configuration item, and obtain the job invocation time of described first task;
According to first task described in dependence condition, the priority of described first task and the job invocation time scheduling of described first task that described first task completes.
5. method according to claim 4, is characterized in that, comprises according to first task described in dependence condition, the priority of described first task and the job invocation time scheduling of described first task that described first task completes:
When the dependence condition of described first task and other tasks is all satisfied, and the priority of described first task higher than other tasks time, first task described in priority scheduling;
When the dependence condition of described first task and other tasks is all satisfied, when the priority of described first task is identical with other tasks, and the job invocation time of described first task prior to other tasks time, first task described in priority scheduling;
When the priority of described first task is identical with other tasks, the dependence condition of described first task meets and the dependence condition of other tasks does not meet time, first task described in priority scheduling.
6. a task scheduling apparatus, is characterized in that, comprising:
Parsing module, obtains task configuration item for resolving from the task configuration file of first task;
Enquiry module, for inquiring about the dependence condition that described first task completes according to described task configuration item;
Scheduler module, for first task described in the dependence conditional scheduling that completes according to described first task.
7. device according to claim 6, it is characterized in that, when the dependence condition that described first task completes comprises the second task, described scheduler module, for first task according to the node state scheduling of described second task, when the state of described second task is not for performing, do not perform described first task; When the state of described second task is for completing unsuccessfully, do not perform described first task; When the state of described second task is for completing successfully, judging other dependence conditions that described first task completes, when other dependence conditions meet, performing described first task.
8. device according to claim 7, is characterized in that, described device also comprises:
Monitoring module, for monitoring the execution result of described first task;
Module is set, for after described first task completes, the status indicator of described first task is set for completing, in described task configuration item, comprises described status indicator.
9. device according to claim 7, is characterized in that, described device also comprises:
Acquisition module, for obtaining the priority of priority as described first task of the task family at described first task place according to described task configuration item, and obtains the job invocation time of described first task;
Described scheduler module, for first task described in the priority of the dependence condition completed according to described first task, described first task and the job invocation time scheduling of described first task.
10. device according to claim 9, is characterized in that, described scheduler module, all satisfied for the dependence condition when described first task and other tasks, and the priority of described first task higher than other tasks time, first task described in priority scheduling; When the dependence condition of described first task and other tasks is all satisfied, when the priority of described first task is identical with other tasks, and the job invocation time of described first task prior to other tasks time, first task described in priority scheduling; When the priority of described first task is identical with other tasks, the dependence condition of described first task meets and the dependence condition of other tasks does not meet time, first task described in priority scheduling.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310681601.9A CN104714838A (en) | 2013-12-12 | 2013-12-12 | Task scheduling method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310681601.9A CN104714838A (en) | 2013-12-12 | 2013-12-12 | Task scheduling method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104714838A true CN104714838A (en) | 2015-06-17 |
Family
ID=53414207
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310681601.9A Pending CN104714838A (en) | 2013-12-12 | 2013-12-12 | Task scheduling method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104714838A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105045832A (en) * | 2015-06-30 | 2015-11-11 | 北京奇艺世纪科技有限公司 | Data acquisition method and apparatus |
| CN106293920A (en) * | 2016-08-15 | 2017-01-04 | 北京票之家科技有限公司 | Method for scheduling task and device |
| CN106814993A (en) * | 2015-12-01 | 2017-06-09 | 广州神马移动信息科技有限公司 | The method for determining the task scheduling time is, the method and apparatus for determining task execution time |
| CN106844055A (en) * | 2017-01-25 | 2017-06-13 | 北京百分点信息科技有限公司 | A kind of execution method and apparatus of task |
| CN106970834A (en) * | 2017-01-20 | 2017-07-21 | 阿里巴巴集团控股有限公司 | The method and device of task processing |
| CN109542616A (en) * | 2018-10-26 | 2019-03-29 | 深圳点猫科技有限公司 | The dispatching method and electronic equipment that data calculate between a kind of multinode of educational system |
| CN110134499A (en) * | 2019-03-29 | 2019-08-16 | 新智云数据服务有限公司 | Method for scheduling task, task scheduling system, storage medium and computer equipment |
| CN110837412A (en) * | 2019-11-15 | 2020-02-25 | 中国建设银行股份有限公司 | Method, device, equipment and storage medium for judging operation ready state |
| CN111090665A (en) * | 2019-11-15 | 2020-05-01 | 广东数果科技有限公司 | Data task scheduling method and scheduling system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI221250B (en) * | 2002-03-07 | 2004-09-21 | Toshiba Corp | Multi-processor system |
| WO2010148139A2 (en) * | 2009-06-16 | 2010-12-23 | Applied Materials, Inc. | Counter and timer constraints |
| US20100333109A1 (en) * | 2009-06-30 | 2010-12-30 | Sap Ag | System and method for ordering tasks with complex interrelationships |
| CN102521056A (en) * | 2011-12-28 | 2012-06-27 | 用友软件股份有限公司 | Task allocation device and task allocation method |
| CN102981904A (en) * | 2011-09-02 | 2013-03-20 | 阿里巴巴集团控股有限公司 | Task scheduling method and system |
-
2013
- 2013-12-12 CN CN201310681601.9A patent/CN104714838A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI221250B (en) * | 2002-03-07 | 2004-09-21 | Toshiba Corp | Multi-processor system |
| WO2010148139A2 (en) * | 2009-06-16 | 2010-12-23 | Applied Materials, Inc. | Counter and timer constraints |
| US20100333109A1 (en) * | 2009-06-30 | 2010-12-30 | Sap Ag | System and method for ordering tasks with complex interrelationships |
| CN102981904A (en) * | 2011-09-02 | 2013-03-20 | 阿里巴巴集团控股有限公司 | Task scheduling method and system |
| CN102521056A (en) * | 2011-12-28 | 2012-06-27 | 用友软件股份有限公司 | Task allocation device and task allocation method |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105045832A (en) * | 2015-06-30 | 2015-11-11 | 北京奇艺世纪科技有限公司 | Data acquisition method and apparatus |
| CN105045832B (en) * | 2015-06-30 | 2018-08-07 | 北京奇艺世纪科技有限公司 | A kind of collecting method and device |
| CN106814993A (en) * | 2015-12-01 | 2017-06-09 | 广州神马移动信息科技有限公司 | The method for determining the task scheduling time is, the method and apparatus for determining task execution time |
| CN106814993B (en) * | 2015-12-01 | 2019-04-12 | 广州神马移动信息科技有限公司 | It determines the method for task schedule time, determine the method and apparatus of task execution time |
| CN106293920A (en) * | 2016-08-15 | 2017-01-04 | 北京票之家科技有限公司 | Method for scheduling task and device |
| CN106970834A (en) * | 2017-01-20 | 2017-07-21 | 阿里巴巴集团控股有限公司 | The method and device of task processing |
| CN106844055A (en) * | 2017-01-25 | 2017-06-13 | 北京百分点信息科技有限公司 | A kind of execution method and apparatus of task |
| CN106844055B (en) * | 2017-01-25 | 2020-02-28 | 北京百分点信息科技有限公司 | Task execution method and device |
| CN109542616A (en) * | 2018-10-26 | 2019-03-29 | 深圳点猫科技有限公司 | The dispatching method and electronic equipment that data calculate between a kind of multinode of educational system |
| CN110134499A (en) * | 2019-03-29 | 2019-08-16 | 新智云数据服务有限公司 | Method for scheduling task, task scheduling system, storage medium and computer equipment |
| CN110837412A (en) * | 2019-11-15 | 2020-02-25 | 中国建设银行股份有限公司 | Method, device, equipment and storage medium for judging operation ready state |
| CN111090665A (en) * | 2019-11-15 | 2020-05-01 | 广东数果科技有限公司 | Data task scheduling method and scheduling system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104714838A (en) | Task scheduling method and device | |
| US7331048B2 (en) | Backfill scheduling of applications based on data of the applications | |
| Grover et al. | Data Ingestion in AsterixDB. | |
| WO2014187412A1 (en) | Method and apparatus for controlling message processing thread | |
| US8589929B2 (en) | System to provide regular and green computing services | |
| CN112148455B (en) | Task processing method, device and medium | |
| US20180248934A1 (en) | Method and System for a Scheduled Map Executor | |
| CN103136056A (en) | Cloud computing platform scheduling method | |
| CN106293919A (en) | The built-in tasks dispatching device of a kind of Time Triggered and method | |
| CN104252381A (en) | Method and equipment for scheduling ETL (Extraction-Transform-Loading) task | |
| CN106933736B (en) | Method and system for continuously integrating projects | |
| CN101262498A (en) | A distributed message call method and device | |
| US10083066B2 (en) | Processing data by using simultaneous multithreading | |
| Wang et al. | FLOWPROPHET: Generic and accurate traffic prediction for data-parallel cluster computing | |
| CN104391705A (en) | Distributed automated testing framework applied to high availability cluster software | |
| CN108958789A (en) | A kind of parallel streaming calculation method, electronic equipment, storage medium and system | |
| CN101576831A (en) | Distributed calculating system and realization method | |
| EP3069272B1 (en) | Managing job status | |
| US8812578B2 (en) | Establishing future start times for jobs to be executed in a multi-cluster environment | |
| CN101495978B (en) | Reduction of message flow between bus-connected consumers and producers | |
| CN103631594B (en) | Asynchronous scheduling method and asynchronous scheduling system for general flow | |
| CN104346220A (en) | Task scheduling method and system | |
| CN102591714B (en) | Process calling method, system and application server | |
| CN112099937A (en) | Resource management method and device | |
| CN101308464B (en) | Modularization based mobile proxy grid operation migration method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150617 |
|
| RJ01 | Rejection of invention patent application after publication |