CN102981904B - A kind of method for scheduling task and system - Google Patents

A kind of method for scheduling task and system Download PDF

Info

Publication number
CN102981904B
CN102981904B CN201110259129.0A CN201110259129A CN102981904B CN 102981904 B CN102981904 B CN 102981904B CN 201110259129 A CN201110259129 A CN 201110259129A CN 102981904 B CN102981904 B CN 102981904B
Authority
CN
China
Prior art keywords
task
scheduling
rearmounted
time
previous
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201110259129.0A
Other languages
Chinese (zh)
Other versions
CN102981904A (en
Inventor
乔平
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to CN201110259129.0A priority Critical patent/CN102981904B/en
Publication of CN102981904A publication Critical patent/CN102981904A/en
Priority to HK13104705.3A priority patent/HK1177291A1/en
Application granted granted Critical
Publication of CN102981904B publication Critical patent/CN102981904B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Landscapes

  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

This application provides a kind of method for scheduling task and system, to solve the circumscribed problem of system in task scheduling.Described method includes: first extract scheduling timing time overdue task, then described task is set as rearmounted task, rely on dimension table by the task scheduling of described rearmounted task and calculate the plan execution time of previous task, and rely on dimension table from described task scheduling and read the task number of previous task, then by the execution state in the task number of previous task and the task scheduling operation result table of plan execution time lookup previous task, if running succeeded, then described rearmounted task is placed in task queue, waits that scheduling performs.Method described herein in all cases, as long as determining rearmounted task just can find previous task, determining previous task just to run succeeded and can perform rearmounted task so that task scheduling system is more perfect.Shorten scheduling time, improve dispatching efficiency, decrease error rate.

Description

A kind of method for scheduling task and system
Technical field
The application relates to journaling techniques, particularly relates to a kind of method for scheduling task and system.
Background technology
Daily record refers to affairs or the record of operation that relevant computer system occurs.In various application systems, in banking system, portal website's system, e-commerce system, in-company financial system etc. application system, all occurred in system operation will be recorded as daily record, and generate journal file, then by the parsing to journal file, it is possible to understand the whole service process of system in detail.
Need daily record to perform operations during daily record resolves, the described operation performing daily record can be regarded as a task, such as, download log can be regarded as a task, this task of download log to be performed is accomplished by scheduler task, then by system command, task being sent to operating system, operating system performs the operation of download log again.
In task scheduling, task can be divided into rearmounted task and previous task, and previous task is finished and could perform rearmounted task.The configuration of common task scheduling system is the task of dispatching the same day same day, and the rearmounted task that the same day performs can be automatically found the previous task on the same day, it is judged that the previous task on the same day is finished, then can perform the rearmounted task on the same day.But due to the limitation of system configuration, in some special cases, if previous task and rearmounted task are not performing on the same day.Such as, rearmounted task is the task on the same day, and the previous task of described rearmounted task is the task of the previous day.Now do not performing on the same day due to previous task and rearmounted task, make cannot be automatically found the previous task of the previous day when performing the rearmounted task on the same day, now it is accomplished by the previous task that manual configuration searches the previous day of rearmounted task on the described same day, checks whether the previous task of described the previous day is finished.
In sum, owing to the limitation of system configuration causes at some in particular cases, manual configuration is needed to search the previous task of rearmounted task.Therefore the time performing scheduling can be caused long, inefficient, and error rate is higher.
Summary of the invention
The application provides a kind of method for scheduling task and system, to solve the circumscribed problem of system in task scheduling.
In order to solve the problems referred to above, this application discloses a kind of method for scheduling task, including:
Extract scheduling timing time overdue task, and set extracted task as rearmounted task;
Search task scheduling corresponding to rearmounted task and rely on dimension table, and relied on the plan execution time of previous task corresponding to the described rearmounted task of dimension table calculating by the task scheduling that described rearmounted task is corresponding;
The task number reading previous task dimension table is relied on from described task scheduling, search, with the plan execution time, the task scheduling operation result table that previous task is corresponding by the task number of previous task, wherein said task scheduling operation result table have recorded the execution state of previous task;
Searching whether the execution state of previous task in described task scheduling operation result table is successfully, if running succeeded, then rearmounted task being placed in task queue, wait that scheduling performs.
Preferably, described extraction scheduling timing time overdue task, and set extracted task as after rearmounted task, also include:
The time is performed by the plan of the described rearmounted task of scheduling timing Time Calculation.
Preferably, the described task scheduling corresponding by described rearmounted task relies on dimension table and calculates the plan execution time of previous task corresponding to described rearmounted task, including:
The plan execution time of rearmounted task deducts task scheduling corresponding to rearmounted task and relies on the time migration relying on previous task in dimension table, obtains corresponding value of calculation;
Described value of calculation combining task scheduling corresponding to rearmounted task and relies on the preposition dependence timing in dimension table, the plan being calculated previous task corresponding to described rearmounted task performs the time.
Preferably, described extraction scheduling timing time overdue task, and set extracted task as before rearmounted task, also include:
Starting task scheduling, read parameter, described parameter includes thread pool number n, dispatches natural law x, interval scan time m and time zone across sky, and wherein x, n and m are positive integer;
Start scanning thread, worker thread and monitoring thread, initialization task queue;
Reference clock is adjusted according to time zone.
Preferably, described extraction scheduling timing time overdue task, including:
Scanning thread, according to scheduling timing time overdue task in extracting x days across sky scheduling natural law x, calculates and reaches in task scheduling operation result table to plan the execution time but the most off-duty task list.
Preferably, described rearmounted task is placed in task queue, after waiting that scheduling performs, also includes:
Scanning thread is by semaphore notice worker thread Detection task queue and extracts the rearmounted task in described task queue;
Worker thread detect the task lock of described rearmounted task whether be carrying out the task lock conflict of task, if not conflicting, then performing described rearmounted task, otherwise skipping described rearmounted task;
If performing described task, then before performing described rearmounted task, the parameter in displacement task run order, insert dispatching record to task scheduling operation result table, and execution state is initially is currently running;
Run the task run order in the task scheduling operation result table of described rearmounted task.
Preferably, after the task run order in the task scheduling operation result table of the described rearmounted task of described operation, also include:
Wait the return result of described task run order, if it is successful, by the execution status modifier in task scheduling operation result table for running succeeded, and return;
If running unsuccessfully, in conjunction with the failed weight scheduling parameter in task scheduling definition table, revising task run order, and resubmiting worker thread scheduling;
If the frequency of failure exceedes the failed weight scheduling times in task scheduling definition table, by the execution status modifier in task scheduling operation result table for perform failure, and return.
Preferably, described startup worker thread, including:
According to thread pool number n, concurrently start n worker thread.
Preferably, scanning thread is run according to interval scan time m, per interval m.
Preferably, after described startup monitoring thread, also include:
Scan task management and running result table, sends the corresponding order of failed tasks, overtime task and/or timing overtime task.
Accordingly, present invention also provides a kind of task scheduling system, including:
Extraction module, is used for extracting scheduling timing time overdue task, and sets extracted task as rearmounted task;
Second computing module, is relied on dimension table for searching task scheduling corresponding to rearmounted task, and is relied on the plan execution time of previous task corresponding to the described rearmounted task of dimension table calculating by the task scheduling that described rearmounted task is corresponding;
Search module, for relying on the task number reading previous task in dimension table from described task scheduling, search, with the plan execution time, the task scheduling operation result table that previous task is corresponding by the task number of previous task, wherein said task scheduling operation result table have recorded the execution state of each task;
Judge module, for searching whether the execution state of previous task in described task scheduling operation result table is successfully, if running succeeded, is then placed into rearmounted task in task queue, waits that scheduling performs.
Preferably, described system also includes:
First computing module, for performing the time by the plan of the described rearmounted task of scheduling timing Time Calculation.
Preferably, the second described computing module, including:
Subtract each other submodule, deduct, for the plan execution time of rearmounted task, the time migration relying on previous task that task scheduling corresponding to rearmounted task relies in dimension table, obtain corresponding value of calculation;
Calculating sub module, relies on preposition dependence timing in dimension table for described value of calculation combines task scheduling corresponding to rearmounted task, and the plan being calculated previous task corresponding to described rearmounted task performs the time.
Preferably, described system also includes:
Starting module, be used for starting task scheduling, read parameter, described parameter includes thread pool number n, dispatches natural law x, interval scan time m and time zone across sky, and wherein x, n and m are positive integer;
Initial module, is used for starting scanning thread, worker thread and monitoring thread, initialization task queue;
Clock adjusting module, for adjusting reference clock according to time zone.
Preferably, described extraction module, including:
Task list submodule, is used for scanning thread and dispatches scheduling timing time overdue task in natural law x extracts x days according to across sky, calculate and reach in task scheduling operation result table to plan the execution time but the most off-duty task list.
Preferably, described system also includes:
Extract task module, by semaphore notice worker thread Detection task queue and extract the task in described task queue for scanning thread;
Test assignment lock module, for worker thread detect the task lock of described rearmounted task whether be carrying out the task lock conflict of task, if not conflicting, then performing described rearmounted task, otherwise skipping described rearmounted task;
Displacement parameter module, if for performing rearmounted task, then before performing described rearmounted task, the parameter in displacement task run order, insert dispatching record to task scheduling operation result table, and execution state is initially is currently running;
Task run module, the task run order in the task scheduling operation result table running described rearmounted task;
Task rescheduling module, for waiting the return result of described task run order, if it succeeds, by the execution status modifier in task scheduling operation result table for running succeeded, and return;If running unsuccessfully, in conjunction with the failed weight scheduling parameter in task scheduling definition table, revising task run order, and resubmiting worker thread scheduling;If the frequency of failure exceedes the failed weight scheduling times in task scheduling definition table, by the execution status modifier in task scheduling operation result table for perform failure, and return.
Preferably, described startup module, according to thread pool number n, concurrently starts n worker thread;
Described scan module runs scanning thread according to interval scan time m, per interval m.
Preferably, described system also includes:
Monitoring module, for scan task management and running result table, sends the corresponding order of failed tasks, overtime task and/or timing overtime task.
Compared with prior art, the application includes advantages below:
The application provides a kind of method for scheduling task and system, first scheduling timing time overdue task is extracted, then described task is set as rearmounted task, rely on dimension table by the task scheduling of described rearmounted task and calculate the plan execution time of previous task, and rely on dimension table from described task scheduling and read the task number of previous task, then by the execution state in the task number of previous task and the task scheduling operation result table of plan execution time lookup previous task, if running succeeded, the most rearmounted task meets schedulable condition, described rearmounted task is placed in task queue, wait that scheduling performs.By the dependence between task pre-configured in relying on dimension table in task scheduling, it is determined that whether previous task runs succeeded.Method described herein in all cases, as long as determining rearmounted task just can find previous task, determining previous task just to run succeeded and can perform rearmounted task so that task scheduling system is more perfect.Shorten scheduling time, improve dispatching efficiency, decrease error rate.
Secondly, the application passes through unsuccessfully to weigh scheduling parameter, it is possible to achieve failed tasks reschedules automatically, it is to avoid leakage Trigger Problems, reduces artificial participation simultaneously, only just thinks when the frequency of failure exceedes predetermined number of times and dispatch unsuccessfully so that task scheduling speed is faster.And the application supports that history scheduling heavily processes so that can again process the daily record of some day during some special occasions, therefore dispatching patcher is more flexible.
Again, the application comprises monitoring thread, it is provided that auto-alarm function, it is notified that task leader when failed tasks and overtime task being detected, failed tasks, overtime task and timing overtime task can quickly be processed, make task scheduling system run more rapid, more perfect.
Again, task is done the function that parameter is replaced by the application support, and the commonly used time parameter of the task that is scheduled is all to perform Time Transmission by the plan of dispatching patcher.Further, the application adjusts reference clock according to time zone, it is possible to achieve change in the time zone of task, such as can be according to American time scheduler task on the main frame of China.
Again, the application uses multithreading to dispatch, and the number of thread pool and scan period can configure, and multiple threads can concurrently perform task scheduling, and therefore dispatching patcher is more flexible, and the task of mutual exclusion can not concurrently perform so that dispatching patcher is more accurate.
Accompanying drawing explanation
Fig. 1 is a kind of method for scheduling task flow chart described in the embodiment of the present application;
Fig. 2 is a kind of method for scheduling task flow chart described in the application preferred embodiment;
Fig. 3 is worker thread operation task flow chart in a kind of method for scheduling task described in the application preferred embodiment;
Fig. 4 is worker thread task rescheduling flow chart in a kind of method for scheduling task described in the application preferred embodiment;
Fig. 5 is a kind of task scheduling system structure chart described in the embodiment of the present application.
Detailed description of the invention
Understandable for enabling the above-mentioned purpose of the application, feature and advantage to become apparent from, with detailed description of the invention, the application is described in further detail below in conjunction with the accompanying drawings.
In order to solve the problem across sky scheduling, this application provides a kind of method for scheduling task and device, the circumscribed problem of system in task scheduling can be solved.
Daily record refers to affairs or the record of operation that relevant computer system occurs.By the parsing to journal file, the whole service process of system can be understood in detail.Need daily record to perform operations during daily record resolves, the described operation performing daily record can be regarded as a task, such as, download log can be regarded as a task, this task of download log to be performed is accomplished by scheduler task, then by system command, task being sent to operating system, operating system performs the operation of download log again.
Generally, a complete daily record analytic method can be divided into following step:
1) the dimension table configuration of data base being exported to file, Distributed Calculation typically can not directly read data base's dimension table, otherwise needs to calculate node installation database client at each;
2) original log to be analyzed is downloaded by modes such as http/ftp/scp/cp;
3) conventional individual or distributed computing carry out log analysis, export flat file to be put in storage;
4) flat file to be put in storage is put in storage ORACLE or GREENPLUM;
5) rebuild database index rebuild_index, insert ETL notification interface labelling.
During this daily record resolves, it is required for task is scheduling in the tasks such as download log, parsing daily record, specifically discusses method for scheduling task below in conjunction with embodiment.
In method described herein, can apply in linux system, performed by system command system between task scheduling and task, being scheduled of task can not memory-resident, and arrange: return value is 0 expression success, otherwise represents failure.And each task can configure 3 tables in method described herein:
JOBS_DIMT0 Task scheduling definition dimension table
JOBS_RELATION_DIMT0 Task scheduling relies on dimension table
JOBS_FATDT0 Task scheduling operation result table
Table 1
Above two table main definitions tasks, and the dependence between task;A table next is the operation result of task, is specifically addressed below.
Task scheduling definition dimension table is for being defined task, and concrete configuration is as follows:
Table 2
Wherein, scheduling scheme SCHEMA_ID, for one Scheduling System of unique mark, can be website rank, such as Chinese website scheduling, the scheduling of international station;Can also be business-level, such as Chinese website clickstat business, Chinese website CTR exposure business etc..The dependence of task can only in same Scheduling System effectively with mutual exclusion.Described Chinese website mainly towards the Chinese website of China, may be used for processing the business such as domestic trade, generally uses the time on the basis of Beijing time;If international station owner is towards the English website of international particularly foreign country, may be used for processing the business of the trade between the world, the time on the basis of the America eastern time can be used.
The task that task number JOB_ID can be run for mark.
JOB_NAME is only illustrative field, for describing the function of task.
Task order row JOB_COMMAND is task run order, supports the displacement of some parameters.
The order run in linux system, command parameter to be had.Such as download, it is to be appreciated which day is downloaded, the task of which hour;May be from China's server downloading task, it is also possible to from U.S.'s server downloading task, the time of China services device, and the time of U.S.'s server, because the relation in time zone is different, it is possible to be not task on the same day.Therefore when scheduler task, it is only necessary to parameter is carried out displacement and just can run all of task.
Parameter displacement configuration is as follows:
Table 3
Can also carry out the replacement of environmental variable equally, the replacement of environmental variable is convenient migration, and such as $ HOME refers to the root of user, and same catalogue need not amendment, is exactly $ HOME catalogue for order.Can shield concrete catalogue by the configuration of parameter, therefore the root for different catalogues, such as party A-subscriber is /home/app, and the root of party B-subscriber is /home/admin, performs to have only to during scheduling revise the parameter of environmental variable.Such as table 3, the commonly used time parameter of the task that is scheduled is all to transmit the plan execution time by dispatching patcher, i.e. parameter displacement is the value of calculation taking the plan execution time with time migration.
JOB_LOCK field is the task lock of a database tier, the task mutual exclusion that this Configuration Values is equal, it is impossible to concurrently run.In task scheduling, can first detect whether before operation task that task equivalent for JOB_LOCK is currently running, temporarily can not run if there is the most described task, need to wait that the task equivalent with its JOB_LOCK could be run after terminating.
TIME_CRONTAB field is the scheduling timing time, and for setting the time of task scheduling, its configuration is on all four with the time portion of the crontab instrument under Linux system, may determine that, by the scheduling timing time, the time that plan performs.The scheduling timing time is typically the time point not having exact date, and therefore the scheduling timing time can be defined as the scheduling sometime of every day.
Such as, the scheduling timing time of A task is: 0: 40 of 4000***, i.e. every day, then the plan execution time in this day of 2011-7-27 is exactly 2011-7-2700:40.
Certainly, the described therefore scheduling timing time can define certain time scheduling weekly or monthly, it is also possible to is defined as certain time scheduling of certain day, it is also possible to being defined as dispatching at set intervals, this is not limited by the application.
Definition according to the scheduling timing time, it can be seen that the task of above-mentioned task number mark is not unique.Such as, when task number is the scheduling timing time of the task of 001 0: 40 of every day, task of having task number to be 001 behind 0: 40 of yesterday can perform scheduling, and task of also having task number to be 001 behind 0: 40 of today can perform scheduling.
The busy degree of consideration system and the dependence of scheduler task, actual execution time is very likely later than the plan execution time.Concrete configuration is as follows:
Table 4
JOB_RETRY field is unsuccessfully to weigh scheduling times, the maximum times that can reschedule when collocation task scheduling failure, exceedes described number of times then mission failure, and alerts to the person liable of OPERATOR field configuration.
The task that heavily the process time-division is concrete, some tasks can directly reschedule, such as download;Some tasks must do environment cleaning, such as puts in storage.Therefore JOB_REDO_ARGS unsuccessfully weighs scheduling parameter, for identifying whether this task is the most heavily processing of task.
TIME_OFFSET field is time migration, time migration just sets when the task of definition, such as, need when resolving daily record to dispatch this task of download log, therefore, in daily record by the hour resolves, this hour processes the daily record of upper one hour, and daily record is 22 generations, the time of download log is 23 points, then shift time is just set as-1.
Method described herein carries out the scheduling of task according to the dependence between task, and the dependence between task relies on dimension table with reference to task scheduling, and concrete configuration is as follows:
Table 5
Wherein, SCHEMA_ID with JOB_ID is consistent with task scheduling definition dimension table;PRE_JOB_ID identifies previous task, is the task number of previous task;JOB_ID is task number, and the task of its mark is rearmounted task, previous task and rearmounted task and has dependence.
TIME_CRONTAB field is called preposition dependence timing, is similar to the above-mentioned scheduling timing time, and preposition dependence timing does not generally also have a time point at exact date, and the most preposition dependence timing can be defined as certain time scheduling of every day.
Certainly, described preposition dependence timing can also be defined as certain time scheduling weekly or monthly, it is also possible to is defined as certain time scheduling of certain day, it is also possible to being defined as dispatching at set intervals, this is not limited by the application.
TIME_OFFSET field is called the time migration relying on previous task, and the time migration of previous task is identical.Generally, in task scheduling, the time migration of each task can be identical, such as, in resolving in daily record by the hour, this hour processes the daily record of upper one hour.Resolving in this daily record and need to carry out the scheduling of task, usual shift time is set as-1.Certainly the time migration of each task can also differ, and this is not limited by the application.
Discussion of hereinafter illustrating judges whether rearmounted task meets the process of schedulable condition.
Also can be according to SCHEMA_ID when searching the task scheduling result table of previous task, scheduling scheme SCHEMA_ID is for one Scheduling System of unique mark, and the dependence of task can only in same Scheduling System effectively with mutual exclusion.
Even two tasks meet the dependence task across sky and can also be scheduling.
One task whether be scheduled and be scheduled after execution state, can by task scheduling operation result table understand, concrete configuration such as table 6.
JOBS_FATDT0 table is for preserving the operation result of task scheduling, it is judged that the execution state of task.It is identical that SCHEMA_ID with JOB_ID runs definition dimension table with task scheduling;TASK_ID is job identification, can be realized by the sequence in the database software of Oracle (ORACLE), and described Sequence is the Serial No. that Database Systems increase automatically according to certain rule;JOB_COMMAND is the value that JOBS_DIMT0 dimension table respective field does after parameter displacement, can directly directly run under Linux system.REAL_START_TIME with REAL_STOP_TIME field is combined weighs the time-consuming of task scheduling, also serves as the standard of task time-out;Execution state STATUS field is for identifying the dispatch state of task, and wherein 0 representative is carrying out;1 representative runs succeeded;2 representatives perform failure;INSERT_DATE is the insertion time, is the time inserted and perform task record, can be identical with REAL_START_TIME.
When i.e. task is carrying out STATUS=0, and current time deducts actual execution time started REAL_START_TIME and exceedes the task time-out time JOB_RETRY of task scheduling operation definition dimension table JOBS_DIMT0 configuration, then it is assumed that task time-out;When the i.e. tasks carrying failure of STATUS=2, if its record strip number is more than 1, the most unsuccessfully alert, for both of these case, the JOB_COMMAND order of execution can be notified by the mode such as mail, note the person liable of task.
Table 6
With reference to Fig. 1, give a kind of method for scheduling task flow chart described in the embodiment of the present application.
Step 101, extracts scheduling timing time overdue task, and sets extracted task as rearmounted task;
First extract scheduling timing time overdue task, then set being extracted of task as rearmounted task.
Wherein, the described scheduling timing time is the time for setting task scheduling, and its configuration is on all four with the time portion of the crontab instrument under Linux system.The scheduling timing time can be defined as certain time scheduling daily or weekly or monthly, it is also possible to is defined as certain time scheduling of certain day, it is also possible to being defined as dispatching at set intervals, this is not limited by the application.
Described scheduling timing time overdue task refers to meet or exceed the task of scheduling timing time.The scheduling timing time of such as B task is the 19:30 of every day, and time during scanning is 2011-7-2719:30, equal with the scheduling timing time, or, time during scanning is 2011-7-2720:30, exceedes the scheduling timing time, all it is believed that the scheduling timing time of A task expires.
Step 102, is searched task scheduling corresponding to rearmounted task and relies on dimension table, and relied on the plan execution time of previous task corresponding to the described rearmounted task of dimension table calculating by the task scheduling that described rearmounted task is corresponding;
The scheduling timing time overdue task extracted in previous step also sets extracted task as after rearmounted task, rearmounted task is had to the previous task of correspondence, previous task and rearmounted task have dependence, are configured with corresponding task scheduling according to described dependence and rely on dimension table.Search task scheduling corresponding to rearmounted task and rely on dimension table, rely on dimension table by the task scheduling that described rearmounted task is corresponding and can calculate the plan execution time of previous task corresponding to described rearmounted task.The method of the concrete plan execution time calculating previous task can be discussed in detail below.
Step 103, the task number reading previous task dimension table is relied on from described task scheduling, searched the task scheduling operation result table of previous task by the task number of previous task and plan execution time, wherein said task scheduling operation result table have recorded the execution state of each task;
The plan having obtained previous task corresponding to described rearmounted task in above-mentioned steps performs the time, then the task number reading previous task dimension table is relied on from described task scheduling, by the task number of previous task and plan execution time, the task scheduling operation result table that previous task is corresponding can be searched, when the task number in table is coincide with the task number and plan execution time planning execution time and described previous task, i.e. when the task number in table and plan execution time are identical with the task number of described previous task and plan execution time difference, it it is exactly the task scheduling operation result table of previous task.Described task scheduling operation result table have recorded the execution state of each task.
Wherein, described task number is not unique; the a certain moment that scheduling timing set of time is every day would generally be performed; such as; the scheduling timing time is 0: 40 of every day; the task of 0: 40 of 2011-7-26 it is probably, it is also possible to the task of 0: 40 of 2011-7-27 under the most same task number.But it is unique that the plan of each task performs the time, the plan execution time of the task of 0: 40 of the 2011-7-2600:40,2011-7-27 of the plan execution time of the task of 0: 40 of 2011-7-26 is 2011-7-2700:40.But the task of same plan execution time is also possible to not be same task number, it is therefore desirable to jointly search the task scheduling operation result table of described previous task according to task symbol and the plan execution time of previous task.
Step 104, whether the execution state of previous task in described task scheduling operation result table of searching is successfully, if running succeeded, then rearmounted task is placed in task queue, waits that scheduling performs.
Search the task scheduling operation result table of previous task, check the execution state in described task scheduling operation result table.If execution state is for running succeeded, then it is believed that described rearmounted task meets schedulable condition, the rearmounted task meeting schedulable condition is placed in task queue, waits that scheduling performs.
If execution state is not for running succeeded, then it is assumed that described rearmounted task is unsatisfactory for schedulable condition, it is impossible to described rearmounted task be placed in task queue, scheduling the most temporarily can not be performed.
Can also be scheduling even if task meets the task that schedulable condition is not the same day.Then the described task of meeting schedulable condition being placed in task queue, the task in task queue is daily ranked up with task priority, the most daily sorts, and natural law task priority more early is the highest, and i.e. the task priority of yesterday is higher than the task of today;Task on the same day is by the prioritization in task scheduling definition dimension table.Carry out next step operation the most again.
In sum, the application provides a kind of method for scheduling task and system, first scheduling timing time overdue task is extracted, then described task is set as rearmounted task, rely on dimension table by the task scheduling of described rearmounted task and calculate the plan execution time of previous task, then the task number reading previous task dimension table is relied on from described task scheduling, then by the execution state in the task number of previous task and the task scheduling operation result table of plan execution time lookup previous task, if running succeeded, then rearmounted task is placed in task queue, wait that scheduling performs.Method described herein in all cases, as long as determining rearmounted task just can find previous task, determining previous task just to run succeeded and can perform rearmounted task so that task scheduling system is more perfect.
Further, extract scheduling timing time overdue task, and set extracted task as after rearmounted task, also include:
The time is performed by the plan of the described rearmounted task of scheduling timing Time Calculation.
The plan that can be calculated rearmounted task by the scheduling timing time of rearmounted task performs the time.The scheduling timing time is typically the time point not having exact date, is therefore combined the exact date of scheduling timing time and execution this day of scheduling and just can calculate the plan execution time of task.
Such as, the scheduling timing time of A task is: 0: 40 of 4000***, i.e. every day, then the plan execution time in this day of 2011-7-27 is exactly 2011-7-2700:40.
Further, the described task scheduling corresponding by described rearmounted task relies on dimension table and calculates the plan execution time of previous task corresponding to described rearmounted task, including:
First, the plan execution time of rearmounted task deducts task scheduling corresponding to rearmounted task and relies on the time migration relying on previous task in dimension table, obtains corresponding value of calculation;
The plan having calculated rearmounted task in above-mentioned steps performs the time, relies on the field being provided with expression shift time in dimension table in the task scheduling that rearmounted task is corresponding, is called the shift time relying on previous task.Deduct the described shift time relying on previous task with the plan execution time of rearmounted task, be calculated corresponding difference, then carry out next step computing.
Such as, the plan execution time of above-mentioned A task is 2011-7-2700:40, and the shift time relying on previous task in the task scheduling dependence dimension table of A task is-1, represents previous hour.Therefore the plan execution time of A task deduct the difference that the shift time of described dependence previous task obtains is 2011-7-2623:40.
Secondly, described value of calculation combining task scheduling corresponding to rearmounted task and relies on preposition dependence timing in dimension table, the plan being calculated previous task corresponding to described rearmounted task performs the time.
Being configured with a field in task scheduling corresponding to rearmounted task relies on dimension table and be preposition dependence timing, similar with the above-mentioned scheduling timing time, preposition dependence timing does not generally also have a time point at exact date.Exact date in the difference calculated in above-mentioned steps is combined with described preposition dependence timing and calculates the plan execution time of previous task corresponding to described rearmounted task.
Such as, above-mentioned calculated difference is that to rely on preposition dependence timing in dimension table be 3023*** for the task scheduling of 2011-7-2623:40, A task, 23: 30 of every day, if the previous task of A task is B task, so it is 2011-7-2623:30 that the plan of B task performs the time.
In above-mentioned example, assume B task task scheduling definition dimension table the scheduling timing time be: 3023***, the task scheduling of A task relies on the preposition dependence timing in dimension table also: scheduling timing time and the described preposition dependence timing of 3023***, B task are equal.
Another situation, A task is identical with above-mentioned hypothesis, the scheduling timing time of A task is: 4000*** (0: 40 every day), it is assumed that B task is to run for each hour, then in the task scheduling definition dimension table of B task, the scheduling timing time is: 30**** (30 points of each hour, such as 1: 30,2: 30 graded).
The task scheduling of A task relies on the preposition dependence timing in dimension table still: 3023***, computational methods are identical with upper example.
But now, the scheduling timing time of the task scheduling definition dimension table of B task is: the preposition dependence timing in 30****, with the task scheduling of A task dependence dimension table is unequal.
Certainly, having dependence between described task can also be the dependence between multiple task, such as A depends on B, A also relies on C, now the task scheduling at A relies in dimension table, just having the configuration of two previous tasks having dependence with A, now the execution state of B and C is intended to run successfully, and A just meets schedulable condition and can perform.
With reference to Fig. 2, give a kind of method for scheduling task flow chart described in the application preferred embodiment.
Step 201, starts task scheduling, reads parameter, and described parameter includes thread pool number n, dispatches natural law x, interval scan time m and time zone across sky, and wherein n, x and m are positive integer;
First starting task scheduling system, read the parameter in the configuration file of task scheduling system, wherein said parameter includes thread pool number n, dispatches natural law x, interval scan time m and time zone across sky, and n, x and m are positive integer.
Described parameter can also include scheduling scheme, determine task data to be processed by scheduling scheme, after starting data base, extract corresponding data according to scheduling scheme.
Step 202, starts scanning thread, worker thread and monitoring thread, initialization task queue;
First have to start scanning thread, worker thread and monitoring thread, task queue is initialized, the task list that wherein scanning thread can be run by task queue alternately with worker thread, i.e. scanning thread will have dependence and meets the task of schedulable condition and be put into task queue in task list, task is taken out from task queue and is scheduling by worker thread, if the success of described tasks carrying, the most described task will not be put in task list again, the most constantly updating task list, detailed process can describe in the content below.
Step 203, adjusts reference clock according to time zone.
In method described herein, the time of task scheduling is all based on fiducial time rather than host time.
Generally in number of site, such as, BeiJing, China's time resolution is pressed in the above-mentioned daily record in Chinese website, and the daily record in international station was resolved by the America eastern time, and the problem that described international station there is also U.S.'s daylight-saving time.Daily record needs task scheduling in resolving, and in method described herein, can process the task in described Chinese website and international station in United Dispatching system.It is thus desirable to adjust reference clock according to time zone, have only to revise the time zone of dispatching patcher, without the time zone revising task one by one, just can solve scheduling problem interval time each, task scheduling system on the main frame of China can make task scheduling more flexible according to American time scheduler task.
Preferably, described extraction scheduling timing time overdue task, including:
According to scheduling timing time overdue task in extracting x days across sky scheduling natural law x, calculate and task scheduling operation result table reaches plan the execution time but the most off-duty task list.
First can be according to scheduling timing time overdue task in taking x days across sky scheduling natural law x, therefore the scheduling timing time task of expiring within x days can be carried out scheduling.Then look up the plan execution time that reached in task scheduling operation result table, but without bringing into operation of task.
The described task of planning the execution time that reaches is similar to above-mentioned scheduling timing time overdue task, refer to meet or exceed the task of plan execution time, such as, it it is 2 days across sky natural law, the plan execution time of A task is 2011-7-2719:30, the plan execution time of B task is 2011-7-2819:30, the time performing scheduling is 2011-7-2819:30, A task and B task are not the most run, then the plan execution time of B task just reaches, can be placed in task queue, the plan execution time of A task exceedes, it is considered as reaching the plan execution time, therefore can also be placed in task queue.
One task really starts to perform, just can generate task scheduling operation result table, the task of not starting to described in therefore run refers to that described task does not has task scheduling operation result table, task scheduling operation result table does not i.e. have any record about described task, described task is made a task list.Wherein whether task scheduling operation result table record task is scheduled, and the execution state etc. after scheduling.
It is typically the task on Automatic dispatching same day on the same day in the prior art; but there are some special circumstances; such as; some external environmental factorss such as database failure, main shut-down system maintenance suspend scheduling; if fixed a breakdown, host maintenance completed on the same day, then the task on the same day can be scheduling automatically.If but just completing after failture evacuation, host maintenance are several days, then the task within the past few days is the most all not carried out scheduling.Having some in task within the past few days is not the task on the same day, prior art cannot perform scheduling automatically, dispatch these tasks and will task be added in task list manual configuration, then configure their previous task, check whether previous task runs succeeded.
In method described herein, scheduling timing time overdue task in x can be automatically extracted, be up to plan execution time but the most off-duty task and be placed in task list.Task in task list is set to rearmounted task, whether then looks at the execution state of previous task corresponding to described rearmounted task for running succeeded, if running succeeded, described rearmounted task being placed in task queue, waits execution to be scheduled.
The application configuration by parameter, can dispatch the task of x days, it is achieved history scheduling heavily processes so that can again process the daily record of some day during some special occasions, therefore dispatching patcher is more flexible.
With reference to Fig. 3, give worker thread operation task flow chart in a kind of method for scheduling task described in the application preferred embodiment.
Preferably, described rearmounted task is placed in task queue, after waiting that scheduling performs, also includes:
Step 301, scanning thread is by semaphore notice worker thread Detection task queue and extracts the rearmounted task in described task queue;
After rearmounted task is put in task queue, scanning thread can notify worker thread Detection task queue by semaphore, after rearmounted task being detected in task queue, worker thread extracts the rearmounted task in task queue, it is the task scheduling of multithreading because described herein, so by semaphore, scanning thread can notify that all of worker thread, now idle worker thread will go Detection task queue.
Wherein, described semaphore, sometimes referred to as signal lights, is a kind of facility used under multi-thread environment, may be employed to ensure that two or more critical code section is not by concurrent invocation.
Step 302, worker thread detect the task lock of described rearmounted task whether be carrying out the task lock conflict of task, if not conflicting, then performing described rearmounted task, otherwise skipping described rearmounted task;
After taking out described rearmounted task, worker thread can be checked the task lock of described rearmounted task and be carrying out the task lock of task, the task of detecting described rearmounted task locks conflict the most phase-locked with the task of certain task being carrying out, described phase conflict can be that the value of task lock is equal, two tasks that described value is equal are mutual exclusions, it is impossible to concurrently perform scheduling.
Having a field in task scheduling result table is task lock, each task lock is configured with corresponding value, in task scheduling, after taking out described rearmounted task from task queue, can first detect the task lock of described rearmounted task, check whether that the task equivalent with its task lock is currently running, i.e. check execution state STATUS=0 in the traffic control result table of the task equivalent with its task lock, if there is then skipping described rearmounted task, the task lock being i.e. carrying out task does not locks equivalent time with the task of described rearmounted task, described rearmounted task could be run.
Such as, A task depends on B task, B task has run succeeded, then A task meets schedulable condition, A task is placed in task queue, worker thread extracts the A task in task queue, but the task lock that detection finds A task is identical with the task drop lock of C task, and C task is currently running, then A task temporarily can not be run, C task run to be waited until terminates, and A task could be run.
Step 303, if performing described rearmounted task, then before performing described rearmounted task, the parameter in displacement task run order, insert dispatching record to task scheduling operation result table, and execution state is initially is currently running;
If judging that described rearmounted task can perform through above-mentioned steps, the then parameter in displacement task run order, then in the task scheduling operation result table of described rearmounted task, insert task run record, i.e. one task is not before running, task scheduling operation result table is empty, the data of relevant field are inserted after displacement parameter, such as JOB_COMMAND, JOB_LOCK, INSERT_DATE and PLAN_START_TIME etc. after SCHEMA_ID, JOB_ID, parameter displacement, and execution state STATUS is initially 1.The parameter that task run order JOB_COMMAND is exemplified below is replaced:
Original task run order: JOB_COMMAND field configuration
$HOME/hadoop/bin/hadoopjar
$HOME/jobs/MultiCtr/HalusStreaming/halus.jar-input
CookieLog/MultiCtr/$YYYY$$MM$$DD$/$HH$-analysisTime02:00:00
-analysisPeriod3600-cmd " logUniparse-ilogUniparse.cfg-j $ SCHEME_ID $-d $ YYYY $ $ MM $ $ DD $ "-env " LOGGER=$ HOME/logLD_LIBRARY_PATH=. "
-reduceNum8-shipFiles
$HOME/jobs/MultiCtr/HalusStreaming/logUniparse/logUniparse
$HOME/jobs/MultiCtr/HalusStreaming/logUniparse/logUniparse.cfg
$HOME/jobs/MultiCtr/HalusStreaming/logUniparse/libctrdump.so
$HOME/jobs/MultiCtr/HalusStreaming/logUniparse/libctrdec.so-output
SqlldrDat/MultiCtr/$YYYY$$MM$$DD$/$HH$-config
$HOME/jobs/MultiCtr/HalusStreaming/halus.xml
Assume task scheduling timing: 102***, then the scheduling timing time of the previous day is: 2011-07-2702:10:00
Parameter after replacing it is:
/home/admin/hadoop/bin/hadoopjar
/home/admin/jobs/MultiCtr/HalusStreaming/halus.jar-input
CookieLog/MultiCtr/20110727/01-logger
/home/admin/jobs/MultiCtr/logs/20110727/halus-analysisTime02:00:00
-analysisPeriod3600-cmd″logUniparse-ilogUniparse.cfg-j26-d20110727″
-env " LOGGER=/home/admin/logLD_LIBRARY_PATH=. "-reduceNum8
-shipFiles/home/admin/jobs/MultiCtr/HalusStreaming/logUniparse/logUniparse
/home/admin/jobs/MultiCtr/HalusStreaming/logUniparse/logUniparse.cfg
/home/admin/jobs/MultiCtr/HalusStreaming/logUniparse/libctrdump.so
/home/admin/jobs/MultiCtr/HalusStreaming/logUniparse/libctrdec.so-output
SqlldrDat/MultiCtr/20110727/01-config
/home/admin/jobs/MultiCtr/HalusStreaming/halus.xml
The order run in linux system, command parameter to be had.Such as download, it is to be appreciated which day to be downloaded, the task of which hour;May be from China's server downloading task, it is also possible to from U.S.'s server downloading task, the time of China services device, and the time of U.S.'s server, because the relation in time zone is different, it is possible to be not task on the same day.Therefore when scheduler task, it is only necessary to parameter is carried out displacement and just can run all of task.
Can also carry out the replacement of environmental variable equally, the replacement of environmental variable is convenient migration, such as goes up the $ HOME in example, refers to the root of user, and same catalogue need not amendment, is exactly $ HOME catalogue for order.Concrete catalogue can be shielded by the configuration of parameter, therefore the root for different catalogues, such as party A-subscriber is /home/app, and the root of party B-subscriber is /home/admin, performing to have only to during scheduling revise the parameter of environmental variable, the configuration of management and running result table is constant.
Method described herein, supports task is done the function that parameter is replaced, and the commonly used time parameter of the task that is scheduled as above is all to transmit the plan execution time by dispatching patcher.
Step 304, runs the task run order in the task scheduling operation result table of described rearmounted task;
Run the task run order in the task scheduling operation result table of described rearmounted task by system command, described task run order is after parameter is replaced.
With reference to Fig. 4, give worker thread task rescheduling flow chart in a kind of method for scheduling task described in the application preferred embodiment.
Preferably, after the task run order in the task run scheduling result table of the described rearmounted task of described operation, also include:
Step 401, waits the return result of described task run order, if it is successful, by the execution status modifier in task scheduling operation result table for running succeeded, and return;
Wait the return result of described task run order, if returning result is successfully, by the execution status modifier in task scheduling operation result table for running succeeded, i.e. STATUS=1, and return step 401 and re-execute worker thread.
Step 402, if running unsuccessfully, in conjunction with the failed weight scheduling parameter in task scheduling definition table, revising task run order, and resubmiting worker thread scheduling;
If the return result of described task run order is failure, can be with some seconds of dormancy, in conjunction with the failed weight scheduling parameter in task scheduling definition table, revise task run order, the data deposited in Clean Up Database, then submit to worker thread to reschedule.
Certainly, some situation such as downloading task, then can directly reschedule.
Step 403, if the frequency of failure exceedes the failed weight scheduling times in task scheduling definition table, by the execution status modifier in task scheduling operation result table for perform failure, and returns.
Task can be rescheduled after tasks carrying failure, but it is absorbed in the problems such as endless loop for anti-locking system, reschedule and can not unlimited number of perform, the failed weight scheduling times during therefore if the frequency of failure exceedes task scheduling definition table, can be by the execution status modifier in task scheduling operation result table for perform failure, i.e. STATUS=2, and return step 301 and re-execute worker thread.
By above-mentioned steps, method described herein passes through unsuccessfully to weigh scheduling parameter, it is possible to achieve failed tasks reschedules automatically, avoid Lou Trigger Problems, reduce artificial participation simultaneously, only just think when the frequency of failure exceedes predetermined number of times and dispatch unsuccessfully so that task scheduling speed is faster.
Preferably, described startup worker thread, including:
According to thread pool number n, concurrently start n worker thread.
Preferably, in method described herein, run scanning thread according to interval scan time m, per interval m.
The application can apply in the system of multithreading, can concurrently start n worker thread according to thread pool number n in task scheduling, it is achieved multi-thread concurrent performs, and uses multithreading scheduling;And can be according to interval scan time m, per interval m runs scanning thread so that scheduling configuration is more flexible.
Preferably, after described startup monitoring thread, also include:
Scan task management and running result table, sends the corresponding order of failed tasks, overtime task and/or timing overtime task.
Scan task management and running result table, will perform failed task, i.e. the corresponding order of the task of STATUS=2 and person liable report and submit Surveillance center, and Surveillance center can notify person liable by modes such as mail, note, phones.
Scan task management and running result table, task by time-out, namely it is carrying out i.e. STATUS=0, and current time is poor with the actual execution time started to be exceeded, the task of the value of the task time-out time in task scheduling definition dimension table, corresponding order and the person liable of described task are reported and submitted Surveillance center, and Surveillance center can notify person liable by modes such as mail, note, phones.
Scan task management and running result table, timing overtime task is i.e. planned the difference of execution time and fiducial time, exceed the task of value of task time-out time in task scheduling definition dimension table, corresponding order and the person liable of described task are reported and submitted Surveillance center, and Surveillance center can notify person liable by modes such as mail, note, phones.
Certain interval of time, such as 15 minutes, repeated execution of steps 402.
Pass through monitoring thread, it is possible to achieve auto-alarm function, be notified that task leader when failed tasks and overtime task being detected so that failed tasks and overtime task can quickly be processed, make task scheduling system run more rapid, more perfect.
Generally daily record analysis program language has python/java/c++ etc., embodiment described herein to relate to ORACLE data base, and python is as development language in employing, but this is not limited by the application.
In currently existing scheme, crontab carries out task scheduling by performing periodically instruction, and this kind of method maintenance cost is high, does not has task to rely on, if main frame is restarted, it is also possible to there is the situation that leakage triggers;Another kind of scheme, Quartz is the job scheduling framework increased income, the most powerful mechanism is provided for carrying out job scheduling in java application, the basic function of scheduling is provided for java application program, but Quartz needs to be realized by the manual configuration of developer for the management of task abnormity and the dependence of task.
By foregoing, and understanding with the contrast of prior art, the application comprises advantages below:
The application provides a kind of method for scheduling task and system, first scheduling timing time overdue task is extracted, then described task is set as rearmounted task, rely on dimension table by the task scheduling of described rearmounted task and calculate the plan execution time of previous task, and search described task scheduling and rely on the task number that dimension table comprises previous task, then by the execution state in the task number of previous task and the task scheduling operation result table of plan execution time lookup previous task, if running succeeded, the most rearmounted task meets schedulable condition.By getting well the dependence between having configured of task in relying on dimension table in task scheduling in advance, it is determined that whether previous task runs succeeded.Method described herein in all cases, as long as determining rearmounted task just can find previous task, determining previous task just to run succeeded and can perform rearmounted task so that task scheduling system is more perfect.Shorten scheduling time, improve dispatching efficiency, decrease error rate.
Secondly, the application passes through unsuccessfully to weigh scheduling parameter, it is possible to achieve failed tasks reschedules automatically, it is to avoid leakage Trigger Problems, reduces artificial participation simultaneously, only just thinks when the frequency of failure exceedes predetermined number of times and dispatch unsuccessfully so that task scheduling speed is faster.And the application supports that history scheduling heavily processes so that can again process the daily record of some day during some special occasions, therefore dispatching patcher is more flexible.
Again, the application comprises monitoring thread, it is provided that auto-alarm function, it is notified that task leader when failed tasks and overtime task being detected, failed tasks, overtime task and timing overtime task can quickly be processed, make task scheduling system run more rapid, more perfect.
Again, task is done the function that parameter is replaced by the application support, and the commonly used time parameter of the task that is scheduled is all to perform Time Transmission by the plan of dispatching patcher.Further, the application adjusts reference clock according to time zone, it is possible to achieve change in the time zone of task, such as can be according to American time scheduler task on the main frame of China.
Again, the application uses multithreading to dispatch, and the number of thread pool and scan period can configure, and multiple threads can concurrently perform task scheduling, and therefore dispatching patcher is more flexible, and the task of mutual exclusion can not concurrently perform so that dispatching patcher is more accurate.
Being described as a example by task scheduling in daily record resolving in above-described embodiment, certain method for scheduling task described herein all can use in the task scheduling of linux system, and this is not limited by the application.
Explanation based on said method embodiment, present invention also provides corresponding task scheduling system embodiment, realizes the content described in said method embodiment.
With reference to Fig. 5, give a kind of task scheduling system structure chart described in the embodiment of the present application.
Described task scheduling system can include extraction module 103, the second computing module 105, searches module 106 and judge module 107, wherein,
Extraction module 103, is used for extracting scheduling timing time overdue task, and sets extracted task as rearmounted task;
Second computing module 105, is relied on dimension table for searching task scheduling corresponding to rearmounted task, and is relied on the plan execution time of previous task corresponding to the described rearmounted task of dimension table calculating by the task scheduling that described rearmounted task is corresponding;
Search module 106, for relying on the task number reading previous task in dimension table from described task scheduling, search, with the plan execution time, the task scheduling operation result table that previous task is corresponding by the task number of previous task, wherein said task scheduling operation result table have recorded the execution state of each task;
Judge module 107, for searching whether the execution state of previous task in described task scheduling operation result table is successfully, if running succeeded, is then placed into rearmounted task in task queue, waits that scheduling performs.
Further, described system also includes:
First computing module 104, for performing the time by the plan of the described rearmounted task of scheduling timing Time Calculation.
Further, the second described computing module 105, including:
Subtract each other submodule 1051, deduct the time migration relying on previous task in the task scheduling dependence dimension table that rearmounted task is corresponding for the plan execution time of rearmounted task, obtain corresponding value of calculation;
Calculating sub module 1052, relies on preposition dependence timing in dimension table for described value of calculation combines task scheduling corresponding to rearmounted task, and the plan being calculated previous task corresponding to described rearmounted task performs the time.
Preferably, described system also includes:
Starting module 100, be used for starting task scheduling, read parameter, described parameter includes thread pool number n, dispatches natural law x, interval scan time m and time zone across sky, and wherein x, n and m are positive integer;
Initial module 101, is used for starting scanning thread, worker thread and monitoring thread, initialization task queue;
Clock adjusting module 102, for adjusting reference clock according to time zone.
Preferably, described extraction module 103, including:
Task list submodule 1031, is used for scanning thread and dispatches scheduling timing time overdue task in natural law x extracts x days according to across sky, calculate and reach in task scheduling operation result table to plan the execution time but the most off-duty task list.
Preferably, described system also includes:
Extract task module 108, by semaphore notice worker thread Detection task queue and extract the task in described task queue for scanning thread;
Test assignment lock module 109, for worker thread detect the task lock of described rearmounted task whether be carrying out the task lock conflict of task, if not conflicting, then performing described rearmounted task, otherwise skipping described rearmounted task;
Displacement parameter module 110, when being used for performing described rearmounted task, the parameter in displacement task run order, insert dispatching record to task scheduling operation result table, and execution state is initially is currently running;
Task run module 111, the task run order in the task scheduling operation result table running described rearmounted task;
Task rescheduling module 112, for waiting the return result of described task run order, if it succeeds, by the execution status modifier in task scheduling operation result table for running succeeded, and return;If running unsuccessfully, in conjunction with the failed weight scheduling parameter in task scheduling definition table, revising task run order, and resubmiting worker thread scheduling;If the frequency of failure exceedes the failed weight scheduling times in task scheduling definition table, by the execution status modifier in task scheduling operation result table for perform failure, and return.
Preferably, described startup module 105, according to thread pool number n, concurrently start n worker thread.
Preferably, described scan module 107, runs scan module according to interval scan time m, per interval m.
Preferably, described task scheduling system, also include:
Monitoring module 113, is used for starting monitoring thread;Scan task management and running result table, sends the corresponding order of failed tasks and/or the corresponding order of overtime task.
For system embodiment, due to itself and embodiment of the method basic simlarity, so describe is fairly simple, relevant part sees the part of embodiment of the method and illustrates.
Each embodiment in this specification all uses the mode gone forward one by one to describe, and what each embodiment stressed is the difference with other embodiments, and between each embodiment, identical similar part sees mutually.
Above to a kind of method for scheduling task provided herein and system, it is described in detail, principle and the embodiment of the application are set forth by specific case used herein, and the explanation of above example is only intended to help and understands the present processes and core concept thereof;Simultaneously for one of ordinary skill in the art, according to the thought of the application, the most all will change, in sum, this specification content should not be construed as the restriction to the application.

Claims (14)

1. a method for scheduling task, it is characterised in that including:
Extracting scheduling timing time overdue task, and set extracted task as rearmounted task, described scheduling timing time overdue task refers to meet or exceed the task of scheduling timing time;
The time is performed by the plan of the described rearmounted task of scheduling timing Time Calculation;
Search task scheduling corresponding to rearmounted task and rely on dimension table, and relied on the plan execution time of previous task corresponding to the described rearmounted task of dimension table calculating by the task scheduling that described rearmounted task is corresponding;
The task number reading previous task dimension table is relied on from described task scheduling, search, with the plan execution time, the task scheduling operation result table that previous task is corresponding by the task number of previous task, wherein said task scheduling operation result table have recorded the execution state of previous task;
Searching whether the execution state of previous task in described task scheduling operation result table is successfully, if running succeeded, then rearmounted task being placed in task queue, wait that scheduling performs;
Wherein, the described task scheduling corresponding by described rearmounted task relies on dimension table and calculates the plan execution time of previous task corresponding to described rearmounted task, including:
The plan execution time of rearmounted task deducts task scheduling corresponding to rearmounted task and relies on the time migration relying on previous task in dimension table, obtains corresponding value of calculation;
Described value of calculation combining task scheduling corresponding to rearmounted task and relies on the preposition dependence timing in dimension table, the plan being calculated previous task corresponding to described rearmounted task performs the time.
Method the most according to claim 1, it is characterised in that extract scheduling timing time overdue task, and set extracted task as before rearmounted task, also include:
Starting task scheduling, read parameter, described parameter includes thread pool number n, dispatches natural law x, interval scan time m and time zone across sky, and wherein n, x and m are positive integer, and the linear module of interval scan time is unit of time;
Start scanning thread, worker thread and monitoring thread, initialization task queue;
Reference clock is adjusted according to time zone.
Method the most according to claim 2, it is characterised in that described extraction scheduling timing time overdue task, including:
Scanning thread, according to scheduling timing time overdue task in extracting x days across sky scheduling natural law x, calculates and reaches in task scheduling operation result table to plan the execution time but the most off-duty task list.
Method the most according to claim 2, it is characterised in that described rearmounted task be placed in task queue, after waiting that scheduling performs, also includes:
Scanning thread is by semaphore notice worker thread Detection task queue and extracts the rearmounted task in described task queue;
Worker thread detect the task lock of described rearmounted task whether be carrying out the task lock conflict of task, if not conflicting, then performing described rearmounted task, otherwise skipping described rearmounted task;
If performing described task, then before performing described rearmounted task, the parameter in displacement task run order, insert dispatching record to task scheduling operation result table, and execution state is initially is currently running;
Run the task run order in the task scheduling operation result table of described rearmounted task.
Method the most according to claim 4, it is characterised in that after the task run order in the task scheduling operation result table of the described rearmounted task of described operation, also include:
Wait the return result of described task run order, if it is successful, by the execution status modifier in task scheduling operation result table for running succeeded, and return;
If running unsuccessfully, in conjunction with the failed weight scheduling parameter in task scheduling definition table, revising task run order, and resubmiting worker thread scheduling;
If the frequency of failure exceedes the failed weight scheduling times in task scheduling definition table, by the execution status modifier in task scheduling operation result table for perform failure, and return.
Method the most according to claim 2, it is characterised in that described startup worker thread, including:
According to thread pool number n, concurrently start n worker thread.
Method the most according to claim 2, it is characterised in that:
Scanning thread is run according to interval scan time m, per interval m.
Method the most according to claim 2, it is characterised in that after starting monitoring thread, also include:
Scan task management and running result table, sends the corresponding order of failed tasks, overtime task and/or timing overtime task.
9. a task scheduling system, it is characterised in that including:
Extraction module, is used for extracting scheduling timing time overdue task, and sets extracted task as rearmounted task, and described scheduling timing time overdue task refers to meet or exceed the task of scheduling timing time;
First computing module, for performing the time by the plan of the described rearmounted task of scheduling timing Time Calculation;
Second computing module, is relied on dimension table for searching task scheduling corresponding to rearmounted task, and is relied on the plan execution time of previous task corresponding to the described rearmounted task of dimension table calculating by the task scheduling that described rearmounted task is corresponding;
Search module, for relying on the task number reading previous task in dimension table from described task scheduling, search, with the plan execution time, the task scheduling operation result table that previous task is corresponding by the task number of previous task, wherein said task scheduling operation result table have recorded the execution state of each task;
Judge module, for searching whether the execution state of previous task in described task scheduling operation result table is successfully, if running succeeded, is then placed into rearmounted task in task queue, waits that scheduling performs;
Wherein, the second described computing module, including:
Subtract each other submodule, deduct, for the plan execution time of rearmounted task, the time migration relying on previous task that task scheduling corresponding to rearmounted task relies in dimension table, obtain corresponding value of calculation;
Calculating sub module, relies on preposition dependence timing in dimension table for described value of calculation combines task scheduling corresponding to rearmounted task, and the plan being calculated previous task corresponding to described rearmounted task performs the time.
System the most according to claim 9, it is characterised in that also include:
Starting module, be used for starting task scheduling, read parameter, described parameter includes thread pool number n, dispatches natural law x, interval scan time m and time zone across sky, and wherein x, n and m are positive integer, and the linear module of interval scan time is unit of time;
Initial module, is used for starting scanning thread, worker thread and monitoring thread, initialization task queue;
Clock adjusting module, for adjusting reference clock according to time zone.
11. systems according to claim 10, it is characterised in that described extraction module, including:
Task list submodule, is used for scanning thread and dispatches scheduling timing time overdue task in natural law x extracts x days according to across sky, calculate and reach in task scheduling operation result table to plan the execution time but the most off-duty task list.
12. systems according to claim 10, it is characterised in that also include:
Extract task module, by semaphore notice worker thread Detection task queue and extract the task in described task queue for scanning thread;
Test assignment lock module, for worker thread detect the task lock of described rearmounted task whether be carrying out the task lock conflict of task, if not conflicting, then performing described rearmounted task, otherwise skipping described rearmounted task;
Displacement parameter module, if for performing rearmounted task, then before performing described rearmounted task, the parameter in displacement task run order, insert dispatching record to task scheduling operation result table, and execution state is initially is currently running;
Task run module, the task run order in the task scheduling operation result table running described rearmounted task;
Task rescheduling module, for waiting the return result of described task run order, if it succeeds, by the execution status modifier in task scheduling operation result table for running succeeded, and return;If running unsuccessfully, in conjunction with the failed weight scheduling parameter in task scheduling definition table, revising task run order, and resubmiting worker thread scheduling;If the frequency of failure exceedes the failed weight scheduling times in task scheduling definition table, by the execution status modifier in task scheduling operation result table for perform failure, and return.
13. systems according to claim 10, it is characterised in that:
Described startup module, according to thread pool number n, concurrently starts n worker thread;
Scan module runs scanning thread according to interval scan time m, per interval m.
14. systems according to claim 10, it is characterised in that also include:
Monitoring module, for scan task management and running result table, sends the corresponding order of failed tasks, overtime task and/or timing overtime task.
CN201110259129.0A 2011-09-02 2011-09-02 A kind of method for scheduling task and system Active CN102981904B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201110259129.0A CN102981904B (en) 2011-09-02 2011-09-02 A kind of method for scheduling task and system
HK13104705.3A HK1177291A1 (en) 2011-09-02 2013-04-18 Method and system for task dispatching

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201110259129.0A CN102981904B (en) 2011-09-02 2011-09-02 A kind of method for scheduling task and system

Publications (2)

Publication Number Publication Date
CN102981904A CN102981904A (en) 2013-03-20
CN102981904B true CN102981904B (en) 2016-08-03

Family

ID=47855962

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201110259129.0A Active CN102981904B (en) 2011-09-02 2011-09-02 A kind of method for scheduling task and system

Country Status (2)

Country Link
CN (1) CN102981904B (en)
HK (1) HK1177291A1 (en)

Families Citing this family (50)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104714838A (en) * 2013-12-12 2015-06-17 中国移动通信集团四川有限公司 Task scheduling method and device
CN103761148B (en) * 2014-01-26 2017-04-05 北京京东尚科信息技术有限公司 The control method of cluster timer-triggered scheduler task
CN103942104A (en) * 2014-04-23 2014-07-23 北京金山网络科技有限公司 Task managing method and device
CN104407847B (en) * 2014-10-29 2019-05-07 中国建设银行股份有限公司 A kind of method and device of batch processing
CN104331323A (en) * 2014-10-31 2015-02-04 北京思特奇信息技术股份有限公司 Scheduling method and system for journal work
CN105808619B (en) * 2014-12-31 2019-08-06 华为技术有限公司 Method, impact analysis computing device and the key reset apparatus that task based on impact analysis is reformed
CN104935658B (en) * 2015-06-17 2018-05-08 交通银行股份有限公司 A kind of bank's method for allocating tasks and device
CN105893122A (en) * 2015-08-25 2016-08-24 乐视网信息技术(北京)股份有限公司 Data processing operation scheduling method, device and system
CN105138666B (en) * 2015-09-06 2019-04-26 深圳市创梦天地科技有限公司 A kind of method and terminal of data processing
CN105354083B (en) * 2015-09-30 2018-12-04 北京京东尚科信息技术有限公司 A kind of method and device checking scheduler task precondition
CN105468500B (en) * 2015-11-16 2018-11-16 中国建设银行股份有限公司 timed task monitoring method and device
CN105701257B (en) * 2016-03-31 2019-05-21 北京奇虎科技有限公司 Data processing method and device
CN106095540B (en) * 2016-05-31 2020-01-03 上海萌果信息科技有限公司 Task flow processing method based on Quartz frame
CN107783843B (en) * 2016-08-31 2021-08-06 中国移动通信有限公司研究院 Cloud data processing method and device
CN108108231A (en) * 2016-11-24 2018-06-01 阿里巴巴集团控股有限公司 Generation, processing method, device, system and the electronic equipment of user's request
CN106647359B (en) * 2016-11-24 2019-02-05 哈工大机器人集团上海有限公司 A kind of dispatching method
CN106775977B (en) * 2016-12-09 2020-06-02 北京小米移动软件有限公司 Task scheduling method, device and system
CN106970834A (en) * 2017-01-20 2017-07-21 阿里巴巴集团控股有限公司 The method and device of task processing
CN106874090A (en) * 2017-01-23 2017-06-20 北京思特奇信息技术股份有限公司 Job scheduling method and system based on cloud system
CN108509257B (en) * 2017-02-28 2022-07-22 苏宁易购集团股份有限公司 Message processing method and device based on multithreading
CN107168260B (en) * 2017-06-05 2019-06-11 国网江苏省电力公司电力科学研究院 A kind of store management scheduling system of electric-power metering production
CN107368360B (en) * 2017-06-05 2021-01-26 和宇健康科技股份有限公司 Timed task execution method and timed task server
CN108287756A (en) * 2018-01-25 2018-07-17 联动优势科技有限公司 A kind of method and device of processing task
CN110162531B (en) * 2018-02-28 2021-04-16 山东衡昊信息技术有限公司 Distributed concurrent data processing task decision method
CN109725785A (en) * 2018-05-08 2019-05-07 中国平安人寿保险股份有限公司 Task execution situation method for tracing, device, equipment and readable storage medium storing program for executing
CN108958915A (en) * 2018-06-28 2018-12-07 中国建设银行股份有限公司 Timed task executes method and device
CN109308219B (en) * 2018-08-23 2021-08-10 创新先进技术有限公司 Task processing method and device and distributed computer system
CN109656693A (en) * 2018-10-25 2019-04-19 中国平安人寿保险股份有限公司 Method for scheduling task, device, server and storage medium
CN109669767B (en) * 2018-11-30 2022-09-23 河海大学 Task packaging and scheduling method and system for multi-type context dependence
CN109815087B (en) * 2019-01-07 2022-05-20 平安科技(深圳)有限公司 Task processing process monitoring method and device, computer equipment and storage medium
CN109684229B (en) * 2019-01-08 2022-03-15 杭州雅拓信息技术有限公司 Chain dependency analysis method and system of directed acyclic graph
CN109901920A (en) * 2019-01-18 2019-06-18 中国平安财产保险股份有限公司 Method for scheduling task and device, electric terminal
CN110096345B (en) * 2019-03-16 2024-04-12 平安科技(深圳)有限公司 Intelligent task scheduling method, device, equipment and storage medium
CN110134499A (en) * 2019-03-29 2019-08-16 新智云数据服务有限公司 Method for scheduling task, task scheduling system, storage medium and computer equipment
CN110109740B (en) * 2019-05-08 2021-08-10 宜人恒业科技发展(北京)有限公司 Crontab command execution method and device
CN110120833A (en) * 2019-05-14 2019-08-13 中国电子科技集团公司第五十四研究所 A kind of star ground cotasking management system
CN110362362A (en) * 2019-07-24 2019-10-22 北京明略软件系统有限公司 Method for scheduling task and device, storage medium, electronic device
CN110647387B (en) * 2019-08-29 2022-04-01 武汉天喻教育科技有限公司 Education cloud big data task scheduling method and system
CN110851245A (en) * 2019-09-24 2020-02-28 厦门网宿有限公司 Distributed asynchronous task scheduling method and electronic equipment
CN111080104A (en) * 2019-12-05 2020-04-28 成都飞机工业(集团)有限责任公司 Periodic task management scheduling method based on test flight management and control service
CN111290911A (en) * 2020-01-21 2020-06-16 上海悦易网络信息技术有限公司 Method and equipment for detecting mobile terminal task
CN111580939B (en) * 2020-04-01 2023-09-01 微梦创科网络科技(中国)有限公司 Method and device for processing transactions in hierarchical and asynchronous mode
CN111562974A (en) * 2020-04-30 2020-08-21 苏州亿歌网络科技有限公司 Method, device and equipment for acquiring offline data and storage medium
CN111881148A (en) * 2020-06-18 2020-11-03 北京三快在线科技有限公司 Object combination attribute determination method and device and electronic equipment
CN111858013A (en) * 2020-06-19 2020-10-30 中科院计算所西部高等技术研究院 Workflow job scheduling control method
CN111881143A (en) * 2020-07-30 2020-11-03 四川长虹电器股份有限公司 Method for managing ETL task
CN112328392B (en) * 2020-10-27 2024-05-28 金蝶软件(中国)有限公司 Data processing method and related equipment
CN112965793B (en) * 2021-01-21 2023-11-21 中国互联网络信息中心 Identification analysis data-oriented data warehouse task scheduling method and system
CN113312357A (en) * 2021-06-23 2021-08-27 中国农业银行股份有限公司 Data loading method, device, equipment and storage medium
CN115061807A (en) * 2022-07-27 2022-09-16 中建电子商务有限责任公司 Method for generating and scheduling execution of tasks as required

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030225470A1 (en) * 2002-01-30 2003-12-04 Demetriou John Savvas Method and system for making a product
CN101241567A (en) * 2008-03-04 2008-08-13 浪潮集团山东通用软件有限公司 Background automatic business accomplishing method based on Windows service technology
CN101853182A (en) * 2010-05-05 2010-10-06 中兴通讯股份有限公司 Task execution method and device based on database
CN102054231A (en) * 2010-12-30 2011-05-11 用友软件股份有限公司 Method and device for generating bill of material
CN102298540A (en) * 2011-06-22 2011-12-28 航天东方红卫星有限公司 Method for scheduling task with preferential comprehensive benefit

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030225470A1 (en) * 2002-01-30 2003-12-04 Demetriou John Savvas Method and system for making a product
CN101241567A (en) * 2008-03-04 2008-08-13 浪潮集团山东通用软件有限公司 Background automatic business accomplishing method based on Windows service technology
CN101853182A (en) * 2010-05-05 2010-10-06 中兴通讯股份有限公司 Task execution method and device based on database
CN102054231A (en) * 2010-12-30 2011-05-11 用友软件股份有限公司 Method and device for generating bill of material
CN102298540A (en) * 2011-06-22 2011-12-28 航天东方红卫星有限公司 Method for scheduling task with preferential comprehensive benefit

Also Published As

Publication number Publication date
CN102981904A (en) 2013-03-20
HK1177291A1 (en) 2013-08-16

Similar Documents

Publication Publication Date Title
CN102981904B (en) A kind of method for scheduling task and system
CN109684053B (en) Task scheduling method and system for big data
US9311064B1 (en) Systems and methods for automated centralized build/merge management
US7340491B2 (en) Methods and apparatus for data preservation and software distribution within an enterprise system
US8938421B2 (en) Method and a system for synchronizing data
US9002777B1 (en) Systems and methods for handling data
CN111125444A (en) Big data task scheduling management method, device, equipment and storage medium
US20190238708A1 (en) System and method for monitoring and optimizing a document capture system
CN101876943A (en) The system and method for the relation in the identification mainframe environment between a plurality of related application
CN1615484A (en) Method and system for integrated asset management
CN108491254A (en) A kind of dispatching method and device of data warehouse
US11366713B2 (en) System and method for automatically identifying and resolving computing errors
CN111400011B (en) Real-time task scheduling method, system, equipment and readable storage medium
CN108710532A (en) Across dependence implementation method, device, equipment and the storage medium of dispatching platform
CN111190892B (en) Method and device for processing abnormal data in data backfilling
US11429572B2 (en) Rules-based dataset cleaning
CN111145374A (en) Attendance management method, device and equipment and storage medium
CN109298929A (en) Timing task carrying-out time recommended method, device, equipment and storage medium
US20170076296A1 (en) Prioritizing and planning issues in automation
CN106815223B (en) Mass picture management method and device
CN101206649A (en) Multitask analysis system and method
CN112559641A (en) Processing method and device of pull chain table, readable storage medium and electronic equipment
EP3407191A1 (en) Managing job schedules
CN111353751A (en) Batch card supplementing recovery method and device
US20180046966A1 (en) System and method for analyzing and prioritizing issues for automation

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 1177291

Country of ref document: HK

C14 Grant of patent or utility model
GR01 Patent grant
REG Reference to a national code

Ref country code: HK

Ref legal event code: GR

Ref document number: 1177291

Country of ref document: HK