CN113268318A - Task scheduling method and distributed system - Google Patents
Task scheduling method and distributed system Download PDFInfo
- Publication number
- CN113268318A CN113268318A CN202110371640.3A CN202110371640A CN113268318A CN 113268318 A CN113268318 A CN 113268318A CN 202110371640 A CN202110371640 A CN 202110371640A CN 113268318 A CN113268318 A CN 113268318A
- Authority
- CN
- China
- Prior art keywords
- task
- execution
- trigger
- module
- tasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 230000001419 dependent effect Effects 0.000 claims abstract description 30
- 238000011144 upstream manufacturing Methods 0.000 claims abstract description 10
- 238000012795 verification Methods 0.000 claims description 4
- 238000012544 monitoring process Methods 0.000 claims description 3
- 238000001514 detection method Methods 0.000 abstract description 3
- 230000010365 information processing Effects 0.000 abstract description 2
- 238000012360 testing method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 101150098959 MON1 gene Proteins 0.000 description 2
- 101100291875 Neurospora crassa (strain ATCC 24698 / 74-OR23-1A / CBS 708.71 / DSM 1257 / FGSC 987) apg-13 gene Proteins 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/48—Indexing scheme relating to G06F9/48
- G06F2209/484—Precedence
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The invention discloses a task scheduling method and a distributed system, belonging to the technical field of information processing, wherein the method comprises the following steps: acquiring a trigger mode, wherein the trigger mode comprises the following trigger conditions or a combination thereof: an upstream task, a trigger time, and a trigger file; when the triggering condition is met, judging whether the current task has a dependent task; if yes, detecting and judging whether the dependent task is completed; if the task is finished, submitting the current task to a queue; and sequentially sending the tasks in the queue to the task execution cluster according to the priority of the tasks. The trigger conditions of the tasks are configured through various trigger modes, the detection of the dependent tasks is controlled through the trigger conditions, the times of circularly detecting the dependent operations are reduced, the resource consumption of a system is reduced, and the task scheduling efficiency is improved.
Description
Technical Field
The invention relates to the technical field of information processing, in particular to a task scheduling method and a distributed system.
Background
The scheduling execution of the big data is carried out, along with the high-speed development of mobile services, the continuous development of big data platforms and the establishment of enterprise-level data centers, various operators gather massive data, and various systems need to process the data aiming at the massive data so as to realize the creation of the maximum value of massive data assets. In actual production, when data is processed, it is impossible to include only one task (MR), which is generally a multi-task, and it is also possible to include multiple Java tasks or HDFS tasks, even shell operation tasks, etc., since Oozie is powerful and the types of supported scheduling tasks are many, including shell, MapReduce, Pig, live, sqoop, spark, etc., actual production usually uses Oozie to complete the scheduling of these tasks.
Oozie is a workflow engine-based open source framework contributed by Cloudera to Apache for workflow scheduling for distributed (Hadoop) platforms. The big data management and control platform (such as a data workshop) of the current production system utilizes Oozie scheduling to undertake various task scheduling of big data cluster tasks, wherein the number of the tasks is as many as ten thousand.
However, due to the limitation of Oozie, Oozie triggers tasks based on time conditions, one task schedules regularly, checks whether dependence conditions are met or not every several minutes, and circulates all the time until the dependence conditions are met, and the tasks are really called to run until the dependence conditions are met.
Disclosure of Invention
Aiming at the technical problems in the prior art, the invention provides a task scheduling method and a distributed system, which can be used for scheduling tasks in multiple triggering modes, thereby improving the task scheduling efficiency and saving system resources.
The invention discloses a task scheduling method, which comprises the following steps:
acquiring a trigger mode, wherein the trigger mode comprises the following trigger conditions or a combination thereof: an upstream task, a trigger time, and a trigger file;
when the triggering condition is met, judging whether the current task has a dependent task;
if yes, detecting and judging whether the dependent task is completed;
if the task is finished, submitting the current task to a queue;
and sequentially sending the tasks in the queue to the task execution cluster according to the priority of the tasks.
Preferably, the method of the present invention further comprises a method of task re-execution:
judging whether the task runs successfully or not;
and if the operation fails, generating warning information and executing the task again.
Preferably, the method for triggering the file includes:
a provider of the trigger file provides the trigger file and a control file of the trigger file;
and when the control file is monitored, judging that the upstream trigger condition is met.
Preferably, the control file includes one or a combination of the following information:
data source, file name, number of files, expected size of file, file size, expected number of rows, and file verification result.
Preferably, the trigger condition is configured by a policy, and the policy includes a policy ID and the trigger condition.
Preferably, the policy further includes an execution cluster, and the task is sent to the corresponding execution cluster for execution according to the execution cluster.
Preferably, the method of the present invention further includes a method for performing task scheduling by multiple execution clusters:
the multiple execution clusters are deployed with a cluster management platform, and execution components of Oozie are deployed in the execution clusters;
the strategy comprises an execution cluster ID, and the task is sent to a corresponding execution cluster to be executed according to the cluster ID.
Preferably, the method for sending the task to the corresponding execution cluster for execution includes:
monitoring the task and the cluster ID executed by the task;
and sending the task to a corresponding execution cluster to be executed according to the execution cluster ID.
The invention also provides a distributed system for realizing the task scheduling method, which comprises a triggering module, a dependence module, a queue module, a task sending module and an execution module,
the trigger module is used for detecting the trigger condition of the trigger mode according to the acquired trigger mode, detecting the condition that the current task depends on the task through the dependency module when the trigger condition is met, and submitting the current task to a queue through the queue module when the dependency task is completed;
the task sending module is used for sequentially sending the tasks in the queue to the task execution cluster according to the priority of the tasks;
and the execution module of the task execution cluster is used for executing the task.
Preferably, the distributed system further includes a task redo module, the task sending module is configured to monitor the tasks in the queue and send the tasks to the corresponding execution cluster for execution, and the execution module includes an execution component of Oozie;
the task redoing module judges whether the task is successfully operated; and if the operation fails, generating warning information and executing the task again.
Compared with the prior art, the invention has the beneficial effects that: the triggering conditions of the tasks are configured in various triggering modes, and the detection of the dependent tasks is controlled through the triggering conditions, so that the times of circularly detecting the dependent operations are reduced, the resource consumption of a system is reduced, and the task scheduling efficiency is improved; through the priority of the tasks, the tasks can be flexibly scheduled.
Drawings
FIG. 1 is a flowchart of a task scheduling method of the present invention;
FIG. 2 is a logical block diagram of tasks sent to execution in a corresponding execution cluster;
FIG. 3 is a logical block diagram of a distributed system of the present invention;
fig. 4 is a logic block diagram of embodiment 1.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
The invention is described in further detail below with reference to the attached drawing figures:
a method of task scheduling, as shown in fig. 1, the method comprising:
step 101: acquiring a trigger mode, wherein the trigger mode comprises the following trigger conditions or a combination thereof: an upstream task, a trigger time, and a trigger file. A plurality of trigger modes may be set, and when any trigger condition is satisfied, it is determined that the trigger condition is satisfied.
Step 102: and when the triggering condition is met, judging whether the current task has a dependent task.
If yes, go to step 103: and detecting and judging whether the dependent task is completed.
If not, go to step 104.
If the dependent task is completed, execute step 104: the current task is submitted to the queue.
If the dependent task is not completed, step 103 is executed periodically.
Step 105: and sequentially sending the tasks in the queue to the task execution cluster according to the priority of the tasks. The task execution cluster is used for executing the task.
The triggering conditions of the tasks are configured in various triggering modes, and the detection of the dependent tasks is controlled through the triggering conditions, so that the times of circularly detecting the dependent operations are reduced, the resource consumption of a system is reduced, and the task scheduling efficiency is improved; through the priority of the tasks, the tasks can be flexibly scheduled. By the method, a developer or operation and maintenance personnel can flexibly schedule which task is executed and when the task is executed.
The method of the invention also comprises a task re-execution method:
step 111: and judging whether the task runs successfully.
Step 112: and if the task is not successful, generating warning information and executing the task again. The condition or frequency of re-execution may be set, for example, after the operation fails, the tasks are re-executed at certain time intervals, and it should be noted that an upper limit of the number of times of loop execution of each task should be set to prevent the infinite repeated execution from causing waste of system resources.
In a specific embodiment, the method for executing the task again includes: adding the failed task into a failure queue; and after analyzing the warning information and eliminating failure factors, moving the failed task from the failure queue to the queue, setting high priority and performing priority execution. Therefore, in implementation, operation and maintenance personnel can conveniently execute failed tasks repeatedly, and the whole task chain is prevented from executing failure. The pressure and the cost of maintenance and operation are reduced, and the dispatching operation of the comprehensive management and control system is facilitated.
The trigger time may be implemented by developing a timer configuration function, including configuration of a timer and configuration of a timing trigger, where in a specific embodiment, the timer configuration includes: timer name, start time, end time, timing mode (simple/scheduled), number of repetitions, interval time, cycle period, whether to enable and discontinue completions, etc.
The upstream task may be an upstream dependent task, for example, the execution of task a depends on the completion of task B and task C, then task B may be set as the upstream task, i.e., a trigger condition, task C is set as the dependent task, and after task B is completed, it is detected whether task C is completed.
The file trigger is used for triggering a trigger condition of the file, and a data file provided by a file provider is used as a trigger file. In specific implementation, when a provider of the trigger file provides the trigger file, the provider of the trigger file provides a control file of the trigger file; and when the control file is monitored, judging that the upstream trigger condition is met. However, the content of the control file may be set, for example, the contents of the data source, the file name, the file size, the file expected size, the creation time, the number of files, the expected number of lines, and the file verification result may be verified, and it is determined that the trigger condition is satisfied when the verification is passed.
For example, the source data provider of the task TB _ TEST _ MON provides the trigger file TB _ TEST _ MON1.txt and also provides the control file dir.tb _ TEST _ MON1, and the task TB _ TEST _ MON satisfies the trigger condition when the control file is monitored.
The task schedule may be configured by policy: the trigger conditions or trigger modes, execution clusters, and priorities are configured. The content of the policy may include a policy ID, a policy name, a policy description, a trigger condition, an enable flag, an execution cluster, a priority, and the like. It is convenient to control the task scheduling by modifying the parameters in the policy.
The invention can adopt an execution component of Oozie as a task execution engine, and the execution component is deployed in an execution cluster. Multiple execution clusters may be deployed for task scheduling: the multiple execution clusters are deployed with a unified cluster management platform, and execution components of Oozie are deployed in the execution clusters; the strategy comprises an execution cluster ID, and the task is sent to a corresponding execution cluster to be executed according to the cluster ID. Therefore, multi-cluster distributed (hadoop) task scheduling or job scheduling is realized. But is not limited to such, other task execution engines, such as custom task execution engines, may also be employed.
Specifically, as shown in fig. 2, the method for sending the task to the corresponding execution cluster to be executed includes:
step 201: and monitoring the task and the cluster ID executed by the task. A listening process may be deployed on a host of task scheduling.
Step 202: and sending the task to a corresponding execution cluster to be executed according to the execution cluster ID.
The invention also provides a distributed system for realizing the method, which comprises a triggering module 1, a relying module 2, a queue module 3, a task sending module 4 and an execution module 5,
the triggering module 1 is used for detecting a triggering condition of the triggering mode according to the acquired triggering mode, detecting the condition that the current task depends on the task through the relying module 2 when the triggering condition is met, and submitting the current task to the queue 11 through the queue module 3 when the relying task is completed;
the task sending module 4 is configured to send the tasks in the queue 11 to the task execution cluster 12 in sequence according to the priorities of the tasks;
the execution modules 5 of the task execution cluster 12 are used to execute the tasks.
The task sending module 4 is configured to monitor the tasks in the queue 11 and send the tasks to the corresponding execution cluster 12 for execution, where the execution module includes an execution component of Oozie. The task sending module 4 may include a plurality of listening processes, each listening process is responsible for sending a task of the execution cluster, and the listening processes send the task according to the policy, so that the policy may be modified before sending the task, thereby modifying the parameter of the task.
The system of the invention can also comprise a task redoing module 6, and the task redoing module 6 judges whether the task is successfully operated; and if the operation fails, generating warning information and executing the task again.
In a specific embodiment, the triggering module 1, the relying module 2, the queue module 3 and the task sending module 4 are used as a proxy system and can be deployed on a task scheduling host; the execution module and the task redo module 6 are deployed in the execution cluster, the execution cluster may be deployed on the multiple execution hosts, the trigger module may acquire the trigger condition in the execution cluster, and the specific execution module 5 is deployed on a node of the execution cluster. After the Oozie is deployed successfully, the Oozie is set to be submitted, namely the running condition is met, namely only the task execution component of the Oozie is used, so that the resource pressure of the execution cluster is reduced; the distributed system adopts a remote scheduling method, and improves concurrency performance and system stability.
Example 1
Task description: the dependent jobs of job 1 include job 00, job 01, job 02, job 03 and job 04, that is, if job 01 is to be executed, the five jobs dependent thereon are all completed, and job 01 satisfies the execution condition.
Scheduling conditions: as shown in fig. 4, job 00 whose completion time is the latest is taken as a trigger, and job 02, job 03, and job 04 are taken as dependent tasks; when the trigger condition is met, namely job 00 is completed, job 01 is activated, and dependent tasks are periodically detected: whether or not job 02, job 03, and job 04 are completed; after all dependent tasks are completed, job 01 is placed into a queue, the priority of job 01 is set to be high, and job 01 is sent to the execution cluster A to be executed in preference to other jobs. And if the execution of the job 01 fails, generating warning information and executing the task again.
Compared with a timing triggering mode, the method shortens the time for regularly detecting the dependent tasks and improves the execution efficiency.
Example 2
Task description: the dependent jobs of job 1 include job 10, job 11, job 12, job 13, and job 14, i.e., if job 11 is to be executed, the five jobs dependent thereon are all completed, and job 11 satisfies the condition of execution, wherein job 10 generates a data file.
Scheduling conditions: as shown in fig. 4, a data file generated by job 10 is taken as a trigger, and job 12, job 13, and job 14 are taken as dependent tasks;
after the execution of the job 10 is completed, a data file and a control file are generated, when the control file is monitored, it is judged that the trigger condition is met, the job 11 is activated, and the dependent task is periodically detected: whether job 12, job 13, and job 14 are completed;
after all dependent tasks are completed, job 11 is placed in a queue, the priority of job 11 is set to high, and job 11 is sent to execution cluster a for execution in preference to other jobs. If it is determined that the execution of the job 11 fails, warning information is generated, and the task is executed again.
Compared with a timing triggering mode, the method shortens the time for regularly detecting the dependent tasks and improves the execution efficiency.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes will occur to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of task scheduling, the method comprising:
acquiring a trigger mode, wherein the trigger mode comprises the following trigger conditions or a combination thereof: an upstream task, a trigger time, and a trigger file;
when the triggering condition is met, judging whether the current task has a dependent task;
if yes, detecting and judging whether the dependent task is completed;
if the task is finished, submitting the current task to a queue;
and sequentially sending the tasks in the queue to the task execution cluster according to the priority of the tasks.
2. The method for task scheduling according to claim 1, further comprising a method for task re-execution:
judging whether the task runs successfully or not;
and if the task is not successful, generating warning information and executing the task again.
3. The method for task scheduling according to claim 1, wherein the triggering method for the trigger file comprises:
a provider of the trigger file provides the trigger file and a control file of the trigger file;
and when the control file is monitored, judging that the upstream trigger condition is met.
4. A method for task scheduling according to claim 3, wherein the control file comprises one or a combination of the following information:
data source, file name, number of files, expected size of file, file size, expected number of rows, and file verification result.
5. The method of claim 2, wherein the trigger condition is configured by a policy, and wherein the policy comprises a policy ID and the trigger condition.
6. The method of task scheduling according to claim 5, wherein the policy further comprises an execution cluster, and the task is sent to the corresponding execution cluster for execution according to the execution cluster.
7. The method for task scheduling according to claim 6, further comprising a method for task scheduling by a plurality of execution clusters:
the multiple execution clusters are deployed with a cluster management platform, and execution components of Oozie are deployed in the execution clusters;
the strategy comprises an execution cluster ID, and the task is sent to a corresponding execution cluster to be executed according to the cluster ID.
8. The method of task scheduling according to claim 7, wherein the method of sending the task to the corresponding execution cluster for execution comprises:
monitoring the task and the cluster ID executed by the task;
and sending the task to a corresponding execution cluster to be executed according to the execution cluster ID.
9. A distributed system for implementing the task scheduling method according to any one of claims 1 to 8, comprising a triggering module, a dependency module, a queue module, a task sending module and an execution module,
the trigger module is used for detecting the trigger condition of the trigger mode according to the acquired trigger mode, detecting the condition that the current task depends on the task through the dependency module when the trigger condition is met, and submitting the current task to a queue through the queue module when the dependency task is completed;
the task sending module is used for sequentially sending the tasks in the queue to the task execution cluster according to the priority of the tasks;
and the execution module of the task execution cluster is used for executing the task.
10. The distributed system according to claim 9, further comprising a task redo module, wherein the task send module is configured to listen to the tasks in the queue and send the tasks to the corresponding execution cluster for execution, and the execution module includes an execution component of Oozie;
the task redoing module judges whether the task is successfully operated; and if the operation fails, generating warning information and executing the task again.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110371640.3A CN113268318A (en) | 2021-04-07 | 2021-04-07 | Task scheduling method and distributed system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110371640.3A CN113268318A (en) | 2021-04-07 | 2021-04-07 | Task scheduling method and distributed system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113268318A true CN113268318A (en) | 2021-08-17 |
Family
ID=77228775
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110371640.3A Pending CN113268318A (en) | 2021-04-07 | 2021-04-07 | Task scheduling method and distributed system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113268318A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115421898A (en) * | 2022-11-07 | 2022-12-02 | 杭州比智科技有限公司 | Big data task scheduling management system and method based on quartz framework |
| CN115599519A (en) * | 2022-10-17 | 2023-01-13 | 安芯网盾（北京）科技有限公司（Cn） | Job scheduling method and device |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104965754A (en) * | 2015-03-31 | 2015-10-07 | 腾讯科技（深圳）有限公司 | Task scheduling method and task scheduling apparatus |
| CN105824697A (en) * | 2016-03-23 | 2016-08-03 | 浪潮通信信息系统有限公司 | Distributed multi-stage scheduling method based on queues |
| CN106293971A (en) * | 2016-08-15 | 2017-01-04 | 张家林 | A kind of method and apparatus of distributed task dispatching |
| CN106980543A (en) * | 2017-04-05 | 2017-07-25 | 福建智恒软件科技有限公司 | The distributed task dispatching method and device triggered based on event |
| US20170329632A1 (en) * | 2014-11-21 | 2017-11-16 | Sanechips Technology Co., Ltd. | Device scheduling method, task manager and storage medium |
| CN109445926A (en) * | 2018-11-09 | 2019-03-08 | 杭州玳数科技有限公司 | Data task dispatching method and data task dispatch system |
| CN110704173A (en) * | 2019-09-29 | 2020-01-17 | 深圳前海环融联易信息科技服务有限公司 | Task scheduling method, scheduling system, electronic device and computer storage medium |
| CN110888721A (en) * | 2019-10-15 | 2020-03-17 | 平安科技（深圳）有限公司 | Task scheduling method and related device |
| CN111309464A (en) * | 2020-02-12 | 2020-06-19 | 杭州涂鸦信息技术有限公司 | Task scheduling method and system |
| CN111338770A (en) * | 2020-02-12 | 2020-06-26 | 咪咕文化科技有限公司 | Task scheduling method, server and computer readable storage medium |
| WO2020186809A1 (en) * | 2019-03-19 | 2020-09-24 | 深圳壹账通智能科技有限公司 | Big data platform-based hive task scheduling method and apparatus, device, and storage medium |
| CN111950988A (en) * | 2020-08-18 | 2020-11-17 | 北京字节跳动网络技术有限公司 | Distributed workflow scheduling method and device, storage medium and electronic equipment |
| CN112286661A (en) * | 2020-10-30 | 2021-01-29 | 海通证券股份有限公司 | Task scheduling method and device, storage medium and terminal |
-
2021
- 2021-04-07 CN CN202110371640.3A patent/CN113268318A/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170329632A1 (en) * | 2014-11-21 | 2017-11-16 | Sanechips Technology Co., Ltd. | Device scheduling method, task manager and storage medium |
| CN104965754A (en) * | 2015-03-31 | 2015-10-07 | 腾讯科技（深圳）有限公司 | Task scheduling method and task scheduling apparatus |
| CN105824697A (en) * | 2016-03-23 | 2016-08-03 | 浪潮通信信息系统有限公司 | Distributed multi-stage scheduling method based on queues |
| CN106293971A (en) * | 2016-08-15 | 2017-01-04 | 张家林 | A kind of method and apparatus of distributed task dispatching |
| CN106980543A (en) * | 2017-04-05 | 2017-07-25 | 福建智恒软件科技有限公司 | The distributed task dispatching method and device triggered based on event |
| CN109445926A (en) * | 2018-11-09 | 2019-03-08 | 杭州玳数科技有限公司 | Data task dispatching method and data task dispatch system |
| WO2020186809A1 (en) * | 2019-03-19 | 2020-09-24 | 深圳壹账通智能科技有限公司 | Big data platform-based hive task scheduling method and apparatus, device, and storage medium |
| CN110704173A (en) * | 2019-09-29 | 2020-01-17 | 深圳前海环融联易信息科技服务有限公司 | Task scheduling method, scheduling system, electronic device and computer storage medium |
| CN110888721A (en) * | 2019-10-15 | 2020-03-17 | 平安科技（深圳）有限公司 | Task scheduling method and related device |
| CN111309464A (en) * | 2020-02-12 | 2020-06-19 | 杭州涂鸦信息技术有限公司 | Task scheduling method and system |
| CN111338770A (en) * | 2020-02-12 | 2020-06-26 | 咪咕文化科技有限公司 | Task scheduling method, server and computer readable storage medium |
| CN111950988A (en) * | 2020-08-18 | 2020-11-17 | 北京字节跳动网络技术有限公司 | Distributed workflow scheduling method and device, storage medium and electronic equipment |
| CN112286661A (en) * | 2020-10-30 | 2021-01-29 | 海通证券股份有限公司 | Task scheduling method and device, storage medium and terminal |
Non-Patent Citations (3)
| Title |
|---|
| APACHE SOFTWARE FOUNDATION: ""Oozie, Workflow Engine for Apache Hadoop"", Retrieved from the Internet <URL:https://oozie.apache.org/docs/5.2.1/index.html> * |
| 孙伟东;马宗民;: "基于数据依赖和触发器的简单子任务调度算法", 辽宁大学学报(自然科学版), no. 02 * |
| 黄丽达等: ""事件触发关键级提升的实时任务可调度性分析"", 《计算机研究与发展》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115599519A (en) * | 2022-10-17 | 2023-01-13 | 安芯网盾（北京）科技有限公司（Cn） | Job scheduling method and device |
| CN115421898A (en) * | 2022-11-07 | 2022-12-02 | 杭州比智科技有限公司 | Big data task scheduling management system and method based on quartz framework |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8522207B1 (en) | Systems and methods for automated centralized build/merge management | |
| CN107016480B (en) | Task scheduling method, device and system | |
| CN113569987A (en) | Model training method and device | |
| CN113268318A (en) | Task scheduling method and distributed system | |
| CN106789141B (en) | Gateway equipment fault processing method and device | |
| CN110611707B (en) | Task scheduling method and device | |
| CN108089915B (en) | Method and system for business control processing based on message queue | |
| CN105159769A (en) | Distributed job scheduling method suitable for heterogeneous computational capability cluster | |
| CN111526049B (en) | Operation and maintenance system, operation and maintenance method, electronic device and storage medium | |
| CN103365708A (en) | Method and device for scheduling tasks | |
| CN111190732A (en) | Timed task processing system and method, storage medium and electronic device | |
| US10733084B2 (en) | Early test breakage detection using presubmit runs | |
| CN112181621A (en) | Task scheduling system, method, equipment and storage medium | |
| CN112905339B (en) | Task scheduling execution method, device and system | |
| CN111147541A (en) | Node processing method, device and equipment based on parameter server and storage medium | |
| CN112817706A (en) | Distributed task scheduling system and method | |
| US10430300B2 (en) | Predictive analysis, scheduling and observation system for use with loading multiple files | |
| CN113672452A (en) | Method and system for monitoring operation of data acquisition task | |
| CN111538738B (en) | Service upgrading method and device, electronic equipment and storage medium | |
| CN112925648A (en) | Service policy issuing method and device | |
| CN111082964B (en) | Distribution method and device of configuration information | |
| CN110287159B (en) | File processing method and device | |
| CN112148420B (en) | Abnormal task processing method based on container technology, server and cloud platform | |
| CN108255515B (en) | Method and device for realizing timer service | |
| US20230297970A1 (en) | Intelligent scheduling of maintenance tasks to minimize downtime |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |