CN113821322A - Loosely-coupled distributed workflow coordination system and method - Google Patents
Loosely-coupled distributed workflow coordination system and method Download PDFInfo
- Publication number
- CN113821322A CN113821322A CN202111061941.2A CN202111061941A CN113821322A CN 113821322 A CN113821322 A CN 113821322A CN 202111061941 A CN202111061941 A CN 202111061941A CN 113821322 A CN113821322 A CN 113821322A
- Authority
- CN
- China
- Prior art keywords
- workflow
- task
- distributed
- coordinator
- worker
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 19
- 238000012545 processing Methods 0.000 claims abstract description 23
- 239000010453 quartz Substances 0.000 claims abstract description 13
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N silicon dioxide Inorganic materials O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 claims abstract description 13
- 230000002085 persistent effect Effects 0.000 claims abstract description 6
- 230000006870 function Effects 0.000 claims description 9
- 238000007726 management method Methods 0.000 claims description 4
- 230000001419 dependent effect Effects 0.000 claims description 3
- 238000012913 prioritisation Methods 0.000 claims description 3
- 239000012141 concentrate Substances 0.000 abstract 1
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000012163 sequencing technique Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 230000002688 persistence Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000006996 mental state Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/48—Indexing scheme relating to G06F9/48
- G06F2209/484—Precedence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5011—Pool
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5021—Priority
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a loosely-coupled distributed workflow coordination system and a loosely-coupled distributed workflow coordination method.A user defines, uploads, operates and maintains a workflow and the like by calling an interface service API; the distributed workflow Coordinator (Coordinator) schedules the workflow in a timing manner and adds the workflow to a workflow dispatching distributed Message Queue (MQ) through an integrated distributed timing engine (Quartz), receives the workflow and processes the task dependency relationship of the workflow, and adds a service type task to be executed after coordination to the task dispatching distributed Message Queue (MQ); the distributed task executor Worker receives and executes each business type task from the task dispatching distributed message queue MQ, and the task execution result is called back to the distributed workflow Coordinator through the task call-back distributed message queue MQ; and finally, the Coordinator stores the task execution result in a database in a persistent mode for feeding back the user. The invention concentrates the Coordinator on logic coordination processing, ensures the sufficient decoupling of the workflow coordination processing and the task execution, and improves the throughput, expansibility and flexibility of the system.
Description
Technical Field
The invention relates to the technical field of big data development and processing, in particular to a loosely-coupled distributed workflow coordination system and a loosely-coupled distributed workflow coordination method.
Background
In a data development project, a plurality of workflows (tasks are taken as nodes and DAGs of dependency relationships are satisfied) are usually required to be formulated, and each workflow is periodically executed according to corresponding timing (Cron expressions); such services are typically implemented through a workflow scheduling system.
Conventional workflow scheduling systems are generally composed of: API interface service, Manager workflow Manager, Worker task executor, RDBMS data storage.
API interface service: the system is responsible for the creation of workflow and task thereof, the creation of timing and the on-line of workflow timing;
manager: the system is responsible for timing execution of a workflow, conversion of a workflow DAG into a task linear queue, task queue execution context management, task assignment to a corresponding worker, task execution result callback monitoring, task state judgment and subsequent task assignment;
a Worker task processor: and is responsible for executing various business tasks.
The RDBMS database stores: the system is responsible for storing the metadata of the workflow and issuing the queue of the workflow;
the disadvantages of the conventional architecture are:
the Manager does not support priority sequencing when processing the workflow and the task queue thereof, the linear sequence of the workflow or the task without clear precedence relationship is random, and the scheduling is not accurate enough;
the coupling degree of the Manager and the Worker is high, and the system expansion and maintenance are not facilitated;
the Manager memory manages a large number of workflow instances and task team contexts thereof, so that the resource consumption is high, and the system throughput is limited; in addition, because the workflow context is cached inside the Manager instance, one workflow instance can only be completed inside one Manager instance, and the distributed processing granularity of the Manager is limited to the workflow instance.
Disclosure of Invention
The invention aims to provide a loosely-coupled distributed workflow coordination method and system aiming at the defects of the prior art, wherein a Coordinator is concentrated in logic coordination processing, the workflow coordination processing and task execution are fully decoupled, and the throughput, expansibility and flexibility of the system are improved.
The purpose of the invention is realized by the following technical scheme: a loosely-coupled distributed workflow coordination system, the system comprising: the system comprises an interface service API, a distributed workflow Coordinator, a distributed task executor Worker, a high-availability Cache, a distributed message queue MQ, a database RDBMS and a distributed coordination service Zookeeper.
The interface service API monitors the service conditions of a Coordinator of the distributed workflow and a Worker of a distributed task executor through a Zookeeper;
the distributed message queue MQ comprises a workflow dispatch distributed message queue MQ (workflow-MQ), a task dispatch distributed message queue MQ (task-dispatch-worker group-MQ) and a task callback distributed message queue MQ (task-callback-MQ);
the distributed workflow Coordinator performs the following functions: integrating a distributed timing engine Quartz, scheduling workflow at regular time and sending the workflow to a workflow-MQ; receiving the workflow from the workflow-MQ for processing; processing a DAG topology and prioritization of workflows to task queues; creating workflow instances and task instances thereof, and persistently storing data of state update to a database; caching the context of the workflow and the task queue thereof into a high available cache redis; the service type task is distributed to a distributed task executor Worker through a task-dispatch-Workergroup-MQ, and the logic type task is directly executed on a Coordinator; the task instance execution result is obtained through a task-callback-MQ; evaluating the utilization rates of a host memory, computing resources and a thread pool so as to reasonably receive the workflow;
the distributed task executor Worker is responsible for receiving, executing and calling back all service type tasks; and evaluating the utilization rate of the memory, the computing resources and the thread pool of the host computer so as to reasonably receive tasks.
Further, the interface service API implements the following functions: defining and creating a workflow and tasks thereof, and creating the workflow at regular time; timely uploading the workflow; manually operating and complementing the workflow; re-running, pausing, stopping, etc. control of the workflow instance.
Further, the distributed workflow Coordinator and the distributed task executor Worker both integrate Zookeeper to perform service registration, heartbeat, fault tolerance and distributed locking.
Furthermore, timing on-line is allowed only after the workflow is on-line, the workflow is added to the Quartz engine as timing operation by the timing on-line, and execution is performed after timing trigger is waited.
Further, the system includes an RDBMS for storing workflow schedule and timing metadata.
Further, the system also comprises an HDFS used for storage management of file resources which are depended by tasks.
The invention also provides a loosely-coupled distributed workflow coordination method, which comprises the following steps:
(1) a user initiates an HTTP request, an online workflow and the timing thereof by calling an interface service API;
(2) the distributed workflow Coordinator (Coordinator) schedules the workflow in a timing mode through an integrated distributed timing engine Quartz and sends the workflow to a workflow-MQ; and receiving and processing the workflow from the workflow-MQ, and specifically realizing the following functions:
a. implementing DAG topology and priority ordering of workflows to task queues;
b. caching the context of the workflow and the task queue thereof into a high available cache redis;
c. directly executing the logic type task; the service type task is sent to a task-dispatch-Workergroup-MQ, and a task executor Worker to be distributed receives and executes the task type task; receiving a task instance execution result through the task-callback-MQ;
e. creating workflow instances and task instances thereof, persisting data of state updating, and storing the data in a database;
(3) the distributed workflow Coordinator dispatches the service type tasks to be executed to the distributed task executor Worker through the task-dispatch-Worker group-MQ, the distributed task executor Worker receives and executes the service type tasks, and the task execution results are called back to the distributed workflow Coordinator through the task-callback-MQ; and finally, persistently storing the task execution result to a database through a distributed workflow Coordinator and feeding back the result to the user.
The invention has the beneficial effects that:
the method comprises the following steps that 1, priority is configured by an API when a workflow is on line Quartz at regular time, and the priority sequence of the workflow is guaranteed; when the workflow is converted into a node task queue from DAG, the Coordinator performs topological sorting and priority sorting to ensure the priority sequence of the task queue in the workflow; the scheduling accuracy is improved;
2. according to the invention, a high-availability cache redis is added to cache the workflow instance and the context of the task queue thereof, and the independent context cache enables the internal task callback and dispatch processing of the workflow instance to be carried out across coordinators, so that the distributed capability of coordinators is realized to a greater extent, and the service processing capability of the system is improved;
3. the invention adds MQ for the message queue of workflow and task dispatch and callback, and improves the throughput of the system; and the distributed consumption of MQ ensures that the workflow and the task are not repeatedly consumed;
4. loosely coupling the Coordinator and the Worker through MQ; ensuring that the Coordinator and the Worker have single and definite responsibility and lighter service; MQs are friendly to system changes or adaptations of more types of Worker.
Drawings
FIG. 1 is a block diagram of a distributed workflow scheduling system architecture according to the present invention;
FIG. 2 is a flow chart of the workflow timed online process of the present invention;
FIG. 3 is a schematic diagram of coordination process of Coordinator in accordance with the present invention;
FIG. 4 is a topology-bound prioritization of Coordinator's workflows by DAG through task priority queues;
Detailed Description
The following describes embodiments of the present invention in further detail with reference to the accompanying drawings.
As shown in fig. 1, the present invention provides a loosely-coupled distributed workflow coordination system, which includes: the system comprises an interface service API, a distributed workflow Coordinator, a distributed task executor Worker, a high-availability cache Redis, a distributed message queue MQ, an RDBMS and a Zookeeper.
The interface service API monitors the service conditions of the distributed workflow Coordinator and the distributed task executor Worker through the Zookeeper as shown in fig. 2; the interface service API implements the following functions: defining and creating a workflow and tasks thereof, and creating the workflow at regular time; timely uploading the workflow; manually operating and complementing the workflow; re-running, pausing, stopping, etc. control of the workflow instance.
The distributed message queue MQ comprises a workflow dispatching distributed message queue MQ, a task dispatching distributed message queue MQ and a task callback distributed message queue MQ;
as shown in fig. 3, the distributed workflow Coordinator performs the following functions: integrating a distributed timing engine Quartz, and scheduling a work class to send a distributed message queue MQ at regular time; receiving a workflow from a workflow dispatch distributed message queue MQ; DAG topology and priority ordering of workflows to task queues, as shown in fig. 4, creation of workflow instances and task instances thereof, and data persistence of state updates are saved to a database; caching the context of the workflow and the task queue thereof into a high available memory database redis; the service type task is dispatched to a distributed task executor Worker through a task dispatching distributed message queue MQ, and the logic type task is directly executed in a coordinator; the task instance execution result is obtained through a task callback distributed message queue MQ; evaluating the utilization rates of a host memory, computing resources and a thread pool so as to reasonably receive the workflow;
the distributed task executor Worker is responsible for receiving, executing and calling back all service type tasks; and evaluating the utilization rate of the memory, the computing resources and the thread pool of the host computer so as to reasonably receive tasks.
The distributed workflow Coordinator and the distributed task executor Worker are integrated with a Zookeeper to perform service registration, heartbeat, fault tolerance and distributed locking.
And the RDBMS is used for storing the workflow metadata. The HDFS is used for storage management of file resources which are depended by tasks.
The invention also provides a loosely-coupled distributed workflow coordination method, which comprises the following steps:
(1) a user initiates an HTTP request, an online workflow and tasks thereof by calling an interface service API;
(2) the distributed workflow Coordinator dispatches a distributed message queue MQ through an integrated distributed timing engine Quartz timing scheduling workflow, receives the workflow from the workflow dispatch distributed message queue MQ and processes the workflow, and specifically realizes the following functions:
a. realizing DAG topology and priority sequencing from the workflow to the task queue, creating the workflow instance and the task instance thereof, and persisting the data of state updating, and storing the data in a database;
a. implementing DAG topology and priority ordering of workflows to task queues;
b. caching the context of the workflow and the task queue thereof into a high available cache redis;
c. directly executing the logic type task; the service type task is dispatched to a task dispatching distributed message queue MQ, and a to-be-distributed task executor Worker receives and executes the task; receiving a task instance execution result through a task callback distributed Message Queue (MQ);
e. creating workflow instances and task instances thereof, persisting data of state updating, and storing the data in a database;
(3) the distributed workflow Coordinator dispatches the service type tasks to a distributed task executor Worker through a task dispatching distributed message queue MQ, the distributed task executor Worker receives and executes each service type task, and the task execution results are called back to the distributed workflow Coordinator through a task call-back distributed message queue MQ; and finally, persistently storing the task execution result to a database through a distributed workflow Coordinator and feeding back the result to the user.
Examples
Examples of the present invention include: API, Coordinator, Worker, and RDBMS, Cache, MQ and Zookeeper.
The API terminal is responsible for processing an HTTP request of a user for WEB UI operation; carrying out workflow and tasks thereof, wherein the tasks need to specify Worker groups (Workergroup), wherein the tasks are divided into logic tasks (including sub-workflow, dependent workflow and conditions) and definition creation, timing creation (based on cron expression) of business tasks (such as shell, http, procedure, spark and the like), and data persistence to RDBMS; allowing the workflow to be on line regularly only after the workflow is on line, adding the workflow as a timing operation to a Quartz engine by the timing on line, and executing after waiting for timing triggering;
an API end stores workflow task dependent resource files through an HDFS;
through the rerun operation, the API terminal can rejoin the workflow instance which is terminated in failure into a workflow distribution type message queue MQ (workflow-MQ) to wait for execution; setting a running workflow instance to be in pause ready/stop ready through pause/stop operation, after waiting for a Coordinator to process a latest task callback of the workflow instance, setting the workflow instance to be in pause/stop state according to the pause ready/stop ready state, ending the workflow instance and clearing a context cache of the workflow instance from a redis;
the API terminal obtains and monitors the conditions of the Coordinator and the Worker service instance through the Zookeeper.
A Coordinator end integrates a distributed timing engine Quartz, the Quartz triggers workflow operation added by API according to timing, and an operation executor sends corresponding workflow to a workflow-MQ;
the Coordinator end monitors workflow-MQ and reasonably receives the workflow according to the resource condition of the host and the allowance of the workflow thread pool; after receiving the workflow, the Coordinator end gives the workflow processor thread to process: firstly, creating a workflow instance and persisting, then sequencing tasks of the workflow by combining DAG topology with priority, filtering out forbidden tasks and tasks which are already executed, and caching the workflow instance and a task queue thereof to redis; and then, processing the task at the head of the task queue, if the task is a service-type task, sending the task to a corresponding task sending distributed message queue MQ (task-dispatch-Workergroup-MQ) to wait for the worker under the corresponding Workergroup to receive and execute, if the task is a logic-type task, handing the task to a task processor for processing, and sending all task processing results to the task callback distributed message queue MQ (task-callback-MQ).
Integrating a Zookeeper at a Coordinator end to perform self-service registration and heartbeat; and the Coordinator monitors the state of the service instance, processes fault tolerance for the service instance losing the mental state, and realizes the distributed lock through the Coordinator.
The Worker end is provided with a Worker group to which the Worker end belongs, monitors task-dispatch-Worker group-MQ corresponding to the Worker group, and reasonably receives tasks according to the calculation of the host resource condition and the available threads of the task processing thread pool; the Worker end receives the task and sends the task-in-operation callback to the task-callback-MQ, if the task is overtime in the execution process, the task fails or alarms according to the overtime strategy, if the task fails, the task is retried or ended according to the retry strategy, and after the task is completely ended, the task ending state is sent to the task-callback-MQ.
And integrating the Zookeeper at the Worker end to perform self registration and heartbeat, thereby realizing distributed locking.
The Coordinator end monitors a task-callback-MQ and reasonably receives task callbacks according to the resource condition of a host and the available threads of a task thread pool; the received task call-back updates the task instance state to a context cache redis and an RDBMS; then if the task instance is finished, judging whether the parent workflow instance is finished, if not, processing the task to dequeue from the queue and processing the next task; if the parent workflow is finished, finishing the parent workflow; and judging whether the volt workflow is a sub workflow or not, if so, ending the sub workflow task instance, and entering the processing of the workflow instance and the task queue of the sub workflow task.
The distributed interface service API, the distributed workflow Coordinator and the distributed task executor Worker realize service module decoupling through a distributed scheduling engine and a distributed message queue, and the three are completely in a loose coupling state.
The above-described embodiments are intended to illustrate rather than to limit the invention, and any modifications and variations of the present invention are within the spirit of the invention and the scope of the appended claims.
Claims (7)
1.A loosely-coupled distributed workflow coordination system, the system comprising: the system comprises an interface service API, a distributed workflow Coordinator, a distributed task executor Worker, a high-availability Cache, a distributed message queue MQ, a database RDBMS and a distributed coordination service Zookeeper.
The interface service API monitors the service conditions of a Coordinator of the distributed workflow and a Worker of a distributed task executor through a Zookeeper;
the distributed message queue MQ comprises a workflow dispatch distributed message queue MQ (workflow-MQ), a task dispatch distributed message queue MQ (task-dispatch-worker group-MQ) and a task callback distributed message queue MQ (task-callback-MQ);
the distributed workflow Coordinator performs the following functions: integrating a distributed timing engine Quartz, scheduling workflow at regular time and sending the workflow to a workflow-MQ; receiving the workflow from the workflow-MQ for processing; processing a DAG topology and prioritization of workflows to task queues; creating workflow instances and task instances thereof, and persistently storing data of state update to a database; caching the context of the workflow and the task queue thereof into a high available cache redis; the service type task is distributed to a distributed task executor Worker through a task-dispatch-Workergroup-MQ, and the logic type task is directly executed on a Coordinator; the task instance execution result is obtained through a task-callback-MQ; evaluating the utilization rates of a host memory, computing resources and a thread pool so as to reasonably receive the workflow;
the distributed task executor Worker is responsible for receiving, executing and calling back all service type tasks; and evaluating the utilization rate of the memory, the computing resources and the thread pool of the host computer so as to reasonably receive tasks.
2. The loosely-coupled distributed workflow coordination system of claim 1, wherein said interface services API performs the following functions: defining and creating a workflow and tasks thereof, and creating the workflow at regular time; timely uploading the workflow; manually operating and complementing the workflow; re-running, pausing, stopping, etc. control of the workflow instance.
3. The loosely-coupled distributed workflow coordination system of claim 1, wherein said distributed workflow Coordinator and distributed task executor Worker each integrate Zookeeper for service registration, heartbeat, fault tolerance, and distributed locking.
4. The loosely-coupled distributed workflow coordination system of claim 1, wherein timed up-line is allowed only after the workflow is up-line, wherein timed up-line adds the workflow as a timed operation to the Quartz engine, and wherein the timed up-line waits for a timed trigger before executing the workflow.
5. The loosely-coupled distributed workflow coordination system of claim 1, further comprising an RDBMS for storing workflow scheduling and timing metadata.
6. The loosely-coupled distributed workflow coordination system of claim 1, further comprising HDFS for storage management of task-dependent file resources.
7. A distributed workflow coordination method based on the loosely coupled distributed workflow coordination system of any of claims 1 to 6, characterized in that the method comprises the steps of:
(1) a user initiates an HTTP request, an online workflow and the timing thereof by calling an interface service API;
(2) the distributed workflow Coordinator (Coordinator) schedules the workflow in a timing mode through an integrated distributed timing engine Quartz and sends the workflow to a workflow-MQ; and receiving and processing the workflow from the workflow-MQ, and specifically realizing the following functions:
a. implementing DAG topology and priority ordering of workflows to task queues;
b. caching the context of the workflow and the task queue thereof into a high available cache redis;
c. directly executing the logic type task; the service type task is sent to a task-dispatch-Workergroup-MQ, and a task executor Worker to be distributed receives and executes the task type task; receiving a task instance execution result through the task-callback-MQ;
e. creating workflow instances and task instances thereof, persisting data of state updating, and storing the data in a database;
(3) the distributed workflow Coordinator dispatches the service type tasks to be executed to the distributed task executor Worker through the task-dispatch-Worker group-MQ, the distributed task executor Worker receives and executes the service type tasks, and the task execution results are called back to the distributed workflow Coordinator through the task-callback-MQ; and finally, persistently storing the task execution result to a database through a distributed workflow Coordinator and feeding back the result to the user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111061941.2A CN113821322A (en) | 2021-09-10 | 2021-09-10 | Loosely-coupled distributed workflow coordination system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111061941.2A CN113821322A (en) | 2021-09-10 | 2021-09-10 | Loosely-coupled distributed workflow coordination system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113821322A true CN113821322A (en) | 2021-12-21 |
Family
ID=78922080
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111061941.2A Pending CN113821322A (en) | 2021-09-10 | 2021-09-10 | Loosely-coupled distributed workflow coordination system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113821322A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114661438A (en) * | 2022-03-23 | 2022-06-24 | 杭州杰竞科技有限公司 | Distributed workflow scheduling system and method based on event driving |
| CN117093384A (en) * | 2023-08-23 | 2023-11-21 | 北京志凌海纳科技有限公司 | Universal back-end reliable execution method, system, equipment and readable medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150067028A1 (en) * | 2013-08-30 | 2015-03-05 | Indian Space Research Organisation | Message driven method and system for optimal management of dynamic production workflows in a distributed environment |
| US20160260040A1 (en) * | 2013-11-14 | 2016-09-08 | Huawei Technologies Co., Ltd. | Computer Device, Method, and Apparatus for Scheduling Business Flow |
| US20190138375A1 (en) * | 2017-11-03 | 2019-05-09 | Dell Products L. P. | Optimization of message oriented middleware monitoring in heterogenenous computing environments |
| CN110825535A (en) * | 2019-10-12 | 2020-02-21 | 中国建设银行股份有限公司 | Job scheduling method and system |
| CN112162841A (en) * | 2020-09-30 | 2021-01-01 | 重庆长安汽车股份有限公司 | Distributed scheduling system, method and storage medium for big data processing |
| CN112559159A (en) * | 2021-01-05 | 2021-03-26 | 广州华资软件技术有限公司 | Task scheduling method based on distributed deployment |
-
2021
- 2021-09-10 CN CN202111061941.2A patent/CN113821322A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150067028A1 (en) * | 2013-08-30 | 2015-03-05 | Indian Space Research Organisation | Message driven method and system for optimal management of dynamic production workflows in a distributed environment |
| US20160260040A1 (en) * | 2013-11-14 | 2016-09-08 | Huawei Technologies Co., Ltd. | Computer Device, Method, and Apparatus for Scheduling Business Flow |
| US20190138375A1 (en) * | 2017-11-03 | 2019-05-09 | Dell Products L. P. | Optimization of message oriented middleware monitoring in heterogenenous computing environments |
| CN110825535A (en) * | 2019-10-12 | 2020-02-21 | 中国建设银行股份有限公司 | Job scheduling method and system |
| CN112162841A (en) * | 2020-09-30 | 2021-01-01 | 重庆长安汽车股份有限公司 | Distributed scheduling system, method and storage medium for big data processing |
| CN112559159A (en) * | 2021-01-05 | 2021-03-26 | 广州华资软件技术有限公司 | Task scheduling method based on distributed deployment |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114661438A (en) * | 2022-03-23 | 2022-06-24 | 杭州杰竞科技有限公司 | Distributed workflow scheduling system and method based on event driving |
| CN117093384A (en) * | 2023-08-23 | 2023-11-21 | 北京志凌海纳科技有限公司 | Universal back-end reliable execution method, system, equipment and readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TW406242B (en) | System and method for maximizing usage of computer resources in scheduling of application tasks | |
| US8745628B2 (en) | Execution order management of multiple processes on a data processing system by assigning constrained resources to the processes based on resource requirements and business impacts | |
| CN112000445A (en) | Distributed task scheduling method and system | |
| CN113821322A (en) | Loosely-coupled distributed workflow coordination system and method | |
| US8538793B2 (en) | System and method for managing real-time batch workflows | |
| Panagos et al. | Reducing escalation-related costs in WFMSs | |
| CN110377406A (en) | A kind of method for scheduling task, device, storage medium and server node | |
| CN112905339B (en) | Task scheduling execution method, device and system | |
| CN112181621A (en) | Task scheduling system, method, equipment and storage medium | |
| CN111580990A (en) | Task scheduling method, scheduling node, centralized configuration server and system | |
| CN114138434B (en) | Big data task scheduling system | |
| CN112559159A (en) | Task scheduling method based on distributed deployment | |
| CN113535362B (en) | Distributed scheduling system architecture and micro-service workflow scheduling method | |
| CN112162841A (en) | Distributed scheduling system, method and storage medium for big data processing | |
| CN107316124B (en) | Extensive affairs type job scheduling and processing general-purpose system under big data environment | |
| CN110611707A (en) | Task scheduling method and device | |
| CN110955506A (en) | Distributed job scheduling processing method | |
| CN116010064A (en) | DAG job scheduling and cluster management method, system and device | |
| CN113032125A (en) | Job scheduling method, device, computer system and computer-readable storage medium | |
| Mangler et al. | Rule-based synchronization of process activities | |
| CN111240819A (en) | Dispatching task issuing system and method | |
| CN109614222A (en) | A kind of multithreading resource allocation methods | |
| CN111124651B (en) | Method for concurrently scheduling multiple threads in distributed environment | |
| CN116302423A (en) | Distributed task scheduling method and system for cloud management platform | |
| CN111913784A (en) | Task scheduling method and device, network element and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |