CN118069314A - Batch task and stream task integrated scheduling processing method, system, equipment and medium - Google Patents
Batch task and stream task integrated scheduling processing method, system, equipment and medium Download PDFInfo
- Publication number
- CN118069314A CN118069314A CN202410136081.1A CN202410136081A CN118069314A CN 118069314 A CN118069314 A CN 118069314A CN 202410136081 A CN202410136081 A CN 202410136081A CN 118069314 A CN118069314 A CN 118069314A
- Authority
- CN
- China
- Prior art keywords
- task
- batch
- running
- node
- directed acyclic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 10
- 238000004458 analytical method Methods 0.000 claims abstract description 67
- 230000004044 response Effects 0.000 claims abstract description 25
- 238000000034 method Methods 0.000 claims description 46
- 238000012544 monitoring process Methods 0.000 claims description 14
- 238000012545 processing Methods 0.000 claims description 13
- 238000011084 recovery Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 3
- 230000010354 integration Effects 0.000 description 13
- 230000008569 process Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 238000012423 maintenance Methods 0.000 description 6
- 239000000725 suspension Substances 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 239000000306 component Substances 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 125000002015 acyclic group Chemical group 0.000 description 4
- 238000007726 management method Methods 0.000 description 3
- 230000002829 reductive effect Effects 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000003203 everyday effect Effects 0.000 description 2
- 230000002441 reversible effect Effects 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 239000008358 core component Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002147 killing effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application provides a batch task and stream task integrated scheduling processing method, a system, equipment and a medium, comprising the following steps: in response to the task execution indication, performing a copy operation, the copy operation including: copying the current directed acyclic graph in the primary workflow task scheduling platform to form a first directed acyclic graph and a second directed acyclic graph; executing stream task analysis operation to determine a first task operation forbidden task node, a first operable task node and a first task operation sequence corresponding to the first operable task node; executing a streaming task operation to operate tasks corresponding to the first operable node according to the first task operation sequence; executing batch task analysis operation to determine a second task operation forbidden task node, a second executable task node and a second task operation sequence corresponding to the second executable task node; and executing batch task operation to operate the task corresponding to the second executable node according to the second task operation sequence. The batch flow integrated dispatching function is satisfied.
Description
Technical Field
The application relates to the technical field of big data, in particular to a batch task and stream task integrated scheduling processing method, a system, equipment and a medium.
Background
In the existing workflow task scheduling system with an open source, for example DolphinScheduler, various tasks such as Shell, MR and the like are scheduled and monitored regularly according to the sequence of DAG (DIRECTED ACYCLIC GRAPH ), so that the operation and maintenance cost of big data development is reduced; the task execution flow is generally shown in fig. 1, and after the clicking operation, the tasks will be executed according to the corresponding sequence. The specific flow is as follows: after the operation A is successful, the operation B, C, D is performed, and the C mark is used for prohibiting the execution, so that the operation will be skipped, and only B, D is performed successfully, the E node is performed successfully, and the whole DAG is performed successfully. If the middle node fails to operate, execution will stop, marked as failure.
In a typical scenario of big data, as shown in fig. 2, log data is continuously generated, collected into Kafka by real-time streaming collection tasks, and then synchronized into Hive database by real-time synchronous streaming tasks, which do not stop. In the early morning, the batch task is analyzed and summarized by scheduling offline at regular time, and the data collected every day is analyzed and summarized, and the calculation result may be written into the Hive database and may be written into Kafka (the data written into Kafka is written into Hive again through the real-time synchronous stream task). The task runs longer and eventually stops. However, in an actual scene, more than one real-time streaming task and offline batch task depend on the related flow direction of data, and a batch-stream integrated task scheduling mode is required to meet the task scheduling of the type. However, existing DolphinScheduler does not support scheduling streaming tasks, but only temporarily configuring blood-edge dependencies between streaming tasks and batch tasks. If the DAG contains a stream task node, the node can only be skipped to execute, and the corresponding stream task is started manually, so that the batch-stream integrated task scheduling in the data integration analysis scene can not be met.
Therefore, a task processing method for integrating batch tasks and stream tasks in a data integration analysis scene is needed to solve the above technical problems.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a batch task and stream task integrated scheduling processing method, system, device and medium, so as to solve the foregoing technical problems.
In a first aspect, the present application provides a method for scheduling and processing batch tasks and stream tasks in an integrated manner, where the method includes:
In response to the task execution indication, performing a copy operation, the copy operation comprising: copying the current directed acyclic graph in the primary workflow task scheduling platform to form a first directed acyclic graph and a second directed acyclic graph;
Executing a stream task analysis operation to determine a first task operation prohibition node, a first operable task node and a first task operation sequence corresponding to the first operable node in the first directed acyclic graph;
executing a streaming task operation to operate the task corresponding to the first operable node according to the first task operation sequence;
executing batch task analysis operation to determine a second task operation forbidden task node, a second operable task node and a second task operation sequence corresponding to the second operable task node in the second directed acyclic graph;
And executing batch task operation to operate the task corresponding to the second executable node according to the second task operation sequence.
In some embodiments, the performing a streaming task parsing operation includes:
marking all batch task nodes in the first directed acyclic graph as first forbidden task nodes and marking all stream task nodes in the first directed acyclic graph as first operable task nodes;
Acquiring and reversing a first current task operation sequence in the first directed acyclic graph, and updating the reversed first current task operation sequence to be a first task operation sequence;
the executing stream task running operation includes:
According to a first task operation sequence of the first directed acyclic graph, operating a flow task corresponding to a first operable task node in the first directed acyclic graph;
Monitoring whether the flow task is submitted to run successfully or not;
and after the success of the submitting operation of the streaming task is monitored, the streaming task of the next first executable task node is operated according to the first task operation sequence, and the first forbidden task node is skipped.
In some embodiments, the performing a batch task parsing operation includes:
marking all flow task nodes in the second directed acyclic graph as second forbidden task nodes and marking all batch task nodes in the second directed acyclic graph as second runnable task nodes;
Acquiring a second current task operation sequence in the second directed acyclic graph and updating the second current task operation sequence to a second task operation sequence;
The executing batch task running operation comprises the following steps:
According to a second task operation sequence of the second directed acyclic graph, operating batch tasks corresponding to second operable task nodes in the first directed acyclic graph;
And monitoring the running state of the batch task, and after the batch task is successfully run, running the batch task of the next second runnable task node according to the second task running sequence and skipping over the second task node which is forbidden to run.
In some embodiments, the method further comprises:
responding to the task stop instruction, and judging whether the currently running task contains batch tasks or not;
If the current running task comprises a batch task, stopping running the batch task, analyzing a first directed acyclic graph, and stopping running the running stream task according to the running sequence of the first current task;
If the current running task does not contain batch tasks, directly analyzing the first directed acyclic graph, and stopping running the running streaming task according to the running sequence of the first current task;
Responding to a resume stop task instruction, and sequentially executing the copy operation, the stream task analysis operation, the batch task analysis operation and the stream task running operation;
Judging whether the task corresponding to the task node which stops running last time is a batch task or not;
if the task is a batch task, starting to run the batch task from a task node which stops running last time according to a second current task running sequence in a second directed acyclic graph which is generated at present;
And if the task is not a batch task, starting to run the batch task from a first task node in the current task running sequence according to the second current task running sequence.
In some embodiments, the method comprises:
responding to the task pause instruction, and judging whether the currently running task contains batch tasks or not;
If the current running task comprises a batch task, stopping running after the running of the batch task running at present is finished;
If the current running task does not contain batch tasks, after the task pause indication confirms that the running task needs to be paused, analyzing the first directed acyclic graph and stopping running the running streaming task according to the running sequence of the first current task;
In response to resuming the suspending task indication, performing the copy operation;
If no running streaming task exists, executing the streaming task analysis operation and the streaming task running operation;
if the running streaming task exists, executing the batch task analysis operation, and judging whether the task corresponding to the task node which is suspended to run last time is a batch task or not;
if the task is a batch task, starting to run the batch task from a task node which stops running last time according to a second current task running sequence in a second directed acyclic graph which is generated at present;
And if the task is not a batch task, starting to run the batch task from a first task node in the second current task running sequence according to the second current task running sequence.
In some embodiments, the method comprises:
responsive to a recovery failure task indication, performing the copy operation;
Judging whether the failed task is a streaming task or not;
if the failed task is a streaming task, executing the streaming task analysis operation and the streaming task operation;
and if the failed task is not the streaming task, executing the batch task analysis operation and the batch task running operation.
In some embodiments, the method comprises:
setting one or more batch task operation time periods, wherein the batch task operation time periods comprise a starting time and an ending time;
Responsive to the start time, performing a copy operation, a batch task parsing operation, and a batch task running operation;
And stopping executing the batch task running operation in response to the ending time.
In a second aspect, the present application provides a batch task and stream task integrated processing system, the system comprising:
a copy module for performing a copy operation in response to a task execution instruction, the copy operation comprising: copying the current directed acyclic graph in the primary workflow task scheduling platform to form a first directed acyclic graph and a second directed acyclic graph;
the analysis module is used for executing flow task analysis operation to determine a first task operation forbidden task node, a first operable task node and a first task operation sequence corresponding to the first operable task node in the first directed acyclic graph;
The running module is used for executing a streaming task running operation to run tasks corresponding to the first runnable node according to the first task running sequence;
the analysis module is further configured to perform batch task analysis operations to determine a second task operation prohibition node, a second task node capable of being operated, and a second task operation sequence corresponding to the second task node capable of being operated in the second directed acyclic graph;
the operation module is further configured to perform a batch task operation to operate a task corresponding to the second executable node according to the second task operation order.
In a third aspect, the present application provides an electronic device, including:
One or more processors;
And a memory associated with the one or more processors, the memory for storing program instructions that, when read for execution by the one or more processors, perform the following:
In response to the task execution indication, performing a copy operation, the copy operation comprising: copying the current directed acyclic graph in the primary workflow task scheduling platform to form a first directed acyclic graph and a second directed acyclic graph;
Executing a stream task analysis operation to determine a first task operation prohibition node, a first operable task node and a first task operation sequence corresponding to the first operable node in the first directed acyclic graph;
executing a streaming task operation to operate the task corresponding to the first operable node according to the first task operation sequence;
executing batch task analysis operation to determine a second task operation forbidden task node, a second operable task node and a second task operation sequence corresponding to the second operable task node in the second directed acyclic graph;
And executing batch task operation to operate the task corresponding to the second executable node according to the second task operation sequence.
In a fourth aspect, the present application also provides a computer-readable storage medium having stored thereon a computer program that causes a computer to perform the operations of:
In response to the task execution indication, performing a copy operation, the copy operation comprising: copying the current directed acyclic graph in the primary workflow task scheduling platform to form a first directed acyclic graph and a second directed acyclic graph;
Executing a stream task analysis operation to determine a first task operation prohibition node, a first operable task node and a first task operation sequence corresponding to the first operable node in the first directed acyclic graph;
executing a streaming task operation to operate the task corresponding to the first operable node according to the first task operation sequence;
executing batch task analysis operation to determine a second task operation forbidden task node, a second operable task node and a second task operation sequence corresponding to the second operable task node in the second directed acyclic graph;
And executing batch task operation to operate the task corresponding to the second executable node according to the second task operation sequence.
The beneficial effects achieved by the application are as follows:
The application provides a batch task and stream task integrated scheduling processing method, which comprises the following steps: in response to the task execution indication, performing a copy operation, the copy operation comprising: copying the current directed acyclic graph in the primary workflow task scheduling platform to form a first directed acyclic graph and a second directed acyclic graph; executing a stream task analysis operation to determine a first task operation prohibition node, a first operable task node and a first task operation sequence corresponding to the first operable node in the first directed acyclic graph; executing a streaming task operation to operate the task corresponding to the first operable node according to the first task operation sequence; executing batch task analysis operation to determine a second task operation forbidden task node, a second operable task node and a second task operation sequence corresponding to the second operable task node in the second directed acyclic graph; and executing batch task operation to operate the task corresponding to the second executable node according to the second task operation sequence. The application supports the visual configuration of the batch-stream integrated task directed acyclic graph by modifying DolphinScheduler, expands DolphinScheduler functions according to the sequence of the directed acyclic graph, and supports stream task scheduling according to the data integration scene. The method meets the integral dispatching function of batch flow, and the data flow direction can be clearly seen through configuring and dispatching the directed acyclic graph flow direction. For operation and maintenance developers, the operation notice that the streaming task needs to be started from back to front and stopped from front to back is not needed, and the scheduled task can be better scheduled and managed only by configuring the corresponding DAG according to the data flow direction; the operation, maintenance, management and scheduling costs of the data integration task are reduced, and data backlog and data loss caused by improper scheduling of the streaming task are avoided.
Drawings
For a clearer description of the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, it being obvious that the drawings in the description below are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art, wherein:
FIG. 1 is a directed acyclic graph provided by an embodiment of the present application;
Fig. 2 is a schematic diagram of a data integration analysis scenario provided in an embodiment of the present application;
FIG. 3 is a flow chart of a batch flow integrated task operation provided by an embodiment of the present application;
FIG. 4 is a schematic diagram of a batch integrated task architecture according to an embodiment of the present application;
FIG. 5 is a flow chart of a batch flow integrated task stop provided by an embodiment of the present application;
FIG. 6 is a flow chart of a batch flow integrated task resume and stop provided by an embodiment of the application
FIG. 7 is a flow chart of a batch flow integrated task suspension procedure provided by an embodiment of the present application
FIG. 8 is a flow chart of a batch flow integrated task resume and pause provided by an embodiment of the application
FIG. 9 is a flow chart of a batch flow unified task recovery failure provided by an embodiment of the application
FIG. 10 is a schematic diagram of task monitoring provided by an embodiment of the present application;
FIG. 11 is a block diagram of a processing system architecture integrating batch tasks and stream tasks provided by an embodiment of the present application;
fig. 12 is a block diagram of an electronic device according to an embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
It should be understood that throughout this specification and the claims, unless the context clearly requires otherwise, the words "comprise", "comprising", and the like, are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is, it is the meaning of "including but not limited to".
It should also be appreciated that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Furthermore, in the description of the present application, unless otherwise indicated, the meaning of "a plurality" is two or more.
It should be noted that the terms "S1", "S2", and the like are used for the purpose of describing the steps only, and are not intended to be construed to be specific as to the order or sequence of steps, nor are they intended to limit the present application, which is merely used to facilitate the description of the method of the present application, and are not to be construed as indicating the sequence of steps. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present application.
As described in the background, currently in data integration scenarios; the dependency existing between the batch tasks is strong (the analysis of the result of the previous task is not carried out, the execution of the next task is certainly wrong), the dependency between the stream tasks is also strong (the downstream stream tasks are not started, the problem of data backlog, data loss and the like can be caused by the fact that the data comes on the downstream stream tasks), the dependency existing between the batch tasks is strong (the upstream stream tasks are not started, the analysis of the downstream batch tasks cannot be carried out), the dependency existing between the stream tasks and the batch tasks is not strong (the analysis of the result of the upstream batch tasks is not carried out, the operation of the downstream stream tasks is not influenced, the downstream stream tasks can be operated until the result of the batch tasks is carried out, and the result of the stream tasks is calculated. The existing DolphinScheduler architecture does not support scheduling streaming tasks, but only temporarily configuring blood-edge dependencies between streaming tasks and batch tasks. If the DAG contains a stream task node, the node can only be skipped to execute, and the corresponding stream task is started manually, so that the batch-stream integrated task scheduling in the data integration analysis scene can not be met. In the existing data integration scene, a large number of streaming tasks need to be continuously operated, data is summarized through a large number of batch tasks in the early morning every day, summarized data can be associated with real-time service data, and summarized analysis is performed through the real-time tasks again. Based on the existing Dolphinscheduler functions, batch tasks can only be scheduled to be executed according to the sequence of the DAG; for the streaming task, the operation of the streaming task can be manually managed, scheduled and monitored, and the operation sequence of the streaming task can not be managed, and if the starting sequence of the streaming task is configured incorrectly, data backlog and even data loss can be caused.
The application provides a DolphinScheduler-based batch task and stream task integrated scheduling processing method, which is used for uniformly scheduling batch and stream tasks by modifying the scheduling mode of DolphinScheduler, expanding DolphinScheduler functions and being used for data integration analysis scene batch and stream integrated scheduling functions.
Example 1
The embodiment of the application provides a batch task and stream task integrated scheduling processing method which is used for a data integration analysis scene, and the task is scheduled by the method disclosed by the embodiment of the application, and comprises the following contents:
After the directed acyclic graph is configured according to the data flow direction, the directed acyclic graph analysis logic is redesigned, the batch-flow integrated directed acyclic graph is split into two directed acyclic graphs, the two directed acyclic graphs are respectively scheduled to run according to different task types, and when the directed acyclic graph runs, as shown in fig. 3, the method comprises the following steps:
In response to the task execution indication, performing a copy operation, the copy operation including: the current directed acyclic graph within the once workflow task scheduling platform is replicated to form a first directed acyclic graph and a second directed acyclic graph. In some of these embodiments, as shown in FIG. 4, in a batch flow unified directed acyclic graph, there are both batch and flow task nodes in one directed acyclic graph. When the execution is scheduled, the directed acyclic graph is copied into two directed acyclic graphs, namely a first directed acyclic graph (i.e. a stream task directed acyclic graph) and a second directed acyclic graph (i.e. a batch task directed acyclic graph). It will be appreciated that the task running indication, and the task stop indication, resume stop task indication, task pause indication, resume pause task indication, and resume failure task indication mentioned below are all user initiated operations, and the user selects what operation needs to be performed on the workflow task scheduling platform.
Executing a stream task parsing operation to determine a first prohibited task node, a first runnable task node, and a first task running order corresponding to the first runnable task node in the first directed acyclic graph; specifically, the executing the streaming task parsing operation includes: marking all batch task nodes in the first directed acyclic graph as first forbidden task nodes and marking all stream task nodes in the first directed acyclic graph as first runnable task nodes; and acquiring and reversing the first current task operation sequence in the first directed acyclic graph, and updating the reversed first current task operation sequence to be the first task operation sequence. The application changes the sequence of the task operation in the first directed acyclic graph, firstly operates the flow task behind the directed acyclic graph, and then operates the previous flow task, thereby avoiding the problem of data loss caused by the fact that a large amount of flow data backlog is caused after the flow data is directly processed and cannot be processed in time.
Executing a streaming task operation to operate tasks corresponding to the first operable node according to the first task operation sequence; specifically, the executing the streaming task running operation includes:
According to a first task operation sequence of the first directed acyclic graph, operating a flow task corresponding to a first operable task node in the first directed acyclic graph; monitoring whether the submitting operation of the streaming task is successful; and after the success of the submitting operation of the streaming task is monitored, the streaming task of the next first executable task node is operated according to the first task operation sequence, and the first forbidden task node is skipped. Aiming at the streaming task, the application provides different streaming task operation modes, the streaming task is submitted and operated successfully, and the task is successful and starts to execute the next node. Such as successful start of a jump process, successful submission of a streaming task to a cluster, successful start of a task, the task execution time of such a directed acyclic graph associated with a streaming task is typically short. And finally, issuing the processed first directed acyclic graph to a worker for execution, wherein the worker is a core component in the thread pool and is used for executing tasks submitted to the thread pool.
Executing batch task analysis operation to determine a second task operation forbidden task node, a second operable task node and a second task operation sequence corresponding to the second operable node in the second directed acyclic graph; specifically, the executing batch task parsing operation includes: marking all flow task nodes in the second directed acyclic graph as second forbidden task nodes and marking all batch task nodes in the second directed acyclic graph as second executable task nodes; and acquiring a second current task operation sequence in the second directed acyclic graph and updating the second current task operation sequence to the second task operation sequence.
And executing batch task operation to operate the task corresponding to the second executable node according to the second task operation sequence. Specifically, the executing batch task running operation includes: running batch tasks corresponding to second executable task nodes in the first directed acyclic graph according to a second task running sequence of the second directed acyclic graph; and monitoring the running state of the batch task, and after the batch task is monitored to run successfully, running the batch task of the next second runnable task node according to the reversed second task running sequence and skipping over the second forbidden task node. For batch tasks, the application also provides a batch task running mode, namely batch task running is completed and successful, the task is successful, the execution of the next node is started, and the task running time of the directed acyclic graph is longer. For different task types, different scheduling monitoring methods are adopted, so that the monitoring result is more accurate, and the rationality of overall task scheduling is further improved.
The application also provides a method for stopping and recovering the flow task and the batch task in the dispatching process.
For task stopping, the task type corresponding to the node where the task runs needs to be considered, and the stopping logic comprises: the first directed acyclic graph and the second directed acyclic graph obtained by splitting the task starting operation are based on the above; in response to the task stop indication, determining whether the currently running task contains a batch task (i.e., confirming in which directed acyclic graph the running task is); if the currently running task comprises a batch task (i.e. the running batch task also exists in the second directed acyclic graph and the streaming task is running all the time), stopping running the batch task, analyzing the first directed acyclic graph without reversing the running sequence of the first current task in the first directed acyclic graph, and stopping running the running streaming task according to the running sequence of the first current task; if the current running task does not contain batch tasks (i.e. there is no running task in the second directed acyclic graph, and the streaming task is running all the time), stopping running the running streaming task directly according to the running sequence of the first current task, and if there are streaming tasks on some nodes that have stopped, skipping the stopped streaming tasks.
In some implementations, as shown in fig. 5, the method includes the steps of:
1. confirming in which directed acyclic graph the running task is;
a) Executing step 3 by the first directed acyclic graph;
b) Executing step 2 by the second directed acyclic graph;
2. according to the second current task operation sequence of the second directed acyclic graph, killing the task of the second directed acyclic graph, and executing the fourth step;
3. Analyzing the first directed acyclic graph, and marking batch tasks as forbidden operations;
4. stopping running the first directed acyclic graph, stopping the streaming task according to the first current task running sequence in the first directed acyclic graph of step 3, and skipping the task if the task has stopped.
In the application, the stopping logic is redesigned, and in the stopping process, a front-to-back stopping mode is adopted aiming at the stopped streaming task, thereby avoiding data loss.
The logic for resuming stopping is similar to that for starting operation, and the difference is that according to the condition of the stopping node, the execution mode of the batch task in the second directed acyclic graph is not very the same, and specifically includes:
Responding to the instruction of stopping task, and sequentially executing copy operation, stream task analysis operation, batch task analysis operation and stream task operation; judging whether the task corresponding to the task node which stops running last time is a batch task or not; if the task is a batch task, starting to run the batch task from a task node which stops running last time according to a second current task running sequence in a second directed acyclic graph which is generated at present; and if the task is not a batch task, starting to run the batch task from the first task node in the second current task running sequence according to the second current task running sequence.
As shown in fig. 6, in some implementations, the method includes:
1. performing a copying operation, namely copying the current directed acyclic graph into two directed acyclic graphs, namely a first directed acyclic graph and a second directed acyclic graph;
2. And executing stream task analysis operation: marking batch task nodes as first task nodes which are forbidden to run, acquiring and reversing a first current task running sequence in a first directed acyclic graph, and updating the reversed first current task running sequence into a first task running sequence;
3. executing stream task operation: the successful start of the streaming task submission is successful, and the streaming task of the next node is executed after the success;
4. executing batch task parsing operation: the marked flow task node is a second task node forbidden to run, and the second current task running sequence in the second directed acyclic graph is acquired and updated to be the second task running sequence.
5. Judging whether the type of the last task stop node is a batch task node, if yes, executing 6, and if not, executing 7
6. Starting from the last stop node according to the second task operation sequence, and operating batch tasks in the second directed acyclic graph;
7. and running the batch tasks in the second directed acyclic graph according to the second task running order.
The application also provides a method for suspending and resuming suspending the stream task and the batch task in the dispatching process.
For the pause operation, the type of task node running the task, that is, the directed acyclic graph where the task node running the task is located, needs to be considered, and whether to stop the streaming task in the first directed acyclic graph needs to be selected, so that the pause method comprises the following steps: responding to the task pause instruction, judging whether the currently running task contains batch tasks (namely judging whether the type of task nodes running by the task comprises batch tasks or not, if so, storing the task nodes running by the task in the second directed acyclic graph, and if not, storing the task nodes running by the task in the second directed acyclic graph; if the current running task comprises a batch task, stopping running after waiting for the running of the current running batch task to be completed; if the current running task does not contain batch tasks, after the task pause indication confirms that the streaming task needs to be paused, the first directed acyclic graph is analyzed, and the running streaming task is stopped according to the running sequence of the first current task. The task pause instruction is an operation initiated by a user, and when the user selects the pause operation, a prompt box is automatically generated to enable the user to select whether to stop the streaming task.
In some implementations, as shown in fig. 7, the method includes the following steps:
1. judging whether the type of task nodes running the task comprises batch tasks or not, if so, executing the step 2, and if not, executing the step 3;
2. According to the second current task operation sequence in the second directed acyclic graph, waiting for completion of batch task execution in the second directed acyclic graph, and executing step 3;
3. judging whether the streaming task in the first directed acyclic graph needs to be stopped, if the user selects to stop the streaming task, executing the step 4, otherwise, ending;
4. Executing stream task analysis operation;
5. Stopping the streaming task in the second directed acyclic graph: and stopping the streaming task in the first directed acyclic graph according to the first current task running sequence in the first directed acyclic graph, and skipping the task if the task has stopped.
The application redesigns the pause logic, supports to select whether to pause all stream tasks in the pause process, and avoids data loss by adopting a front-to-back pause mode if the all stream tasks are paused.
The process of resuming the suspension is similar to the process of starting the operation, and the difference is that according to the condition of stopping the node, the execution mode of the batch task in the second directed acyclic graph is not very the same, including: in response to resuming the pause task indication, performing a copy operation; if the running streaming task does not exist, executing streaming task analysis operation and streaming task running operation; if the running streaming task exists, executing batch task analysis operation, and judging whether the task corresponding to the task node which pauses running last time is a batch task or not; if the task is a batch task, starting to run the batch task from a next task node of the task node which stops running last time according to a second current task running sequence in a second directed acyclic graph which is generated at present; and if the task is not a batch task, starting to run the batch task from the first task node in the second current task running sequence according to the second current task running sequence.
In some implementations, as shown in fig. 8, the method includes the following steps:
1. performing a copy operation;
2. judging whether the flow task in the first directed acyclic graph runs or not, if so, executing the step 3, otherwise, executing the step 5;
3. Executing stream task analysis operation;
4. executing a streaming task operation;
5. Executing batch task analysis operation;
6. Judging whether the task corresponding to the task node which pauses operation last time is a batch task, if so, executing the step 7, otherwise, executing the step 8;
7. Starting to run batch tasks in the second directed acyclic graph from the node after the last pause according to a second current task running sequence in the second directed acyclic graph;
8. And starting to run batch tasks from the first task node in the current task running sequence according to the second current task running sequence.
The application also provides a method for recovering the tasks which fail to run when the streaming tasks and the batch tasks are scheduled, comprising the following steps: responsive to the recovery failure task indication, performing a copy operation; judging whether the failed task is a streaming task or not; if the failed task is a streaming task, executing streaming task analysis operation and streaming task operation; and if the failed task is not the streaming task, executing batch task analysis operation and batch task running operation.
In some implementations, as shown in fig. 9, the method includes the following steps:
1. performing a copy operation;
2. Judging whether the failed task is a streaming task of the first directed acyclic graph, if so, executing the step 3, otherwise, executing the step 5;
3. Executing stream task analysis operation;
4. executing a streaming task operation;
5. Executing batch task analysis operation;
6. And executing batch task analysis operation.
In some embodiments, the present application further proposes to perform a timing scheduling process for a batch task, specifically by setting one or more batch task operation time periods, where the batch task operation time periods include a start time and an end time; responsive to the start time, performing a copy operation, a batch task parsing operation, and a batch task running operation; in response to the end time, execution of the batch task run operation is stopped. The batch task operation time period is set by a worker according to actual requirements, and the application is not limited to this. The application redesigns the timing scheduling logic, namely, only operates the batch task nodes of the second directed acyclic graph when in timing operation, simplifies the timing logic and is convenient for operation and maintenance personnel to check.
In addition, as shown in fig. 10, the application also provides for monitoring the status of batch tasks and monitoring the status of streaming tasks. For the batch task, since the batch task is stopped finally, the batch task thread is started to manage and monitor the state of the batch task until the batch task is completed, and report the state of the batch task. For the streaming task, as the streaming task cannot stop, after the streaming task is successfully operated, the thread for starting the streaming task ends to exit, and meanwhile, the streaming task is added into a monitoring queue to monitor the streaming task, and the specific flow is as follows: after the streaming task is successfully operated, information such as a host name, IP, PID and the like of the streaming task operation is put into a streaming task information queue; starting a timing thread, wherein the timing time can be configured, the flow task information is regularly taken out from the flow task information queue, a host computer running the flow task is connected, and whether the PID exists or not is checked, or the flow task state is detected through other information; if the PID exists, the running state of the streaming task is put into a running queue, and if the streaming task is not running any more, the streaming task is put into a reporting queue; starting a reporting thread, extracting running information data of the streaming task in the streaming task reporting queue, such as PID (proportion integration differentiation) and task ID (identity), and sending the running information data to a Master; the Master updates the stopped streaming task state to the database for persistent storage, so that the streaming task state is displayed on the workflow task scheduling platform interface.
The application supports the visual configuration of the batch-stream integrated task directed acyclic graph by modifying DolphinScheduler, expands DolphinScheduler functions according to the sequence of the directed acyclic graph, and supports stream task scheduling according to the data integration scene. The method meets the integral dispatching function of batch flow, and the data flow direction can be clearly seen through configuring and dispatching the directed acyclic graph flow direction. For operation and maintenance developers, the operation notice that the streaming task needs to be started from back to front and stopped from front to back is not needed, and the scheduled task can be better scheduled and managed only by configuring the corresponding DAG according to the data flow direction; and the operation, maintenance, management and scheduling costs of the data integration task are reduced.
Example two
Corresponding to the first embodiment, as shown in fig. 2, an embodiment of the present application further provides a batch task and stream task integrated processing system, including:
A copy module 1101, configured to perform a copy operation in response to a task execution instruction, the copy operation including: copying the current directed acyclic graph in the primary workflow task scheduling platform to form a first directed acyclic graph and a second directed acyclic graph;
The parsing module 1102 is configured to perform a streaming task parsing operation to determine a first task running order corresponding to a first task node capable of running, and a first task node capable of running in the first directed acyclic graph;
an operation module 1103, configured to perform a streaming task operation to operate a task corresponding to the first executable node according to the first task operation order;
The parsing module 1102 is further configured to perform a batch task parsing operation to determine a second prohibited task node, a second executable task node, and a second task running order corresponding to the second executable task node in the second directed acyclic graph;
the running module 1103 is further configured to perform a batch task running operation to run the task corresponding to the second executable node according to the second task running order.
In some implementations, the parsing module 1102 is further configured to mark all batch task nodes in the first directed acyclic graph as first no-run task nodes and mark all stream task nodes in the first directed acyclic graph as first runnable task nodes; the parsing module 1102 is further configured to obtain and reverse a first current task operation order in the first directed acyclic graph, and update the reverse first current task operation order to a first task operation order; the operation module 1103 is further configured to operate, according to a first task operation order of the first directed acyclic graph, a flow task corresponding to a first executable task node in the first directed acyclic graph; monitoring whether the flow task is submitted to run successfully or not; and after the success of the submitting operation of the streaming task is monitored, the streaming task of the next first executable task node is operated according to the first task operation sequence, and the first forbidden task node is skipped.
In some implementations, the parsing module 1102 is further configured to mark all flow task nodes in the second directed acyclic graph as second forbidden task nodes and mark all batch task nodes in the second directed acyclic graph as second runnable task nodes; acquiring a second current task operation sequence in the second directed acyclic graph and updating the second current task operation sequence to a second task operation sequence; the operation module 1103 is further configured to operate a batch task corresponding to a second executable task node in the first directed acyclic graph according to a second task operation sequence of the second directed acyclic graph; and monitoring the running state of the batch task, and after the batch task is successfully run, running the batch task of the next second runnable task node according to the second task running sequence and skipping over the second task node which is forbidden to run.
In some implementations, the system further includes a task stop processing module 1104 (not shown) that determines whether the currently running task includes a batch task in response to the task stop indication; if the current running task comprises a batch task, stopping running the batch task, analyzing a first directed acyclic graph, and stopping running the running stream task according to the running sequence of the first current task; if the current running task does not contain batch tasks, directly analyzing the first directed acyclic graph, and stopping running the running streaming task according to the running sequence of the first current task; the task stop processing module 1104 is further configured to sequentially execute the copy operation, the stream task analysis operation, the batch task analysis operation, and the stream task running operation in response to a resume stop task instruction; judging whether the task corresponding to the task node which stops running last time is a batch task or not; if the task is a batch task, starting to run the batch task from a task node which stops running last time according to a second current task running sequence in a second directed acyclic graph which is generated at present; and if the task is not a batch task, starting to run the batch task from a first task node in the current task running sequence according to the second current task running sequence.
In some implementation scenarios, the system further includes a task suspension processing module 1105 (not illustrated) configured to determine, in response to the task suspension indication, whether the currently running task includes a batch task; if the current running task comprises a batch task, stopping running after the running of the batch task running at present is finished; if the current running task does not contain batch tasks, after the task pause indication confirms that the running task needs to be paused, analyzing the first directed acyclic graph and stopping running the running streaming task according to the running sequence of the first current task; the task suspension processing module 1105 is further configured to perform the copy operation in response to a resume suspension task instruction; if no running streaming task exists, executing the streaming task analysis operation and the streaming task running operation; if the running streaming task exists, executing the batch task analysis operation, and judging whether the task corresponding to the task node which is suspended to run last time is a batch task or not; if the task is a batch task, starting to run the batch task from a task node which stops running last time according to a second current task running sequence in a second directed acyclic graph which is generated at present; and if the task is not a batch task, starting to run the batch task from a first task node in the second current task running sequence according to the second current task running sequence.
In some implementations, the system further includes a task failure processing module 1106 (not shown) for performing the copy operation in response to a recovery failure task indication; judging whether the failed task is a streaming task or not; if the failed task is a streaming task, executing the streaming task analysis operation and the streaming task operation; and if the failed task is not the streaming task, executing the batch task analysis operation and the batch task running operation.
In some implementations, the system further includes a timing module 1107 (not shown) for setting one or more batch task run time periods including a start time and an end time; responsive to the start time, performing a copy operation, a batch task parsing operation, and a batch task running operation; and stopping executing the batch task running operation in response to the ending time.
Example III
Corresponding to all the embodiments described above, an embodiment of the present application provides an electronic device, including: one or more processors; and a memory associated with the one or more processors, the memory for storing program instructions that, when read for execution by the one or more processors, perform the following:
In response to the task execution indication, performing a copy operation, the copy operation comprising: copying the current directed acyclic graph in the primary workflow task scheduling platform to form a first directed acyclic graph and a second directed acyclic graph;
Executing a stream task analysis operation to determine a first task operation prohibition node, a first operable task node and a first task operation sequence corresponding to the first operable node in the first directed acyclic graph;
executing a streaming task operation to operate the task corresponding to the first operable node according to the first task operation sequence;
executing batch task analysis operation to determine a second task operation forbidden task node, a second operable task node and a second task operation sequence corresponding to the second operable task node in the second directed acyclic graph;
And executing batch task operation to operate the task corresponding to the second executable node according to the second task operation sequence.
Fig. 12 illustrates an architecture of an electronic device, which may include a processor 1210, a video display adapter 1211, a disk drive 1212, an input/output interface 1213, a network interface 1214, and a memory 1220, among others. The processor 1210, the video display adapter 1211, the disk drive 1212, the input/output interface 1213, the network interface 1214, and the memory 1220 may be communicatively connected by a bus 1230.
The processor 1210 may be implemented by a general-purpose CPU (Central Processing Unit ), a microprocessor, an Application SPECIFIC INTEGRATED Circuit (ASIC), or one or more integrated circuits, etc. for executing related programs to implement the technical solution provided by the present application.
The Memory 1220 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory ), static storage, dynamic storage, etc. The memory 1220 may store an operating system 1221 for controlling execution of the electronic device 1200, and a Basic Input Output System (BIOS) 1222 for controlling low-level operation of the electronic device 1200. In addition, a web browser 1223, a data storage management system 1224, an icon font processing system 1225, and the like may also be stored. The icon font processing system 1225 may be an application program that implements the operations of the foregoing steps in the embodiment of the present application. In general, when implemented in software or firmware, the relevant program code is stored in memory 1220 and executed by processor 1210.
The input/output interface 1213 is used to connect with an input/output module to enable information input and output. The input/output module may be configured as a component in a device (not shown) or may be external to the device to provide corresponding functionality. Wherein the input devices may include a keyboard, mouse, touch screen, microphone, various types of sensors, etc., and the output devices may include a display, speaker, vibrator, indicator lights, etc.
The network interface 1214 is used to connect communication modules (not shown) to enable communication interactions of the present device with other devices. The communication module may implement communication through a wired manner (such as USB, network cable, etc.), or may implement communication through a wireless manner (such as mobile network, WIFI, bluetooth, etc.).
Bus 1230 includes a path to transfer information between components of the device (e.g., processor 1210, video display adapter 1211, disk drive 1212, input/output interface 1213, network interface 1214, and memory 1220).
In addition, the electronic device 1200 may also obtain information of specific acquisition conditions from the virtual resource object acquisition condition information database, for performing condition judgment, and so on.
It is noted that although the above devices illustrate only the processor 1210, video display adapter 1211, disk drive 1212, input/output interface 1213, network interface 1214, memory 1220, bus 1230, etc., the device may include other components necessary to achieve proper execution in an implementation. Furthermore, it will be appreciated by those skilled in the art that the apparatus may include only the components necessary to implement the present application, and not all of the components shown in the drawings.
From the above description of embodiments, it will be apparent to those skilled in the art that the present application may be implemented in software plus a necessary general hardware platform. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for causing a computer device (which may be a personal computer, a cloud server, or a network device, etc.) to execute the method described in the embodiments or some parts of the embodiments of the present application.
Example IV
Corresponding to all the above embodiments, the embodiments of the present application further provide a computer-readable storage medium, characterized in that it stores a computer program that causes a computer to perform the operations of:
In response to the task execution indication, performing a copy operation, the copy operation comprising: copying the current directed acyclic graph in the primary workflow task scheduling platform to form a first directed acyclic graph and a second directed acyclic graph;
Executing a stream task analysis operation to determine a first task operation prohibition node, a first operable task node and a first task operation sequence corresponding to the first operable node in the first directed acyclic graph;
executing a streaming task operation to operate the task corresponding to the first operable node according to the first task operation sequence;
executing batch task analysis operation to determine a second task operation forbidden task node, a second operable task node and a second task operation sequence corresponding to the second operable task node in the second directed acyclic graph;
And executing batch task operation to operate the task corresponding to the second executable node according to the second task operation sequence.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for a system or system embodiment, since it is substantially similar to a method embodiment, the description is relatively simple, with reference to the description of the method embodiment being made in part. The systems and system embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
The foregoing description of the preferred embodiments of the application is not intended to limit the application to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and scope of the application are intended to be included within the scope of the application.
Claims (10)
1. A batch task and stream task integrated scheduling processing method, the method comprising:
in response to the task execution indication, performing a copy operation, the copy operation comprising: copying the current directed acyclic graph in the primary workflow task scheduling platform to form a first directed acyclic graph and a second directed acyclic graph;
Executing a stream task analysis operation to determine a first task operation prohibition node, a first operable task node and a first task operation sequence corresponding to the first operable node in the first directed acyclic graph;
executing a streaming task operation to operate the task corresponding to the first operable node according to the first task operation sequence;
executing batch task analysis operation to determine a second task operation forbidden task node, a second operable task node and a second task operation sequence corresponding to the second operable task node in the second directed acyclic graph;
And executing batch task operation to operate the task corresponding to the second executable node according to the second task operation sequence.
2. The method of claim 1, wherein performing a streaming task parsing operation comprises:
marking all batch task nodes in the first directed acyclic graph as first forbidden task nodes and marking all stream task nodes in the first directed acyclic graph as first operable task nodes;
Acquiring and reversing a first current task operation sequence in the first directed acyclic graph, and updating the reversed first current task operation sequence to be a first task operation sequence;
the executing stream task running operation includes:
According to a first task operation sequence of the first directed acyclic graph, operating a flow task corresponding to a first operable task node in the first directed acyclic graph;
Monitoring whether the flow task is submitted to run successfully or not;
and after the success of the submitting operation of the streaming task is monitored, the streaming task of the next first executable task node is operated according to the first task operation sequence, and the first forbidden task node is skipped.
3. The method of claim 2, wherein performing a batch task parsing operation comprises:
marking all flow task nodes in the second directed acyclic graph as second forbidden task nodes and marking all batch task nodes in the second directed acyclic graph as second runnable task nodes;
Acquiring a second current task operation sequence in the second directed acyclic graph and updating the second current task operation sequence to a second task operation sequence;
The executing batch task running operation comprises the following steps:
According to a second task operation sequence of the second directed acyclic graph, operating batch tasks corresponding to second operable task nodes in the first directed acyclic graph;
And monitoring the running state of the batch task, and after the batch task is successfully run, running the batch task of the next second runnable task node according to the second task running sequence and skipping over the second task node which is forbidden to run.
4. A method according to claim 3, characterized in that the method further comprises:
responding to the task stop instruction, and judging whether the currently running task contains batch tasks or not;
If the current running task comprises a batch task, stopping running the batch task, analyzing a first directed acyclic graph, and stopping running the running stream task according to the running sequence of the first current task;
If the current running task does not contain batch tasks, directly analyzing the first directed acyclic graph, and stopping running the running streaming task according to the running sequence of the first current task;
Responding to a resume stop task instruction, and sequentially executing the copy operation, the stream task analysis operation, the batch task analysis operation and the stream task running operation;
Judging whether the task corresponding to the task node which stops running last time is a batch task or not;
if the task is a batch task, starting to run the batch task from a task node which stops running last time according to a second current task running sequence in a second directed acyclic graph which is generated at present;
And if the task is not a batch task, starting to run the batch task from a first task node in the current task running sequence according to the second current task running sequence.
5. A method according to claim 3, characterized in that the method further comprises:
responding to the task pause instruction, and judging whether the currently running task contains batch tasks or not;
If the current running task comprises a batch task, stopping running after the running of the batch task running at present is finished;
If the current running task does not contain batch tasks, after the task pause indication confirms that the running task needs to be paused, analyzing the first directed acyclic graph and stopping running the running streaming task according to the running sequence of the first current task;
In response to resuming the suspending task indication, performing the copy operation;
If no running streaming task exists, executing the streaming task analysis operation and the streaming task running operation;
if the running streaming task exists, executing the batch task analysis operation, and judging whether the task corresponding to the task node which is suspended to run last time is a batch task or not;
if the task is a batch task, starting to run the batch task from a task node which stops running last time according to a second current task running sequence in a second directed acyclic graph which is generated at present;
And if the task is not a batch task, starting to run the batch task from a first task node in the second current task running sequence according to the second current task running sequence.
6. The method according to claim 1, wherein the method further comprises:
responsive to a recovery failure task indication, performing the copy operation;
Judging whether the failed task is a streaming task or not;
if the failed task is a streaming task, executing the streaming task analysis operation and the streaming task operation;
and if the failed task is not the streaming task, executing the batch task analysis operation and the batch task running operation.
7. The method according to claim 6, characterized in that the method comprises:
setting one or more batch task operation time periods, wherein the batch task operation time periods comprise a starting time and an ending time;
Responsive to the start time, performing a copy operation, a batch task parsing operation, and a batch task running operation;
And stopping executing the batch task running operation in response to the ending time.
8. A batch and streaming task integrated processing system, the system comprising:
a copy module for performing a copy operation in response to a task execution instruction, the copy operation comprising: copying the current directed acyclic graph in the primary workflow task scheduling platform to form a first directed acyclic graph and a second directed acyclic graph;
the analysis module is used for executing flow task analysis operation to determine a first task operation forbidden task node, a first operable task node and a first task operation sequence corresponding to the first operable task node in the first directed acyclic graph;
The running module is used for executing a streaming task running operation to run tasks corresponding to the first runnable node according to the first task running sequence;
the analysis module is further configured to perform batch task analysis operations to determine a second task operation prohibition node, a second task node capable of being operated, and a second task operation sequence corresponding to the second task node capable of being operated in the second directed acyclic graph;
the operation module is further configured to perform a batch task operation to operate a task corresponding to the second executable node according to the second task operation order.
9. An electronic device, the electronic device comprising:
One or more processors;
And a memory associated with the one or more processors, the memory for storing program instructions that, when read for execution by the one or more processors, perform the method of any of claims 1-7.
10. A computer-readable storage medium, characterized in that it stores a computer program, which causes a computer to perform the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410136081.1A CN118069314A (en) | 2024-01-31 | 2024-01-31 | Batch task and stream task integrated scheduling processing method, system, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410136081.1A CN118069314A (en) | 2024-01-31 | 2024-01-31 | Batch task and stream task integrated scheduling processing method, system, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118069314A true CN118069314A (en) | 2024-05-24 |
Family
ID=91094512
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410136081.1A Pending CN118069314A (en) | 2024-01-31 | 2024-01-31 | Batch task and stream task integrated scheduling processing method, system, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118069314A (en) |
-
2024
- 2024-01-31 CN CN202410136081.1A patent/CN118069314A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113569987A (en) | Model training method and device | |
| JP5365051B2 (en) | Management program, management apparatus and management method | |
| CN107491346B (en) | Application task processing method, device and system | |
| US7383470B2 (en) | Method, system, and apparatus for identifying unresponsive portions of a computer program | |
| CN109656782A (en) | Visual scheduling monitoring method, device and server | |
| US20140297355A1 (en) | Workflow control apparatus and method therefor | |
| KR20200078328A (en) | Systems and methods of monitoring software application processes | |
| CN109144701A (en) | A kind of task flow management method, device, equipment and system | |
| US12106143B2 (en) | Scheduling complex jobs in a distributed network | |
| CN110109741B (en) | Method and device for managing circular tasks, electronic equipment and storage medium | |
| CN113157569B (en) | Automated testing method, apparatus, computer device and storage medium | |
| JP2017016507A (en) | Test management system and program | |
| CN117252559A (en) | Business process processing method, device, computer equipment and storage medium | |
| CN118069314A (en) | Batch task and stream task integrated scheduling processing method, system, equipment and medium | |
| CN115080093B (en) | Distributed system upgrading method, device, server and medium | |
| CN116319758A (en) | Data migration method, device, electronic equipment and readable storage medium | |
| CN105868957A (en) | Continuous integration method and device | |
| JP5387083B2 (en) | Job management system and method | |
| CN115437766A (en) | Task processing method and device | |
| JP2005266919A (en) | System analysis device and analysis system | |
| CN113127162B (en) | Automatic task execution method and device, electronic equipment and computer storage medium | |
| CN114328090A (en) | Program monitoring method and device, electronic equipment and storage medium | |
| CN111858234A (en) | Task execution method, device, equipment and medium | |
| JP2006260528A (en) | Computer control method, management computer, and its processing program | |
| JPH1196046A (en) | Journal acquiring device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |