CN117971412A

CN117971412A - Task dependency scheduling method supporting dependency arrangement

Info

Publication number: CN117971412A
Application number: CN202311741073.1A
Authority: CN
Inventors: 张进; 曹银通; 李迪砺; 苏春园
Original assignee: Hangzhou Guanyuan Data Co ltd
Current assignee: Hangzhou Guanyuan Data Co ltd
Priority date: 2023-12-18
Filing date: 2023-12-18
Publication date: 2024-05-03

Abstract

The application relates to a task dependency scheduling method supporting dependency scheduling, which can enable a background developer to clearly know which task nodes have the labels of dependency tasks through task nodes and task configuration based on Maven project directory structure scheduling, and simultaneously track the node position IDs of the task nodes where the subtasks of the dependency tasks exist according to the IDs of the labels, and automatically track the dependency nodes of the task nodes according to a workflow to execute the dependency tasks, so that the dependency nodes where the dependency tasks exist can be monitored, tracked and executed in time, manual intervention is replaced, the time period of judging the dependency tasks by the manual intervention is shortened, and development time is shortened. After checking whether each task node depends on other tasks, the background can execute priority setting, execute according to the priority, widen the task arrangement function of data stream arrangement, and enhance the arrangement capability of the platform.

Description

Task dependency scheduling method supporting dependency arrangement

Technical Field

The disclosure relates to the technical field of data development, in particular to a task dependency scheduling method, a system and electronic equipment supporting dependency scheduling.

Background

The task node types supported by the workflow include: SQL/Python multiple language development nodes, FILE type cloud data storage acquisition nodes, data quality inspection nodes and extensible Java plug-in nodes, and provide workflow management capability for the nodes.

The technical architecture of Universe (Platform) data development platform as shown in figure 1 enables workflow and data stream development on the platform. The front-end user performs work management corresponding workflow (such as financial docking data, accounting data, business management data and single-store single-product management movable sales data) and can be managed and displayed by a Universe (Platform) data development platform deployed in the background, the front-end user can log in the background to enter a Universe (Platform) data development platform, and task scheduling, data management and the like of the workflow are realized through a basic service module.

In the scheduling work with the dependent tasks, when a certain scheduling workflow is executed, whether the dependent tasks are stored or not needs to be judged, and if yes, whether the dependent tasks are executed or not needs to be judged. If the execution monitoring of the dependent tasks is not available, the arrangement workflow may be interrupted and the arrangement may be disabled.

In task scheduling, a developer is required to actively intervene in judging a dependent task of a certain workflow node (task node of a workflow), and when a dependent task exists in a certain workflow node, the dependent task on the node needs to be called and executed. The mode of manually intervening dependent tasks is delayed, and the development period is increased.

Therefore, good task-dependent scheduling flows are also smooth conditions to ensure a orchestration workflow.

Disclosure of Invention

In order to solve the problems, the application provides a task dependency scheduling method, a system and electronic equipment for supporting dependency programming.

In one aspect of the present application, a task dependency scheduling method supporting dependency scheduling is provided, including the following steps:

Arranging task nodes of the workflow based on a Maven item directory structure, and recording and storing node position IDs of the task nodes on the Maven item directory structure to the background;

The background configures corresponding subtasks for each task node, and marks the node position ID of the task node where the subtask with the dependent task is located;

When workflow is arranged, each task node reports the respective task state to the background in real time, and the background judges whether the node position ID of the current task node has a task-dependent mark or not according to the task state:

If the mark of the dependent task is provided, scheduling and executing the dependent task;

otherwise, the next subtask is performed.

As an optional embodiment of the present application, optionally, arranging task nodes of the workflow based on a Maven item directory structure, and recording and saving node position IDs of the task nodes on the Maven item directory structure to a background, including:

analyzing the scheduled task and determining each task root node of the scheduled task;

determining father-son relation of subtasks according to task flow of workflow, and setting corresponding father-son task nodes for the task root nodes;

According to the task root node and the associated father-son task nodes, arranging the task nodes of the workflow based on a Maven project directory structure, and recording the node position IDs of each task root node and the associated father-son task nodes;

and reporting the node position IDs to a background.

As an optional embodiment of the present application, optionally, configuring, by the background, a corresponding subtask for each task node, and marking a node location ID of the task node where the subtask where the dependent task exists, including:

Traversing each task node in the Maven item directory structure according to the node position ID of each task node;

configuring corresponding subtasks for each task node according to the attribute of the task node, and binding the subtasks with the corresponding node position IDs;

Traversing the subtasks bound under the node position ID, and judging whether associated dependent tasks exist in the subtasks of each task node based on a text recognition algorithm:

if the dependent task exists, marking the node position ID of the task node where the subtask with the dependent task exists, and marking the mark with the dependent task;

and otherwise, giving up.

As an optional embodiment of the present application, optionally, the background configures a corresponding subtask for each task node, marks a node location ID of the task node where the subtask where the dependent task exists, and further includes:

according to the requirement of task priority, setting the task priorities of the subtasks and the dependent tasks associated with the subtasks for the subtasks with the dependent tasks respectively: d1 and D2, and saving the set task priority in the background.

As an optional embodiment of the present application, optionally, after the background determines that the node location ID of the current task node has a task-dependent flag according to the task state, the method further includes:

judging whether the subtasks of the current task node have the set task priority or not:

If the subtask of the current task node has the set task priority, further judging the task priorities of the subtask and the dependent tasks associated with the subtask: relationship of D1 and D2:

If:

D1≤D2，

Preferentially executing the subtasks;

If:

D1＞D2，

Preferentially executing the dependent task;

And if the subtasks of the current task node do not have the set task priority, preferentially executing the dependent tasks and then executing the subtasks.

And providing complex scheduling strategy configuration such as scheduling priority, failure retry and the like by means of scheduling arrangement functions.

Task dependent scheduling is an algorithm for determining the execution order of tasks to avoid conflicts between tasks and to ensure proper execution of tasks.

In another aspect of the present application, a system for implementing the task dependency scheduling method supporting dependency orchestration is provided, including:

The task node configuration module is used for arranging task nodes of the workflow based on a Maven project directory structure, and recording and storing node position IDs of the task nodes on the Maven project directory structure to the background;

The node marking module is used for configuring corresponding subtasks for each task node in the background and marking the node position ID of the task node where the subtask with the dependent task is located;

The task scheduling module is used for reporting the task state of each task node to the background in real time when the workflow is arranged, and judging whether the node position ID of the current task node has a task-dependent mark or not according to the task state by the background:

otherwise, the next subtask is performed.

In another aspect, the present application further provides an electronic device, including:

A processor;

a memory for storing processor-executable instructions;

Wherein the processor is configured to implement the one task dependency scheduling method supporting dependency orchestration when executing the executable instructions.

The invention has the technical effects that:

According to the application, workflow arrangement is carried out through the Maven project directory structure, and through task nodes and task configuration based on the Maven project directory structure arrangement, a background developer can clearly know which task nodes have the marks of dependent tasks, meanwhile, according to the ID of the marks, the node position ID of the task node where the subtask with the dependent task exists can be tracked, and the execution of the dependent task is carried out by automatically tracking the dependent node of the task node according to the workflow, so that monitoring tracking and task execution can be carried out on the dependent node where the dependent task exists in time, manual intervention is replaced, the time period of judging the dependent task by manual intervention is shortened, and development time is shortened.

In the scheme, in the process of task arrangement of the workflow, the system supports task priority based on sub-workflows and dependent nodes, tasks executed in parallel are scheduled to be executed, and after checking whether each task node depends on other tasks or not, the background can execute priority setting. The dependent tasks and the dependent subtasks in the scheme have task priorities, the priority level between the dependent tasks and the dependent subtasks is determined according to the dependent tasks to be scheduled, and the dependent tasks and the dependent subtasks are executed respectively according to the priorities, so that the task arrangement function of data stream arrangement can be widened, and the arrangement capacity of a platform is enhanced.

Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments, features and aspects of the present disclosure and together with the description, serve to explain the principles of the disclosure.

FIG. 1 is a schematic diagram of a technical architecture of a data development platform of the present invention;

FIG. 2 shows a schematic flow chart of an embodiment of the invention;

FIG. 3 is a schematic diagram of a Maven project directory structure of the present invention;

FIG. 4 is a schematic diagram of task nodes corresponding to a Maven programming link in the present invention;

fig. 5 shows a schematic application diagram of the electronic device of the present invention.

Detailed Description

Various exemplary embodiments, features and aspects of the disclosure will be described in detail below with reference to the drawings. In the drawings, like reference numbers indicate identical or functionally similar elements. Although various aspects of the embodiments are illustrated in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

In addition, numerous specific details are set forth in the following detailed description in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, well known means, elements, and circuits have not been described in detail so as not to obscure the present disclosure.

Example 1

As shown in fig. 2, in one aspect of the present application, a task dependency scheduling method supporting dependency scheduling is provided, including the following steps:

s1, arranging task nodes of the workflow based on a Maven item directory structure, and recording and storing node position IDs of the task nodes on the Maven item directory structure to the background;

S2, configuring corresponding subtasks for each task node by the background, and marking the node position ID of the task node where the subtask with the dependent task is located;

S3, when workflow is arranged, reporting the respective task state to a background in real time by each task node, and judging whether the node position ID of the current task node has a task-dependent mark or not by the background according to the task state:

otherwise, the next subtask is performed.

In order to enable background developers to clearly know the layout data of the development task, when a workflow is laid out, task nodes of the workflow are laid out based on a Maven project directory structure, and node position IDs of the task nodes on the Maven project directory structure are recorded and stored to the background. The background developer is enabled to arrange the task nodes of the workflow through the task structure tree of the Maven project directory structure, the background can clearly know and judge whether the dependent tasks are stored on each development task node or not, the dependent tasks are executed timely, the subtasks executed by each task node are monitored through the background, and whether the subtasks are associated with the corresponding dependent tasks or not is judged, so that the danger that the workflow arrangement is interrupted and the workflow arrangement is abolished due to lack of monitoring on the execution of the dependent tasks is avoided.

The main goal of Maven is to allow developers to understand that the development work is complete in the shortest time. The construction process is simplified, a unified construction system is provided, high-quality project information is provided, and better development practice is encouraged.

As shown in fig. 3, the following functions can be achieved through the data flow development management of the Maven project directory structure:

1. Output management

Maven can manage the product of project construction and add it to the user library. This function can be used for delivery actions between the project team and other departments.

2. Dependency relationship

The maven carries out careful analysis and division on the characteristics of the dependency relationship, avoids the dependence confusion and mutual pollution behavior in the development process, can carry out association binding on subtasks executed by a certain task node and the carried dependent tasks, and enables a background to clearly know which task node position IDs have the mark of the dependent tasks in a traversing structure mode of each node: and if the mark of the dependent task is provided, scheduling and executing the dependent task. Therefore, the manual intervention can be replaced, and the tracking and addressing time of the dependent tasks is shortened.

3. Document and construction results

The site command of maven supports the release of various document information including various outputs of the build process, java doc, product documents, etc.

4. Project relationship

A large project is typically made up of several small items or modules, which can be easily managed by maven.

5. Portability management

Maven can output different kinds of output results aiming at different development scenes.

6. Contract is superior to configuration

Although the established dependent node performs preferentially over the dependent subtasks (i.e., having associated task nodes and its dependent nodes, the dependent node should be executed first).

While in maven they are not completely unmodified, they are just default values for some configurations. But need not be modified to suit those conventions unless necessary. In the maven default file storage structure, a strategy of 'contract preferred over configuration' can be adopted, so that a background user can carry out self-defined preferential configuration, the workload of configuration modification can be reduced, and the learning cost can be reduced.

Specific please understand in conjunction with the following description of task priorities.

and reporting the node position IDs to a background.

The task scheduling method comprises the steps that a background developer can specifically determine the task scheduling of the workflow, the workflow flow of the task scheduling can be obtained by analyzing a task scheduling data packet issued at the time, and each task node of the task is determined according to the flow.

In one scheduling task, the method comprises a plurality of root nodes and a plurality of subtasks below each root node, namely, each root node and each task node form a parent-child relationship, so that the associated subtasks behind each root node are hung to configure different subtasks. In a certain orchestration subtask, a task node executes a management task, and other subtasks below the task node are used for realizing other subtask tasks, such as relying on tasks, and each task node is determined according to task requirements required by a process.

And on the data development platform, arranging and constructing each task node according to the Maven project directory structure and carrying out association. After each task node is arranged and set according to the Maven project directory structure, node position IDs of each node can be uniformly recorded and stored in the background, so that a background developer can conveniently identify subtasks of each task node in a mode of traversing the node position IDs.

The specific traversal may be performed using a traversal process or other plug-in tool.

A workflow node arrangement diagram is shown in fig. 4. According to the task arrangement, links to be constructed are determined first, corresponding root nodes are configured, and each root node executes a main task.

If other subtasks need to be set under a certain main task, corresponding task nodes for executing the subtasks are set under the corresponding root nodes, for example, a packing link in the corresponding task nodes needs to be implemented by two task nodes for executing the subtasks.

If the task node where the packing link is located needs to generate a dependent task, for example, before jar packing, the dependent task needs to be executed by depending on other nodes, a dependent node can be added on the task node for executing a corresponding dependent task, for example, a dependent node is constructed on the task node where the Java engineering is located for preprocessing jar packets, and the like.

and otherwise, giving up.

The background developer can traverse each task node in the Maven project directory structure through the background, traverse through the position information of the node position ID, know the attribute of each task node according to the obtained node information of each task node, and judge the scheduling task adapted by each task node.

The background developer can store all the main tasks and the subtasks in the task in the database in advance, and after the attribute of each task node is obtained by the background, the corresponding subtasks are automatically configured for each task node according to the attribute.

The master task where each root node is located can be manually configured by a background developer.

And for the subtasks required to be executed by each task node behind each root node, carrying out automatic configuration by the background, automatically configuring the corresponding subtasks for the task node according to the acquired attribute of each task node, and binding the configured subtasks with the node position IDs of the corresponding task nodes, so that the corresponding dependency relationship can be conveniently obtained from Maven in subsequent visualization.

The specific subtasks can be set corresponding task data packages and the like by a background developer according to the scheduled tasks. The corresponding attribute labels can be pressed, and after the attributes of the task nodes are obtained through identification, the corresponding subtasks can be automatically configured for the task nodes according to the matching of the attributes and the attribute labels. For example, in the workflow compiling process, attribute matching is performed on task nodes for executing workflow compiling, and compiling tasks are configured for the task nodes.

It is possible to include dependent tasks in the subtasks that need to be performed. Therefore, in order to facilitate the subsequent quick finding out whether each task node has a dependent task, the background obtains the subtasks of each task node by traversing each node position ID.

After the subtasks are obtained, task data text in the subtasks can be obtained, and the background judges whether the text data of the subtasks contain the associated dependent tasks or not through text recognition. Specifically, the background can call a text recognition plug-in, recognize the text data in each subtask depending on the task keywords based on a text recognition algorithm, and judge whether the depending keywords exist or not:

If the corresponding keyword exists, the subtask is determined to contain the dependent task.

The dependency keywords corresponding to the specific dependency tasks can be set in the keyword database in advance by a background developer.

The background judges whether the corresponding dependent tasks exist or not in a text recognition and keyword comparison mode, if yes, the background marks the corresponding task nodes, then the background knows that the task nodes have the dependent labels, a background developer can quickly recognize which task nodes have the dependent tasks through a Maven directory structure, and the dependent tasks of the task nodes are automatically and preferentially executed.

Based on the Maven editing directory structure, the scheme gives a certain priority setting authority to the background personnel, so that the background personnel can set the priority to the subtasks with dependent tasks according to the demands of task priorities, and the priority can be higher than the priority of the dependent tasks, thereby changing the priority of the default dependent tasks over the subtasks.

Specifically, the method is carried out according to the needs.

Through the editing directory structure of Maven, the editing capability of the workflow can be expanded, more tasks and editing functions are carried out, and the development system is enriched.

If:

D1≤D2，

Preferentially executing the subtasks;

If:

D1＞D2，

Preferentially executing the dependent task;

And the scheduling-dependent scheduling function provides complex scheduling policy configuration such as scheduling priority, failure retry and the like.

By determining the task priority of the subtask and the dependent task associated therewith: and D1 and D2, determining task dependent scheduling execution priority, so as to determine the execution sequence of the tasks, avoid conflict among the tasks and ensure correct execution of the tasks.

Task priorities of the subtasks and the dependent tasks associated therewith: d1 and D2 are specifically set by a background developer according to the development work executed by each task node.

It should be apparent to those skilled in the art that implementing all or part of the above-described embodiments may be accomplished by computer programs to instruct related hardware, and the programs may be stored in a computer readable storage medium, which when executed may include the processes of the embodiments of the controls described above. It will be appreciated by those skilled in the art that implementing all or part of the above-described embodiments may be accomplished by computer programs to instruct related hardware, and the programs may be stored in a computer readable storage medium, which when executed may include the processes of the embodiments of the controls described above. The storage medium may be a magnetic disk, an optical disk, a Read-only memory (ROM), a random access memory (RandomAccessMemory, RAM), a flash memory (flash memory), a hard disk (HARDDISKDRIVE, abbreviated as HDD), a Solid state disk (Solid-state STATEDRIVE, SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.

Example 2

Based on the implementation principle of embodiment 1, in another aspect, the present application proposes a system for implementing the task dependent scheduling method supporting dependency orchestration, where the system includes:

otherwise, the next subtask is performed.

The above modules are described in conjunction with embodiment 1, and the functions and interactions are not described herein. The task scheduling module can refer to the development platform in fig. 1 for application.

The modules or steps of the invention described above may be implemented in a general-purpose computing system, they may be centralized in a single computing system, or distributed across a network of computing systems, where they may alternatively be implemented in program code executable by a computing system, where they may be stored in a memory system and executed by a computing system, where they may be separately fabricated into individual integrated circuit modules, or where multiple modules or steps within them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

Example 3

As shown in fig. 5, in still another aspect, the present application further provides an electronic device, including:

A processor;

a memory for storing processor-executable instructions;

Embodiments of the present disclosure provide for an electronic device that includes a processor and a memory for storing processor-executable instructions. Wherein the processor is configured to implement any one of the task dependency scheduling methods described above that support dependency orchestration when executing the executable instructions.

Here, it should be noted that the number of processors may be one or more. Meanwhile, in the electronic device of the embodiment of the disclosure, an input system and an output system may be further included. The processor, the memory, the input system, and the output system may be connected by a bus, or may be connected by other means, which is not specifically limited herein.

The memory is a computer-readable storage medium that can be used to store software programs, computer-executable programs, and various modules, such as: a task dependency scheduling method supporting dependency orchestration according to an embodiment of the present disclosure corresponds to a program or module. The processor executes various functional applications and data processing of the electronic device by running software programs or modules stored in the memory.

The input system may be used to receive an input digital or signal. Wherein the signal may be a key signal generated in connection with user settings of the device/terminal/server and function control. The output system may include a display device such as a display screen.

The foregoing description of the embodiments of the present disclosure has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the technical improvement of the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A task dependency scheduling method supporting dependency arrangement is characterized by comprising the following steps:

otherwise, the next subtask is performed.

2. The task dependency scheduling method supporting dependency scheduling according to claim 1, wherein the task nodes of the workflow are scheduled based on a Maven project directory structure, and node position IDs of the task nodes on the Maven project directory structure are recorded and stored to the background, and the method comprises:

and reporting the node position IDs to a background.

3. The task dependency scheduling method supporting dependency scheduling according to claim 2, wherein the background configures corresponding subtasks for each task node, marks a node location ID of the task node where the subtask where the dependent task exists, and includes:

and otherwise, giving up.

4. The task dependency scheduling method supporting dependency scheduling according to claim 1, wherein the task nodes of the workflow are scheduled based on a Maven project directory structure, and node position IDs of the task nodes on the Maven project directory structure are recorded and stored to the background, and the method comprises:

and reporting the node position IDs to a background.

5. The task dependency scheduling method supporting dependency scheduling according to claim 1, wherein the configuring, in the background, a corresponding subtask for each task node, and marking a node location ID of the task node where the subtask where the dependent task exists, includes:

and otherwise, giving up.

6. The task dependency scheduling method supporting dependency scheduling according to claim 5, wherein the background configures corresponding subtasks for each task node, marks a node location ID of the task node where the subtask where the dependent task exists, and further comprising:

7. The task dependency scheduling method supporting dependency scheduling according to claim 6, further comprising, after the background determines that the node location ID of the current task node has a flag of a dependent task according to the task state:

If:

D1≤D2，

Preferentially executing the subtasks;

If:

D1＞D2，

Preferentially executing the dependent task;

8. A system for implementing a task dependent scheduling method supporting dependency orchestration according to any one of claims 1-7, comprising:

otherwise, the next subtask is performed.

9. An electronic device, comprising:

A processor;

a memory for storing processor-executable instructions;

Wherein the processor is configured to implement the task dependent scheduling method supporting dependency orchestration of any one of claims 1-7 when executing the executable instructions.