WO2020232951A1

WO2020232951A1 - Task execution method and device

Info

Publication number: WO2020232951A1
Application number: PCT/CN2019/110466
Authority: WO
Inventors: 夏勇; 李凌
Original assignee: 深圳前海微众银行股份有限公司
Priority date: 2019-05-17
Filing date: 2019-10-10
Publication date: 2020-11-26
Also published as: CN110147273A

Abstract

The present invention discloses a task execution method and device. Said method comprises: a first processing device acquiring a predefined task, the predefined task comprising N steps, N being an integer greater than 1; the first processing device generating, according to a dependency relationship existing between the N steps, an execution directed graph corresponding to the predefined task, the execution directed graph being a directed graph indicating an execution order of the N steps determined according to the dependency relationship, and in the case of any two steps among said N steps without said dependency relationship, the directed graph indicating that said any two steps are executed in parallel; and the first processing device executing the predefined task according to the step execution order indicated by the execution directed graph. When applied to the field of finance technology (Fintech), said method improves the efficiency of financial transactions.

Description

Task execution method and device

Cross references to related applications

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 201910413986.8, and the application name is "a task execution method and device" on May 17, 2019, the entire content of which is incorporated into this application by reference .

Technical field

The present invention relates to the field of task processing and financial technology (Fintech), in particular to a task execution method and device.

Background technique

With the development of computer technology, more and more technologies (Big Data, Distributed, Blockchain, Artificial Intelligence, etc.) are applied in the financial field, and the traditional financial industry is gradually changing to Finteh. At present, in the field of financial technology, in order to achieve a certain requirement or purpose, a series of instruction combinations that a processing device needs to execute is called a task. For example, the combination of data insertion and data modification on a data table in the database is called a data processing task. A task consists of one or more steps.

In the prior art, since the task execution requires high stability and reliability in the process of financial transactions, it is necessary to execute step by step sequentially until all steps are completed. Obviously, in this step execution mode, each step can only be executed after the previous step is executed, and the waiting time is longer. Once the previous step fails to execute again, it will waste more time. In this way, the execution time of the entire task is longer and the execution efficiency is lower.

Therefore, in the prior art, a long task execution time and low execution efficiency are a problem to be solved urgently.

Summary of the invention

The embodiments of the present application provide a task execution method and device, which solve the problems of long execution time and low execution efficiency of tasks in the prior art.

An embodiment of the present application provides a task execution method, including: a first processing device acquires a predefined task; the predefined task includes N steps; N is an integer greater than 1, and the first processing device according to the N The dependency relationship between steps is generated, and the execution directed graph corresponding to the predefined task is generated; the execution directed graph is a directed graph indicating the execution order of the N steps determined according to the dependency relationship, if If any two steps of the dependency relationship do not exist among the N steps, the directed graph indicates that any two steps are executed in parallel; the first processing device executes the steps indicated by the directed graph The execution sequence is to execute the predefined tasks.

Optionally, the first step is any one of the N steps, and the first processing device executes the first step in the following manner: if the first step includes only one sub-step, then directly execute the Sub-step; if the first step includes M sub-steps, execute the M sub-steps in parallel, and M is an integer greater than 1.

Optionally, the method further includes: if any step that has the dependency relationship with the first step is not defined in the predefined task, executing the predefined task first when the first processing device executes the predefined task The first step.

Optionally, the method further includes: the first processing device determining whether the predefined task has sent heartbeat data within a preset time period, if not, and determining that the running state of the predefined task is running, Then, a notification message indicating to host the predefined task is sent to the second processing device.

Optionally, the method further includes: if the first processing device determines that there are at least two failed execution steps among the N steps, re-executing the at least two failed execution steps in parallel.

In the embodiment of the present application, the steps may be executed by defining the execution directed graph in a predefined task. The execution directed graph is a directed graph indicating the execution order of the N steps determined according to the dependency relationship. Figure, if any two steps of the dependency relationship do not exist in the N steps, the directed graph indicates that the any two steps are executed in parallel. In addition, the execution directed graph also indicates the dependency relationship existing among the N steps, thereby ensuring that tasks are processed normally according to the preset execution sequence. Therefore, the execution time of each step in any two steps overlaps, and they are not affected by execution failure or not, thereby shortening the execution time of the entire task while ensuring the stable execution of the predefined task. Improve the efficiency of task execution.

The embodiment of the present application provides a task execution device, which includes: an acquisition module for acquiring a predefined task; the predefined task includes N steps; N is an integer greater than 1, and a processing module is used for The dependency relationship between steps is generated, and the execution directed graph corresponding to the predefined task is generated; the execution directed graph is a directed graph indicating the execution order of the N steps determined according to the dependency relationship, if If any two steps of the dependency relationship do not exist among the N steps, the directed graph indicates that any two steps are executed in parallel; the execution of the steps is in the order indicated by the executed directed graph. Pre-defined tasks.

Optionally, the first step is any one of the N steps, and the processing module is specifically configured to: execute the first step in the following manner: if the first step includes only one sub-step, directly Perform this sub-step; if the first step includes M sub-steps, execute the M sub-steps in parallel, and M is an integer greater than 1.

Optionally, the processing module is further configured to: if any step that has the dependency relationship with the first step is not defined in the predefined task, execute the first step first when executing the predefined task. step.

Optionally, the processing module is further configured to: determine whether the predefined task has sent heartbeat data within a preset period of time, if not, and if it is determined that the running state of the predefined task is running, then send the pre-configuration The device sends a notification message indicating to host the predefined task.

Optionally, the processing module is further configured to: if the first processing device determines that there are at least two failed execution steps among the N steps, re-execute the at least two failed execution steps in parallel.

The embodiment of the present application provides a computer device, including a program or instruction. When the program or instruction is executed, a task execution method and optional method provided in the embodiment of the present application are executed.

The embodiment of the present application provides a storage medium including a program or instruction. When the program or instruction is executed, a task execution method and optional method provided in the embodiment of the present application are executed.

Description of the drawings

FIG. 1 is a schematic diagram of a predefined task in an embodiment of this application;

2 is a schematic flowchart of a task execution method in an embodiment of this application;

FIG. 3 is a schematic structural diagram of a task execution device in an embodiment of the application.

Detailed ways

In order to better understand the above technical solutions, the above technical solutions will be described in detail below with reference to the drawings and specific implementations of the specification. It should be understood that the embodiments of the application and the specific features in the embodiments are detailed to the technical solutions of the application. Note, rather than limiting the technical solution of the present application, the embodiments of the present application and the technical features in the embodiments can be combined with each other if there is no conflict.

With the development of computer technology, more and more technologies (Big Data, Distributed, Blockchain, Artificial Intelligence, etc.) are applied in the financial field, and the traditional financial industry is gradually changing to Finteh. At present, in the field of financial technology, in order to achieve a certain requirement or purpose, a processing device often needs to perform a series of steps. The combination of this series of steps is called a task. For example, the task of modifying data table A in the database to data table B includes a series of steps of data insertion and data modification to the data table, and the whole task can be called a data processing task. For another example, collect the running logs of a computer within a certain period of time and perform statistics on these running logs to analyze the running status of this computer. The whole task can be called a log statistics task.

The current way of executing the task is to execute each step in the task serially, that is, step by step and proceed sequentially until all steps are completed. For example, task one has 5 steps: step one, step two, step three, step four, step five, regardless of whether there are necessary constraints between step one, step two, step three, step four, and step five. Relationship, steps 1 to 5 are executed in series. Among them, the necessary sequential execution constraint relationship refers to the sequential execution sequence relationship that must be followed when the two steps are executed. For example, to modify a piece of data in a data table, you must first find and locate this data, and then modify it. Obviously, in this way of serially executing steps, each step must be executed after the previous step is executed, and the waiting time is longer. If the previous step fails to execute and needs to be executed again, it will waste more time. In this way, the execution time of the entire task is longer, especially when there are many steps, the execution efficiency is lower.

To this end, in order to solve the problems of long execution time and low execution efficiency of tasks in the prior art, an embodiment of the present application provides a task execution method.

The embodiment of the application provides a task processing model based on a directed graph. Directed graph: In mathematics, a graph is a method of representing the relationship between objects and objects, and is the basic research object of graph theory. A graph appears to be composed of small dots (called vertices or nodes) and straight lines or curves (called edges) connecting these dots. If a direction is specified for each edge of the graph, the resulting graph is called a directed graph.

Before the task is executed, the task needs to be defined first, which can be divided into three parts: task definition, step definition and step dependency definition. According to the definition, a task model of directed graph can be generated. Among them, the task definition is to name the task, such as data processing task 1, task Q, etc. The name does not limit the character type; the step definition includes the definition of each step in the task and the specific execution method of each step. Some steps are also available. There are several sub-steps, which are also given in the step definition. In addition, when the step is defined, you can define the step name, implementation class, how to split data, execute thread size, read data size, commit transaction size, log level, etc. ; The step dependency is: in the N steps, before each step is executed, the constraint relationship of the execution results of other steps needs to be obtained. Specifically, for any two steps: step A and step B, if step A and step B meet any of the following conditions, it is said that there is a dependency between step A and step B: (1) Step A must be executed before step A is executed B, and the next step to be executed after step B is executed is step A, step A needs to be executed on the basis of the execution result of step B, also called step A depends on step B; (2) step B must be executed before step B is executed A, and the next step that needs to be executed after step A is executed is step B, and step B needs to be executed on the basis of the execution result of step A, which is called step B depends on step A. Obviously, there can be multiple steps that depend on step A or step B in the task. It should be noted that if a step in the task does not need to be executed before any other steps in the task are executed, then this step is called the apex step of the task, that is, the first step executed when the task is executed.

Take a specific example to illustrate the definition of a task:

Task definition: task X.

Step definition: Task X includes step X1, step X2, step X3, step X4, step X5, and step X6. Wherein, step X3 includes two sub-steps: step X3-1, step X3-2; step X6 includes three sub-steps: step X6-1, step X6-2, and step X6-3.

Step dependency definition: step X1 is the vertex step; step X2, step X3 needs to be executed before step X1; step X4 and step X5 need to be executed before step X2; step X6 needs to be executed before step X3, step X4 and step X5, Therefore, step X6 is the last step executed by task X. In addition, the step dependency can be defined by storing the step dependency definition table in the database. If the step dependency needs to be adjusted between steps, you only need to modify the step dependency definition table, which improves the flexibility of task definition.

The following takes FIG. 1 as an example to describe in detail the task definition in the embodiment of the present application. As shown in Figure 1, it is a schematic diagram of a task definition in an embodiment of this application. The task shown in FIG. 1 is named task Y and includes step Y1, step Y2, step Y3, step Y4, step Y5, and step Y6. According to the task definition, a corresponding directed graph as shown in Figure 1 can be generated.

It can be seen from Figure 1 that step Y1 is the vertex step of task Y; step Y1 needs to be executed before step Y2, step Y3, and step Y5 are executed; and in task Y, after step Y2 and step Y3 are executed, the next step that needs to be executed The step is step Y4, and the prerequisite for the execution of step Y4 is that both steps Y2 and Y3 are executed; after both steps Y4 and Y5 are executed, the next step to be executed is step Y6, and the premise of step Y6 is step Y4 , Step Y5 has been executed, and step Y6 is divided into step Y6-1 and step Y6-2.

The following describes in detail the execution steps of task Y shown in FIG. 1 with reference to FIG. 2. As shown in FIG. 2, it is a schematic flowchart of a task execution method in an embodiment of this application. The task execution method shown in FIG. 2 can be applied to Figure 1 shows task Y. It should be noted that during the execution of each step in the task, the execution status is defined, and the execution status is updated according to the current execution status of the step, which specifically includes: execution failed, completed, executed, scheduled, and not started. Take the first processing device as an example to illustrate the execution process of the steps in the predefined task. It should be noted that the first processing device is any device that can execute instructions and is not limited. For example, the first processing device is a computer device; predefined tasks are tasks that have been defined in advance, such as task X and task Y.

The first processing device executes each step defined in the predefined task in the following manner:

S201: The first processing device obtains a predefined task.

The predefined task includes N steps, and N is an integer greater than 1.

S202: The first processing device generates an execution directed graph corresponding to the predefined task according to the dependency relationship existing between the N steps.

The execution directed graph is a directed graph indicating the execution order of the N steps determined according to the dependency relationship. If any two steps of the dependency relationship do not exist in the N steps, then the The directed graph indicates that any two steps are executed in parallel.

It should be emphasized that FIG. 2 only uses the first processing device as an example to illustrate the execution process of the task. In fact, each step or substep can be executed on different processing devices to facilitate concurrent processing. As a result, the number of processing devices can be expanded horizontally to improve computing processing capabilities, and therefore, to maximize the concurrent processing capabilities between and within steps. In addition, in order to improve performance, task definitions can be loaded into memory when the system starts.

S203: The first processing device executes the predefined task according to the execution order of the steps indicated by the execution directed graph.

In S201, the first step is any step defined in the predefined task; the precursor dependency indication: in the predefined task, and the N second steps are all successfully executed, it is allowed Perform the first step. It should be noted that since each step in the predefined task defines a state, after the first step is executed once, the execution may succeed or fail. When the execution of the first step fails, the execution will be retried. The first step Skip after successful execution.

For example, when the first step is step Y4 in task Y, then N is 2, and the two second steps are step Y2 and step Y3. Step Y2 and step Y3 will be executed in parallel first. After step Y2 and step Y3 are all successfully executed, the next step to be executed is step Y4, and step Y2 and step Y3 are all executed successfully before step Y4 is allowed to be executed. In addition, step Y5 may be a long time step. Step Y2, step Y3, and step Y4 can be executed in parallel with step Y5. After step Y4 is completed, it will check whether step Y5 is successfully executed, and then step Y6 can be executed. Improve the efficiency of task execution.

In addition, during the execution of a predefined task, an execution log will be generated, which is divided into three parts: task execution log, step execution log, and sub-step execution log. The sub-step log and the step log have a one-to-many relationship. If the step is not split, there is one sub-step log; if there is a split, then there are multiple sub-step logs. The sub-step execution log includes: first, the data of the definition table is included, the task name, step name, implementation class, how to split the data, the execution thread size, the read data size, the commit transaction size, and the log level are included. And, the currently processed data pieces, the total number of pending items, the number of read items, the number of processed items, the number of ignored items, the processing Internet address (Internet protocol, IP), creation time, start time, completion time and other information.

In S202, an optional implementation manner is that, if any step that has the dependency relationship with the first step is not defined in the predefined task, the first processing device first executes the predefined task Perform the first step.

In other words, when the first step is the apex step, the first step will be executed first when the predefined task is executed. For example, the first step is step Y1. When step Y is executed, step Y1 is executed first.

In S203, an optional implementation manner is that the first processing device executes the first step in the following manner:

(1) If the first step includes only one sub-step, the sub-step is directly executed.

(2) If the first step includes M sub-steps, the M sub-steps are executed in parallel, and M is an integer greater than 1.

For example, when the first step is step Y4 in task Y, and Y4 includes only one substep, then this substep is directly executed; when the first step is step Y6 in task Y, M is 2, and Y6 includes, 2. Sub-steps, then execute these two sub-steps in parallel.

In S203, an optional implementation manner is that, if the first processing device determines that there are at least two failed execution steps among the N steps, re-execute the at least two failed execution steps in parallel.

For example, when the first step is step Y4 in task Y, the two second steps are step Y2 and step Y3. Therefore, if step Y2 and step Y3 fail to execute, step Y2 and step Y3 will be executed again in parallel .

In S201 to S203, an optional implementation manner is that the first processing device determines whether the predefined task has sent heartbeat data within a preset period of time, if not, and determines that the running status of the predefined task is During operation, a notification message indicating to host the predefined task is sent to the second processing device. Through the above method, the running status of the task can be obtained in time, so that the task that is not running normally can be managed, saving computing resources.

The hosting is: a. When the machine executes the task, the task is being processed due to hardware failures and other conditions, but it is not actually processed. The machine b judges whether machine a is unavailable according to the heartbeat time updated by each machine. If there is no heartbeat for more than the preset time, it is considered unavailable. Then machine b modifies the predefined task to machine b to execute, and executes this task again to achieve high availability the goal of. Simply put, that is, a normally operating machine can perform the tasks of a faulty machine.

In S203, when any one of the at least one step that has a dependency relationship in the first step is executed successfully, it will be checked whether it is the last step in the at least one step that has successfully executed, and if so, the first step is executed. One step.

The embodiments of this application have the following advantages: (1) Define task step dependencies based on the directed graph, and each step only cares about whether the dependent step is completed and then can start; (2) A single step can be split into multiple sub-steps according to the definition, all When the sub-step is completed, the current step is completed; (3) Realization and configuration are separated, and different configuration combinations form different functional combinations.

In the embodiment of the present application, the steps may be executed by defining the execution directed graph in a predefined task. The execution directed graph is a directed graph indicating the execution order of the N steps determined according to the dependency relationship. Figure, if any two steps of the dependency relationship do not exist in the N steps, the directed graph indicates that the any two steps are executed in parallel. In addition, the execution directed graph also indicates the dependency relationship existing among the N steps, thereby ensuring that tasks are processed normally according to the preset execution sequence. Therefore, the execution time of each step in any two steps overlaps, and they are not affected by execution failure or not, so that the execution time of the entire task is shortened while ensuring the stable execution of the predefined task. Improve the efficiency of task execution.

As shown in FIG. 3, it is a schematic structural diagram of a task execution device in an embodiment of this application.

An embodiment of the present application provides a task execution device, including: an acquisition module 301, configured to acquire a predefined task; the predefined task includes N steps; N is an integer greater than 1; a processing module 302, configured to The dependency relationship between the N steps is generated, and the execution directed graph corresponding to the predefined task is generated; the execution directed graph is a directed graph indicating the execution order of the N steps determined according to the dependency relationship , If any two steps of the dependency relationship do not exist in the N steps, the directed graph indicates that the any two steps are executed in parallel; the execution order of the steps indicated by the executed directed graph is executed The predefined task.

Optionally, the first step is any one of the N steps, and the processing module 302 is specifically configured to: execute the first step in the following manner: if the first step includes only one sub-step, then This sub-step is directly executed; if the first step includes M sub-steps, the M sub-steps are executed in parallel, and M is an integer greater than 1.

Optionally, the processing module 302 is further configured to: if any step that has the dependency relationship with the first step is not defined in the predefined task, execute the first step first when executing the predefined task One step.

Optionally, the processing module 302 is further configured to: determine whether the predefined task has sent heartbeat data within a preset period of time; if not, and if it is determined that the running state of the predefined task is running, then send the pre-defined task to the pre-defined task. The configuration device sends a notification message indicating to host the predefined task.

Optionally, the processing module 302 is further configured to: if the first processing device determines that there are at least two failed execution steps among the N steps, re-execute the at least two failed execution steps in parallel.

Finally, it should be noted that those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program codes.

This application is described with reference to flowcharts and/or block diagrams of methods, equipment (systems), and computer program products according to this application. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment are generated It is a device that realizes the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

Obviously, those skilled in the art can make various changes and modifications to the application without departing from the scope of the application. In this way, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application also intends to include these modifications and variations.

Claims

A task execution method, characterized in that it comprises:

The first processing device acquires a predefined task; the predefined task includes N steps; N is an integer greater than 1;

The first processing device generates an execution directed graph corresponding to the predefined task according to the dependency relationship existing between the N steps; the execution directed graph indicates that the N steps are based on the dependency A directed graph of an execution sequence determined by a relationship, if any two steps of the dependency relationship do not exist in the N steps, the directed graph indicates that any two steps are executed in parallel;

The first processing device executes the predefined task according to the execution sequence of the steps indicated by the execution directed graph.
The method according to claim 1, wherein the first step is any one of the N steps, and the first processing device executes the first step in the following manner:

If the first step includes only one sub-step, then directly execute the sub-step;

If the first step includes M sub-steps, the M sub-steps are executed in parallel, and M is an integer greater than 1.
The method of claim 2, wherein the method further comprises:

If any step that has the dependency relationship with the first step is not defined in the predefined task, the first step is first executed when the first processing device executes the predefined task.
The method according to any one of claims 1 to 3, wherein the method further comprises:

The first processing device determines whether the predefined task has sent heartbeat data within a preset period of time, and if not, and determines that the running state of the predefined task is running, it sends an instruction to the second processing device Describes notification messages for predefined tasks.
The method according to any one of claims 1 to 3, wherein the method further comprises:

If the first processing device determines that there are at least two failed execution steps among the N steps, re-execute the at least two failed execution steps in parallel.
A task execution device is characterized by comprising:

The obtaining module is used to obtain a predefined task; the predefined task includes N steps; N is an integer greater than 1;

The processing module is configured to generate an execution directed graph corresponding to the predefined task according to the dependency relationship existing between the N steps; the execution directed graph indicates that the N steps are based on the dependency relationship A directed graph with a determined execution order. If any two steps of the dependency relationship do not exist in the N steps, the directed graph indicates that any two steps are executed in parallel; The execution sequence of the steps indicated in the figure is to execute the predefined task.
The device according to claim 6, wherein the first step is any one of the N steps, and the processing module is specifically configured to:

Perform the first step as follows:

If the first step includes only one sub-step, then directly execute the sub-step;

If the first step includes M sub-steps, the M sub-steps are executed in parallel, and M is an integer greater than 1.
The device according to claim 7, wherein the processing module is further configured to:

If any step that has the dependency relationship with the first step is not defined in the predefined task, the first step is executed first when the predefined task is executed.
A computer device characterized by comprising a program or an instruction, and when the program or an instruction is executed, the method according to any one of claims 1 to 5 is executed.
A storage medium, characterized by comprising a program or instruction, and when the program or instruction is executed, the method according to any one of claims 1 to 5 is executed.