WO2022135079A1

WO2022135079A1 - Data processing method for task flow engine, and task flow engine, device and medium

Info

Publication number: WO2022135079A1
Application number: PCT/CN2021/134268
Authority: WO
Inventors: 曹军; 吴礼蔚; 程善伯; 王明轩; 李磊
Original assignee: 北京有竹居网络技术有限公司
Priority date: 2020-12-25
Filing date: 2021-11-30
Publication date: 2022-06-30
Also published as: CN112685154A

Abstract

A data processing method for a task flow engine, and a task flow engine, a device and a medium, which relate to the technical field of computer data processing. The data processing method for a task flow engine comprises: converting a task flow described by a DSL into a graphical node dependency relationship (S11); and on the basis of the node dependency relationship, executing an operator logic of each node among a plurality of nodes, so as to process data in a task flow engine (S12).

Description

Data processing method of task flow engine, task flow engine, device and medium

This application claims the priority of the Chinese Patent Application No. 202011564198.8 filed with the China Patent Office on December 25, 2020, the entire contents of which are incorporated herein by reference.

technical field

The present disclosure relates to the technical field of computer data processing, for example, to a data processing method of a task flow engine, a task flow engine, a device and a medium.

Background technique

The task flow of a complex machine learning system can usually be abstracted as a Directed Acyclic Graph (DAG), and the execution of task nodes is completed through a special scheduling engine and a heterogeneous execution engine.

However, there is a problem of low efficiency in the process of executing task nodes with specialized scheduling engines and heterogeneous execution engines.

SUMMARY OF THE INVENTION

The present disclosure provides a data processing method of a task flow engine, a task flow engine, a device and a medium, so as to improve the development efficiency of a task flow system.

The present disclosure provides a data processing method for a task flow engine, the method comprising:

Convert the task flow in the task flow engine described by Domain-Specific Language (DSL) into graphical node dependencies;

The operator logic of each node in the plurality of nodes is executed based on the node dependency, so as to process the data in the task flow engine.

The present disclosure also provides a task flow engine, the task flow engine includes:

The conversion module is set to convert the task flow in the task flow engine described by the DSL into a graphical node dependency;

The data processing module is configured to execute the operator logic of each node in the plurality of nodes based on the node dependency, so as to process the data in the task flow engine.

The present disclosure also provides a task flow engine device, including:

one or more processors;

memory, arranged to store one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned data processing method of the task flow engine.

The present disclosure also provides a medium, where the medium stores a computer program, and when the computer program is executed by a processor, implements the data processing method of the above task flow engine.

Description of drawings

1 is a flowchart of a data processing method for a task flow engine provided by an embodiment of the present disclosure;

2 is a schematic diagram of a graphical node dependency provided by an embodiment of the present disclosure;

3 is a flowchart of another data processing method of a task flow engine provided by an embodiment of the present disclosure;

4 is a schematic diagram of a graphical state monitoring interface provided by an embodiment of the present disclosure;

5 is a schematic diagram of another graphical node dependency relationship provided by an embodiment of the present disclosure;

6 is a structural diagram of a task flow engine provided by an embodiment of the present disclosure;

FIG. 7 is a structural diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, the present disclosure may, however, be embodied in various forms and should not be construed as limited to the embodiments set forth herein, which are provided for a more thorough and complete understanding this disclosure. The figures and examples of the present disclosure are for illustrative purposes only.

The multiple steps described in the method embodiments of the present disclosure may be performed in different orders and/or in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this regard.

As used herein, the term "including" and variations thereof are open-ended inclusions, ie, "including but not limited to". The term "based on" is "based at least in part on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below.

Concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of functions performed by these devices, modules or units relation.

Modifications of "a" and "a plurality" mentioned in the present disclosure are illustrative rather than limiting, and those skilled in the art should understand that unless the context indicates otherwise, they should be construed as "one or more".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of these messages or information.

FIG. 1 is a flowchart of a data processing method of a task flow engine provided by an embodiment of the present disclosure. This embodiment is applicable to the situation of arranging and processing data task flow. The method can be executed by a task flow engine. The task flow engine described above can be implemented by means of software and/or hardware. The task flow engine can be integrated, for example, in a terminal device.

As shown in FIG. 1 , the data processing method of the task flow engine provided in this embodiment mainly includes steps S11 and S12.

S11. Convert the task flow in the task flow engine described by the DSL into a graphical node dependency.

DSL is also known as domain-specific language, DSL is only used in some specific fields. For example, the Hyper Text Markup Language (HTML) used to display web pages, and the Emac Locator Identifier Separation Protocol (LISP) language used by Emacs.

The task flow (workflow) is the calculation model of the task flow, that is, the logic and rules of how the work in the task flow is organized together before and after are represented in the computer with an appropriate model and the calculation is performed. The main problem to be solved by task flow is: in order to achieve a business goal, among multiple participants, the computer is used to automatically transfer according to a predetermined rule.

Mapping and modeling can be included in the implementation of task flows. Mapping is the first step whose primary task is to identify and document all existing manual and automated business processes within the organization; modeling is the development of a model that helps streamline business processes.

The task flow engine is also called the workflow engine. The task flow engine can take the task flow as a part of the application system and provide the application system with a core solution that has a decisive effect on the application system. The division of labor and different conditions determine the information delivery route, content level, etc. The task flow engine generally includes important functions such as process node management, flow direction management, and process sample management.

The task flow in the embodiments of the present disclosure mainly refers to a task flow used for data processing, and the task flow includes multiple nodes executed in a certain order and dependencies between the multiple nodes.

The dependency relationship between multiple nodes can be understood as the execution sequence relationship set for multiple nodes in the task flow. For example, a sequential task flow can be set. The execution process of the sequential task flow is a continuous sequence of steps. In the sequential task flow, after the logical operation of one node is completed, the logical operation of the next node will be executed.

The task flow described by the DSL refers to the task flow described using the DSL language or the task flow defined using the DSL language. That is to use DSL language programming to describe the operator logic in each node in the task flow and the dependencies between multiple nodes.

Graphical node dependencies can be understood as expressing the dependencies between multiple nodes in a graphical manner, so as to realize the visualization of the dependencies between multiple established tasks.

FIG. 2 is a schematic diagram of a graphical node dependency provided by an embodiment of the present disclosure; as shown in FIG. 2 , the dependency between multiple nodes is that the node can be executed only after the operator logic in node A is executed. The operator logic in node B can only be executed after the operator logic in node C is executed; the operator logic in node D and node B can be executed only after the operator logic in node D and node B are executed. sub logic.

FIG. 2 is only an exemplary illustration of the node dependency, but not a limitation. In practical applications, node dependencies are usually much more complex. For example, the first node needs to execute a web crawler to obtain massive data from the network, and the second node performs model training based on the obtained massive data.

S12. Execute the operator logic of each node in the multiple nodes based on the node dependency, so as to process the data in the task flow engine.

Each node contains operator logic that needs to be executed, and operator logic refers to the logic of data processing in the node. For example, the operator logic may be the operation of performing the sum of two input data; another example: the operator logic may be the logical operation of performing the exclusive OR of the two input data. In this embodiment, only the operator logic is described, but not limited.

In this embodiment, the scheduling layer of the task flow engine determines the current node based on the node dependency, and sends the node execution command to the execution layer of the task flow engine, and the execution layer executes the operator logic of the node based on the node execution command.

The execution layer of the task flow engine first executes the first layer node of the task flow, that is, the node without other predecessor dependencies. Generate new event messages to drive the operation of lower-level nodes.

The scheduling layer of the task flow engine sends node execution commands to the execution layer through the Remote Procedure Call (RPC) protocol.

The scheduling layer of the task flow engine determines the current node based on the node dependency relationship, including: after receiving the successful execution information of the predecessor node sent by the execution layer, the scheduling layer determines the current node based on the node dependency relationship.

After the scheduling layer receives the successful execution information of the predecessor node sent by the execution layer, the scheduling layer determines the current node based on the node dependency, and then the scheduling layer determines whether the current node execution condition is satisfied based on the predecessor node of the current node; condition, then execute the operation of sending the node execution command to the execution layer.

The number of predecessor nodes of the current node is determined based on the node dependency. If the number of predecessor nodes is one, the execution success information of one predecessor node is received, and the operation of sending the node execution command to the execution layer. If the number of precursor nodes is more than one, after receiving the successful execution information of the precursor nodes and the number of precursor nodes equal to the number of predecessor nodes, the operation of sending the node execution command to the execution layer.

In this embodiment, a node in the execution layer executes the operator logic in the node based on the received input parameters to obtain output parameters.

The data processing method provided by this embodiment includes: converting the task flow described by the DSL into a graphical node dependency; executing the operator logic of each node in the multiple nodes based on the node dependency, so as to control the flow of tasks in the task flow engine. data is processed. The embodiments of the present disclosure reduce the complexity of task flow arranging, debugging and running, and improve the efficiency of task flow system development by using a DSL to describe the data task flow and convert it into a graphically executable scheduling logic.

On the basis of the above embodiment, before converting the task flow described by the DSL into a graphical node dependency, the method further includes: using the DSL to describe the task flow for processing data in the task flow engine.

In this embodiment, the DSL is used to describe the task flow. Compared with the related art, a flowchart editing tool is used to edit the task flow, which reduces the complexity of task arrangement, debugging and running, and improves the development efficiency of the task flow system.

In the task flow engine development system, the definition layer of the task flow engine development system uses a DSL to describe the task flow for processing data in the task flow engine.

You can use the DSL language editor to describe the task flow, and directly import the described task flow into the definition layer of the engine development system. You can also use the DSL language to edit the task flow directly in the definition layer of the engine development system. In this embodiment, only the editing position is described, but not limited.

Use a DSL to describe the task flow for processing data, including: using a DSL to describe the input parameters of each node, output parameters, and dependencies between multiple nodes.

The above-mentioned input parameters refer to the parameters required to execute the operator logic in the node, and the input parameters of a node may be one or multiple. The input parameters are determined by the operator logic in the node. The above output parameters refer to the parameters to be output after the node executes the operator logic. The input parameters of a node can be one or more. The output parameters are determined by the operator logic in the node.

FIG. 3 is a flowchart of another data processing method of a task flow engine provided by an embodiment of the present disclosure. As shown in FIG. 3 , the data processing method provided in this embodiment mainly includes steps S21 , S22 , S23 , S24 and S25 .

S21. Convert the task flow in the task flow engine described by the DSL into a graphical node dependency.

S22. For each node, the scheduling layer in the task flow engine determines the current node based on the node dependency, and sends a node execution command to the execution layer in the task flow engine.

Node execution commands include the current node ID. The current node refers to the node that is about to execute the operator logic, and the node identifier refers to the characteristic value that uniquely identifies the node, which can be the node name, node code, etc. The current node identifier refers to the characteristic value that uniquely identifies the current node. The node execution command refers to a command or instruction instructing the execution layer to execute the operator logic in the node.

The scheduling layer can be understood as the layer where the task flow engine development system performs computer resource allocation or task allocation. The execution layer can be understood as the layer that performs logical operations in the task flow engine development system.

As shown in Figure 2, the scheduling layer determines the current node based on the node dependency. It can be understood that the scheduling layer determines that the current node is node B after the operator logic in node A is executed; or the scheduling layer determines the operator in node C. After the logic is executed, it is determined that the current node is node D.

The execution layer first executes the first layer node of the task flow, that is, the node without other predecessor dependencies. Generate new event messages to drive the operation of lower-level nodes.

The scheduling layer sends the node execution command to the execution layer, including: the scheduling layer sends the node execution command to the execution layer through the RPC protocol.

The RPC protocol refers to a protocol that requests services from a remote computer program over a network without knowing the underlying network technology. That is, the scheduling layer calls an object that exists on the execution layer without knowing the details of the call, just like calling an object in the local application.

RPC implementations include: Dubbo, Thrift, GRPC, Hetty, etc. In this embodiment, any one of the RPC protocols may be selected, which is not limited in this embodiment.

In this embodiment, the scheduling layer is completely decoupled from the execution layer based on the RPC protocol, and does not depend on each other, thereby improving the security of the task flow system.

The scheduling layer determining the current node based on the node dependency includes: after receiving the successful execution information of the predecessor node sent by the execution layer, the scheduling layer determines the current node based on the node dependency.

In this embodiment, the predecessor node refers to the node of the operator logic executed before the current node. As shown in FIG. 2 , node A is the predecessor node of node B, node C is the predecessor node of node D, node D and node B is the predecessor node of node E.

The message that the predecessor node executes successfully refers to the message that the predecessor node executes the operator logic and successfully outputs the parameters. The execution layer executes the operator logic in the current node, and returns a successful execution message to the scheduling layer after successful execution. In this embodiment, the scheduling layer processes task node dependencies based on a message-driven mechanism.

For example, after the scheduling layer receives the successful execution information of the predecessor node A sent by the execution layer, the scheduling layer determines that the current node is node B based on the node dependency; The layer determines that the current node is node D based on the node dependencies.

After the scheduling layer determines the current node based on the node dependency relationship, the method further includes: the scheduling layer judges whether the current node execution condition is satisfied based on the predecessor node of the current node; if the current node execution condition is satisfied, executes sending a node execution command to the execution layer operation.

The execution conditions of the current node refer to the conditions that the system needs to meet to execute the operator logic in the current node. The current node execution condition includes: the number of predecessor nodes of the current node is equal to the number of the received predecessor node execution success information.

In this embodiment, when the number of precursor-dependent nodes of a node is greater than 1, since the sequence of the completion time of the precursor nodes cannot be determined, message synchronization needs to be performed to ensure that the current node collects the same number of messages as the number of precursor nodes. .

S23, the execution layer executes the operator logic in the current node.

In this embodiment, the logical operator in the node refers to the logical relationship between predefined data.

In one embodiment, the execution layer executes the operator logic in the current node, including: the current node in the execution layer obtains output parameters based on the received input parameters and the operator logic in the current node.

In this embodiment, for the received input parameters, the operator logic in the node is executed to obtain the output parameters. For example: two input parameters are received, which are 1 and 2 respectively; the operator logic is summation, then 1 and 2 are summed, and the output parameter of the current node is 3.

An operator can be an abstraction of the computing process on any computing platform, so it is independent of the engine.

After the current node obtains the output parameters based on the received input parameters and the operator logic in the current node, the method further includes: saving the output parameters through the hash signature of the input parameters, and using the saved output parameters as the corresponding input parameters cached results.

Hash signature is the most important digital signature method, also known as Digital Digest or Digital Finger Print. Digital digest is to use a single Hash function to "digest" the plaintext to be encrypted into a string of fixed-length (127-bit) ciphertext. This string of ciphertext is also called digital fingerprint, which has a fixed length, and different plaintext digests are Ciphertext, the result is always different, and the digest of the same plaintext must be consistent. Common hash algorithms include: Message Digest Algorithm 2, MD2, version 2, Message Digest Algorithm 4, MD4, version 4, Message Digest Algorithm 5, MD5, version 5 ), HAVAL, Secure Hash Algorithm (SHA), etc.

In this embodiment, the execution result saved by using the hash signature of the input parameter, that is, the output parameter is used as the cached result corresponding to the input parameter of the node. When the current input parameter is the same as a historical input parameter of the node, the task node directly outputs the cached result corresponding to the historical input parameter as the current output parameter.

In this implementation, a cache switch can also be set. If the cache switch is turned on, if the current input parameter is the same as a historical input parameter of the node, the task node directly outputs the cached result corresponding to the historical input parameter as the current output parameter. If the cache switch is closed, the task node performs calculations based on the current input parameters and operator logic to determine the current output parameters.

In another embodiment, the execution layer executes the operator logic in the current node, including: the current node of the execution layer receives the input parameter; detects whether there is a cache parameter that is the same as the input parameter; If the same cache parameter is used, the cache result corresponding to the cache parameter is directly determined as the output parameter.

In this embodiment, after receiving the input parameter, you can first query whether there is a cache parameter that is the same as the input parameter. If there is a cache parameter that is the same as the input parameter, it means that the node has executed the operator logic based on the input parameter. Get the output parameter, which is cached as the cached result. At this time, the cached result corresponding to the cached parameter can be directly read, and outputted as the output parameter corresponding to the input parameter. In this way, frequent execution of operator logic by nodes can be avoided, resulting in an excessively large amount of data at the execution layer, reducing execution efficiency, and improving system operation efficiency.

S24. The execution layer judges whether the execution of the operator logic in the current node ends.

The completion of the execution of the operator logic in the current node includes one or more of the following: the execution of the operator logic of the current node is successful; the execution duration of the operator logic of the current node exceeds a preset duration; the operator logic of the current node is executed. The number of repeated executions of the sub-logic exceeds the preset number of times.

S25. If the execution of the operator logic in the current node ends, the execution layer sends a message that the execution of the current node ends to the scheduling layer.

The message that the execution of the current node ends includes: a message that the operator logic is successfully executed and a message that the operator logic fails to be executed.

Successful execution can be understood as the node executing the operator logic and outputting the correct result. If it is determined that the execution of the operator logic of the current node is successful, it is determined that the execution of the operator logic in the current node ends, and a message that the execution of the current node is successful is sent to the scheduling layer.

Execution failure can be understood as the node executing the operator logic and failing to output the correct result. If the execution time of the operator logic of the current node exceeds the preset time, or the number of repeated executions of the operator logic of the current node exceeds the preset number of times, it is determined that the execution of the operator logic in the current node has ended, and the execution failure of the current node is sent to the scheduling layer. information. The preset duration and the preset number of times can be set according to actual conditions.

If the message that the current node executes successfully is sent to the scheduling layer, after the scheduling layer receives the successful execution information of the predecessor node sent by the execution layer, the scheduling layer determines the current node based on the node dependency. If the current node execution failure message is sent to the scheduling layer, the scheduling layer detects the cause of the failure based on the execution failure message, and tries to repair it.

In this embodiment, the operator logic in the node is automatically executed by configuring the preset duration and the preset number of times by the user, so that errors in the node and the problem of executing the operator logic countless times can be avoided, and the robustness of the system can be improved. .

On the basis of the above embodiment, after executing the operator logic in each node based on the node dependency, the method further includes: acquiring the execution state of each node, and displaying the execution state of each node on a graph on each node described in the state monitoring interface.

As shown in Figure 4, the execution status of each node is displayed in the graphical status monitoring interface. For example, the execution status of the server node is listening, and the execution status of the Data Export node is ending ( finished), the execution status of the Data Augmentation node is finished, and the execution status of the Preprocess node is running. This allows the staff to easily understand the execution status of each node.

In an application example, in order to implement a task flow such as e=(a+b)×(c-d), three operators need to be defined using DSL language: namely, add, subtract, multiply ). Define the input parameters and output parameters of each operator. Dependency of operators: multiply needs to depend on two operators, add and subtract; and the mapping relationship between the input parameters of each operator and the execution results of the operators that the operator depends on.

By parsing the task flow description defined by the DSL language into a graphical representation, as shown in Figure 5. Driven by the initial configuration message, the engine first executes the first layer node of the task flow, that is, the node without other precursor dependencies, such as the add node and the subtract node in Figure 5. After the first layer node is executed, a new event message is generated to drive the operation of the lower layer node (such as multiply).

When the number of precursor dependent nodes of a node is greater than one, since the sequence of the completion time of the predecessor nodes cannot be determined, message synchronization is required to ensure that the current node collects the same number of messages as the number of predecessor nodes.

The task node in the execution layer usually includes an operator, which runs the operator logic through the parameters passed in by RPC, and returns the agreed parameter results to the scheduling layer.

FIG. 6 is a structural diagram of a task flow engine provided by an embodiment of the present disclosure. This embodiment is applicable to the situation of arranging and processing data task flow, and the task flow engine can be generated by software and/or hardware. accomplish. The task flow engine can be integrated, for example, in a terminal device.

As shown in FIG. 6 , the task flow engine provided in this embodiment mainly includes a conversion module 61 and a data processing module 62 .

The conversion module 61 is configured to convert the task flow in the task flow engine described by the DSL into a graphical node dependency; the data processing module 62 is configured to execute the operator of each node in the multiple nodes based on the node dependency Logic to process data in the task flow engine.

The embodiments of the present disclosure provide a task flow engine, which performs the following operations: converts the task flow described by the DSL into a graphical node dependency; data in for processing. The embodiments of the present disclosure reduce the complexity of task flow arranging, debugging and running, and improve the efficiency of task flow system development by using a DSL to describe the data task flow and convert it into a graphically executable scheduling logic.

Before converting the task flow described by the DSL into graphical node dependencies, it also includes: using the DSL to describe the task flow for processing data.

The using the DSL to describe the task flow for processing data includes: using the DSL to describe the input parameters, output parameters and dependencies between multiple nodes of each node in the task flow.

Based on the node dependency relationship, executing the operator logic of each node in the plurality of nodes includes: for each node, the scheduling layer in the task flow engine determines the current node based on the node dependency relationship, and reports to the node. The execution layer in the task flow engine sends a node execution command, wherein the node execution command includes the current node identifier; the execution layer executes the operator logic in the current node.

The scheduling layer sending the node execution command to the execution layer includes: the scheduling layer sends the node execution command to the execution layer through the RPC protocol.

After the node dependency relationship of the scheduling layer determines the current node, the method further includes: the scheduling layer judges whether the execution condition of the current node is satisfied based on the predecessor node of the current node; if the execution condition of the current node is satisfied, executing the execution condition of the scheduling layer. The layer sends the action of the node to execute the command.

The current node execution condition includes: the number of predecessor nodes of the current node is equal to the number of the received predecessor node execution success information.

The execution layer executes the operator logic in the current node, including: the current node in the execution layer obtains output parameters based on the received input parameters and the operator logic in the current node.

After the current node obtains the output parameters based on the received input parameters and the operator logic in the current node, the method further includes: the current node in the execution layer saves the output parameters through the hash signature of the input parameters, and saves the saved output parameters. The output parameter is used as the cached result corresponding to the input parameter.

The execution layer executes the operator logic in the current node, including: the current node of the execution layer receives the input parameter; detects whether there is a cache parameter that is the same as the input parameter; if there is a cache parameter that is the same as the input parameter, directly The cached result corresponding to the cached parameter is determined as the output parameter.

After the execution layer executes the operator logic in the current node, the method further includes: the execution layer determines whether the execution of the operator logic in the current node ends; if the execution of the operator logic in the current node ends, the execution The layer sends a message that the execution of the current node is over to the scheduling layer.

The execution end of the operator logic in the current node includes one or more of the following:

The execution of the operator logic of the current node is successful; the execution duration of the operator logic of the current node exceeds the preset duration; the repeated execution times of the operator logic of the current node exceeds the preset number of times.

After executing the operator logic of each node in the plurality of nodes based on the node dependency, the method further includes: acquiring the execution state of each node, and displaying the execution state of each node in a graphical state on each node in the monitoring interface.

The task flow engine provided in this embodiment can execute the data processing method of the task flow engine provided by any embodiment of the present disclosure, and has functional modules and effects corresponding to executing the data processing method of the task flow engine.

Referring next to FIG. 7 , it shows a schematic structural diagram of an electronic device (eg, a terminal device or a server in FIG. 7 ) 700 suitable for implementing an embodiment of the present disclosure. Terminal devices in the embodiments of the present disclosure may include, but are not limited to, such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (Personal Digital Assistants, PDAs), tablet computers (PADs), and portable multimedia players (Portable Media Players). , PMP), in-vehicle terminals (eg, in-vehicle navigation terminals), and other mobile terminals, as well as fixed terminals such as digital (Television, TV), desktop computers, and the like. The electronic device shown in FIG. 7 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.

As shown in FIG. 7 , the electronic device 700 may include a processing device (eg, a central processing unit, a graphics processor, etc.) 701, which may be based on a program stored in a read-only memory (Read-Only Memory, ROM) 702 or from a storage device 707 programs loaded into Random Access Memory (RAM) 703 to perform various appropriate actions and processes. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An Input/Output (I/O) interface 705 is also connected to the bus 704 .

Typically, the following devices may be connected to the I/O interface 705: input devices 708 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a Liquid Crystal Display (LCD) Output device 707 , speaker, vibrator, etc.; storage device 708 , including, for example, magnetic tape, hard disk, etc.; and communication device 709 . Communication means 709 may allow electronic device 700 to communicate wirelessly or by wire with other devices to exchange data. Although FIG. 7 shows an electronic device 700 having various means, it is not required to implement or have all of the illustrated means. More or fewer devices may alternatively be implemented or provided.

According to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication device 709, or from the storage device 708, or from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.

The computer-readable medium described above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. Examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, RAM, ROM, Erasable Programmable Read-Only Memory (EPROM) or flash memory), optical fiber, portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . The program code embodied on the computer-readable medium may be transmitted by any suitable medium, including but not limited to: electric wire, optical fiber cable, radio frequency (RF), etc., or any suitable combination of the above.

In some embodiments, the client and server can use any currently known or future developed network protocols such as HyperText Transfer Protocol (HTTP) to communicate, and can communicate with digital data in any form or medium. Data communications (eg, communications networks) interconnect. Examples of communication networks include Local Area Networks (LANs), Wide Area Networks (WANs), the Internet (eg, the Internet), and peer-to-peer networks (eg, ad hoc peer-to-peer networks), as well as any currently Known or future developed networks.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or may exist alone without being assembled into the electronic device.

The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device:

The task flow described by the DSL is converted into a graphical node dependency; the operator logic of each node in the multiple nodes is executed based on the node dependency, so as to process the data in the task flow engine.

Computer program code for performing operations of the present disclosure may be written in one or more programming languages, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and This includes conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. Where a remote computer is involved, the remote computer may be connected to the user's computer through any kind of network, including a LAN or WAN, or may be connected to an external computer (eg, using an Internet service provider to connect through the Internet).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The units involved in the embodiments of the present disclosure may be implemented in a software manner, and may also be implemented in a hardware manner. Among them, the name of the unit does not constitute a limitation of the unit itself in one case.

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (Application Specific Standard Products) Standard Parts, ASSP), system on chip (System on Chip, SOC), complex programmable logic device (Complex Programmable Logic Device, CPLD) and so on.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. Examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, RAM, ROM, EPROM or flash memory, optical fibers, CD-ROMs, optical storage devices, magnetic storage devices, or Any suitable combination of the above.

According to one or more embodiments of the present disclosure, a data processing method/task flow engine/device/medium of a task flow engine is provided, including:

The task flow in the task flow engine described by the DSL is converted into a graphical node dependency; the operator logic of each node in the multiple nodes is executed based on the node dependency to process the data in the task flow engine.

According to one or more embodiments of the present disclosure, there is provided a data processing method/task flow engine/device/medium of a task flow engine, in which the task flow described in the DSL is converted into a graphical one in the task flow engine Before node dependencies, also include:

The task flow for processing data in the task flow engine is described using a DSL.

According to one or more embodiments of the present disclosure, there is provided a data processing method/task flow engine/device/medium of a task flow engine, which describes the method for processing data in the task flow engine using a DSL Task flow, including:

A DSL is used to describe the input parameters of each node in the task flow, the output parameters and the dependencies among the plurality of nodes.

According to one or more embodiments of the present disclosure, there is provided a data processing method/task flow engine/device/medium for a task flow engine, wherein based on the node dependency relationship, a computation of each node in a plurality of nodes is performed. Sub logic, including:

The scheduling layer in the task flow engine determines the current node based on the node dependency, and sends a node execution command to the execution layer in the task flow engine, wherein the node execution command includes the current node identifier; the execution The layer executes the operator logic in the current node.

According to one or more embodiments of the present disclosure, there is provided a data processing method/task flow engine/device/medium of a task flow engine, where a scheduling layer in the task flow engine reports to an execution layer in the task flow engine Send node execution commands, including:

The scheduling layer sends the node execution command to the execution layer through the RPC protocol.

According to one or more embodiments of the present disclosure, a data processing method/task flow engine/device/medium of a task flow engine is provided, wherein the scheduling layer determines the current node based on the node dependency, including:

After receiving the successful execution information of the predecessor node sent by the execution layer, the scheduling layer determines the current node based on the node dependency.

According to one or more embodiments of the present disclosure, a data processing method/task flow engine/device/medium of a task flow engine is provided, after the scheduling layer determines the current node based on the node dependency, further comprising:

The scheduling layer judges whether the execution condition of the current node is satisfied based on the predecessor node of the current node; if the execution condition of the current node is satisfied, the operation of sending the execution command of the node to the execution layer by the scheduling layer is performed.

According to one or more embodiments of the present disclosure, a mail processing method/device/system is provided, and the current node execution condition includes:

The number of predecessor nodes of the current node is equal to the number of the received predecessor node execution success information.

According to one or more embodiments of the present disclosure, a data processing method/task flow engine/device/medium of a task flow engine is provided, wherein the execution layer executes the operator logic in the current node, including:

The current node in the execution layer obtains output parameters based on the received input parameters and operator logic in the current node.

According to one or more embodiments of the present disclosure, a mail processing method/device/system is provided, after the current node obtains output parameters based on the received input parameters and the operator logic in the current node, further include:

The current node in the execution layer saves the output parameter through the hash signature of the input parameter, and uses the saved output parameter as a cached result corresponding to the input parameter.

The current node of the execution layer receives the input parameter; detects whether there is a cache parameter that is the same as the input parameter; if there is a cache parameter that is the same as the input parameter, directly determines the cache result corresponding to the cache parameter as the output parameter.

According to one or more embodiments of the present disclosure, a data processing method/task flow engine/device/medium of a task flow engine is provided. After executing the operator logic in the current node, the execution layer further includes:

The execution layer judges whether the execution of the operator logic in the current node ends; if the execution of the operator logic in the current node ends, the execution layer sends a message of the execution end of the current node to the scheduling layer.

According to one or more embodiments of the present disclosure, a data processing method/task flow engine/device/medium of a task flow engine is provided, where the completion of the execution of the operator logic in the current node includes one or more of the following:

According to one or more embodiments of the present disclosure, there is provided a data processing method/task flow engine/device/medium for a task flow engine, wherein based on the node dependency relationship, a computation of each node in a plurality of nodes is performed. After the sub logic, also include:

The execution status of each node is acquired, and the execution status of each node is displayed on each node in the graphical status monitoring interface.

Additionally, although operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while the above discussion contains several implementation details, these should not be construed as limitations on the scope of the present disclosure. Some features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Claims

A data processing method for a task flow engine, comprising:

Convert the task flow in the task flow engine described by the domain-specific language DSL into a graphical node dependency;

The operator logic of each node in the plurality of nodes is executed based on the node dependency, so as to process the data in the task flow engine.
The method according to claim 1, before converting the task flow in the task flow engine described by the DSL into a graphical node dependency, further comprising:

The task flow for processing data in the task flow engine is described using a DSL.
3. The method of claim 2, wherein said using a DSL to describe the task flow for processing data in the task flow engine comprises:

A DSL is used to describe the input parameters of each node in the task flow, the output parameters and the dependencies among the plurality of nodes.
The method of claim 1, wherein, based on the node dependencies, executing the operator logic of each node in the plurality of nodes comprises:

The scheduling layer in the task flow engine determines the current node based on the node dependency, and sends a node execution command to the execution layer in the task flow engine, wherein the node execution command includes the current node identifier;

The execution layer executes the operator logic in the current node.
The method according to claim 4, wherein the scheduling layer in the task flow engine sends a node execution command to the execution layer in the task flow engine, comprising:

The scheduling layer sends the node execution command to the execution layer through the remote procedure call RPC protocol.
The method of claim 4, wherein the scheduling layer determines the current node based on the node dependency, comprising:

After receiving the successful execution information of the predecessor node sent by the execution layer, the scheduling layer determines the current node based on the node dependency.
The method according to claim 4, after the scheduling layer determines the current node based on the node dependency, further comprising:

The scheduling layer judges whether the current node execution condition is satisfied based on the predecessor node of the current node;

In response to satisfying the current node execution condition, an operation of the scheduling layer sending the node execution command to the execution layer is performed.
The method according to claim 7, wherein the current node execution condition comprises:

The number of predecessor nodes of the current node is equal to the number of the received predecessor node execution success information.
The method according to claim 4, wherein the execution layer executes the operator logic in the current node, comprising:

The current node in the execution layer obtains output parameters based on the received input parameters and operator logic in the current node.
The method according to claim 9, after the current node in the execution layer obtains output parameters based on the received input parameters and the operator logic in the current node, further comprising:

The current node in the execution layer saves the output parameter through the hash signature of the input parameter, and uses the saved output parameter as a cached result corresponding to the input parameter.
The method according to claim 4, wherein the execution layer executes the operator logic in the current node, comprising:

The current node of the execution layer receives input parameters;

Detecting whether there is a cache parameter that is the same as the input parameter;

In response to the existence of the same cache parameter as the input parameter, the cache result corresponding to the cache parameter is directly determined as the output parameter.
The method according to claim 4, wherein after the execution layer executes the operator logic in the current node, it further comprises:

The execution layer judges whether the execution of the operator logic in the current node ends;

In response to a situation in which the execution of the operator logic in the current node ends, the execution layer sends a message that the execution of the current node ends to the scheduling layer.
The method according to claim 12, wherein the completion of the execution of the operator logic in the current node comprises at least one of the following:

The operator logic of the current node is successfully executed;

The execution duration of the operator logic of the current node exceeds the preset duration;

The number of repeated executions of the operator logic of the current node exceeds the preset number of times.
The method according to claim 1, wherein after executing the operator logic of each node in the plurality of nodes based on the node dependency, the method further comprises:

Acquire the execution status of each node, and display the execution status of each node on each node in the graphical status monitoring interface.
A task flow engine including:

The conversion module is set to convert the task flow in the task flow engine described by the domain-specific language DSL into a graphical node dependency;

The data processing module is configured to execute the operator logic of each node in the plurality of nodes based on the node dependency, so as to process the data in the task flow engine.
A data processing device for a task flow engine, comprising:

at least one processor;

a memory, arranged to store at least one program;

When the at least one program is executed by the at least one processor, the at least one processor implements the data processing method of the task flow engine according to any one of claims 1-14.
A computer storage medium storing a computer program, when the computer program is executed by a processor, the data processing method of the task flow engine according to any one of claims 1-14 is implemented.