WO2023004805A1

WO2023004805A1 - Workflow modeling implementation system and method, and storage medium

Info

Publication number: WO2023004805A1
Application number: PCT/CN2021/109876
Authority: WO
Inventors: 周冠兴; 张洪洋; 唐秉超
Original assignee: 西门子股份公司; 西门子（中国）有限公司
Priority date: 2021-07-30
Filing date: 2021-07-30
Publication date: 2023-02-02

Abstract

Disclosed are a workflow modeling implementation method and system, and a storage medium. The system comprises: a node library, which is configured with nodes for constructing various behavior trees; a graphical interface module, which is used for providing a graphical human-computer interaction interface for a user to construct and configure a behavior tree on the basis of nodes in a node storage module; an editing engine module, which is used for maintaining a current behavior tree as an extendable tag language file in response to a behavior tree creation operation performed by the user by means of the graphical human-computer interaction interface, and which generates the current behavior tree corresponding to a current workflow; and a behavior tree engine execution module, which is used for running the current behavior tree by accessing the extendable tag language file according to a running or debugging instruction of the user, and which implements the current workflow corresponding to the current behavior tree. The technical solution in the embodiments of the present invention may achieve general workflow modeling and implementation.

Description

Workflow modeling implementation system, method and storage medium

technical field

The invention relates to the field of industrial technology, in particular to a general workflow modeling implementation system, method and storage medium.

Background technique

Workflow can be simply defined as a description of a series of operations. Workflow is widely used in automation systems, artificial intelligence, robotics and other fields. For example, the workflow of a product sorting line in an automated system can be simply described as starting, taking pictures, sorting and moving the products to the target location. The model deployment workflow in the field of artificial intelligence can be described as data collection, data labeling, model training, and model deployment.

But at present, these workflows only have text descriptions. If users want to execute such workflows, they need to follow the text descriptions and may use a variety of engineering tools. However, these tools are almost irrelevant and provide completely different user operation behaviors. It is not only a challenge to users, but also greatly increases costs, reduces efficiency, and limits flexibility due to the long development cycle. For example, in the field of artificial intelligence, users need to use a tool for data collection, manually or use other tools for data labeling, write python scripts for model training, and deploy tools for deployment.

Contents of the invention

In view of this, on the one hand, the embodiment of the present invention proposes a workflow modeling implementation method, and on the other hand, proposes a workflow modeling implementation system and a computer-readable storage medium to realize general workflow creation and implementation.

A workflow modeling implementation method proposed in an embodiment of the present invention includes: a node library, in which nodes for building various behavior trees are set, and the nodes include a root node, a logic control node and an action node; wherein, one The behavior tree is used to represent a workflow, the logic control node is used to realize the logic control in the workflow, and the action node is used to realize the business operation in the workflow, and the action node includes: General action nodes encapsulated by common operations in different industrial fields, and proprietary action nodes oriented to specific industrial fields by encapsulating non-general operations in workflows in different industrial fields; the graphical interface module is used to provide users with The nodes in the node storage module are used to construct and configure the graphical human-computer interaction interface of the behavior tree; the editing engine module is used to respond to the behavior tree creation operation performed by the user through the graphical human-computer interaction interface, and generate a corresponding current work The current behavior tree of the flow, keeping the current behavior tree as an extensible markup language file; the behavior tree engine execution module is used to run the current behavior by accessing the extensible markup language file according to the user's running or debugging instruction tree, and implement the current workflow corresponding to the current behavior tree.

In one embodiment, each action node is composed of an interface part and an implementation part; for the implementation part, each action node is an application program; for the interface part, each action node is a Graphical elements for dragging and dropping, connection and property configuration; the interface part of each action node is set in the node library, and the implementation part of each node is set in a node service module; the behavior tree engine execution module includes : a behavior tree engine module and an executor module; the behavior tree engine module is used to run the current behavior tree, and when running to an action node, package the interface part of the action node to the executor module, The executor module parses it and encapsulates it into a corresponding execution request, sends it to the runtime server provided with the implementation part of the action node, and receives the execution of the implementation part of the action node by the runtime server The returned execution result is fed back to the behavior tree engine module, and the behavior tree engine module presents it to the user through the graphical interface module.

In one embodiment, the editing engine module is further configured to respond to the user's operation of maintaining the current behavior tree as a sub-behavior tree node and/or behavior tree template through the graphical human-computer interaction interface, and edit the The current behavior tree is stored as a node in the node library as a sub-behavior tree node and/or a behavior tree template, so as to provide the user with direct calling when creating a behavior tree.

In one embodiment, it further includes: a knowledge graph module, configured to construct a behavior tree knowledge graph based on the grammar of the knowledge graph for all nodes of the behavior tree to be constructed for storage; and to construct the current behavior tree into a knowledge graph based on the grammar of the knowledge graph Graph example; the behavior tree knowledge graph and the knowledge graph instance include: nodes respectively representing each node of the behavior tree, and multiple edges representing the relationship between nodes or node attributes.

In one embodiment, the knowledge graph module is further used to recommend control nodes and/or action nodes for the user based on the behavior tree knowledge graph and historical knowledge graph instances when the user constructs the current behavior tree , error checking, and smart mapping of node attributes.

In one embodiment, the action nodes in the current behavior tree include: the following action nodes executed sequentially: model trainer and application deployer; or include: the following action nodes executed sequentially: model trainer, model converter and application deployer; wherein, the model trainer is used to retrain a training model with a new data source or training parameters; the model converter is used to convert the trained model from the source format to the target format; the application deployer is used to convert the The model is deployed to the specified target device.

The workflow modeling implementation method proposed in the embodiment of the present invention includes: providing a graphical human-computer interaction interface for users to construct and configure behavior trees based on preset nodes used to construct various behavior trees; The behavior tree creation operation performed by the graphical human-computer interaction interface generates the current behavior tree corresponding to the current workflow, and keeps the current behavior tree as an extensible markup language file; the nodes include root nodes, logic control nodes and actions node; wherein, a behavior tree is used to represent a workflow, the logic control node is used to realize the logic control in the workflow, and the action node is used to realize the business operation in the workflow, and the action node includes: different General action nodes encapsulated by general operations in workflows in industrial fields, and proprietary action nodes oriented to specific industrial fields by encapsulating non-general operations in workflows in different industrial fields; according to the user's operation or debugging An instruction for running the current behavior tree by accessing the extensible markup language file, and realizing the current workflow corresponding to the current behavior tree.

In one embodiment, the current behavior tree is: a behavior tree built by the user by adding, connecting and configuring root nodes, required logic control nodes and required action nodes; The workflow type is a sub-behavior tree node selected from the sub-behavior tree node library, and the sub-behavior tree node is a stored historical behavior tree instance; the user selects the corresponding behavior tree from the behavior tree template library according to the required workflow type After the template, configure the behavior tree based on the behavior tree template and adjust the nodes or not adjust the behavior tree. The behavior tree template is a pre-stored behavior tree template or a behavior tree template dumped according to the user's historical behavior tree instance .

In one embodiment, each action node is composed of an interface part and an implementation part; for the implementation part, each action node is an application program; for the interface part, each action node is an Graphical elements of drag and drop, connection and attribute configuration; when each action node in the current behavior tree is executed, the input in the attribute configuration of the interface part of the action node will be read and transmitted to the action node The implementation part, and after the implementation part completes the corresponding operation, the operation result will be converted into the output in the attribute configuration of the action node interface part.

In one embodiment, it further includes: saving the current behavior tree as a sub-behavior tree node representing a behavior tree instance in the sub-behavior tree node library; and/or, using the frame of the current behavior tree as a behavior Tree templates are saved to the behavior tree template library.

In one embodiment, it further includes: constructing all nodes of the behavior tree into a behavior tree knowledge graph based on the grammar of the knowledge graph for storage; constructing the current behavior tree into a knowledge graph instance based on the grammar of the knowledge graph; the behavior The tree knowledge graph and the instance of the knowledge graph include: nodes respectively representing the nodes of the behavior tree, and multiple edges representing the relationship between the nodes or the attributes of the nodes.

In one embodiment, it further includes: when the user builds the current behavior tree, recommending logic control nodes and/or action nodes, error checking and Smart mapping of node attributes.

Another workflow modeling implementation system proposed in the embodiments of the present invention includes: at least one memory and at least one processor, wherein: the at least one memory is used to store computer programs; the at least one processor is used to call the The computer program stored in the at least one memory executes the workflow modeling implementation method described in any one of the above implementation manners.

A computer-readable storage medium proposed in an embodiment of the present invention, on which a computer program is stored; the computer program can be executed by a processor and implement a general workflow modeling method as described in any of the above implementation modes .

It can be seen from the above scheme that in the embodiment of the present invention, the behavior tree diagram is used to represent the operation process of the workflow, that is, the related operations in the workflow of different industrial fields are encapsulated into independent nodes, so that the nodes are reusable. It realizes the decoupling of specific business and engineering platforms, and then organizes nodes in the form of behavior tree to generate an intuitive workflow operation process from development to implementation, thereby reducing the complexity of workflow modeling and implementation.

In addition, by using the knowledge graph to represent and save the nodes and instances of the behavior tree in the construction of the behavior tree, and based on this, the recommendation function of the node, attribute or workflow can be further reduced, which can further reduce the complexity of workload modeling and implementation Spend.

Further, by storing the constructed behavior tree as a sub-behavior tree node or a behavior tree template, the complexity of workload modeling and implementation can be further reduced.

Description of drawings

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, so that those of ordinary skill in the art will be more aware of the above-mentioned and other features and advantages of the present invention. In the accompanying drawings:

FIG. 1 is an exemplary structural diagram of a general workflow modeling implementation system in an embodiment of the present invention.

FIG. 2A is a schematic diagram of an encapsulated model trainer node in an example of the present invention.

FIG. 2B is a schematic diagram of a human-computer interaction interface in an example of the present invention.

FIG. 2C is a schematic diagram of a behavior tree storage format in an example of the present invention.

FIG. 2D is a schematic diagram of a behavior tree in an artificial intelligence application in an example of the present invention.

FIG. 2E is a schematic diagram of a behavior tree in an application of a quality inspection production line in an example of the present invention.

Fig. 3 is a schematic diagram of a behavior tree knowledge map in an example of the present invention.

Fig. 4A is a schematic diagram of a behavior tree in an example of the present invention.

FIG. 4B is a schematic diagram of converting the behavior tree shown in FIG. 4A into a knowledge graph example.

FIG. 5 is a schematic diagram of node recommendation when a user builds a behavior tree in an example of the present invention.

Fig. 6 is an exemplary flow chart of a method for implementing workflow modeling in an embodiment of the present invention.

Fig. 7 is an exemplary flowchart of another system for implementing workflow modeling in an embodiment of the present invention.

Wherein, the reference signs are as follows:

label

meaning

110110	接口库 interface library
120120	图形化界面模块 Graphical Interface Module
130130	编辑引擎模块 Edit Engine Module
140140	行为树引擎执行模块Behavior tree engine execution module
141141	行为树引擎模块Behavior Tree Engine Module
142142	执行器模块 actuator module
150150	知识图谱模块 Knowledge graph module
160160	节点服务模块 Node service module
601～604601～604	步骤 step
7171	存储器 memory
7272	处理器 processor
7373	显示器 monitor
7474	总线bus

Detailed ways

In the embodiment of the present invention, in order to improve the efficiency, flexibility and scalability of workflow development, it is considered to provide a unified behavioral operation for different industrial fields such as automation systems, artificial intelligence, robots, etc. to construct corresponding tasks. A common workflow modeling platform for streams, which uses a behavior tree diagram to represent each workflow. Specifically, general operations in workflows oriented to different industrial fields can be packaged as general nodes, and non-general operations in workflows can be packaged as dedicated nodes oriented to specific fields. These nodes are reusable and can realize the decoupling of specific business and engineering platforms. Organizing nodes in the form of behavior trees allows for an intuitive workflow from development to execution. In addition, each behavior tree instance can be represented and saved in the form of a knowledge graph, and on this basis, the recommendation function of nodes, attributes or workflows can be realized. Furthermore, a behavior tree instance or a behavior tree template can also be integrated into the system as a node, so that users can select a behavior tree corresponding to their workflow tasks instead of creating a workflow from scratch.

In order to make the purpose, technical solution and advantages of the present invention clearer, the following examples are given to further describe the present invention in detail.

FIG. 1 is an exemplary structural diagram of a general workflow modeling implementation system in an embodiment of the present invention. As shown by the solid line in FIG. 1 , the system may include: a node library 110 , a graphical interface module 120 , an editing engine module 130 and a behavior tree engine execution module 140 . In addition, as shown by the dotted line in Figure 1, in another embodiment, the system may further include: a knowledge graph module 150.

Among them, the node library 110 is provided with nodes for constructing various behavior trees, and the nodes include root nodes, logic control nodes and action nodes; wherein, a behavior tree is used to represent a workflow, and logic control nodes are used to implement work flow. The logic control in the flow, the action node is used to realize the business operation in the workflow, and the action node includes: the general action node that encapsulates the general operation in the workflow of different industrial fields, and the work of different industrial fields The non-universal operations in the flow are encapsulated into proprietary action nodes for specific industrial fields.

Among them, the root node is the start node of the behavior tree, and there is only one root node in a behavior tree.

The logic control node is used to realize the logic control in the workflow, and the logic control node is independent of the specific business of the industrial application. Through logical control nodes, users can create various workflows according to their needs. In this embodiment, the logical control nodes may include: basic control nodes, decorator nodes and conditional nodes. In other implementation manners, the logic control nodes may only include some of the basic control nodes, decorator nodes and condition nodes. For example, only base control nodes and condition nodes are included, or only base control nodes and decorator nodes are included, or although all three types of nodes are included, some types include only a part of them. Specifically, it may be determined according to actual needs, and is not limited here.

The above-mentioned nodes are briefly described below.

1. Basic control node: it defines how to execute branches in the behavior tree, and is used to implement branch logic control in the workflow, such as sequence, parallelism, selector, etc. Each base control node can have one or more child nodes. Basic control nodes mainly include:

1) Sequential node, which will trigger routing from the left to each child node in turn, until it finds a child node that returns failure or operation, and returns the failure or operation to its parent node; if and only if all child nodes of the node When all nodes return success, the node returns success to the parent node.

2) Select a node, and the node will trigger routing to each child node from the left until a child node that returns success or movement is found, and then the success or operation is returned to its parent node. A node returns failure only if all of the node's children return failure.

3) Parallel node, which will trigger routing from the left to all its child nodes. If M child nodes return success, return success; if N-M+1 child nodes return failure, return failure, otherwise return to run. where N is the number of child nodes and M<=N is a user-defined threshold.

2. Decorator node: It determines whether a branch or even a single node in the behavior tree can be executed, and is used to control workflow interruption and jump control. Each decorator node can only have one child node. Decorator nodes mainly include:

1) Inverter (Inverter) node is used to invert child nodes. Trigger the child node once, if the child node fails, it returns success; if the child node succeeds, it returns failure. If the child node returns running, the node also returns running.

2) Force Success (Force Success) node, if the child node returns running, the node also returns running; otherwise, the node will always return success.

3) Force Failure (Force Failure) node, if the child node returns to run, the node also returns to run; otherwise, the node will always return to fail.

4) A Repeat node can trigger its child nodes up to N times. As long as the child node returns success, N will be passed as the input port; if the child node returns failure, the cycle will be interrupted and failure will also be returned.

5) The Retry node can trigger its child nodes up to N times. As long as the child node returns failure, N will be passed as the input port; if the child node returns success, the loop will be interrupted and success will also be returned.

Execution nodes are the leaves of the behavior tree and have no children. It falls into two categories:

3. Condition node, used to execute conditional judgment and return the judgment result. It returns success or failure depending on whether the condition is true. Conditional nodes never return a running state.

The condition node and the action node described below are the leaves of the behavior tree and have no child nodes. The two can also be collectively referred to as an execution node.

Action nodes are used to execute commands and implement specific business operations in the workflow. Action nodes have no output connections. Returns success if the operation completed correctly, or failure if the operation failed. Returns running while the operation is in progress.

These action nodes include: reusable independent action nodes that encapsulate relevant business operations in various industrial fields. Among them, nodes that encapsulate general operations in workflows in different industrial fields are general action nodes, and nodes that encapsulate non-general operations in workflows in different industrial fields for specific industrial fields are proprietary actions node. Through action nodes, encapsulated work can be reused in different workflows.

Some example generic action nodes are listed below:

(1) Data collection nodes, which may include:

File Reader: Used to read data from files in CSV, TDMS, etc. formats.

Database connector: supports connection with common databases such as MS SQL and MongoDB.

OPC-UA client: used to obtain data from OPC-UA servers commonly used in the field of automation.

(2) Data processing nodes, which may include:

Data Cleaner: Used to format source data, delete abnormal data and duplicate data, etc.

Data Converter: Used to convert source data from source format to target format.

Data enhancer: used to perform spatial transformation, color transformation and other processing on the data to expand the data set.

The above nodes are general action nodes that may be involved in different industrial fields. Of course, in other implementation manners, there are more general action nodes, which are not listed here. In addition, there are some proprietary action nodes for specific industrial fields, and these proprietary action nodes can be used as extension nodes for specific industrial fields in the action nodes. Here are two simple categories:

(1) Model management nodes for artificial intelligence (AI), which may include:

Model Importer: This node is used to import a trained model from a model storage area such as a model library or a local PC.

Model Trainer: Used to retrain the imported training model with new data sources or training parameters. Figure 2A shows a schematic diagram of a model trainer node that encapsulates model training operations. As shown in Figure 2A, this node belongs to the model trainer node in the action node, the protocol adopted is RESTful API, the URL of the AI model is: https://IP.port/api/v1/train, and the input is: training data The package file of the set, the output is: the trained model file.

Model Converter: Used to convert a trained model from a source format to a target format. For example, from Tensorflow frozen pb(.pb) to ONNX(.ONNX).

Model Adapter: It is used to realize automatic feature extraction, model selection, parameter optimization and other model adaptation processes based on automatic machine learning according to the currently determined AI application type, that is, to automatically select the appropriate AI model from the model library.

Application Deployer: Used to package the AI model and its peripheral programs into AI applications, and then deploy them to specified devices.

In specific implementation, all AI frameworks (such as TensorFlow, Scikit-learn, ONNX, Kubeflow, and AutoML) can be encapsulated as AI action nodes in the behavior tree.

...

(2) Production line quality inspection nodes for industrial automation, which may include:

Conveyor belt controller: used to control the start and stop of the conveyor belt.

Camera controller: used to control the camera's camera operation.

Robot picking: used to pick up the corresponding products from the current production line.

Quality inspector: It is used for quality inspection of the products picked up by the robot, placing qualified products in the first position, and placing unqualified products in the second position.

Robot pick-up: used to put the products after quality inspection to the corresponding position.

Motion controller: used to position products on the production line.

...

In addition, there can be more proprietary action nodes for other industrial fields. They are not listed here. Among them, nodes oriented to different industrial fields can be stored as extended nodes of action nodes, and users can also expand nodes and keep them in the node library.

In this embodiment, each action node may consist of an interface part and an implementation part. For the implementation part, each action node will be an application (e.g., a containerized application) that contains functional code and runtime dependencies. The application can run independently and be exposed through specific network communication interfaces, such as RESTful API and RPC. For the interface part, each action node is a graphic element that can be dragged and dropped in the human-computer interaction interface, and each action node has a property panel for configuring the properties of the action node, such as input and output.

Each action node can be configured and executed individually. When the action node is executed, the input configured by the user in the interface part of the action node will be read and converted to the corresponding application. After completing a specific operation such as model conversion, the operation result such as the converted model will be converted back to the output of the interface part of the action node.

In this embodiment, the interface part and implementation part of each motion node can be stored separately. For example, the interface library 110 may only store the interface part of the action node, and the implementation part of the action node may be stored in a node service module 160, and the node service module 160 may be located in a server or locally.

In summary, the status of the node mainly includes: success, failure and operation.

Further, in this embodiment, in order to improve the efficiency of user modeling, users are also allowed to store the created behavior tree as a behavior tree instance as a sub-behavior tree node, which is convenient for the user or other users to directly transfer.

In addition, in order to improve the efficiency of user modeling and allow users to modify and edit as needed, behavior tree templates for different workflow types can also be set for users, and the behavior tree templates can also be stored as a node, which is convenient for users It can generate the desired behavior tree by directly invoking and making necessary parameter configurations and node adjustments during modeling. In addition, the behavior tree created by the user can also be stored as a behavior tree template node. The difference between this behavior tree template node and the aforementioned sub-behavior tree nodes is that the behavior tree template only stores the nodes in the behavior tree and the logic between nodes The behavior tree frame formed does not store specific configuration data; while the sub-behavior tree nodes not only store the nodes in the behavior tree and the logic between nodes, but also store the configuration data of each node.

The graphical interface module 120 is used to provide a graphical human-computer interaction interface for users to construct and configure behavior trees based on the nodes in the node storage module. In one embodiment, it may include a node library presentation interface, a behavior tree editing interface, a node configuration interface, a variable configuration interface, a data blackboard interface, and an operation monitoring interface.

Wherein, the node library presentation interface is used for presenting the nodes stored in the node library 110 to the user for selection by the user. For example, the left side of FIG. 2B shows a schematic diagram of a node library presentation interface in an example.

The behavior tree editing interface is an interface for providing users with behavior tree construction operations. Among them, the behavior tree construction operation may include the drag operation of nodes, the association operation between nodes, etc. For example, the middle of FIG. 2B shows a schematic diagram of the behavior tree editing interface in an example.

The node configuration interface can display the properties of the node when the user clicks on the node to be configured, including input parameters, output parameters, etc. The user can also set the input parameters and output parameters of the node. Among them, in addition to configuring corresponding values for input parameters and output parameters, they can also be associated with other nodes for input and output. For example, the input value of node 3 comes from the output value of node 1.

The variable configuration interface is used to provide users with configuration operations on global variables when global variables need to be configured.

The data blackboard interface is used to present the recorded node instance list, and the node instance list presents each node in the behavior tree in the form of a list or a tree. Users can modify the properties of nodes through the blackboard interface, and monitor the status of all node instances when the behavior tree is running. For example, the right side of FIG. 2B shows a schematic diagram of the data blackboard interface in an example.

The running monitoring interface is used for users to set the nodes or custom data required for monitoring, and present the intermediate or final result data of the behavior tree during debugging or running.

The editing engine module 130 is used to respond to the behavior tree creation operation performed by the user based on the graphical human-computer interaction interface, and generate a current behavior tree corresponding to the current workflow, and keep the current behavior tree as an Extensible Markup Language (XML, Extensible Markup Language) file.

In this embodiment, the editing engine module 130 is used to respond to the node drag and drop operation performed by the user based on the graphical human-computer interaction interface to add a node in the behavior tree; to respond to the user's node deletion operation to delete the node from the behavior tree; to respond to the user The node attribute configuration operation stores the configuration parameters in the configuration file; saves, loads or exits the behavior tree in response to the user's save or load or exit behavior tree operation. Wherein, when loading the behavior tree, if there is a corresponding system configuration file, the system configuration file is loaded at the same time to initialize the corresponding nodes.

In this embodiment, an XML file format can be defined to save the current behavior tree. As an example, for the behavior tree shown in Figure 2C, it can be stored as a format file as shown below:

In addition, the editing engine module 130 is also used to store the current behavior tree as a child behavior tree node and/or behavior tree template in response to the user's operation of keeping the current behavior tree (usually a behavior tree that has been debugged or successfully run) as a child behavior tree node and/or a behavior tree template. Behavior tree nodes and/or behavior tree templates. Correspondingly, the editing engine module 130 is also configured to load the sub-behavior tree nodes and/or behavior tree templates into the behavior tree editor of the graphical interface module 120 in response to the user's operation of loading the sub-behavior tree nodes and/or behavior tree templates. presented to the user on the interface.

Fig. 2D shows a schematic diagram of a behavior tree in an artificial intelligence application in an example. As shown in FIG. 2D , the behavior tree includes the following action nodes executed sequentially: model trainer (referred to as training in FIG. 2D ) and application deployer (referred to as deployment in FIG. 2D ). In other implementation manners, the following action nodes executed sequentially may also be included: model trainer, model converter, and application deployment. Or include: the following action nodes executed sequentially: model importer, model trainer, and application deployer, or include the following action nodes executed sequentially: model importer, model trainer, model converter, and application deployer; or Contains the following action nodes executed sequentially: Model Adapter, Model Trainer, and Application Deployer. In addition, it may further include: a data collector and a data preprocessor. Specifically, it may be determined according to actual needs, and is not limited here.

FIG. 2E shows a schematic diagram of a behavior tree in an example of a quality inspection production line application.

The behavior tree engine execution module 140 is configured to run the current behavior tree by accessing the XML file according to the user's running or debugging instruction, and realize the current workflow corresponding to the current behavior tree.

In this embodiment, the behavior tree engine execution module 140 can start to run from the root node of the behavior tree, and the root node can generate a signal allowing the node to be triggered at a given frequency, and the signal is sent to its child nodes. Each node executes only when said trigger signal is received. If it is executing, the child node immediately returns to run to the parent node; if the goal is achieved, it returns success; otherwise, it returns failure. It has all or some of the following functions:

1) Run: run the behavior tree, if the node returns the expected value, the result of the behavior tree is success, otherwise it is failure.

2) Debugging: The debugging function is similar to the running function, and is mainly used to debug complex tasks before officially executing the workload. In order to monitor the debugging process conveniently, you can set breakpoints in the properties of the action nodes that need to be monitored. After setting the breakpoint, when debugging the behavior tree, it will pause at the action node twice, one is for the user to check the input value before executing the action node, and the other is for the user to check the output value after the action node is executed.

3) Monitoring: When running or debugging a behavior tree, users can add some nodes or custom data to the running monitoring interface for monitoring. For example, when a robot is running, you can add a custom data such as "joint position" to monitor the current robot joint position.

4) Error handling: Nodes can have error attributes, including error IDs and error messages. When a node fails to execute, detailed error information can be obtained through the node's error attributes.

5) System log record: In this embodiment, a system log mechanism may be supported. Important information can be recorded, which is very useful for analyzing some complex problems.

In one embodiment, the behavior tree engine execution module 140 may include: a behavior tree engine module 141 and an executor module 142 . Wherein, the behavior tree engine module 141 is used to run the behavior tree, and when it runs to an action node, it provides the interface part of the action node as a package to the executor module 142, and the executor module 142 executes it Parsing and packaging as a corresponding execution request is sent to the node service module 160 provided with the implementation part of the action node, and receiving the execution result returned by the node service module 160 after executing the implementation part of the action node, and converting the The execution result is fed back to the behavior tree engine module 141, and the behavior tree engine module 141 presents it to the user through the operation monitoring interface of the graphical interface module 120.

The executor module 142 may be an executor supporting different protocols, for example, a RESTful API executor, an SDK executor, a GRPC executor, or a ROS executor.

The knowledge map module 150 is used to construct all nodes of the behavior tree based on the grammar of the knowledge map into a behavior tree knowledge map for storage; the behavior tree knowledge map includes: nodes representing each node of the construction behavior tree, and nodes representing nodes multiple edges between relationships or node attributes. The knowledge graph module 150 is also used for recommending control nodes and/or action nodes, error checking, and intelligent mapping of node attributes for the user based on the behavior tree knowledge graph when the user constructs the current behavior tree.

During specific implementation, the knowledge graph module 150 may include four sub-modules: a data acquisition sub-module, a knowledge fusion sub-module, a knowledge processing sub-module, and a knowledge graph database.

Among them, the data collection sub-module supports structured data, and can obtain all nodes for constructing a behavior tree from the node library 110 . In addition, when the user finishes constructing each behavior tree, the node composition and association relationship in the behavior tree can also be collected.

The knowledge fusion sub-module is used for coreference resolution and entity disambiguation processing. Among them, coreference resolution is used to solve the problem that multiple references correspond to the same entity object. By using common reference resolution techniques, these referenced items can be associated (merged) to the correct entity object. For example, for the situation that different behavior trees include the same node, the node can be associated with the correct node through the knowledge fusion technology. Entity disambiguation is a special technique to solve the problem of ambiguity of entities with the same name. Through entity disambiguation, entity links can be accurately established according to the current context.

Knowledge processing sub-module is used to realize knowledge reasoning, quality evaluation and ontology extraction. Among them, knowledge reasoning starts from the existing entity relationship data in the knowledge graph database, and establishes new associations between entities through computer reasoning, thereby expanding and enriching the knowledge network. Quality evaluation is an important part of knowledge graph database construction technology. The significance of this part is to quantify the credibility of knowledge and ensure the quality of knowledge graph database by discarding low confidence knowledge. Ontology extraction is an important part of knowledge processing, which is used to represent the relationship between knowledge itself and knowledge, and this model needs to continuously update the knowledge in the model.

The knowledge graph database is used to save and construct the behavior tree knowledge graph for further use.

The knowledge map grammar in this embodiment can be as follows:

1) Define main classes, Table 1 shows the classes defined in an example.

Table 1

2) Define data attributes. Table 2 shows the data attributes defined in an example.

数据属性data attribute
输入enter
输出output
取反Negate
强制成功force success
强制失败force failure
重复repeat
重试Retry

Table 2

3) Define the object properties.

Sequence: (A, sequence, B), which means that class B is executed only after class A is successfully executed.

Choice: (A, choice, B), means to choose A or B to be executed. For example, (data collection A, selection, data collection B), wherein, data collection A is to collect data from the camera, and data collection B is to collect data from the Internet. At this time, only one needs to be selected according to the needs of the user.

Parallel: (A, parallel, B), means that A and B are executed at the same time and do not depend on each other.

Table 3 shows an example of the object properties defined.

table 3

After defining classes, data attributes, and object attributes, a behavior tree knowledge graph as shown in Figure 3 can be constructed.

In this embodiment, the behavior tree constructed by the user can be further constructed into a knowledge graph instance based on the grammar of the knowledge graph for storage, so that when the user constructs the current behavior tree, the knowledge graph instance based on the behavior tree knowledge graph and history is The user performs recommendation of logical control nodes and/or action nodes, error checking, and intelligent mapping of node attributes.

Fig. 4A is a schematic diagram of a behavior tree in an example of the present invention. FIG. 4B is a schematic diagram of converting the behavior tree shown in FIG. 4A into a knowledge graph example.

As shown in FIG. 5 , it shows a schematic diagram of node recommendation when a user constructs a behavior tree in an example of the present invention. As shown in the gray dashed box in Figure 5, when creating an action node for a sequence node, the system recommends Action 5, Action 6, and Action 7, etc., and gives matching degrees of 75%, 50%, and 20%, respectively. To facilitate the user to choose.

The general implementation system of workflow modeling in the embodiment of the present invention has been described in detail above, and the general implementation method of workflow modeling in the embodiment of the present invention will be described in detail below. The workflow modeling implementation method in the embodiment of the present invention can be implemented on the workflow modeling implementation system in the embodiment of the present invention. For the details not disclosed in the method embodiment of the present invention, please refer to the system embodiment of the present invention The corresponding descriptions are not repeated here.

Fig. 6 is an exemplary structural diagram of a method for implementing workflow modeling in an embodiment of the present invention. As shown in the solid line part in Figure 6, the method may include the following steps:

Step 601 , setting up a general workflow creation platform with a graphical human-computer interaction interface as shown in FIG. 1 , where nodes for building various behavior trees are set. The nodes may include: a root node, a logic control node and an action node; wherein, a behavior tree is used to represent a workflow, the logic control node is used to realize the logic control in the workflow, and the action node is used to realize the workflow in the workflow. Business operations, and the action nodes include: a general action node that encapsulates general operations in workflows in different industrial fields, and a specific industry-oriented node that encapsulates non-general operations in workflows in different industrial fields Proprietary action node for .

In this embodiment, the status of the node mainly includes: success, failure and running.

These action nodes include: reusable independent action nodes that encapsulate relevant business operations in various industrial fields. Through action nodes, encapsulated work can be reused in different workflows.

In this embodiment, each action node may consist of an interface part and an implementation part. For the implementation part, each action node will be an application that contains functional code and runtime dependencies. The application can run independently and be exposed through specific network communication interfaces, such as RESTful API and remote procedure call (RPC) interface. For the interface part, each action node is a graphic element that can be dragged and dropped in the workflow editor of the human-computer interaction interface, and each node has a property panel for configuring the properties of the action node, such as input and output. Each action node can be configured and executed individually. When the action node is executed, the input configured by the user in the interface part of the action node will be read and converted to the corresponding application. After completing a specific operation such as model conversion, the operation result such as the converted model will be converted back to the output of the interface part of the action node.

In this embodiment, the interface part and implementation part of each motion node can be stored separately. For example, the interface part can be stored in the interface library, and the implementation part can be stored in a node service module, which can be located on a runtime server or locally. In addition, the interface library is also used to store logic control nodes.

Step 602: Generate a current behavior tree corresponding to the current workflow in response to the behavior tree creation operation performed by the user through the graphical human-computer interaction interface, and save the current behavior tree as an XML file.

In this embodiment, the current behavior tree is: a behavior tree built by the user by adding, connecting and configuring root nodes, required logic control nodes and required action nodes; The flow type is a sub-behavior tree node selected from the sub-behavior tree node library, and the sub-behavior tree node is a stored historical behavior tree instance; the user selects the corresponding behavior tree template from the behavior tree template library according to the required workflow type Finally, configure the behavior tree based on the behavior tree template and adjust the nodes or not adjust the behavior tree, the behavior tree template is a pre-stored behavior tree template or a behavior tree template dumped according to the user's historical behavior tree instance.

Wherein, the sub-behavior tree node library and the behavior tree template library can be juxtaposed with the node library, or can also be a part of the node library.

The general execution logic in the embodiment of the present invention is from left to right and from top to bottom, and the specific execution logic is controlled by the logic control node.

In this embodiment, each node added in the current behavior tree can be further converted into a node instance and added to a node instance list for storage or viewing, and the node instance list presents all nodes in list form or tree form. Describes all nodes in the current behavior tree.

Step 603: According to the user's running or debugging instruction, access the XML file to run the current behavior tree, and implement the current workflow corresponding to the current behavior tree.

As shown before, when an action node is executed, the input configured by the user in the interface part of the action node will be read and converted to the implementation part of the action node. After the implementation part completes a specific operation, the result of the operation is converted back to the output of the interface part of the action node.

During specific implementation, the intermediate results and final results during the execution process can be displayed.

In some implementations, for the same application scenario, for example, deploying the same type of AI model to the same type of target device and implementing the same type of AI application, in order to avoid the waste of manpower and time for repeatedly building behavior trees For the user, the current behavior tree that has been built, preferably debugged or run successfully, is saved as a behavior tree instance, for example, it is saved as a sub-behavior tree node representing the behavior tree instance in the sub-behavior tree node library. When it is necessary to build the same behavior tree again, just call the behavior tree instance directly.

In addition, for similar application scenarios, such as deploying different types of AI models to the same type of target device and implementing the same type of AI application, in order to avoid the waste of manpower and time in repeatedly building behavior trees, for users who have already It is best to save the framework of the current behavior tree that has been debugged or successfully run as a behavior tree template after the construction is completed. Of course, behavior tree templates corresponding to different workflows can also be preset. These Behavior Tree Templates can be stored in the Behavior Tree Template Library. When users need to build a similar behavior tree, they can select the corresponding behavior tree template and perform necessary configurations to obtain a behavior tree for realizing the required workflow.

In addition, in order to further increase the convenience of behavior tree construction, when users create behavior trees, node recommendation, error checking and intelligent mapping of node attributes can be carried out for users, which can further

The behavior tree instance is converted into a knowledge graph instance for saving. Correspondingly, as shown in the dotted line part in Fig. 6, the following steps are further included:

Step 604, building all nodes of the behavior tree into a behavior tree knowledge graph based on the grammar of the knowledge graph for storage; in addition, the current behavior tree can be further constructed into a knowledge graph instance based on the grammar of the knowledge graph; the behavior tree The knowledge graph and the instance of the knowledge graph include: nodes respectively representing the nodes of the behavior tree, and multiple edges representing the relationship between the nodes or the attributes of the nodes.

Correspondingly, when a user creates a behavior tree, the recommendation of control nodes and/or action nodes, error checking, and intelligent mapping of node attributes can be performed for the user based on the stored behavior tree knowledge graph and historical knowledge graph instances. Similarity calculation and comparison, deep learning, etc. can be performed in this embodiment.

FIG. 7 is a schematic structural diagram of another system for implementing workflow modeling in an embodiment of the present application. The system can be used to implement the method shown in FIG. 6 or implement the system shown in FIG. 1 . As shown in FIG. 7 , the system may include: at least one memory 71 , at least one processor 72 and at least one display 73 . In addition, some other components may also be included, such as communication ports and the like. These components communicate via bus 74 .

Among them, at least one memory 71 is used to store computer programs. In one embodiment, the computer program can be understood as including various modules of the workflow modeling implementation system shown in FIG. 1 . In addition, at least one memory 71 can also store an operating system and the like. The operating system includes but is not limited to: Android operating system, Symbian operating system, Windows operating system, Linux operating system and so on.

At least one processor 72 is configured to invoke a computer program stored in at least one memory 71 to execute the workflow modeling implementation method described in the embodiment of the present application. The processor 72 may be a CPU, a processing unit/module, an ASIC, a logic module or a programmable gate array, and the like. It can receive and send data through the communication port.

At least one display 73 is used to display the human-computer interaction interface.

Specifically, at least one processor 72 is configured to invoke a computer program stored in at least one memory 71 to enable the system to execute the operations in the workflow modeling implementation method in any of the foregoing implementation manners.

It should be noted that not all steps and modules in the above-mentioned processes and structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as required. The division of each module is only to facilitate the description of the functional division adopted. In actual implementation, one module can be divided into multiple modules, and the functions of multiple modules can also be realized by the same module. These modules can be located in the same device. , or on a different device.

It can be understood that the hardware modules in the foregoing embodiments may be implemented mechanically or electronically. For example, a hardware module may include specially designed permanent circuits or logic devices (such as special-purpose processors, such as FPGAs or ASICs) to perform specific operations. Hardware modules may also include programmable logic devices or circuits (including, for example, general-purpose processors or other programmable processors) temporarily configured by software to perform particular operations. As for implementing the hardware module in a mechanical way, using a dedicated permanent circuit, or using a temporarily configured circuit (such as configured by software) to realize the hardware module, it can be decided according to cost and time considerations.

In addition, the embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and the computer program can be executed by a processor to implement the workflow modeling implementation method described in the embodiment of the present application . Specifically, a system or device equipped with a storage medium may be provided, on which the software program code for realizing the functions of any implementation manner in the above-mentioned embodiments is stored, and the computer (or CPU or MPU of the system or device) ) to read and execute the program code stored in the storage medium. In addition, an operating system or the like operated on a computer may also complete part or all of the actual operations through instructions based on program codes. It is also possible to write the program code read from the storage medium into the memory set in the expansion board inserted into the computer or into the memory set in the expansion unit connected to the computer, and then based on the instructions of the program code, the memory installed in the The expansion board or the CPU on the expansion unit executes part or all of the actual operations, so as to realize the functions of any one of the above-mentioned implementation manners. Embodiments of storage media for providing program codes include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), Tape, non-volatile memory card, and ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of the present invention. within the scope of protection.

Claims

The workflow modeling implementation system is characterized by including:

A node library (110), wherein nodes for constructing various behavior trees are provided, and the nodes include root nodes, logic control nodes and action nodes; wherein, a behavior tree is used to represent a workflow, and the logic control nodes use To realize the logic control in the workflow, the action nodes are used to realize the business operations in the workflow, and the action nodes include: general action nodes that encapsulate common operations in workflows in different industrial fields, and different Proprietary action nodes for specific industrial fields encapsulated by non-universal operations in industrial workflows;

A graphical interface module (120), configured to provide a graphical human-computer interaction interface for users to construct and configure behavior trees based on the nodes in the node storage module;

The editing engine module (130) is used to respond to the behavior tree creation operation performed by the user through the graphical human-computer interaction interface, and generate a current behavior tree corresponding to the current workflow, and maintain the current behavior tree as an extensible markup language document;

A behavior tree engine execution module (140), configured to run the current behavior tree by accessing the extensible markup language file according to a user's running or debugging instruction, and realize the current workflow corresponding to the current behavior tree.
The workflow modeling implementation system according to claim 1, wherein each action node is composed of an interface part and an implementation part; for the implementation part, each action node is an application program; for the interface part, each The action node is a graphic element that can be dragged and dropped, connected and configured on the human-computer interaction interface;

The interface part of each action node is set in the node library, and the implementation part of each node is set in a node service module;

The behavior tree engine execution module (140) includes: a behavior tree engine module (141) and an executor module (142); the behavior tree engine module (141) is used to run the current behavior tree, and when running to an action node, the interface part of the action node is packaged and provided to the executor module (142), and the executor module (142) parses it and encapsulates it as a corresponding execution request and sends it to the device configured with the action In the runtime server of the implementation part of the node, and receive the execution result returned by the runtime server after executing the implementation part of the action node, feed back the execution result to the behavior tree engine module (141), and the The behavior tree engine module (141) is presented to the user through the graphical interface module (120).
The workflow modeling implementation system according to claim 1, characterized in that, the editing engine module (130) is further configured to respond to the user through the graphical human-computer interaction interface to maintain the current behavior tree as a child The operation of the behavior tree node and/or the behavior tree template, storing the current behavior tree as a node in the node library (110) as a sub-behavior tree node and/or behavior tree template, to provide the user with behavior tree creation when called directly.
The workflow modeling implementation system according to claim 1, further comprising: a knowledge graph module (150), configured to construct a behavior tree knowledge graph for storing all nodes of the behavior tree based on the syntax of the knowledge graph ; Construct the current behavior tree into a knowledge graph instance based on the syntax of the knowledge graph; the behavior tree knowledge graph and the knowledge graph instance include: respectively represent the nodes of each node of the behavior tree, and represent the relationship between nodes or Multiple edges for node attributes.
The workflow modeling implementation system according to claim 4, wherein the knowledge graph module (150) is further configured to, when the user constructs the current behavior tree, based on the behavior tree knowledge graph and historical knowledge graph Examples include recommending control nodes and/or action nodes, error checking, and intelligent mapping of node attributes for the user.
The workflow modeling implementation system according to any one of claims 1 to 5, wherein the action nodes in the current behavior tree include:

The following action nodes executed sequentially: model trainer and application deployer; or include:

The following action nodes are executed sequentially: model trainer, model converter, and application deployer;

Wherein, the model trainer is used for retraining a training model with a new data source or training parameters;

The model converter is used to convert the trained model from the source format to the target format;

The application deployer is used to deploy the model to the specified target device.
The implementation method of workflow modeling is characterized by comprising:

Provide the user with a graphical human-computer interaction interface for constructing and configuring behavior trees based on the preset nodes used to construct various behavior trees; in response to the behavior tree creation operation performed by the user through the graphical human-computer interaction interface, generate a corresponding current The current behavior tree of the workflow, and keep the current behavior tree as an extensible markup language file; the nodes include root nodes, logic control nodes and action nodes; wherein, a behavior tree is used to represent a workflow, and the The logic control node is used to implement the logic control in the workflow, and the action node is used to implement the business operation in the workflow, and the action node includes: a general action node that encapsulates the general operations in the workflow in different industrial fields , and the proprietary action nodes for specific industrial fields that encapsulate non-universal operations in workflows in different industrial fields;

According to the user's running or debugging instruction, the current behavior tree is executed by accessing the extensible markup language file, and the current workflow corresponding to the current behavior tree is realized.
The implementation method of workflow modeling according to claim 7, wherein the current behavior tree is constructed by the user by adding, connecting and configuring root nodes, required logic control nodes and required action nodes Behavior tree for ; or for:

The sub-behavior tree node selected by the user from the sub-behavior tree node library according to the required workflow type, and the sub-behavior tree node is a stored historical behavior tree instance;

After the user selects the corresponding behavior tree template from the behavior tree template library according to the required workflow type, configure the behavior tree based on the behavior tree template and perform node adjustment or no adjustment to obtain the behavior tree. The behavior tree template is a pre-stored Behavior tree templates or behavior tree templates dumped based on the user's historical behavior tree instances.
The implementation method of workflow modeling according to claim 7, wherein each action node is composed of an interface part and an implementation part; for the implementation part, each action node is an application program; for the interface part, each The action node is a graphic element that can be dragged and dropped, connected and configured on the human-computer interface;

When each action node in the current behavior tree is executed, the input in the attribute configuration of the interface part of the action node will be read and transmitted to the implementation part of the action node, and the corresponding After the operation, the operation result will be transformed into the output in the attribute configuration of the interface part of the action node.
The workflow modeling implementation method according to claim 7, further comprising: saving the current behavior tree as a sub-behavior tree node representing a behavior tree instance in the sub-behavior tree node library; and/or , saving the framework of the current behavior tree as a behavior tree template in the behavior tree template library.
The workflow modeling implementation method according to any one of claims 7 to 10, further comprising: constructing all nodes of the behavior tree into a behavior tree knowledge graph based on the syntax of the knowledge graph for storage; The current behavior tree is constructed into a knowledge graph instance based on the grammar of the knowledge graph; the behavior tree knowledge graph and the knowledge graph instance include: nodes representing each node of the constructed behavior tree, and nodes representing the relationship between nodes or node attributes Multiple sides.
The workflow modeling implementation method according to claim 11, further comprising: when the user builds the current behavior tree, performing logic control for the user based on the behavior tree knowledge graph and historical knowledge graph instances Recommendations for nodes and/or action nodes, error checking, and intelligent mapping of node attributes.
A workflow modeling implementation system, comprising: at least one memory and at least one processor, wherein:

said at least one memory is for storing computer programs;

The at least one processor is configured to call a computer program stored in the at least one memory to execute the workflow modeling implementation method according to any one of claims 7 to 12.
A computer-readable storage medium, on which a computer program is stored; characterized in that, the computer program can be executed by a processor to implement the workflow modeling implementation method according to any one of claims 7 to 12.