CN112463143A - DAG-based dynamic rearrangement method - Google Patents
DAG-based dynamic rearrangement method Download PDFInfo
- Publication number
- CN112463143A CN112463143A CN202011386785.2A CN202011386785A CN112463143A CN 112463143 A CN112463143 A CN 112463143A CN 202011386785 A CN202011386785 A CN 202011386785A CN 112463143 A CN112463143 A CN 112463143A
- Authority
- CN
- China
- Prior art keywords
- module
- processing
- module set
- flow
- interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/34—Graphical or visual programming
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
- G06F8/44—Encoding
- G06F8/445—Exploiting fine grain parallelism, i.e. parallelism at instruction level
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The invention discloses a dynamic rearranging method based on DAG. The dynamic rearrangement method comprises two steps, wherein the first step is the construction and function realization of the dynamic rearrangement framework, and the second step is the processing of realizing the business process based on the constructed framework. The method is based on the directed acyclic graph, adopts a mode of processing module hierarchical cascade to realize the processing flow of source data, and solves the key problem that different data can be processed only by reconfiguring the arrangement mode of the hierarchical components without modifying the codes of the hierarchical modules after the complex data is changed by abstracting and unifying the processing flow and designing a standardized interface of the hierarchical modules. And by designing an event trigger mechanism in the module, different data services can be processed in a branched manner when corresponding events occur.
Description
Technical Field
The invention relates to a traditional software development and internet technology, in particular to a DAG-based dynamic rearrangement method which is used for dynamically changing the scene of a business processing flow.
Background
The current stream processing technology is mainly realized by customizing according to different service scenes, the processing flow is fixed when programming is finished, and the flow rearrangement requirements of various scenes cannot be realized. Still other processing flows can only provide a linear processing mode, and cannot perform branch processing on the flows, and in addition, each stage of processing of the flows cannot be changed after the program is written, so that coding and debugging must be performed again when the data processing flow is changed, and the workload of development, debugging and deployment is remarkably increased.
For example, the current asynchronous event-driven Java open source network application framework Netty adopts a pipeline to process a service flow, the service can be processed bidirectionally in the pipeline, a plurality of service processing handles can be bound in each direction, the service processing handles are sequentially arranged in the pipeline from front to back, and the front and back flows cannot be adjusted when recoding is not performed; the pipeline processing process has no branch and cannot be realized when the data needs different branch processing.
Disclosure of Invention
In view of the problems of the prior art, the present invention provides a dynamic reconfigurable method based on DAG. The method is based on the directed acyclic graph, adopts a mode of processing module hierarchical cascade to realize the processing flow of source data, and solves the key problem that different data can be processed only by reconfiguring the arrangement mode of the hierarchical components without modifying the codes of the hierarchical modules after the complex data is changed by abstracting and unifying the processing flow and designing a standardized interface of the hierarchical modules. And by designing an event trigger mechanism in the module, different data services can be processed in a branched manner when corresponding events occur.
The technical scheme adopted by the invention is as follows: a DAG-based dynamic rearrangeable method, comprising the steps of:
firstly, building and function realization of dynamic reconfigurable frame
The dynamic rearrangeable integral framework is divided into three layers: the system comprises a display layer, a service processing layer and a data persistence layer; the display layer comprises an interface module used for displaying the current processing progress and interface operation; the business processing layer comprises a flow processing engine module and a module/module set management module, wherein the flow processing engine module is used for processing the data to be analyzed in parallel and delivering the final processing result to the interface layer for display; the data persistence layer comprises a data storage module and is used for recording the connection relation of the modules/module sets and data analysis results.
The functional implementation of the dynamic re-arrangeable framework comprises the following steps:
1. the module/module set management module realizes the definition and realization of the module, the registration interface, the de-registration interface and the module set connection relation interface of the module set.
2. The module/module management module realizes the verification of the business processing flow; the verification comprises the following steps:
(1) checking whether the business processing flow is an effective directed acyclic graph or not;
(2) and checking whether the triggering events in the module set meet the triggering requirements.
3. The flow processing engine module realizes the parallel processing of the business flow; the parallel processing comprises the following steps:
(1) parallel processing of a plurality of flows;
(2) different branch processes of the same flow.
Second, business process processing based on framework
And (3) completing dynamic rearrangement according to a specific scene based on the frame built in the first step, wherein the specific steps are as follows:
the method comprises the steps of defining a business processing flow, dividing the business processing flow into a plurality of sub-flows according to functions, wherein each sub-flow is designed into a high-cohesion low-coupling module set and is used for node operation in a dynamic rearrangement process.
Secondly, after the business process is fully understood, a module set is modeled according to the sub-process functions split in the first step, whether the module set needs to be split into modules with smaller granularity is determined according to the complexity, and the final process consists of the module set formed by integrating a plurality of modules; firstly, respectively designing a module set and an interface of a module/module set management module, wherein a module set service data interface is used for transmitting data among the module sets, and a module set event trigger interface is used for branch processing among the module sets; the module/module set management module data processing interface is used for managing data processing, and the module/module set management module event trigger interface is used for business process branch processing.
(1) Module/Module set management Module interface definition
a. Existing modules/module sets are registered into the system, and the interfaces are defined as follows:
int RegistProtocolModule2Manager(const unsigned short& mainSubModuleType, void* pProtocolModuleInstance);
wherein the interface parameters are defined as follows:
mainSubModuleType: the master-slave type is used for uniquely identifying one module/module set;
pProtocolModuleInstance: module/module set handling classes.
b. Unregisters unneeded modules/module sets, and the interface is defined as follows:
int UnrRegistProtocolModule2Manager(const unsigned short& mainSubModuleType);
wherein the interface parameters are defined as follows:
mainSubModuleType: the master-slave type is used for uniquely identifying one module/module set;
(2) the module set service data interface is defined as follows:
virtual int ProcessData(const void* lpInContext, void* lpOutContext)；
wherein the interface parameters are defined as follows:
lpinContext: context data of the superior module/module set specifically includes a processing result of the superior module/module set and data required by the current module/module set;
lpinContext: the context data of the subordinate modules/module sets specifically includes processing results of the current modules/module sets and data required by the subordinate modules/module sets.
(3) The module set event trigger interface is defined as follows:
virtual int TriggerEvent(const int eventType, const void* lpOutContext);
wherein the interface parameters are defined as follows:
eventType: the event type is used for setting branch events needing to be triggered;
lpOutContext: the context data of the subordinate modules/module sets specifically includes processing results of the current modules/module sets and data required by the subordinate modules/module sets.
Thirdly, the process processing engine module models the relationship between the module sets according to the scene requirements and stores the relationship into a relationship coefficient database or an XML file; modeling the module, the module set and the module set relation through an XML file or a database configuration table, and forming a DAG model for processing a business process after the modeling is finished; when the flow is changed, processing according to a new flow after the XML configuration file or the database configuration table is changed and the verification is carried out again; constructing a module/module set by utilizing a directed acyclic graph in a graph theory after the module/module set is configured according to a database; each vertex of the directed acyclic graph is a module set, data are circulated in the module set of the directed acyclic graph, and a result is finally processed.
Fourthly, the module/module set management module verifies the relation between the module sets to ensure that no cyclic operation exists in the process, and corresponding process processing is carried out after the verification is passed; when the flow is changed, the XML configuration file or the database configuration table is changed, and the verification is carried out again; the check comprises the check of the service data processing flow and the check of the trigger branch processing.
After the verification is passed, the flow processing engine module carries out parallel processing on the business flow, the business flow processing adopts a mode of carrying out parallel processing after calculating branch links, when branches need to be processed in parallel, branch tasks are put into a thread pool, and if the current thread is idle, the processing can be carried out; if no idle thread exists in the current thread pool, the branch task is firstly put into a task queue and is processed after a subsequent thread is idle; firstly, designing a flow processing interface definition of a flow processing engine module:
(1) the main process processing interface is defined as follows:
int ProcessData(const int& moduleSetLineNum, const short& mainSubModuleType, void* inContext, void* OutContext);
wherein the parameters have the following meanings:
moduleSetLineNum: the line number of the branch flow being processed;
mainSubModuleType: unique identification of the module set;
inContext: inputting context data;
OutContext: the context data is output.
(2) The branch flow processing interface is defined as follows:
int ProcessAutoProcessPipeLineBranch(XXModuleSetInfo firstModuleSetInfo, void* inContext);
wherein the parameters have the following meanings:
firstModuleSetInfo: first module set information;
inContext: context data is input.
Sixthly, the module/module set management module adjusts the relationship among the module sets according to the needs, processes the re-service scene, rearranges the relationship among the module sets according to the service needs, and inserts a new processing flow into the current service flow for data processing; and after the rearrangement is finished, the fourth step row check is required to be executed, and the subsequent data processing can be carried out after the check is passed.
In substep 2 of the first step, the module/module group management module executes the following operations to the directed acyclic graph checking flow:
the first step, delete all vertices with degree < =1 and associated edges, and subtract 1 from the degrees of other vertices associated with the deleted edge.
Second, re-search queues the vertices with degree 1.
Thirdly, judging whether a vertex with the degree of 1 exists in the queue, if so, taking the vertex with the degree of 1 out of the queue and returning to the first step; otherwise, the remaining vertexes are stored in the queue.
Fourthly, judging whether a vertex with the undeleted degree < =1 exists or not, and if so, recording that the current vertex is a cyclic graph; otherwise, no cyclic graph exists; the routine is ended.
In substep 2 of the first step, the module/module set management module performs the following operations on the trigger event checking process in the module set:
(1) firstly, traversing the module set, and searching all the module sets with triggers.
(2) Judging whether a module set with a trigger event exists or not, and if so, acquiring a module set list with a trigger; otherwise the procedure ends directly.
(3) And judging whether a module set exists in the module set list, if so, executing the step (4), if not, recording all results, and ending the program.
(4) And reading the trigger event list of the current module set.
(5) Judging whether a trigger event exists in the module set trigger event list or not, if so, executing the step (6), and if not, returning to the step (3) to obtain the module set with the trigger event.
(6) And judging whether the taken trigger event stores the trigger event of the next module set to respond to the trigger event of the current module set, if so, returning to the step (5), and if not, returning to the step (3) after recording an error log.
The invention is based on C + + 11 standard modularization development, can be applied to the flow based on pipeline processing and branch processing. The algorithm firstly designs a dynamic re-arranging frame, then carries out standardized design on each module, encapsulates the module in a component module through a uniform interface, can carry out component declaration through database registration or XML configuration files, and finally achieves the aim of finishing data processing without modifying codes. The algorithm can conveniently change the processing flow, flexibly cope with scenes that different types of data need different processing requirements, and can be conveniently applied to the technical fields of traditional software development and the Internet. The steps of algorithm implementation and the conditions required for each step of the steps, and the relationship between the steps are described below.
The beneficial effects produced by the invention are as follows: the DAG-based dynamic rearrangement method is realized, the encoding complexity is obviously reduced, the working efficiency is improved, and the debugging and deployment cost is saved in the actual specific project application. Because the multi-branch parallel processing is adopted, the data processing efficiency is also obviously improved.
Drawings
FIG. 1 is an overall block diagram of the present invention;
FIG. 2 is a flow chart of checking a directed acyclic graph according to the present invention;
FIG. 3 is a flow chart of the trigger interface verification of the present invention;
FIG. 4 is a flow chart of the business process of the present invention;
fig. 5 is a flow chart of the service processing according to the embodiment of the present invention.
Detailed Description
The invention is further illustrated by the following figures and examples.
First, the concept of the self-defined term in the present invention is briefly described as follows:
a module: the module is a module which processes specified types of data singly, and the module is a minimum complete data analysis unit.
A module set: a module set is a collection of modules that process a type of data.
Module set connection relation: the module set connection relation is used for describing the connection sequence between the modules.
The main type of the module: the main type of the module is the unique number of the module set, the uniqueness of one module set is identified, and the identification is one byte.
Module subtypes: the subtype of a module primarily identifies a module under a set of modules, identified as a byte.
Unique identification of the module: the unique identifier of the module is composed of a module set identifier and a module identifier, specifically, 256 times the module set identifier plus the module identifier.
Directed Acyclic Graph (DAG) DAG, Chinese is directed acyclic graph. "directed" means having a direction, which is exactly the same direction, and "acyclic" means that it is not closed-loop, and the DAG finally constitutes a topological tree structure.
The dynamic rearrangement method is divided into two steps, the first step is the construction and function realization of the dynamic rearrangement framework, and the second step is the processing of realizing the business process based on the constructed framework.
As shown in fig. 1, the overall framework is mainly divided into three layers, a display layer, a service processing layer and a data persistence layer. The data persistence layer records module sets, module/module set connection relations and processes data information mainly through a database and a storage frame; the business processing layer mainly comprises a flow processing engine module and a module/module set management module, wherein the flow processing engine module processes data to be analyzed in parallel by a plurality of branches and sends final processing results to a plurality of interface layers for displaying; the module/module set management module mainly realizes the definition and realization of the module, the registration interface, the de-registration interface and the module set connection relation interface of the module set; the module/module set management module also realizes the verification of the service processing flow, and the main check comprises checking whether the designed flow is an effective directed acyclic graph or not, ensuring that the processing flow has no dead cycle and can ensure that the program is stopped and executed, and the check also comprises checking the module set triggering events, ensuring that the module set triggering events have the response of the subsequent module set, and ensuring that the branch processing can be realized; the display layer completes the display of the current processing progress, dynamically registers and unregisters the existing module/module set, and dynamically arranges the data flow and other interface operations through the interface provided by the flow processing engine module and the module/module set management module.
Based on the framework designed in the step one, dynamic rearrangement can be completed, and the following flow is realized according to a specific scene, as shown in fig. 4:
1. defining business processing flow and splitting the business processing flow into a plurality of functional module sets according to functions
Before the design of the business process processing, the business process is divided into a plurality of sub-processes according to functions, and each sub-process is designed into a high-cohesion low-coupling module set and used for node operation in the dynamic rearrangement process. This step is a prerequisite for all other subsequent processing, and needs to be processed first after understanding the service.
2. And splitting the business flow after understanding the business, modeling each split sub-flow into a module set, determining whether the sub-flow needs to be split into modules with smaller granularity according to the complexity of the module set, and finally forming the flow by the module set consisting of a plurality of modules. The module set needs to design and process a service data interface for transmitting data among the module sets, and the module set also needs to define an event trigger interface for branch processing.
For a module set, the module set can be split into a plurality of modules according to different dimensions, and different processing modules with the same dimension are arranged among the modules. The splitting granularity of the module set directly determines the execution efficiency of the dynamic reconfigurable method, so the module set needs to be designed according to the business requirements.
Expandability is fully considered in the design of the module set, a factory mode is adopted for design when a new module set and a module are designed, and codes can be changed as little as possible.
The module design adopts a single-case mode, so that the backup in the system is reduced, and the service performance of the equipment is improved.
The module set and the modules need to inherit the unified interface, and the module set and the modules are convenient to manage and call. The specifically defined interfaces are as follows:
(1) existing modules/module sets are registered into the system, and the interfaces are defined as follows:
int RegistProtocolModule2Manager(const unsigned short& mainSubModuleType, void* pProtocolModuleInstance);
wherein the interface parameters are defined as follows:
mainSubModuleType: the master-slave type is used for uniquely identifying one module/module set;
pProtocolModuleInstance: module/module set handling classes.
(2) Unregisters unneeded modules/module sets, and the interface is defined as follows:
int UnrRegistProtocolModule2Manager(const unsigned short& mainSubModuleType);
wherein the interface parameters are defined as follows:
mainSubModuleType: the master sub-type is used to uniquely identify a module/set of modules.
The data transmission among the module sets is realized through the module set data processing interface, and the module set branch processing is realized through the trigger interface. The module set interface is used for transmitting the specification of service data in the service flow processing process, and ensures that a lower module can perform subsequent processing according to the processing result of an upper module after the upper module set finishes processing the data; the transmission data among the modules is defined as context data, and the processing result of the upper module and the data required by the lower processing are stored in the context data; the trigger interface is used for branch processing between the module sets, and when a branch flow exists between the upper module set and the lower module set, the upper module can trigger a corresponding event to determine to execute a lower module set responding to a trigger event after finishing processing data to obtain a result; the implementation mode of the trigger interface is different according to the selected language, and the implementation mode is different, but the general idea is not changed.
The module/module set management module is used for managing an interface of data processing and an interface of a trigger, and is defined as follows:
(1) the data processing interface is defined as follows:
virtual int ProcessData(const void* lpInContext, void* lpOutContext)；
wherein the interface parameters are defined as follows:
pIContext: context data of the superior module/module set specifically includes a processing result of the superior module/module set and data required by the current module/module set;
lpinContext: the context data of the subordinate modules/module sets specifically includes processing results of the current modules/module sets and data required by the subordinate modules/module sets.
(2) A trigger interface, the interface being defined as follows:
virtual int TriggerEvent(const int eventType, const void* lpOutContext);
wherein the interface parameters are defined as follows:
eventType: the event type is used for setting branch events needing to be triggered;
lpOutContext: the context data of the subordinate modules/module sets specifically includes processing results of the current modules/module sets and data required by the subordinate modules/module sets.
3. Modeling the relation between the module sets according to the scene requirements, and storing the relation in a relation coefficient database or an XML file, wherein the step corresponds to the module/module set management module to record relevant configuration information for the process processing engine module to use when processing data, and can also meet the requirement on data persistence.
And modeling the module, the module set and the module set relation through an XML file or a database configuration table, and forming a DAG model for processing the business process after the modeling is finished. When the flow is changed, the XML configuration file or the database configuration table is changed, and the processing can be carried out according to the new flow after the verification is carried out again. Modules/module sets can be constructed according to database configuration using directed acyclic graphs in graph theory. Each vertex of the directed acyclic graph is a module set (a set of modules), data are circulated in the module set of the directed acyclic graph, and a result is finally processed.
4. And checking the relation among the module sets to ensure that no cyclic operation exists in the process, wherein the step is realized by the module/module set management module.
(1) After the relation between the module sets is determined, the flow needs to be verified, and after the verification is passed, corresponding flow processing can be carried out. When the flow changes, the XML configuration file or the database configuration table is changed, and the verification is carried out again. The check comprises the check of the service data processing flow and the check of the trigger branch processing. The business process needs to ensure that no ring link exists between the module sets; the trigger branch processing verification mainly ensures that each trigger in the module set has a corresponding module set to perform subsequent data processing. The module set verification is crucial in the whole modeling process, the validity of the whole process can be ensured only after the verification is passed, and the module set verification is also a precondition for data processing.
In this step, to avoid the occurrence of a dead cycle, the constructed directed acyclic graph needs to be checked to avoid the occurrence of a loop. The basis is as follows: if the modeling data has a loop, a subgraph must exist, namely a loop. Degree of all vertices in the loop > = 2. As shown in fig. 2, the specific steps are as follows:
the first step, delete all vertices with degree < =1 and associated edges, and subtract 1 from the degrees of other vertices associated with the deleted edge.
Second, re-search queues the vertices with degree 1.
Thirdly, judging whether a vertex with the degree of 1 exists in the queue, if so, taking the vertex with the degree of 1 out of the queue and returning to the first step; otherwise, the remaining vertexes are stored in the queue.
Fourthly, judging whether a vertex with the undeleted degree < =1 exists or not, and if so, recording that the current vertex is a cyclic graph; otherwise, no cyclic graph exists; the routine is ended.
The analysis of the checking method for directed acyclic graphs is as follows: if we assume m related edges, there are n vertices:
(i) if m > = n, then the loop can be judged to exist directly according to the knowledge of graph theory.
(it turns out that if there are no loops, the graph must be k trees k > = 1. depending on the nature of the tree, the number of edges m = n-k.k > =1, so m < n).
(ii) If m < n, it is operated once (at most n times) per vertex with degree 0, or once (at most m times) per vertex with degree 1, according to the above algorithm. The total number of these two operations will not exceed m + n. Since m < n, the algorithm complexity is O (n).
According to the method, V represents the number of vertexes, E represents the number of edges, the complexity of the algorithm is more than O (V), when the degrees of all vertexes are counted at the initial moment, the complexity is (V + E), and even if E > = V in the subsequent cycle, the complexity of the algorithm can only be O (V + E). Secondly, at each cycle, deleting a vertex with degree 1, it is necessary to subtract 1 from the degree of the point connected to this vertex and delete the vertex in the list, where the complexity of the vertex in the lookup list is the length of the vertex list, only to ensure that when the degree of the vertex is 1, there is only one vertex in the vertex list, and then the worst complexity is o (ev).
(2) For a module set needing branch processing, a trigger interface needs to be inherited, and the trigger interface can determine the branch of a lower module set to continue processing according to a trigger condition after data processing is finished, so that a subsequent processing module set defined in the trigger interface needs to exist to ensure subsequent processing. The purpose of the trigger interface verification is to prompt a user if a subsequent processing set does not exist, and the current flow configuration problem needs to be modified and finished.
As shown in fig. 3, the flow of checking the trigger interface is as follows:
(1) firstly, traversing the module set, and searching all the module sets with triggers;
(2) judging whether a module set with a trigger event exists or not, and if so, acquiring a module set list with a trigger; otherwise, the program is directly ended;
(3) judging whether a module set exists in the module set list, if so, executing the step (4), if not, recording all results, and ending the program;
(4) reading a current module set trigger event list;
(5) judging whether a trigger event exists in the trigger event list of the module set, if so, executing the step (6), and if not, returning to the step (3) to obtain the module set with the trigger event;
(6) and judging whether the taken trigger event stores the trigger event of the next module set to respond to the trigger event of the current module set, if so, returning to the step (5), and if not, returning to the step (3) after recording an error log.
5. After the verification is passed, the data can be processed. The service processing adopts a parallel processing mode after calculating the branch link, so that the processing efficiency can be ensured, and the resources are saved. The parallel processing adopts a thread pool implementation mode, when branches need to be processed in parallel, the branch tasks are put into the thread pool, and if the current thread is idle, the processing can be carried out. If no idle thread exists in the current thread pool, the branch task is firstly put into a task queue and is processed after the subsequent thread is idle.
The following interfaces must first be designed in this step:
the process processing interface is used for processing data in each module set of the process, and the specific interface is defined as follows:
(1) main process processing interface
int ProcessData(const int& moduleSetLineNum, const short& mainSubModuleType, void* inContext, void* OutContext);
Wherein the parameter meanings are as follows:
moduleSetLineNum: the line number of the branch flow being processed;
mainSubModuleType: unique identification of the module set;
inContext: inputting context data;
OutContext: the context data is output.
(2) Branch processing flow interface
int ProcessAutoProcessPipeLineBranch(XXModuleSetInfo firstModuleSetInfo, void* inContext);
Wherein the parameter meanings are as follows:
firstModuleSetInfo: first module set information;
inContext: context data is input.
6. The module/module set management module adjusts the relationship between the module sets according to the needs, processes the re-service scene, rearranges the relationship between the module sets according to the service needs, inserts a new processing flow into the current service flow and is used for data processing; and after the rearrangement is finished, the fourth step row check is required to be executed, and the subsequent data processing can be carried out after the check is passed.
The method completes data processing through the six steps of processes on the basis of the main frame, and can dynamically adjust the module set to meet the requirements of the business process if the processing result cannot meet the business requirements after being analyzed.
The method comprises the steps of sorting business requirements, defining business processes, and designing a module/module set according to the business processes; modeling the relation between the module sets according to scene requirements, putting all configuration information into a warehouse, checking DAG validity, checking a module set trigger after the checking is successful, processing flow branches of the service after the checking is completed, analyzing the processed results after gathering, and if rearrangement is needed, modeling the relation of the module sets again and then checking and executing.
The invention relates to the realization of a DAG-based dynamic rearrangement method, which can be applied to scenes with frequently and dynamically changed flow processing according to design. The method overcomes the process that the code needs to be rewritten for development and debugging after the process is changed, directly modifies the configuration dynamic change processing process, can be widely applied to software development and internet technology, and can be used as a universal algorithm for process processing to quickly and efficiently complete the required task requirements.
The invention can also be applied to the scenes of complex processing flow and branches or multiple flows, and can realize efficient parallel processing on different flows and branches, thereby improving the processing efficiency of tasks and improving the use experience.
Example (b): the detailed items are as follows:
1. the project processes data in a certain field, defines a business processing flow and is divided into a plurality of functional module sets according to functions. The business processing flow is totally split into two main business flows, and the first business flow can trigger different branches according to different line identifications to process subsequent businesses, and the specific split flow and module set are shown in fig. 5.
2. The method comprises the steps of modeling a function module set, splitting the function module set into sub-modules, defining a function module set interface for data transmission among the modules, and defining an event trigger interface for branch processing.
Taking a module set as an example: the module design adopts a single-case mode, so that the backup in the system is reduced, and the service performance of the equipment is improved; the expandability is fully considered in the design of the module set, and codes are changed as little as possible when a new module set and sub-modules are added, so that the design is carried out by adopting an engineering mode; for a module set needing branch processing, a trigger interface needs to be inherited, and the trigger interface can determine the branch of a lower module set to continue processing according to a trigger condition after data processing is finished, so that a subsequent processing module set defined in the trigger interface needs to exist to ensure subsequent processing. And the data processing interface and the trigger interface are realized through the two interfaces provided in the step two, so that the service processing and the branch selection are completed.
3. And modeling the relation between the module sets according to the scene requirements, and storing the relation into a relation coefficient database or an XML file.
(1) The module information table mainly defines three parts, namely a module set identifier, a module type and a module name, and the specific definition is shown in table 1:
TABLE 1 Module information Table
(2) The module set information table mainly records three parts, namely a module set type, a business process number to which the module set belongs and a module set name, and the specific definition is shown in table 2:
TABLE 2 Module set information Table
(3) The module set connection information table mainly defines identifiers of front and rear module sets of a line where the module set is located, is used for uniquely marking the position of the module set in the whole business process, and is specifically defined as shown in table 3:
table 3 module set connection information table
4. And constructing by utilizing the directed acyclic graph in the graph theory after the database is configured. And (4) in the construction process, verifying the service flow by using a method for verifying the relation among the module sets in the step (4), ensuring no cycle operation in the flow, taking each vertex of the directed acyclic graph as a set of each module, circulating data in the module set of the directed acyclic graph, and finally processing a result.
5. And (4) executing the parallel processing of the service flow, wherein the main flow processing interface and the branch processing interface in the step (5) are realized by the part, so that the parallel processing of the service is realized.
6. And judging whether the requirements are met according to the results, and if the requirements are not met, re-entering the step 3 to re-rearrange the cascade relation among the module sets.
Through the above example, the rearrangement requirement and branch processing requirement of the business process can be completely met, and the dynamic configuration of the process is realized. The method is applied to a certain system, and the specific application mode is as follows:
(1) building a database according to the script (SQL SERVER 2012 is used for testing at present);
(2) the number of the modules is 9, and the connection relation is tested under the condition that two branches are arranged in the figure 5;
(3) changing or creating a TXT file, and writing DVB or other characters on the first line in the TXT file;
(4) the click starts to test, and test results in different scenes are as follows:
<1> test scenario 1 is as follows: inputting DVB for testing, wherein the testing time is 50.198 seconds;
<2> test scenario 2 is as follows: inputting other texts for testing, wherein the test result is 70.914 seconds;
<3> test scenario 3 is as follows: the conventional test was conducted using DVB as an example, and the test result was 153.923 seconds.
The test of the test scene 1 and the test scene 2 verifies that the algorithm can process multi-path data in parallel and can walk different branches according to different processing results. The comparison of test scenario 2 and test scenario 3 shows that the processing efficiency is improved by 1.17 times under the new test. Therefore, the method has the advantage that the data processing is obviously improved in parallel processing.
For the modification process, the database needs to be reconfigured, the code logic needs to be changed, and the labor cost is saved by 2.13 times by calculation for the same adjustment.
In conclusion, in the specific application realized by the dynamic rearrangement method based on the DAG, the programming can be simpler, the processing efficiency is higher, and the design expectation of the method is met.
Claims (3)
1. A DAG-based dynamic rearrangeable method, comprising the steps of:
construction and function realization of dynamic reconfigurable frame
The dynamic rearrangeable integral framework is divided into three layers: the system comprises a display layer, a service processing layer and a data persistence layer; the display layer comprises an interface module used for displaying the current processing progress and interface operation; the business processing layer comprises a flow processing engine module and a module/module set management module, wherein the flow processing engine module is used for processing the data to be analyzed in parallel and delivering the final processing result to the interface layer for display; the data persistence layer comprises a data storage module and is used for recording the connection relation of the module/module set and the data analysis result;
the functional implementation of the dynamic re-arrangeable framework comprises the following steps:
1. the module/module set management module realizes the definition and realization of the module, the registration interface, the de-registration interface and the module set connection relation interface of the module set;
2. the module/module management module realizes the verification of the business processing flow; the verification comprises the following steps:
(1) checking whether the business processing flow is an effective directed acyclic graph or not;
(2) checking whether the triggering events in the module set meet the triggering requirements;
3. the flow processing engine module realizes the parallel processing of the business flow; the parallel processing comprises the following steps:
(1) parallel processing of a plurality of flows;
(2) different branch processing of the same flow;
second, business process processing based on framework
And (3) completing dynamic rearrangement according to a specific scene based on the frame built in the first step, wherein the specific steps are as follows:
defining a business processing flow, dividing the business processing flow into a plurality of sub-flows according to functions, wherein each sub-flow is designed into a high-cohesion low-coupling module set and is used for node operation in a dynamic rearrangement process;
secondly, after the business process is fully understood, a module set is modeled according to the sub-process functions split in the first step, whether the module set needs to be split into modules with smaller granularity is determined according to the complexity, and the final process consists of the module set formed by integrating a plurality of modules; firstly, respectively designing a module set and an interface of a module/module set management module, wherein a module set service data interface is used for transmitting data among the module sets, and a module set event trigger interface is used for branch processing among the module sets; the module/module set management module data processing interface is used for managing data processing, and the module/module set management module event trigger interface is used for business process branch processing;
(1) Module/Module set management Module interface definition
a. Existing modules/module sets are registered into the system, and the interfaces are defined as follows:
int RegistProtocolModule2Manager(const unsigned short& mainSubModuleType, void* pProtocolModuleInstance);
wherein the interface parameters are defined as follows:
mainSubModuleType: the master-slave type is used for uniquely identifying one module/module set;
pProtocolModuleInstance: module/module set processing class;
b. unregisters unneeded modules/module sets, and the interface is defined as follows:
int UnrRegistProtocolModule2Manager(const unsigned short& mainSubModuleType);
wherein the interface parameters are defined as follows:
mainSubModuleType: the master-slave type is used for uniquely identifying one module/module set;
(2) the module set service data interface is defined as follows:
virtual int ProcessData(const void* lpInContext, void* lpOutContext)；
wherein the interface parameters are defined as follows:
lpinContext: context data of the superior module/module set specifically includes a processing result of the superior module/module set and data required by the current module/module set;
lpinContext: context data of the subordinate module/module set specifically comprises a processing result of the current module/module set and data required by the subordinate module/module set;
(3) the module set event trigger interface is defined as follows:
virtual int TriggerEvent(const int eventType, const void* lpOutContext);
wherein the interface parameters are defined as follows:
eventType: the event type is used for setting branch events needing to be triggered;
lpOutContext: context data of the subordinate module/module set specifically comprises a processing result of the current module/module set and data required by the subordinate module/module set;
thirdly, the process processing engine module models the relationship between the module sets according to the scene requirements and stores the relationship into a relationship coefficient database or an XML file; modeling the module, the module set and the module set relation through an XML file or a database configuration table, and forming a DAG model for processing a business process after the modeling is finished; when the flow is changed, processing according to a new flow after the XML configuration file or the database configuration table is changed and the verification is carried out again; constructing a module/module set by utilizing a directed acyclic graph in a graph theory after the module/module set is configured according to a database; each vertex of the directed acyclic graph is a module set, data are circulated in the module set of the directed acyclic graph, and a result is finally processed;
fourthly, the module/module set management module verifies the relation between the module sets to ensure that no cyclic operation exists in the process, and corresponding process processing is carried out after the verification is passed; when the flow is changed, the XML configuration file or the database configuration table is changed, and the verification is carried out again; the verification comprises the verification of the service data processing flow and the verification of the trigger branch processing;
after the verification is passed, the flow processing engine module carries out parallel processing on the business flow, the business flow processing adopts a mode of carrying out parallel processing after calculating branch links, when branches need to be processed in parallel, branch tasks are put into a thread pool, and if the current thread is idle, the processing can be carried out; if no idle thread exists in the current thread pool, the branch task is firstly put into a task queue and is processed after a subsequent thread is idle; firstly, designing a flow processing interface definition of a flow processing engine module:
(1) the main process processing interface is defined as follows:
int ProcessData(const int& moduleSetLineNum, const short& mainSubModuleType, void* inContext, void* OutContext);
wherein the parameters have the following meanings:
moduleSetLineNum: the line number of the branch flow being processed;
mainSubModuleType: unique identification of the module set;
inContext: inputting context data;
OutContext: outputting the context data;
(2) the branch flow processing interface is defined as follows:
int ProcessAutoProcessPipeLineBranch(XXModuleSetInfo firstModuleSetInfo, void* inContext);
wherein the parameters have the following meanings:
firstModuleSetInfo: first module set information;
inContext: inputting context data;
sixthly, the module/module set management module adjusts the relationship among the module sets according to the needs, processes the re-service scene, rearranges the relationship among the module sets according to the service needs, and inserts a new processing flow into the current service flow for data processing; and after the rearrangement is finished, the fourth step row check is required to be executed, and the subsequent data processing can be carried out after the check is passed.
2. The DAG-based dynamic rearrange method of claim 1, wherein in sub-step 2 of step one, the module/module set management module performs the following operations on the directed acyclic graph checking flow:
deleting the vertexes and related edges of all degrees < =1, and subtracting 1 from the degrees of other vertexes related to the deleted edges;
secondly, re-searching and arranging the top point with the degree of 1 into a queue;
thirdly, judging whether a vertex with the degree of 1 exists in the queue, if so, taking the vertex with the degree of 1 out of the queue and returning to the first step; otherwise, storing the rest vertexes into a queue;
fourthly, judging whether a vertex with the undeleted degree < =1 exists or not, and if so, recording that the current vertex is a cyclic graph; otherwise, no cyclic graph exists; the routine is ended.
3. The DAG-based dynamic rearrange method of claim 1, wherein in sub-step 2 of step one, the module/module set management module performs the following operations on the trigger event check flow in the module set:
(1) firstly, traversing the module set, and searching all the module sets with triggers;
(2) judging whether a module set with a trigger event exists or not, and if so, acquiring a module set list with a trigger; otherwise, the program is directly ended;
(3) judging whether a module set exists in the module set list, if so, executing the step (4), if not, recording all results, and ending the program;
(4) reading a current module set trigger event list;
(5) judging whether a trigger event exists in the trigger event list of the module set, if so, executing the step (6), and if not, returning to the step (3) to obtain the module set with the trigger event;
(6) and judging whether the taken trigger event stores the trigger event of the next module set to respond to the trigger event of the current module set, if so, returning to the step (5), and if not, returning to the step (3) after recording an error log.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011386785.2A CN112463143A (en) | 2020-12-02 | 2020-12-02 | DAG-based dynamic rearrangement method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011386785.2A CN112463143A (en) | 2020-12-02 | 2020-12-02 | DAG-based dynamic rearrangement method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112463143A true CN112463143A (en) | 2021-03-09 |
Family
ID=74805203
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011386785.2A Withdrawn CN112463143A (en) | 2020-12-02 | 2020-12-02 | DAG-based dynamic rearrangement method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112463143A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113434220A (en) * | 2021-07-15 | 2021-09-24 | 上海商汤科技开发有限公司 | Method, device and equipment for constructing workflow and computer storage medium |
| CN113792292A (en) * | 2021-09-14 | 2021-12-14 | 山石网科通信技术股份有限公司 | Response method and device of security script, storage medium and processor |
| CN114022101A (en) * | 2021-10-12 | 2022-02-08 | 东风柳州汽车有限公司 | Flow processing method and device |
| CN115857907A (en) * | 2023-02-06 | 2023-03-28 | 卓望数码技术（深圳）有限公司 | Business flow dynamic assembly system and method |
-
2020
- 2020-12-02 CN CN202011386785.2A patent/CN112463143A/en not_active Withdrawn
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113434220A (en) * | 2021-07-15 | 2021-09-24 | 上海商汤科技开发有限公司 | Method, device and equipment for constructing workflow and computer storage medium |
| CN113792292A (en) * | 2021-09-14 | 2021-12-14 | 山石网科通信技术股份有限公司 | Response method and device of security script, storage medium and processor |
| CN114022101A (en) * | 2021-10-12 | 2022-02-08 | 东风柳州汽车有限公司 | Flow processing method and device |
| CN115857907A (en) * | 2023-02-06 | 2023-03-28 | 卓望数码技术（深圳）有限公司 | Business flow dynamic assembly system and method |
| CN115857907B (en) * | 2023-02-06 | 2023-05-23 | 卓望数码技术（深圳）有限公司 | Business flow dynamic assembly system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10162612B2 (en) | Method and apparatus for inventory analysis | |
| CN112463143A (en) | DAG-based dynamic rearrangement method | |
| US9984059B2 (en) | Evaluating rules applied to data | |
| US5862327A (en) | Activity based long-lived transaction system | |
| US20090144703A1 (en) | Method and system for versioning a software system | |
| US10514898B2 (en) | Method and system to develop, deploy, test, and manage platform-independent software | |
| Remenska et al. | Using model checking to analyze the system behavior of the LHC production grid | |
| US20240143285A1 (en) | Architecture discovery | |
| US8620628B2 (en) | Run-time tunable sample time in programming or modeling environments | |
| Sag et al. | Measuring COSMIC software size from functional execution traces of Java business applications | |
| US7444618B2 (en) | Automatic generation of batch programs with identification, insertion of invariables, declarative statements and variables with the use of place-marks | |
| CN114818565A (en) | Simulation environment management platform, method, equipment and medium based on python | |
| Lemcke et al. | Composing web-service-like abstract state machines (ASMs) | |
| Wojszczyk et al. | Performance analysis of extracting object structure from source code | |
| CN116088809A (en) | Method and system for extensible custom operators | |
| Ciobanu et al. | From TiMo to Event-B: Event-driven timed mobility | |
| Alt et al. | Using high-level petri nets for hierarchical grid workflows | |
| Waszkowski | Reference Low-code Development Platform Architecture: Aurea BPM | |
| CN114207594B (en) | Static analysis and runtime analysis of computer program system | |
| Aoki et al. | An axiomatic formalization of UML models | |
| US20240281737A1 (en) | Data compilation and execution device and data compilation and execution method | |
| Lowe | View abstraction for systems with component identities | |
| Pogrebnoy | Constructing graph models for software system development and analysis | |
| de Boer et al. | Axiomatic characterization of trace reachability for concurrent objects | |
| CN108509330B (en) | Data processing method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210309 |
|
| WW01 | Invention patent application withdrawn after publication |