CN116028038B - Visual pipeline arrangement method based on DAG chart and related components - Google Patents
Visual pipeline arrangement method based on DAG chart and related components Download PDFInfo
- Publication number
- CN116028038B CN116028038B CN202310311926.1A CN202310311926A CN116028038B CN 116028038 B CN116028038 B CN 116028038B CN 202310311926 A CN202310311926 A CN 202310311926A CN 116028038 B CN116028038 B CN 116028038B
- Authority
- CN
- China
- Prior art keywords
- pipeline
- text
- dag
- mode
- plug
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The invention discloses a visual assembly line arrangement method based on a DAG chart and related components, wherein the method comprises the following steps: initializing an arranging assembly of a front-end pipeline, loading plug-in configuration of the pipeline, and generating a front-end form according to the plug-in configuration; loading a pipeline template in a YAML format, and analyzing the pipeline template; acquiring a current arrangement mode of a front-end assembly line; dynamically rendering the DAG chart and the front-end form by combining the plug-in configuration and the analysis result to acquire form change contents; generating a first constraint condition of the form change content through plug-in configuration, and checking; generating corresponding YAML format change content from the form change content and the verification result; and filling the generated YAML format change content into a front-end text field, and completing pipeline arrangement operation according to the filled front-end text field. The invention can automatically generate the pipeline configuration file or convert the pipeline configuration file in real time while realizing the visual arrangement of the pipeline.
Description
Technical Field
The invention relates to the technical field of computer software, in particular to a visual pipeline arrangement method based on a DAG chart and related components.
Background
Unlike an industrial pipeline, a software pipeline (or simply pipeline) refers to a general term for developing services or functions in the field of computers, especially in the field of Development and Operations, which is a group of processes, methods and systems, and a pipeline is an event or a group of Operations connected together in a certain order, which is a series of events or Operations that can be executed. The current visual pipeline arrangement technology is mainly divided into two types: a text field is a pure front-end interface, which provides description and writing of task groups, tasks and other contents, and the contents have a certain format and rules for describing various configurations of a pipeline. It follows that the user needs considerable time to learn the format and rules of the pipeline configuration file, such as Jenkins (open source software project, a method of sequentially executing Jenkins jobs in a defined manner) pipeline, which belongs to the pipeline arrangement mode of plain text fields. And secondly, complete visual editing of the front-end interface, wherein the method comprises task group generation, task group form editing, task group form deletion, task form generation, task form editing, task form deletion and the like, and pipeline configuration data are generated for the rear-end interface after some front-end interactive operations are completed.
However, there are certain drawbacks to both of the above-described visual pipeline orchestration techniques. For example, the front-end form cannot be generated according to the plug-in configuration, and most of the visual pipeline arrangement technologies on the market cannot automatically generate the front-end form according to the plug-in configuration, so that the workload of the DevOps related personnel is increased, and the labor and time cost are increased. In addition, the prior art cannot convert the pipeline configuration file in the YAML format (a format for expressing data serialization) into the visual front-end interface in real time, and meanwhile, the form content in the visual front-end interface cannot generate the pipeline configuration file in the YAML format in real time after being changed. For the Devops related personnel, the method is expected to directly edit the pipeline configuration file in the YAML format and directly generate the pipeline configuration file in the YAML format through the front-end visual interface, but the current pipeline arrangement technology or method on the market cannot simultaneously have the capability of arranging two pipeline contexts and converting in real time.
Disclosure of Invention
The embodiment of the invention provides a visual assembly line arrangement method, a visual assembly line arrangement device, computer equipment and a storage medium based on a DAG chart, which aim to automatically generate an assembly line configuration file or convert the assembly line configuration file in real time while realizing the visual assembly line arrangement.
In a first aspect, an embodiment of the present invention provides a method for arranging a visual pipeline based on a DAG chart, including:
initializing an arranging assembly of a front-end pipeline, loading plug-in configuration of the pipeline, and generating a front-end form according to the plug-in configuration; wherein the orchestration component comprises a code editor and a draggable DAG graph;
loading a pipeline template in a YAML format, and analyzing the pipeline template by utilizing a web worker thread to obtain a corresponding analysis result;
responding to the operation of a user on the front-end pipeline, and acquiring the current arrangement mode of the front-end pipeline; wherein the current programming mode comprises a graphical programming mode based on a DAG chart and a code editing mode based on a code editor;
when the current arrangement mode is a graphical arrangement mode, dynamically rendering the DAG chart and the front-end form by combining the plug-in configuration and the analysis result to obtain form changing contents;
generating a first constraint condition for the form changing content through the plug-in configuration, and checking the form changing content based on the first constraint condition to obtain a corresponding checking result;
The form changing content and the verification result are sent to the web worker thread, and the web worker thread generates corresponding YAML format changing content;
continuously acquiring a current arrangement mode of a front-end pipeline, and continuously generating corresponding YAML format change content if the current arrangement mode is still the graphical arrangement mode until the current arrangement mode is a code editing mode;
and when the current editing mode is a code editing mode, filling the generated YAML format change content into a front-end text field, and completing pipeline editing operation according to the filled front-end text field.
In a second aspect, an embodiment of the present invention provides a visual pipeline arrangement apparatus based on a DAG chart, including:
the initialization unit is used for initializing an arrangement assembly of the front-end pipeline and loading plug-in configuration of the pipeline and generating a front-end form according to the plug-in configuration; wherein the orchestration component comprises a code editor and a draggable DAG graph;
the loading analysis unit is used for loading the assembly line template in the YAML format, and analyzing the assembly line template by utilizing the web worker thread to obtain a corresponding analysis result;
The first mode acquisition unit is used for responding to the operation of a user on the front-end pipeline and acquiring the current arrangement mode of the front-end pipeline; wherein the current programming mode comprises a graphical programming mode based on a DAG chart and a code editing mode based on a code editor;
the dynamic rendering unit is used for dynamically rendering the DAG chart and the front-end form by combining the plug-in configuration and the analysis result when the current arrangement mode is a graphical arrangement mode so as to acquire form changing contents;
the constraint verification unit is used for generating a first constraint condition for the form changing content through the plug-in configuration, and verifying the form changing content based on the first constraint condition to obtain a corresponding verification result;
the change generation unit is used for sending the form change content and the verification result to the web worker thread, and generating corresponding YAML format change content by the web worker thread;
the second mode acquisition unit is used for continuously acquiring a current arrangement mode of the front-end pipeline, and continuously generating corresponding YAML format change content if the current arrangement mode is still the graphical arrangement mode until the current arrangement mode is a code editing mode;
And the filling arrangement unit is used for filling the generated YAML format change content into a front-end text field when the current arrangement mode is a code editing mode, and completing pipeline arrangement operation according to the filled front-end text field.
In a third aspect, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the DAG graph-based visual pipeline arrangement method according to the first aspect when the computer program is executed.
In a fourth aspect, an embodiment of the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a DAG graph-based visual pipeline arrangement method according to the first aspect.
The embodiment of the invention can realize the visual arrangement of the assembly line based on the draggable chart of the DAG and the dynamic rendering technology of the plug-in form, so that a developer can automatically generate the assembly line configuration file or automatically generate a visual interface according to the assembly line configuration file on the premise of not knowing the configuration file of the assembly line, thereby reducing the technical threshold of the developer and improving the interactive experience of the user. Meanwhile, the embodiment of the invention has the characteristics of reducing workload, improving development efficiency of developers, realizing visual assembly line arrangement, dynamic rendering of plug-in forms and the like in development work.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a method for arranging a visual pipeline based on a DAG chart according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of step S102 in a visual pipeline arrangement method based on a DAG chart according to an embodiment of the present invention;
fig. 3 is a schematic flow chart of step S104 in a visual pipeline arrangement method based on a DAG chart according to an embodiment of the present invention;
fig. 4 is a schematic flow chart of step S105 in a visual pipeline arrangement method based on a DAG chart according to an embodiment of the present invention;
fig. 5 is a schematic flow chart of step S106 in a visual pipeline arrangement method based on a DAG chart according to an embodiment of the present invention;
FIG. 6 is another flow chart of step S104 in a method for arranging a visual pipeline based on a DAG chart according to an embodiment of the present invention;
FIG. 7 is a schematic block diagram of a visual pipeline orchestration device based on a DAG graph according to an embodiment of the present invention;
FIG. 8 is a schematic block diagram of a loading parsing unit in a visual pipeline arrangement device based on a DAG chart according to an embodiment of the present invention;
FIG. 9 is a schematic block diagram of a dynamic rendering unit in a visual pipeline orchestration device based on DAG charts according to an embodiment of the present invention;
FIG. 10 is a schematic block diagram of a constraint checking unit in a visual pipeline orchestration device based on a DAG graph according to an embodiment of the present invention;
FIG. 11 is a schematic block diagram of a change generating unit in a visual pipeline arrangement device based on a DAG chart according to an embodiment of the present invention;
FIG. 12 is another schematic block diagram of a dynamic rendering unit in a DAG graph-based visual pipeline orchestration device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be understood that the terms "comprises" and "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
Referring to fig. 1, fig. 1 is a flow chart of a visual pipeline arrangement method based on a DAG chart according to an embodiment of the present invention, which specifically includes: steps S101 to S108.
S101, initializing an arrangement component of a front-end pipeline and loading plug-in configuration of the pipeline, and generating a front-end form according to the plug-in configuration; wherein the orchestration component comprises a code editor and a draggable DAG graph;
s102, loading a pipeline template in a YAML format, and analyzing the pipeline template by utilizing a web worker thread to obtain a corresponding analysis result;
s103, responding to the operation of a user on the front-end pipeline, and acquiring a current arrangement mode of the front-end pipeline; wherein the current programming mode comprises a graphical programming mode based on a DAG chart and a code editing mode based on a code editor;
s104, when the current arrangement mode is a graphical arrangement mode, dynamically rendering the DAG chart and the front-end form by combining the plug-in configuration and the analysis result to acquire form changing contents;
s105, generating a first constraint condition for the form change content through the plug-in configuration, and checking the form change content based on the first constraint condition to obtain a corresponding checking result;
s106, sending the form change content and the verification result to the web worker thread, and generating corresponding YAML format change content by the web worker thread;
S107, continuously acquiring a current arrangement mode of a front-end pipeline, and continuously generating corresponding YAML format change content if the current arrangement mode is still the graphical arrangement mode until the current arrangement mode is a code editing mode;
s108, when the current editing mode is a code editing mode, filling the generated YAML format change content into a front-end text field, and completing pipeline editing operation according to the filled front-end text field.
In this embodiment, the front-end pipeline orchestration component is first initialized, for example, the code editor and DAG graph are initialized, and corresponding plug-in configurations are loaded to build a front-end form for the front-end pipeline. Here, the plug-in configuration may include a pipelining template, so the front-end form may be built using the pipelining template as a form basis. The pipeline template is then loaded and parsed in YAML format by the Web worker thread, which provides a simple way for the Web content to run scripts in the background thread, which can perform tasks without disturbing the user interface. And then, confirming whether the current arrangement mode of the front-end pipeline is a graphical arrangement mode or a code editing mode according to user operation, confirming changed contents in the DAG chart and the front-end form according to plug-in configuration and analysis results when the current arrangement mode is the graphical arrangement mode, dynamically rendering the contents to render and display the form changed contents, generating the form changed contents into a YAML format, and realizing real-time conversion of the YAML format. And when the current programming mode is the code programming mode, filling YAML format change content generated in the graphical programming mode into the front-end text field to complete the visual programming operation of the front-end pipeline.
According to the embodiment, the visual arrangement of the assembly line can be realized based on the draggable DAG chart and the plug-in form dynamic rendering technology, so that a developer can automatically generate the assembly line configuration file or automatically generate a visual interface according to the assembly line configuration file on the premise that the developer does not know the configuration file of the assembly line, the technical threshold of the developer can be reduced, and meanwhile, the interactive experience of a user is improved. Meanwhile, the embodiment of the invention has the characteristics of reducing workload, improving development efficiency of developers, realizing visual assembly line arrangement, dynamic rendering of plug-in forms and the like in development work.
In one embodiment, as shown in fig. 2, the step S102 includes: steps S201 to S204.
S201, extracting a pipeline template text from a pipeline template in a YAML format according to a character string format;
s202, constructing the pipeline template text into a plurality of first syntax dependency graphs by adopting a syntax dependency analysis mode, acquiring node relations in the syntax dependency graphs, and then associating the plurality of first syntax dependency graphs according to the node relations to obtain a second syntax dependency graph;
s203, extracting a semantic characterization vector from the second syntactic dependency graph through a graph convolution neural network, and carrying out semantic error correction on the pipeline template text based on the semantic characterization vector;
S204, analyzing and sorting the pipeline template text through a preset format file template based on the semantic error correction result.
In this embodiment, when the web worker thread is used to parse the pipeline template in the YAML format, the text of the pipeline template is first extracted through the character string format, and then a syntactic dependency graph, that is, the first syntactic dependency graph, is constructed on the extracted text through syntactic dependency analysis. It will be appreciated that the first syntactic dependency constructed is at least one, as there is associated or unassociated text in the pipeline template text. And then, associating each first syntax dependency graph into a second syntax dependency graph according to the node relation, and extracting semantic characterization vectors from the second syntax dependency graph by adopting a graph convolution neural network so as to realize semantic error correction on the pipeline template text, and further converting the pipeline template text into other formats. That is, in this embodiment, before converting the format, error correction processing is performed on the pipeline template text in a semantic error correction manner, so that the conversion accuracy of the pipeline template text can be improved, and text semantic errors in a certain process during arrangement are avoided.
In a specific embodiment, when obtaining a node relationship in the syntax dependency graph, and then associating a plurality of the first syntax dependency graphs according to the node relationship to obtain a second syntax dependency graph, the method includes:
selecting commonly-contained first word nodes, such as subject nodes, predicate nodes, object nodes and the like, from each first syntax dependency graph, and establishing abstract connection for different syntax dependency graphs to which each first word node belongs; or, selecting a second word node which is not contained in other first syntax dependency graphs from each first syntax dependency graph, and establishing abstract connection for noun nodes and/or verb nodes in the first syntax dependency graph to which the second word node belongs.
In one embodiment, as shown in fig. 3, the step S104 includes: steps S301 to S304.
S301, converting the DAG chart and the front-end form into json format arrays;
s302, confirming a changed change array in the json format array by combining the plug-in configuration and the analysis result, and setting a label for the change array;
s303, acquiring the json format array with the tag by adopting a Web data interaction mode;
s304, analyzing the json format array with the label into elements, and inserting the elements into a rendering tree as result nodes so as to realize dynamic rendering.
Dynamic rendering refers to the process of generating images in real-time at runtime. This is different from static rendering, which refers to rendering all image elements in advance and storing them down when making an image, and directly showing them at run-time. Dynamic rendering is commonly used in game development and virtual reality applications because it is capable of exhibiting changes in three-dimensional environments and objects in real time. The process of dynamic rendering requires an efficient Graphics Processing Unit (GPU) to implement, requiring more computing power than static rendering. Dynamic rendering has the advantage that it can show changes in three-dimensional environments and objects in real time, more realistic and realistic, but requires high performance hardware and algorithms to support. Dynamic rendering essentially causes a web page to return a new page structure to the browser according to user behavior or server-side data changes without refreshing the browser. According to the method and the device for dynamically rendering the DAG chart and the front-end form, interactivity of a front-end pipeline can be enhanced, data quantity of data transmitted between a server and a client is reduced, performance and response speed of a front-end page are improved, network bandwidth and resources of the server are saved, user experience is improved, meanwhile, load of the server can be reduced, flow pressure is reduced, and accessibility of the front-end page is improved.
Specifically, the present embodiment first represents the DAG graph and the front-end form in the form of json format arrays, and then determines the changed change array according to the plug-in configuration and the analysis result. Here, the configuration of the inserting sheet and the analysis result can be converted into json format, so that whether the json format array is changed or not and the specific position of the change can be visually compared and confirmed. Then, a label can be set for the change array, and meanwhile, the json format array is parsed into element elements, and the elements are inserted into a rendering tree (Render tree) as nodes according to the label.
In a specific application scenario, the rendering tree may be pre-built, where the building process includes:
traversing each visible node starting from the root node of the DOM (Document Object Model ) tree; some nodes are completely invisible (e.g., script tags, meta tags, etc.) during traversal, as they are not reflected in the rendering results and therefore can be ignored; still other nodes are hidden by CSS (cascading style sheets) and therefore are also ignored in the rendering tree. For example, there is a span node that does not appear in the rendering tree because it has an explicit rule setting display: none attribute;
Finding a corresponding matching CSSOM rule for each visible node and applying the rules;
the visible node is transmitted, along with its content and the calculated pattern.
In one embodiment, as shown in FIG. 4, the plug-in configuration includes a pipeline job template;
the step S105 includes: steps S401 to S405.
S401, setting the pipeline operation template as a first constraint condition;
s402, extracting constraint text for the first constraint condition and extracting change text for the form change content;
s403, extracting constraint vectors of the constraint text through a graph convolution neural network, and extracting change vectors of the change text;
s404, sequentially inputting the constraint vector and the change vector into a mutual attention layer and a self attention layer to obtain a target vector;
s405, carrying out classification prediction on the target vector through a pre-trained full-connection layer, outputting to obtain a prediction probability, and taking the prediction probability as the verification result; the full-connection layer adopts preset check words to complete pre-training.
In this embodiment, when a first constraint condition is generated according to a plug-in configuration and the contents of a form change are verified, firstly, corresponding constraint text and change text are respectively extracted from the first constraint condition and the contents of the form change, then corresponding constraint vector and change vector are lifted from the first constraint condition and the contents of the form change through a graph convolutional neural network, then the constraint vector and the change vector are fused into a target vector through an attention mechanism, and probability prediction is performed on the target vector by using a full-connection layer, so that constraint verification on the contents of the form change is completed according to a probability prediction result.
Specifically, when a constraint vector and a change vector are fused into a target vector through an attention mechanism, the constraint vector and the change vector are input into a mutual attention layer to obtain a first intermediate vector, the first intermediate vector is input into a self-attention layer and output to obtain a second intermediate vector, and then the second intermediate vector and the change vector are input into a feature fusion layer and fused according to the following formula to obtain the target vector:
wherein, the liquid crystal display device comprises a liquid crystal display device,
representing the target vector +.>
And->
All represent trainable parameters, m represents the second middleInter-vector, x represents a change vector, σ is an activation function.
When the target vector is classified and predicted through the full connection layer, the prediction probability is calculated according to the following formula:
wherein, the liquid crystal display device comprises a liquid crystal display device,
、/>
、/>
and->
All mean trainable parameters, v means target vector, < ->
Representing the start probability of the target vector,/->
Representing the termination probability of the target vector.
In one embodiment, as shown in fig. 5, the step S106 includes: steps S501 to S505.
S501, judging whether the verification result passes or not;
s502, if the verification result is not passed, setting the form changing content to be empty;
s503, if the verification result passes, extracting a change text in the form change content;
S504, dividing the changed text into a plurality of text segments by using preset words, and carrying out classification prediction on the plurality of text segments by using a deep learning classification model;
s505, setting a prediction label for each text segment based on a classification prediction result, converting each text segment into json format text according to the prediction label, and then converting the json format text into YAML format change content based on YAML basic grammar rules.
In this embodiment, when the web worker thread is used to generate the corresponding YAML format change content of the form change content, it is first determined whether the verification result passes, if so, the change text in the form change content is continuously extracted, and if not, the form change content is set to be empty, and accordingly, the YAML format change content does not need to be continuously generated. After the change text is extracted, the change text is divided into a plurality of text segments for classification prediction, a prediction label is further set, then the change text can be converted into json format text one by one according to the prediction label, and then the json format text is converted into YAML format change content.
The embodiment specifically includes:
Acquiring a training set containing a change text and a text type, and splicing the change text in the training set according to the text type; and carrying out convolution processing on the spliced change text by adding a deep learning model of a convolution kernel. Here, the improved deep learning model can be optimized and updated by adopting the focal-loss function, so that event classification prediction is performed on the truncated text by using the updated deep learning model.
In one embodiment, before the step S108, the method includes:
judging whether YAML format change content to be filled exists or not;
if the YAML format change content to be filled is judged to exist, filling the YAML format change content into a front-end text field; if the YAML format change content to be filled is judged to not exist, the content in the pipeline template is stored to a front-end pipeline;
the step S108 includes:
generating a second constraint condition for the front-end text field through the plug-in configuration, and checking the front-end text field based on the second constraint condition.
In this embodiment, before filling the front-end text field, it is determined whether YAML format change content is used to fill the front-end text field, if corresponding YAML format change content is present, the YAML format change content may be directly filled into the front-end text field, and if no corresponding YAML format change content is present, the corresponding content in the original pipeline template is continuously maintained. Further, when the YAML format changing content is filled into the front-end text field, the filled text field can be checked by using a second constraint condition generated based on the plug-in configuration to confirm whether each workflow text of the front-end pipeline is expressed correctly. Here, since the foregoing has mentioned that the plug-in configuration includes the pipeline operation template, the pipeline operation template may also be set to the second constraint condition, that is, the second constraint condition is the same as the first constraint condition, and similarly, when the second constraint condition is used for verification, the foregoing process of performing constraint verification with respect to the first constraint condition may also be referred to, which is not described herein again.
In an embodiment, as shown in fig. 6, the step S104 further includes: steps S601 to S604.
S601, acquiring a scheduling dependency relationship and an initial arrangement relationship of the DAG chart;
s602, responding to the operation of a user on the DAG chart, acquiring the middle arrangement relation of the DAG chart, and confirming a target DAG chart with changed position through a chart searching algorithm;
s603, extracting a dependent DAG graph of the target DAG graph by scheduling dependency relationship, and carrying out position adjustment on the intermediate arrangement relationship by combining the position relationship between the dependent DAG graph and the target DAG graph to obtain a final arrangement relationship of the DAG graph;
s604, comparing the initial arrangement relation and the final arrangement relation of the DAG chart, and dynamically rendering the DAG chart with the position change.
In this embodiment, before dynamically rendering a DAG chart, a scheduling dependency relationship and an initial layout relationship of each DAG chart (or DAG icons) in the DAG chart are obtained, that is, the layout relationship of each DAG chart after initialization, then an intermediate layout relationship after operation of the user DAG chart is obtained, and the target DAG chart operated by the user can be confirmed after comparing the initial layout relationship and the intermediate layout relationship by using a chart search algorithm. And continuously acquiring the dependent DAG graph with the dependency relationship for the target DAG graph, judging whether the dependent DAG graph has position change or not, and adjusting the position of the dependent DAG graph according to the scheduling dependency relationship, so that a final arrangement relationship can be obtained, and dynamic rendering is carried out on the DAG graph according to the difference between the final arrangement relationship and the initial arrangement relationship.
In a specific embodiment, when a target DAG graph with a changed position is confirmed by a graph searching algorithm, a depth-first searching algorithm (DFS) is used to search the graph for the initial layout relationship and the intermediate layout relationship, and the DFS is generally implemented by using a data structure such as a stack (first-in last-out), so that the searching is performed by taking an initial starting point as a center, first surrounding points are added to the stack, after the searching of the starting point is completed, one point needs to be selected from the surrounding points to access, however, due to the characteristic of first-in last-out, the last node entering the stack in the surrounding points is accessed. However, the new node also has surrounding points which are pushed onto the stack, so that, as a whole, the points around the starting point have not been accessed, and the new surrounding points are added thereto, and the search is continued in one direction. The DAG graphs in the initial arrangement relation and the intermediate arrangement relation are searched by a depth-first search algorithm, and a search path is established, so that whether the two are different and the positions where the two are different can be judged by combining the search path. Of course, in other embodiments, other graph search algorithms may be employed, such as breadth-first search algorithms, heuristic search algorithms, and the like.
Fig. 7 is a schematic block diagram of a DAG graph-based visual pipeline arrangement apparatus 700 according to an embodiment of the present invention, where the apparatus 700 includes:
an initializing unit 701, configured to initialize an orchestration component of a front-end pipeline, and load a plug-in configuration of the pipeline, and generate a front-end form according to the plug-in configuration; wherein the orchestration component comprises a code editor and a draggable DAG graph;
the loading analysis unit 702 is used for loading a pipeline template in a YAML format, and analyzing the pipeline template by utilizing a web worker thread to obtain a corresponding analysis result;
a first mode obtaining unit 703, configured to obtain a current arrangement mode of the front-end pipeline in response to a user operation on the front-end pipeline; wherein the current programming mode comprises a graphical programming mode based on a DAG chart and a code editing mode based on a code editor;
a dynamic rendering unit 704, configured to dynamically render the DAG chart and the front-end form in combination with the plug-in configuration and the analysis result when the current layout mode is a graphical layout mode, so as to obtain form changing contents;
the constraint verification unit 705 is configured to generate a first constraint condition for the form changing content through the plug-in, and verify the form changing content based on the first constraint condition to obtain a corresponding verification result;
A change generating unit 706, configured to send the form change content and the verification result to the web worker thread, and generate corresponding YAML format change content by the web worker thread;
a second mode obtaining unit 707, configured to continuously obtain a current arrangement mode of the front-end pipeline, and if the current arrangement mode is still the graphical arrangement mode, continuously generate corresponding YAML format change content until the current arrangement mode is a code editing mode;
and a filling arrangement unit 708, configured to fill the generated YAML format change content into a front-end text field when the current arrangement mode is a code editing mode, and complete a pipeline arrangement operation according to the filled front-end text field.
In one embodiment, as shown in fig. 8, the load parsing unit 702 includes:
a text extraction unit 801, configured to extract pipeline template text from a yacl-format pipeline template according to a character string format;
a dependency graph construction unit 802, configured to construct the pipeline template text into a plurality of first syntax dependency graphs by using a syntax dependency analysis manner, obtain node relationships in the syntax dependency graphs, and then associate the plurality of first syntax dependency graphs according to the node relationships to obtain a second syntax dependency graph;
A semantic error correction unit 803, configured to extract a semantic token vector for the second syntactic dependency graph through a graph convolution neural network, and perform semantic error correction on the pipeline template text based on the semantic token vector;
and the parsing and sorting unit 804 is configured to parse and sort the pipeline template text through a preset format file template based on the semantic error correction result.
In one embodiment, as shown in fig. 9, the dynamic rendering unit 704 includes:
a format conversion unit 901, configured to convert the DAG chart and the front-end form into a json format array;
a change confirmation unit 902, configured to confirm a change array that changes in the json format array in combination with the plug-in configuration and the analysis result, and set a tag for the change array;
an array obtaining unit 903, configured to obtain the json format array with the tag by using a Web data interaction manner;
the element parsing unit 904 is configured to parse the json format array with the tag into elements, and insert the elements into a rendering tree as result nodes, so as to implement dynamic rendering.
In one embodiment, the plug-in configuration includes a pipeline job template;
as shown in fig. 10, the constraint checking unit 705 includes:
A constraint setting unit 1001 configured to set the pipeline job template as a first constraint condition;
a constraint extraction unit 1002, configured to extract constraint text for the first constraint condition, and extract change text for the form change content;
a vector extraction unit 1003 configured to extract a constraint vector of the constraint text and extract a change vector of the change text through a graph convolution neural network;
a target output unit 1004, configured to sequentially input the constraint vector and the change vector to a mutual attention layer and a self attention layer, to obtain a target vector;
a probability prediction unit 1005, configured to perform classification prediction on the target vector through a pre-trained full-connection layer, output a prediction probability, and then use the prediction probability as the verification result; the full-connection layer adopts preset check words to complete pre-training.
In one embodiment, as shown in fig. 11, the change generating unit 706 includes:
a result judging unit 1101 for judging whether the verification result passes;
a change setting unit 1102, configured to set the form change content to be empty if the verification result is not passed;
A change extracting unit 1103, configured to extract a change text in the form change content if the verification result passes;
a text classification unit 1104, configured to divide the modified text into a plurality of text segments by using preset words, and perform classification prediction on the plurality of text segments by using a deep learning classification model;
a text conversion unit 1105, configured to set a prediction tag for each text segment based on the classification prediction result, and convert each text segment into json format text according to the prediction tag, and then convert the json format text into the YAML format change content based on the YAML basic grammar rule.
In one embodiment, the DAG graph-based visualization pipeline orchestration device 700 further comprises:
a filling judgment unit for judging whether YAML format change content to be filled exists or not;
a content filling unit, configured to fill YAML format change content into a front-end text field if it is determined that there is YAML format change content to be filled; if the YAML format change content to be filled is judged to not exist, the content in the pipeline template is stored to a front-end pipeline;
the filling arrangement unit 708 includes:
and the text field verification unit is used for generating a second constraint condition for the front-end text field through the plug-in configuration, and verifying the front-end text field based on the second constraint condition.
In an embodiment, as shown in fig. 12, the dynamic rendering unit 704 further includes:
a dependency acquisition unit 1201, configured to acquire a scheduling dependency relationship and an initial scheduling relationship of the DAG chart;
a graph searching unit 1202, configured to obtain an intermediate arrangement relationship of the DAG graph in response to an operation of the DAG graph by a user, and confirm a target DAG graph with a changed position through a graph searching algorithm;
a layout adjustment unit 1203, configured to extract a dependent DAG graph of the target DAG graph according to the scheduling dependency relationship, and perform position adjustment on the intermediate layout relationship by combining the position relationship between the dependent DAG graph and the target DAG graph, so as to obtain a final layout relationship of the DAG graph;
and the arrangement comparison unit 1204 is used for comparing the initial arrangement relation and the final arrangement relation of the DAG chart and dynamically rendering the DAG chart with the position change.
Since the embodiments of the apparatus portion and the embodiments of the method portion correspond to each other, the embodiments of the apparatus portion are referred to the description of the embodiments of the method portion, and are not repeated herein.
The embodiment of the present invention also provides a computer readable storage medium having a computer program stored thereon, which when executed can implement the steps provided in the above embodiment. The storage medium may include: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The embodiment of the invention also provides a computer device, which can comprise a memory and a processor, wherein the memory stores a computer program, and the processor can realize the steps provided by the embodiment when calling the computer program in the memory. Of course, the computer device may also include various network interfaces, power supplies, and the like.
In the description, each embodiment is described in a progressive manner, and each embodiment is mainly described by the differences from other embodiments, so that the same similar parts among the embodiments are mutually referred. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section. It should be noted that it would be obvious to those skilled in the art that various improvements and modifications can be made to the present application without departing from the principles of the present application, and such improvements and modifications fall within the scope of the claims of the present application.
It should also be noted that in this specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Claims (9)
1. A method for visual pipeline orchestration based on DAG charts, comprising:
initializing an arranging assembly of a front-end pipeline, loading plug-in configuration of the pipeline, and generating a front-end form according to the plug-in configuration; wherein the orchestration component comprises a code editor and a draggable DAG graph;
loading a pipeline template in a YAML format, and analyzing the pipeline template by utilizing a web worker thread to obtain a corresponding analysis result;
responding to the operation of a user on the front-end pipeline, and acquiring the current arrangement mode of the front-end pipeline; wherein the current programming mode comprises a graphical programming mode based on a DAG chart and a code editing mode based on a code editor;
when the current arrangement mode is a graphical arrangement mode, dynamically rendering the DAG chart and the front-end form by combining the plug-in configuration and the analysis result to obtain form changing contents;
generating a first constraint condition for the form changing content through the plug-in configuration, and checking the form changing content based on the first constraint condition to obtain a corresponding checking result; the plug-in configuration comprises a pipeline operation template; setting the pipeline operation template as a first constraint condition;
The form changing content and the verification result are sent to the web worker thread, and the web worker thread generates corresponding YAML format changing content;
continuously acquiring a current arrangement mode of a front-end pipeline, and continuously generating corresponding YAML format change content if the current arrangement mode is still the graphical arrangement mode until the current arrangement mode is a code editing mode;
when the current editing mode is a code editing mode, filling the generated YAML format change content into a front-end text field, and completing pipeline editing operation according to the filled front-end text field;
the step of sending the form change content and the verification result to the web worker thread, and generating corresponding YAML format change content by the web worker thread, wherein the method comprises the following steps:
judging whether the verification result passes or not;
if the verification result is not passed, setting the form change content to be empty;
if the verification result passes, extracting a change text in the form change content;
dividing the changed text into a plurality of text segments by using preset words, and carrying out classification prediction on the plurality of text segments by using a deep learning classification model;
And setting a prediction label for each text segment based on the classification prediction result, converting each text segment into json format text according to the prediction label, and then converting the json format text into YAML format change content based on YAML basic grammar rules.
2. The DAG graph-based visual pipeline arrangement method of claim 1, wherein the loading the YAML formatted pipeline template and analyzing the pipeline template by using a web worker thread to obtain a corresponding analysis result comprises:
extracting a pipeline template text from the pipeline template in the YAML format according to the character string format;
constructing the pipeline template text into a plurality of first syntax dependency graphs by adopting a syntax dependency analysis mode, acquiring node relations in the syntax dependency graphs, and then associating the plurality of first syntax dependency graphs according to the node relations to obtain a second syntax dependency graph;
extracting a semantic characterization vector from the second syntactic dependency graph through a graph convolution neural network, and carrying out semantic error correction on the pipeline template text based on the semantic characterization vector;
and analyzing and sorting the pipeline template text through a preset format file template based on the semantic error correction result.
3. The DAG graph-based visualization pipeline orchestration method according to claim 1, wherein when the current orchestration mode is a graphical orchestration mode, dynamically rendering the DAG graph and the front-end form in combination with the plug-in configuration and analysis results, comprising:
converting the DAG chart and the front-end form into json format arrays;
confirming a changed change array in the json format array by combining the plug-in configuration and the analysis result, and setting a label for the change array;
acquiring the json format array with the tag by adopting a Web data interaction mode;
and analyzing the json format array with the tag into elements, and inserting the elements into a rendering tree as result nodes so as to realize dynamic rendering.
4. The DAG graph-based visual pipeline arrangement method of claim 1, wherein the configuring by the plug-in to generate a first constraint condition for the form change content and verifying the form change content based on the first constraint condition to obtain a corresponding verification result includes:
extracting constraint text from the first constraint condition and extracting change text from the form change content;
Extracting constraint vectors of the constraint text through a graph convolution neural network, and extracting change vectors of the change text;
sequentially inputting the constraint vector and the change vector to a mutual attention layer and a self attention layer to obtain a target vector;
classifying and predicting the target vector through a pre-trained full-connection layer, outputting and obtaining a prediction probability, and taking the prediction probability as the verification result; the full-connection layer adopts preset check words to complete pre-training.
5. The DAG graph-based visual pipeline orchestration method according to claim 1, wherein before the step of populating the front-end text field with the generated YAML format change content, comprising:
judging whether YAML format change content to be filled exists or not;
if the YAML format change content to be filled is judged to exist, filling the YAML format change content into a front-end text field; if the YAML format change content to be filled is judged to not exist, the content in the pipeline template is stored to a front-end pipeline;
filling the generated YAML format change content into a front-end text field, and completing pipeline arrangement operation according to the filled front-end text field, wherein the method comprises the following steps:
Generating a second constraint condition for the front-end text field through the plug-in configuration, and verifying the front-end text field based on the second constraint condition; the second constraint is the same as the first constraint.
6. The DAG graph-based visualization pipeline orchestration method according to claim 1, wherein when the current orchestration mode is a graphical orchestration mode, the DAG graph and the front-end form are dynamically rendered in combination with the plug-in configuration and analysis results to obtain form change content, further comprising:
acquiring a scheduling dependency relationship and an initial arrangement relationship of the DAG chart;
responding to the operation of a user on the DAG chart, acquiring an intermediate arrangement relation of the DAG chart, and confirming a target DAG chart with changed position through a chart searching algorithm;
extracting a dependent DAG graph of the target DAG graph by a scheduling dependency relationship, and carrying out position adjustment on the intermediate arrangement relationship by combining the position relationship between the dependent DAG graph and the target DAG graph to obtain a final arrangement relationship of the DAG graph;
and comparing the initial arrangement relation and the final arrangement relation of the DAG chart, and dynamically rendering the DAG chart with the position change.
7. A DAG graph-based visual pipeline orchestration device, comprising:
the initialization unit is used for initializing an arrangement assembly of the front-end pipeline and loading plug-in configuration of the pipeline and generating a front-end form according to the plug-in configuration; wherein the orchestration component comprises a code editor and a draggable DAG graph;
the loading analysis unit is used for loading the assembly line template in the YAML format, and analyzing the assembly line template by utilizing the web worker thread to obtain a corresponding analysis result;
the first mode acquisition unit is used for responding to the operation of a user on the front-end pipeline and acquiring the current arrangement mode of the front-end pipeline; wherein the current programming mode comprises a graphical programming mode based on a DAG chart and a code editing mode based on a code editor;
the dynamic rendering unit is used for dynamically rendering the DAG chart and the front-end form by combining the plug-in configuration and the analysis result when the current arrangement mode is a graphical arrangement mode so as to acquire form changing contents;
the constraint verification unit is used for generating a first constraint condition for the form changing content through the plug-in configuration, and verifying the form changing content based on the first constraint condition to obtain a corresponding verification result; the plug-in configuration comprises a pipeline operation template; setting the pipeline operation template as a first constraint condition;
The change generation unit is used for sending the form change content and the verification result to the web worker thread, and generating corresponding YAML format change content by the web worker thread;
the second mode acquisition unit is used for continuously acquiring a current arrangement mode of the front-end pipeline, and continuously generating corresponding YAML format change content if the current arrangement mode is still the graphical arrangement mode until the current arrangement mode is a code editing mode;
the filling arrangement unit is used for filling the generated YAML format change content into a front-end text field when the current arrangement mode is a code editing mode, and completing pipeline arrangement operation according to the filled front-end text field;
the change generation unit includes:
the result judging unit is used for judging whether the verification result passes or not;
a change setting unit, configured to set the form change content to be empty if the verification result is not passed;
a change extracting unit, configured to extract a change text in the form change content if the verification result passes;
the text classification unit is used for dividing the changed text into a plurality of text segments by using preset words and carrying out classification prediction on the plurality of text segments through a deep learning classification model;
And the text conversion unit is used for setting a prediction label for each text segment based on the classification prediction result, converting each text segment into json format text according to the prediction label, and then converting the json format text into YAML format change content based on YAML basic grammar rules.
8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the DAG graph-based visualization pipeline orchestration method according to any one of claims 1 to 6 when the computer program is executed.
9. A computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, which when executed by a processor, implements the DAG graph-based visualization pipeline orchestration method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310311926.1A CN116028038B (en) | 2023-03-28 | 2023-03-28 | Visual pipeline arrangement method based on DAG chart and related components |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310311926.1A CN116028038B (en) | 2023-03-28 | 2023-03-28 | Visual pipeline arrangement method based on DAG chart and related components |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116028038A CN116028038A (en) | 2023-04-28 |
| CN116028038B true CN116028038B (en) | 2023-06-09 |
Family
ID=86089629
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310311926.1A Active CN116028038B (en) | 2023-03-28 | 2023-03-28 | Visual pipeline arrangement method based on DAG chart and related components |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116028038B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111597143A (en) * | 2020-03-24 | 2020-08-28 | 中电海康集团有限公司 | Method for converting DAG graph data into workflow engine scheduling file |
| CN112083919A (en) * | 2020-08-27 | 2020-12-15 | 远光软件股份有限公司 | YAML template-based target service code generation method and device |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11206229B2 (en) * | 2019-04-26 | 2021-12-21 | Oracle International Corporation | Directed acyclic graph based framework for training models |
| US20210224046A1 (en) * | 2020-01-21 | 2021-07-22 | Ontology-Partners Ltd | Workflow engine for generating code based on visualizations |
| US20220188660A1 (en) * | 2020-12-10 | 2022-06-16 | Coupang Corp. | Systems and methods for processing data for storing in a feature store and for use in machine learning |
| CN112965785B (en) * | 2021-03-05 | 2023-06-13 | 食亨(上海)科技服务有限公司 | Container-based micro-service application development method and development platform |
| CN114594994A (en) * | 2022-01-19 | 2022-06-07 | 浪潮通信信息系统有限公司 | YAML-based Jenkins pipeline configuration method and device |
| CN114912897A (en) * | 2022-05-18 | 2022-08-16 | 阿里巴巴（中国）有限公司 | Workflow execution method, workflow arrangement method and electronic equipment |
-
2023
- 2023-03-28 CN CN202310311926.1A patent/CN116028038B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111597143A (en) * | 2020-03-24 | 2020-08-28 | 中电海康集团有限公司 | Method for converting DAG graph data into workflow engine scheduling file |
| CN112083919A (en) * | 2020-08-27 | 2020-12-15 | 远光软件股份有限公司 | YAML template-based target service code generation method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116028038A (en) | 2023-04-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11734375B2 (en) | Automatic navigation of interactive web documents | |
| US20200125482A1 (en) | Code examples sandbox | |
| US10860778B2 (en) | System, method, and software application for enabling a user to view and interact with a visual map in an external application | |
| CN103597469B (en) | Live browser instrument in IDE | |
| US20080221870A1 (en) | System and method for revising natural language parse trees | |
| US20220050661A1 (en) | Analyzing graphical user interfaces to facilitate automatic interaction | |
| EP1901179A1 (en) | Document processing device, and document processing method | |
| US11461081B2 (en) | Adapting existing source code snippets to new contexts | |
| CN116775183A (en) | Task generation method, system, equipment and storage medium based on large language model | |
| Wielemaker et al. | Using SWISH to realize interactive web-based tutorials for logic-based languages | |
| US20200150937A1 (en) | Advanced machine learning interfaces | |
| KR102561951B1 (en) | Configuration method, device, electronic equipment and computer storage medium of modeling parameters | |
| US20080155479A1 (en) | Method and Apparatus for Building Interactive Software Applications | |
| CN115145568A (en) | Code generation method based on image recognition and related equipment | |
| US10262073B2 (en) | Generating and distributing interactive documents | |
| CN108932225B (en) | Method and system for converting natural language requirements into semantic modeling language statements | |
| US10372744B2 (en) | DITA relationship table based on contextual taxonomy density | |
| CN113704420A (en) | Method and device for identifying role in text, electronic equipment and storage medium | |
| US20200364034A1 (en) | System and Method for Automated Code Development and Construction | |
| CN116028038B (en) | Visual pipeline arrangement method based on DAG chart and related components | |
| US20230153335A1 (en) | Searchable data structure for electronic documents | |
| US9594737B2 (en) | Natural language-aided hypertext document authoring | |
| CN114721932B (en) | Data processing method, device, equipment and storage medium | |
| CN111898762B (en) | Deep learning model catalog creation | |
| CN114021541A (en) | Presentation generation method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |