CN115934097A - Method, device, storage medium and electronic device for generating executable statement - Google Patents
Method, device, storage medium and electronic device for generating executable statement Download PDFInfo
- Publication number
- CN115934097A CN115934097A CN202211573550.3A CN202211573550A CN115934097A CN 115934097 A CN115934097 A CN 115934097A CN 202211573550 A CN202211573550 A CN 202211573550A CN 115934097 A CN115934097 A CN 115934097A
- Authority
- CN
- China
- Prior art keywords
- operator
- topological graph
- operators
- model topological
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Stored Programmes (AREA)
Abstract
The embodiment of the invention provides a method, a device, a storage medium and an electronic device for generating an executable statement, wherein the method for generating the executable statement comprises the following steps: obtaining an operator model topological graph, wherein the operator model topological graph is a topological graph configured in a visualization page, and the operator model topological graph comprises: a data table operator, a target table operator and a system operator; performing reverse verification on the operator model topological graph from a target table operator, wherein the reverse verification comprises connectivity verification and validity verification; and under the condition that the operator model topological graph passes reverse verification, carrying out forward analysis on the operator model topological graph from the data table operator to obtain an executable statement set corresponding to the operator model topological graph. By the invention, the problem of low Fl ink real-time task development efficiency in the related technology is solved.
Description
Technical Field
The embodiment of the invention relates to the field of real-time computing, in particular to a method, a device, a storage medium and an electronic device for generating executable statements.
Background
Since big data is emerging, hadoop does not provide a real-time computing solution, and then real-time computing frameworks such as Storm, spark streaming, and Flink come into play, wherein Flink has advantages in real-time performance, data processing and the like and is widely applied.
At present, the Flink real-time task development mainly adopts a Jar package submission mode, the mode can greatly improve the operation flexibility, but the content of the Jar package is invisible, so that the operation is easily confused, the compiling and packaging process is time-consuming seriously, and the Jar package file is large, occupies more network transmission bandwidth, increases the Flink task deployment time, and reduces the real-time task development efficiency of users.
Aiming at the problem of low efficiency of Flink real-time task development in the related technology, no effective solution is provided at present.
Disclosure of Invention
The embodiment of the invention provides a method, a device, a storage medium and an electronic device for generating executable statements, which at least solve the problem of low efficiency of Flink real-time task development in the related technology.
According to an embodiment of the present invention, there is provided a method of generating an executable statement, including: obtaining an operator model topological graph, wherein the operator model topological graph is a topological graph configured in a visualization page, and the operator model topological graph comprises: a data table operator, a target table operator and a system operator; performing reverse verification on the operator model topological graph from the target table operator, wherein the reverse verification comprises connectivity verification and validity verification; and under the condition that the operator model topological graph passes the reverse check, carrying out forward analysis on the operator model topological graph from the data table operator to obtain an executable statement set corresponding to the operator model topological graph.
In an exemplary embodiment, the reverse checking the operator model topology map starting from the destination table operator comprises: traversing operators in the operator model topological graph from the target table operator, and detecting whether isolated operators exist in the operator model topological graph or not; and under the condition that the isolated operator does not exist in the operator model topological graph, determining that the operator model topological graph passes the connectivity check, otherwise, determining that the operator model topological graph does not pass the connectivity check.
In an exemplary embodiment, traversing the operators in the operator model topological graph from the destination table operator, and detecting whether an orphan operator exists in the operator model topological graph includes: traversing the operators in the operator model topological graph step by step from the target table operator; determining that the orphan operator does not exist in the operator model topological graph under the condition that all operators in the operator model topological graph are traversed; otherwise, determining the operators which are not traversed in the operator model topological graph as the isolated operators.
In an exemplary embodiment, the reversely checking the operator model topology map from the destination table operator further includes: traversing operators in the operator model topological graph from the target table operator; under the condition of traversing to the jth operator of the ith level in the operator model topological graph, acquiring one or more operators which are in directed connection with the jth operator in the (i + 1) th level; detecting whether preset conditions are met between the jth operator and the one or more operators; determining that the jth operator passes the validity check under the condition that the preset condition is met, otherwise determining that the jth operator does not pass the validity check; wherein, the connection direction of the jth operator and the one or more operators is that the jth operator points to the one or more operators, and i is greater than or equal to 1,j and is greater than or equal to 1.
In an exemplary embodiment, detecting whether a preset condition is satisfied between the jth operator and the one or more operators comprises: obtaining the operator type of the jth operator and the operator types of the one or more operators; determining that a preset condition is met between the jth operator and the one or more operators if the operator type of the jth operator and the operator type of the one or more operators meet a preset operator type condition; otherwise, determining that the preset condition is not met between the jth operator and the one or more operators.
In an exemplary embodiment, forward parsing the operator model topological graph from the data table operator to obtain an executable statement set corresponding to the operator model topological graph includes: and analyzing each operator in the model topological graph step by step from the data table operator to obtain an executable statement corresponding to each operator, wherein the executable statement set comprises the executable statements corresponding to each operator in the model topological graph.
In an exemplary embodiment, analyzing each operator in the model topology graph step by step from the data table operator to obtain an executable statement corresponding to each operator, includes: analyzing the mth operator in the nth stage in the model topological graph, wherein n is greater than or equal to 1,m and is greater than or equal to 1, and the method comprises the following steps: acquiring an executable statement template corresponding to the mth operator; parsing target information of the mth operator into an executable statement segment of the mth operator, wherein the target information includes at least one of: input fields, output fields, configuration data; and replacing the corresponding part in the executable statement template with the executable statement segment to obtain the executable statement corresponding to the mth operator.
According to another embodiment of the present invention, there is also provided an apparatus for generating an executable statement, including:
the obtaining module is used for obtaining an operator model topological graph, wherein the operator model topological graph is a topological graph configured in a visualization page, and the operator model topological graph comprises: a data table operator, a target table operator, and a system operator;
the checking module is used for carrying out reverse checking on the operator model topological graph from the target table operator, wherein the reverse checking comprises connectivity checking and validity checking;
and the analysis module is used for carrying out forward analysis on the operator model topological graph from the data table operator to obtain an executable statement set corresponding to the operator model topological graph under the condition that the operator model topological graph passes the reverse check.
According to a further embodiment of the present invention, there is also provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.
According to yet another embodiment of the present invention, there is also provided an electronic device, including a memory in which a computer program is stored and a processor configured to execute the computer program to perform the steps in any of the above method embodiments.
Through the steps, whether effectiveness and connectivity are achieved can be judged through reverse traversal with a target table operator as a starting point, the operator is analyzed into an executable statement through forward traversal with a data table operator as the starting point under the condition that reverse verification is passed, the verification and analysis process of the operator model topological graph can be completed through only one reverse traversal and one forward traversal, the time complexity and the space complexity of conversion of the topological graph into an executable statement set are reduced, a task is submitted according to the generated executable statement set, and the Jar package compiling process is not needed, so that the problem that the Flink real-time task development efficiency is low in the prior art is solved, and the problem of improving the development efficiency is achieved.
Drawings
Fig. 1 is a block diagram of a hardware structure of a mobile terminal of a method of generating an executable sentence according to an embodiment of the present invention;
FIG. 2 is a flow diagram of a method of generating executable statements according to an embodiment of the invention;
FIG. 3 is a schematic diagram of an alternative operator topology according to the present embodiment;
FIG. 4 is an alternative operator model topology diagram with orphan operators, according to embodiments of the invention;
FIG. 5 is an alternative operator model topology that fails the validity check in accordance with the present embodiment;
FIG. 6 is a diagram of executable statements of data source operators, in accordance with an embodiment of the present invention;
FIG. 7 is an analytic schematic of an alternative operator model topology according to embodiments of the invention;
FIG. 8 is a schematic diagram of an alternative configuration data table operator according to an embodiment of the present invention;
FIG. 9 is a schematic diagram of an alternative configuration destination table operator, according to an embodiment of the present invention;
FIG. 10 is a schematic diagram of an alternative configuration operator model topology diagram according to an embodiment of the present invention;
FIG. 11 is a general flow diagram of real-time task development in accordance with a specific embodiment of the present invention;
fig. 12 is a block diagram of an apparatus for generating an executable sentence according to an embodiment of the present invention.
Detailed Description
Embodiments of the present invention will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
The method embodiments provided in the embodiments of the present application may be executed in a mobile terminal, a computer terminal, or a similar computing device. Taking the example of running on a mobile terminal, fig. 1 is a block diagram of a hardware structure of the mobile terminal of the method for generating an executable statement according to the embodiment of the present invention. As shown in fig. 1, the mobile terminal may include one or more (only one shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data, wherein the mobile terminal may further include a transmission device 106 for communication functions and an input-output device 108. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration, and does not limit the structure of the mobile terminal. For example, the mobile terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
The memory 104 may be used for storing computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the method for generating executable statements in the embodiments of the present invention, and the processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, so as to implement the above-mentioned method. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the mobile terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used for receiving or transmitting data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a Network adapter (NIC), which can be connected to other Network devices through a base station so as to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
In the present embodiment, a method for generating an executable statement is provided, and fig. 2 is a flowchart of a method for generating an executable statement according to an embodiment of the present invention, as shown in fig. 2, the flowchart includes the following steps:
step S202, an operator model topological graph is obtained, wherein the operator model topological graph is configured in a visual page and comprises: a data table operator, a target table operator, and a system operator;
step S204, carrying out reverse verification on the operator model topological graph from the target table operator, wherein the reverse verification comprises connectivity verification and validity verification;
and step S206, under the condition that the operator model topological graph passes the reverse check, carrying out forward analysis on the operator model topological graph from the data table operator to obtain an executable statement set corresponding to the operator model topological graph.
In this embodiment, the operator model topological graph is configured in a visualization page, and is a directed acyclic graph constructed by a user in the visualization page by using operator connecting lines, an operator is selected on the left side of the visualization page and dragged into an editing area in the visualization page, and operator nodes in the editing area are connected according to a dependency relationship between operators to generate the operator model topological graph. The operator model topological graph is used for representing the Flink task.
The operator model topological graph comprises three operators: data table operator, target table operator and system operator built in the system. Each operator is composed of at least one item of content such as input, output, parameter configuration and the like, and each item of content is composed of a field name, a type, a description, a parameter value and the like. The data table operators are data source operators defined according to a data format in a data source, the data source operators are positioned at the most upstream node of an operator model topological graph, and flow tables and dimension tables of CSV data source types are supported, wherein the data sources can access data stored in an external storage system (Mysql, hbase and the like), a message system (RabbitMQ, apache Kafka) and a specified coding format file (CSV) and the like. The target table operator is an output source operator used for loading the converted data to a target database or a file and is positioned at the tail end node of the operator model topological graph. The system operator comprises operators such as filtering, conversion, grouping, double-flow Join, an association dimension table, duplication removal and a user-defined function, and is a data processing operator, and field attribute configuration can be carried out according to business logic.
In a complete operator model topological graph, starting from a data table operator, all operators in the operator model topological graph can be traversed according to the connection direction sequence, the operators cannot be traversed to the operators, and finally, the traversal is finished to a target table operator. The input information of the operator in the operator model topological graph can be obtained from the output of the upstream according to the connecting line direction, and the output information of the operator in the operator model topological graph is transmitted to the downstream to be used as the input of the downstream operator.
Fig. 3 is a schematic diagram of an alternative operator topology according to the embodiment, as shown in fig. 3, the flow table operator and the dimension table operator are data table operators, which are denoted as level 4 operators, the JOIN operator and the FILTER operator are system operators, which are denoted as level 3 operators and level 2 operators, respectively, and the destination table operator is located at an end node of the operator model topology diagram, and is denoted as level 1 operator. Optionally, the previous operator of the ith operator is the (i + 1) th operator, the next operator of the ith operator is the (i-1) th operator, when i =1, the ith operator is the target table operator, and the next operator does not exist, and when i = N, the nth operator is the data table operator, and the previous operator does not exist.
The complete operator model topological graph used for representing the Flink task must meet connectivity and validity, therefore, the operator model topological graph is configured by a user in a visual page, errors may exist in the configuration process, and therefore, before the operator model topological graph is used for carrying out the Flink task, whether the operator model topological graph meets the connectivity and the validity needs to be checked. Connectivity means that no isolated operator exists in the topological graph and no repeatedly traversed operator exists, and effectiveness means that connection among operators meets corresponding connection conditions.
Because there is only one target table operator in the complete operator model topological graph, there can be one or more data table operators, therefore, whether the verification operator model topological graph meets the thinking of selecting 'reverse traversal' for connectivity and effectiveness, that is, reverse verification is performed from the target table operator, the verification of the operator model topological graph can be completed only by one traversal process, compared with other verification methods, such as forward verification from the data table operator, the verification speed is accelerated, and the verification complexity is reduced.
Under the condition that the operator model topological graph passes the reverse check, the operator model topological graph is complete and can be used for representing the corresponding Flink task, and each operator is analyzed into an executable statement (SQL statement) by forward traversal with a data table operator in the operator model topological graph as a starting point to obtain an executable statement set.
After the executable statement set is generated, the executable statement set is sent to the Flink client, the Flink client further generates a JobGraph streaming graph according to the executable statements, and configures the operation parameters of the task, so that the task can be submitted and run normally and efficiently.
Through the steps, whether effectiveness and connectivity are achieved can be judged through reverse traversal with a target table operator as a starting point, the operator is analyzed into an executable statement through forward traversal with a data table operator as the starting point under the condition that reverse verification is passed, the verification and analysis process of the operator model topological graph can be completed through only one reverse traversal and one forward traversal, the time complexity and the space complexity of conversion of the topological graph into an executable statement set are reduced, a task is submitted according to the generated executable statement set, and the Jar package compiling process is not needed, so that the problem that the Flink real-time task development efficiency is low in the prior art is solved, and the problem of improving the development efficiency is achieved.
In an optional embodiment, performing a reverse check on the operator model topology map from the destination table operator includes: traversing operators in the operator model topological graph from the target table operator, and detecting whether isolated operators exist in the operator model topological graph or not; and under the condition that the isolated operator does not exist in the operator model topological graph, determining that the operator model topological graph passes the connectivity check, otherwise, determining that the operator model topological graph does not pass the connectivity check.
In this embodiment, the checking on the operator model topological graph is divided into connectivity checking and validity checking, where the performing connectivity checking on the operator model topological graph is to determine whether all the operators in the operator model topological graph can be traversed.
And all the operators which are not traversed are determined as isolated operators, the operator model topological graph does not have connectivity under the condition that the isolated operators exist, namely the operator model topological graph does not pass connectivity verification, and the operator model topological graph has connectivity under the condition that the isolated operators do not exist, namely the operator model topological graph passes connectivity verification.
In an optional embodiment, traversing operators in the operator model topological graph from the destination table operator, and detecting whether an orphan operator exists in the operator model topological graph includes: traversing the operators in the operator model topological graph step by step from the target table operator; determining that the orphan operator does not exist in the operator model topological graph under the condition that all operators in the operator model topological graph are traversed; otherwise, determining the operators which are not traversed in the operator model topological graph as the isolated operators.
In this embodiment, detecting whether an isolated operator exists in the operator model topological graph includes:
firstly, traversing operators in an operator model topological graph step by step from a target table operator, wherein if all the operators can be traversed, no isolated operator exists, and if all the operators cannot be traversed, the isolated operator exists. Fig. 4 is an optional operator model topology diagram with isolated operators according to an embodiment of the present invention, as shown in fig. 4, where operator F is a destination table operator, operators a and B are data table operators, and the other operators (operators C, D, E) are all system operators, and arrows indicate connection directions between the operators. Starting from a target table operator F, performing reverse traversal step by step according to the connection relation between operators, wherein the operator F can be traversed to an operator E, the operator E is connected with the operator C, the operator E can be traversed to the operator C, and then the operator C can be traversed to the operator A; the operator E is not connected to the operator D, and cannot traverse to the operator D from the operator E, and thus cannot traverse to the operator B, so that the operators B and D cannot traverse to the operator D in the operator model topological graph shown in fig. 4, and therefore the operators B and D are determined as isolated operators in the operator model topological graph, that is, the operator topological graph shown in fig. 4 does not have connectivity and cannot pass connectivity verification.
In an optional embodiment, performing a reverse check on the operator model topology map from the destination table operator further includes: traversing operators in the operator model topological graph from the target table operator; under the condition of traversing to the jth operator of the ith level in the operator model topological graph, acquiring one or more operators which are in directed connection with the jth operator in the (i + 1) th level; detecting whether preset conditions are met between the jth operator and the one or more operators; determining that the jth operator passes the validity check under the condition that the preset condition is met, otherwise determining that the jth operator does not pass the validity check; wherein, the connection direction of the jth operator and the one or more operators is that the jth operator points to the one or more operators, and i is greater than or equal to 1,j and is greater than or equal to 1.
In this embodiment, the validity check on the operator model topological graph is to determine whether a connection between operators in the operator topological graph meets a preset condition, where the preset condition is some requirements that the operator connection must meet, for example, the type of output data of an upstream operator must be consistent with the type of input data of a downstream operator, where there is a connection between the operator G and the operator H, and the connection direction is that the operator G points to the operator H, then the output data of the operator G will be used as input data of the operator H, and then the type of output data of the operator G and the type of input data specified by the operator H need to be consistent before being input to the operator H.
During validity check, when traversing to the jth operator of the ith operator, acquiring one or more operators connected with the jth operator in the (i + 1) th stage, if only one operator in the (i + 1) th stage operator is connected with the jth operator, acquiring one operator connected with the jth operator in the (i + 1) th stage, and if m operators in the (i + 1) th stage operator are connected with the jth operator, acquiring m operators connected with the jth operator in the (i + 1) th stage.
And under the condition that preset conditions are met between the jth operator and the one or more operators, the jth operator passes validity check, and under the condition that the preset conditions are not met between the jth operator and the one or more operators, the jth operator does not pass validity check. And under the condition that all system operators and target table operators in the operator model topological graph pass validity verification, determining that the operator model topological graph passes validity verification.
The obtained one or more operators are connected with the jth operator in a directed manner, and the connection direction is that the one or more operators point to the jth operator, namely that the one or more operators are upstream operators of the jth operator.
Optionally, the jth operator in the ith stage is a destination table operator or a system operator, in the case that i =1, only one operator in the 1 st stage, that is, the destination table operator, and in the case that i is not equal to 1, the jth operator in the ith stage refers to the system operator in the ith stage. One or more operators in the (i + 1) th stage may be spreadsheet operators or system operators.
In an optional embodiment, detecting whether a preset condition is satisfied between the jth operator and the one or more operators includes: obtaining an operator type of the jth operator and operator types of the one or more operators; determining that a preset condition is met between the jth operator and the one or more operators if the operator type of the jth operator and the operator type of the one or more operators meet a preset operator type condition; otherwise, determining that the preset condition is not met between the jth operator and the one or more operators.
In this embodiment, when judging whether the j-th operator and the one or more operators satisfy the preset condition, the judgment is performed through the operator type, the type of the j-th operator and the operator types of the one or more operators are obtained, and under the condition that the operator types of the operators satisfy the preset operator type condition, it is determined that the preset condition is satisfied between the j-th operator and the one or more operators, otherwise, it is determined that the preset condition is not satisfied between the j-th operator and the one or more operators.
Fig. 5 is an alternative operator model topology map that does not pass the validity check in the present embodiment, and after traversing to the dimension flow JOIN operator (corresponding to the jth operator in the ith stage), the upstream operator connected to the dimension flow JOIN operator is obtained: a first dimension table operator and a second dimension table operator (corresponding to one or more operators in the i +1 th stage connected to the j operator in the i stage), wherein the operator type of the dimension flow JOIN operator is a system operator of the dimension flow class, the operator types of the first dimension table operator and the second dimension table operator are both data table operators of the dimension table class, and the operator of the dimension flow class needs to use the output data of the dimension table class operator and the flow table class operator as the input data of the operator of the dimension flow class, while in fig. 5, the last trip operator connected to the dimension flow JOIN operator is both operators of the dimension table class, so the input data of the dimension flow JOIN operator is the output data of both dimension table class operators, and therefore the preset subclass operation condition is not satisfied between the dimension flow JOIN operator, the first dimension table operator and the second dimension table operator, i.e. corresponding to the preset condition or the preset condition between the j operator and the j operator. The operator model topology of fig. 5 is valid if the first dimension table operator is modified to a flow table operator.
In an optional embodiment, performing forward parsing on the operator model topological graph from the data table operator to obtain an executable statement set corresponding to the operator model topological graph includes: and analyzing each operator in the model topological graph step by step from the data table operator to obtain an executable statement corresponding to each operator, wherein the executable statement set comprises the executable statements corresponding to each operator in the model topological graph.
In this embodiment, when the operator model topological graph passes the reverse check, the operator model topological graph is converted into an executable statement set, and the executable statement set is submitted to a Flink engine for execution, thereby completing submission of the real-time task.
When the data table operator is converted into the executable statement set, each operator in the operator model topological graph is analyzed step by step according to the forward direction of the connection direction between the operators from the data table operator, executable statements corresponding to each operator are generated, and the executable statement set is formed.
In an optional embodiment, analyzing each operator in the model topological graph step by step from the data table operator to obtain an executable statement corresponding to each operator, includes: analyzing the mth operator in the nth stage in the model topological graph, wherein n is greater than or equal to 1,m and is greater than or equal to 1, and the method comprises the following steps: acquiring an executable statement template corresponding to the mth operator; parsing target information of the mth operator into an executable statement segment of the mth operator, wherein the target information includes at least one of: input fields, output fields, configuration data; and replacing the corresponding part in the executable statement template with the executable statement segment to obtain the executable statement corresponding to the mth operator.
In this embodiment, when one of the operators in the model topological graph is analyzed, the executable statement template corresponding to the operator is obtained, the executable statement templates corresponding to different types of operators are different, an executable statement segment is generated according to the input field, the output field, and the configuration data of the operator, and the corresponding portion in the executable statement template is replaced with the generated executable statement segment, so that the executable statement corresponding to the operator can be obtained, and the mth operator in the nth stage may be any operator in the model topological graph of the operator.
For example, for a data table operator, the data table operator itself has an output field, and an input field does not need to be provided for the data table operator through an upstream operator, so that the data table operator has no upstream operator in an operator model, and an executable statement segment is directly generated according to the field set and the configuration data which can be output.
The executable statement templates of the spreadsheet operators are:
CREATE TABLE${table_clause}(${field_clause})WITH(${properties_clause})；
where $ { table _ clause }, $ { field _ clause }, $ { properties _ clause }, where "$ { table _ clause }" is a table name placeholder, "$ { field _ clause }" is a placeholder for a set of output fields, and "$ { properties _ clause }" represents a different type of data source property placeholder. For example, if there are Kafka-type data table operators, the output structure of each data table operator contains id (big), name (Varchar), and age (big) fields, and the statement fragment is parsed by combining the dependent data source information, and then the corresponding part in the template is replaced, so that the data table operator can be parsed into the executable statement as shown in fig. 6, where fig. 6 is a schematic diagram of the executable statement of the data table operator according to the embodiment of the present invention.
For a system operator and a target table operator, input field information is provided by an upstream operator node, and an output operator is determined jointly according to self parameter configuration.
When the system operator is analyzed, a word nesting mode is used, and the preset executable statement template of the system operator is as follows: INSERT INTO $ { table _ close } $ { select _ status };
wherein, $ { table _ close } and $ { SELECT _ status } are parts that need to be replaced with executable statement fragments, "$ { table _ close }" is a table name placeholder, and "$ { SELECT _ status } is in the list _ close } and" $ { SELECT _ status } is "through a template statement" SELECT … FROM … ", and multiple system operators realize statement concatenation in a forward one-level superposition in a connection direction.
FOR example, fig. 7 is an analytic schematic diagram OF an alternative operator model topology according to an embodiment OF the present invention, where the Join operator in fig. 7 is a flow dimension Join operator, and the upper layer OF the Join operator is a flow table operator and a dimension table operator, a template statement "SELECT … FROM …" OF the Join operator may be represented AS a statement "SELECT id, name, age, extra _ id, id1, name1 FROM di _ source Join v _ table FOR term SYSTEM _ TIME AS program () ON extra _ id = id1", and a FILTER operator is located downstream, and the template statement "SELECT … FROM …" OF the Join operator is used AS a statement input table OF FILTER, such AS: "SELECT t1. > FROM (SELECT id, name, age, extra _ id, id1, name1 FROM di _ source JOIN csv _ table FOR SYSTEM _ TIME AS OF program () ON extra _ id = id 1) t1 WHERE t1 is t1.Id >18". And finally, replacing the packaged and nested statements into template statements to obtain standard, standard and executable FlinkSQL statements, namely executing the statements.
In an optional embodiment, after performing forward parsing on the operator model topological graph from the data table operator to obtain an executable statement set corresponding to the operator model topological graph, the method further includes: and sending the executable statement set to a target client to execute a target task through the target client, wherein the operator model topological graph configured in the visualization page is a topological graph for executing the target task.
In this embodiment, the executable statement set generated after the operator analysis is further processed by the Flink official client to generate a JobGraph flow chart, and then the job parameters of the task (corresponding to the target task) are configured according to the load information of the current Flink cluster, so that the task can be submitted and run normally and efficiently.
In an optional embodiment, before obtaining the operator model topology map, the method further comprises: generating a data table operator according to the pre-configuration information of the data table; generating a target table operator according to the pre-configuration information of the target table; and in response to dragging and connecting operations, dragging the data table operator, the target table operator and the system operator to the visual page, and connecting to obtain an operator model topological graph.
In this embodiment, a user configures a data table and a destination table operator in a visualization page, the data table serves as an input class operator, data source and data structure definitions are provided, and field information of the definitions is processed as an input data stream by a subsequent data operation operator. Fig. 8 is a schematic diagram of an alternative configuration data table operator according to an embodiment of the present invention, and as shown in fig. 8, a user defines information such as name, type, description, and the like of a JSON field in a data table structure according to a message format (JSON) in a data source. Among them, the data source types are pulled down for alternatives RabbitMQ, kafka, CSV, mysql, etc. And after the data table configuration is completed, generating corresponding data table operators, and dragging the operators into the data source of the model when the operators are arranged.
Fig. 9 is a schematic diagram of an alternative configuration destination table operator according to an embodiment of the present invention, and as shown in fig. 9, the destination table is used as a final operator node of the model and is also a unique exit of the model data. Thus, the data table does not require a user-defined data structure that depends on the field information passed from its upstream node, so that the user-defined destination table requires an explicit data source. And after the configuration of the target table operator is finished, generating a corresponding target table operator, and dragging the target table operator into a unique outlet as model data when the operator is arranged.
Fig. 10 is a schematic diagram of an optional configuration operator model topological graph according to an embodiment of the present invention, after a user configures a data table and a destination table, a corresponding operator node (a data table operator and a destination table operator) is automatically generated on the left side of a visualization page, then the data table and the destination table are dragged into a canvas editing area directly in a dragging and pulling manner, and a system operator is selected to be dragged into the canvas editing area as well, each operator is directly connected in a canvas according to a dependency relationship between the operators, and then an attribute of the corresponding system operator is configured according to a service requirement, so that a model logic of the entire operator model topological graph can finally satisfy a target task.
Through the steps, a user can drag out the operator model topological graph meeting the business requirements on the visual page easily, then click to operate, the front end can send the operator model topological graph to the back end service, and the back end service can adopt a strategy of 'reverse verification and forward analysis' to verify and analyze the operator model topological graph.
It is to be understood that the above-described embodiments are only a few, but not all, embodiments of the present invention.
The present invention is specifically illustrated below with reference to examples:
FIG. 11 is a general flowchart of real-time task development according to an embodiment of the present invention, as shown in FIG. 11, including:
step 1102: visualizing a configuration data table and a target table operator;
step 1104: visually configuring a topological graph of the operator model; the method comprises the steps of utilizing an operator of a built-in system and a user-defined operator, dragging the operator to a canvas editing area, and finally connecting into an directed acyclic topological graph which takes a target table operator as a leaf node, namely an operator model topological graph through connecting line arrangement models.
Step 1106: analyzing the operator model topological graph to generate an SQL statement set; the operator model topological graph which is configured can be directly operated by one key, the system can check the connection reasonability of the topological graph and analyze corresponding operator nodes according to a prefabricated Fink SQL template, and finally an executable and standardized Flink SQL statement is generated.
Step 1108: the Flink SQL statement is submitted to perform the real-time task.
Through the description of the foregoing embodiments, it is clear to those skilled in the art that the method according to the foregoing embodiments may be implemented by software plus a necessary general hardware platform, and certainly may also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention or portions thereof contributing to the prior art may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (which may be a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
In this embodiment, an apparatus for generating an executable statement is further provided, and fig. 12 is a block diagram of a structure of an apparatus for generating an executable statement according to an embodiment of the present invention, as shown in fig. 12, the apparatus includes:
an obtaining module 1202, configured to obtain an operator model topological graph, where the operator model topological graph is a topological graph configured in a visualization page, and the operator model topological graph includes: a data table operator, a target table operator, and a system operator;
a checking module 1204, configured to perform reverse checking on the operator model topological graph from the destination table operator, where the reverse checking includes connectivity checking and validity checking;
an analyzing module 1206, configured to perform forward analysis on the operator model topological graph from the data table operator to obtain an executable statement set corresponding to the operator model topological graph, when the operator model topological graph passes the reverse check.
In an optional embodiment, the check module is further configured to traverse operators in the operator model topological graph from the destination table operator, and detect whether an isolated operator exists in the operator model topological graph; and under the condition that the isolated operator does not exist in the operator model topological graph, determining that the operator model topological graph passes the connectivity check, otherwise, determining that the operator model topological graph does not pass the connectivity check.
In an optional embodiment, the check module is further configured to traverse the operators in the operator model topology map step by step from the target table operator; determining that the isolated operator does not exist in the operator model topological graph under the condition of traversing all operators in the operator model topological graph; otherwise, determining the operators which are not traversed in the operator model topological graph as the isolated operators.
In an optional embodiment, the check module is further configured to traverse operators in the operator model topology map from the destination table operator; under the condition of traversing to the jth operator of the ith level in the operator model topological graph, acquiring one or more operators which are in directed connection with the jth operator in the (i + 1) th level; detecting whether preset conditions are met between the jth operator and the one or more operators; determining that the jth operator passes the validity check under the condition that the preset condition is met, otherwise determining that the jth operator does not pass the validity check; wherein, the connection direction of the jth operator and the one or more operators is that the jth operator points to the one or more operators, and i is greater than or equal to 1,j and is greater than or equal to 1.
In an optional embodiment, the checking module is further configured to obtain an operator type of the jth operator and an operator type of the one or more operators; determining that a preset condition is met between the jth operator and the one or more operators if the operator type of the jth operator and the operator type of the one or more operators meet a preset operator type condition; otherwise, determining that the preset condition is not met between the jth operator and the one or more operators.
In an optional embodiment, the parsing module is further configured to parse each operator in the model topological graph from the data table operator step by step to obtain an executable statement corresponding to each operator, where the executable statement set includes the executable statement corresponding to each operator in the model topological graph.
In an optional embodiment, the parsing module is further configured to obtain an executable statement template corresponding to the mth operator; parsing target information of the mth operator into an executable statement fragment of the mth operator, wherein the target information includes at least one of: input fields, output fields, configuration data; and replacing the corresponding part in the executable statement template with the executable statement segment to obtain the executable statement corresponding to the mth operator.
In an optional embodiment, the apparatus is further configured to, after forward parsing is performed on the operator model topological graph from the data table operator to obtain an executable statement set corresponding to the operator model topological graph, send the executable statement set to a target client, so as to execute a target task through the target client, where the operator model topological graph configured in the visualization page is a topological graph used for executing the target task.
In an optional embodiment, the apparatus is further configured to, before obtaining the operator model topological graph, generate the data table operator according to preconfigured information of a data table; generating a target table operator according to the pre-configuration information of the target table; and in response to dragging and connecting operations, dragging the data table operator, the target table operator and the system operator to the visual page, and connecting to obtain an operator model topological graph.
It should be noted that, the above modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the modules are all positioned in the same processor; alternatively, the modules are respectively located in different processors in any combination.
Embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the steps of any of the above-mentioned method embodiments when executed.
In an exemplary embodiment, the computer readable storage medium may include, but is not limited to: various media capable of storing computer programs, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Embodiments of the present invention also provide an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the above method embodiments.
In an exemplary embodiment, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary embodiments, and details of this embodiment are not repeated herein.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented in a general purpose computing device, they may be centralized in a single computing device or distributed across a network of multiple computing devices, and they may be implemented in program code that is executable by a computing device, such that they may be stored in a memory device and executed by a computing device, and in some cases, the steps shown or described may be executed in an order different from that shown or described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps therein may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of generating an executable statement, comprising:
obtaining an operator model topological graph, wherein the operator model topological graph is a topological graph configured in a visualization page, and the operator model topological graph comprises: a data table operator, a target table operator and a system operator;
performing reverse verification on the operator model topological graph from the target table operator, wherein the reverse verification comprises connectivity verification and validity verification;
and under the condition that the operator model topological graph passes the reverse check, carrying out forward analysis on the operator model topological graph from the data table operator to obtain an executable statement set corresponding to the operator model topological graph.
2. The method of claim 1, wherein said inverse checking said operator model topology map starting from said destination table operator comprises:
traversing operators in the operator model topological graph from the target table operator, and detecting whether isolated operators exist in the operator model topological graph or not;
and under the condition that the isolated operator does not exist in the operator model topological graph, determining that the operator model topological graph passes the connectivity check, otherwise, determining that the operator model topological graph does not pass the connectivity check.
3. The method of claim 2, wherein traversing operators in the operator model topology map from the destination table operator, detecting whether orphaned operators are present in the operator model topology map, comprises:
traversing operators in the operator model topological graph step by step from the target table operator;
determining that the isolated operator does not exist in the operator model topological graph under the condition of traversing all operators in the operator model topological graph;
otherwise, determining the operators which are not traversed in the operator model topological graph as the isolated operators.
4. The method of claim 1, wherein said inverse checking said operator model topology map starting from said destination table operator, further comprises:
traversing operators in the operator model topological graph from the target table operator;
under the condition of traversing to the jth operator of the ith level in the operator model topological graph, acquiring one or more operators which are in directed connection with the jth operator in the (i + 1) th level;
detecting whether preset conditions are met between the jth operator and the one or more operators;
determining that the jth operator passes the validity check under the condition that the preset condition is met, otherwise determining that the jth operator does not pass the validity check;
wherein, the connection direction of the jth operator and the one or more operators is that the jth operator points to the one or more operators, and i is greater than or equal to 1,j and is greater than or equal to 1.
5. The method according to claim 4, wherein detecting whether a preset condition is satisfied between the jth operator and the one or more operators comprises:
obtaining the operator type of the jth operator and the operator types of the one or more operators;
determining that a preset condition is met between the jth operator and the one or more operators if the operator type of the jth operator and the operator type of the one or more operators meet a preset operator type condition;
otherwise, determining that the preset condition is not met between the jth operator and the one or more operators.
6. The method of claim 1, wherein forward parsing the operator model topology from the spreadsheet operator to obtain a set of executable statements corresponding to the operator model topology comprises:
and analyzing each operator in the model topological graph step by step from the data table operator to obtain an executable statement corresponding to each operator, wherein the executable statement set comprises the executable statements corresponding to each operator in the model topological graph.
7. The method of claim 6, wherein analyzing each operator in the model topology graph from the data table operator to obtain an executable statement corresponding to each operator comprises:
analyzing the mth operator in the nth stage in the model topological graph, wherein n is greater than or equal to 1,m and is greater than or equal to 1, and the method comprises the following steps:
acquiring an executable statement template corresponding to the mth operator;
parsing target information of the mth operator into an executable statement fragment of the mth operator, wherein the target information includes at least one of: input fields, output fields, configuration data;
and replacing the corresponding part in the executable statement template with the executable statement segment to obtain the executable statement corresponding to the mth operator.
8. An apparatus for generating an executable statement, comprising:
the acquisition module is used for acquiring an operator model topological graph, wherein the operator model topological graph is a topological graph configured in a visual page, and the operator model topological graph comprises: a data table operator, a target table operator and a system operator;
the checking module is used for carrying out reverse checking on the operator model topological graph from the target table operator, wherein the reverse checking comprises connectivity checking and validity checking;
and the analysis module is used for carrying out forward analysis on the operator model topological graph from the data table operator to obtain an executable statement set corresponding to the operator model topological graph under the condition that the operator model topological graph passes the reverse check.
9. A computer-readable storage medium, in which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method as claimed in any of claims 1 to 7 are implemented when the computer program is executed by the processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211573550.3A CN115934097A (en) | 2022-12-08 | 2022-12-08 | Method, device, storage medium and electronic device for generating executable statement |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211573550.3A CN115934097A (en) | 2022-12-08 | 2022-12-08 | Method, device, storage medium and electronic device for generating executable statement |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115934097A true CN115934097A (en) | 2023-04-07 |
Family
ID=86551765
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211573550.3A Pending CN115934097A (en) | 2022-12-08 | 2022-12-08 | Method, device, storage medium and electronic device for generating executable statement |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115934097A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116954587A (en) * | 2023-09-19 | 2023-10-27 | 中电科大数据研究院有限公司 | Front-end intelligent drag engine and method for establishing data processing flow |
| CN117708304A (en) * | 2024-02-01 | 2024-03-15 | 浙江大华技术股份有限公司 | Database question-answering method, equipment and storage medium |
-
2022
- 2022-12-08 CN CN202211573550.3A patent/CN115934097A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116954587A (en) * | 2023-09-19 | 2023-10-27 | 中电科大数据研究院有限公司 | Front-end intelligent drag engine and method for establishing data processing flow |
| CN116954587B (en) * | 2023-09-19 | 2023-12-19 | 中电科大数据研究院有限公司 | Front-end intelligent drag engine and method for establishing data processing flow |
| CN117708304A (en) * | 2024-02-01 | 2024-03-15 | 浙江大华技术股份有限公司 | Database question-answering method, equipment and storage medium |
| CN117708304B (en) * | 2024-02-01 | 2024-05-28 | 浙江大华技术股份有限公司 | Database question-answering method, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115934097A (en) | Method, device, storage medium and electronic device for generating executable statement | |
| US9576037B2 (en) | Self-analyzing data processing job to determine data quality issues | |
| CN110704518B (en) | Business data processing method and device based on Flink engine | |
| US10915508B2 (en) | Data linking | |
| CN109582289B (en) | Method, system, storage medium and processor for processing rule flow in rule engine | |
| CN104156313A (en) | Method for automatically generating Web service testing case | |
| CN110427188B (en) | Configuration method, device, equipment and storage medium of single-test assertion program | |
| Gómez-Martínez et al. | ArgoSPE: Model-based software performance engineering | |
| CN113986241A (en) | Configuration method and device of business rules based on knowledge graph | |
| CN115794106A (en) | Method and system for analyzing configuration of binary protocol data of rail transit | |
| CN111104181A (en) | Webpage data filling system for visually editing task flow | |
| CN108629124B (en) | Method for automatically generating simulation parameter data based on active graph path | |
| CN106909435B (en) | Method and device for analyzing command line of network security equipment | |
| CN111984826B (en) | XML-based data automatic warehousing method, system, device and storage medium | |
| CN113703739A (en) | Cross-language fusion computing method, system and terminal based on omiga engine | |
| CN110188432B (en) | System architecture verification method, electronic device and computer-readable storage medium | |
| CN111767161A (en) | Remote calling depth recognition method and device, computer equipment and readable storage medium | |
| CN111193631B (en) | Information processing method, system, and computer-readable storage medium | |
| CN112632082B (en) | Method and device for creating Flink job | |
| CN111159203B (en) | Data association analysis method, platform, electronic equipment and storage medium | |
| CN113591448A (en) | Report generation method and device and storage medium | |
| CN112015372B (en) | Heterogeneous service arranging method, processing method and device and electronic equipment | |
| CN113885880A (en) | Multi-rule matching method, device and equipment | |
| CN115599881A (en) | Workflow creation method, system, electronic device, and computer-readable storage medium | |
| CN113515285A (en) | Method and device for generating real-time calculation logic data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |