CN113515285A - Method and device for generating real-time calculation logic data - Google Patents
Method and device for generating real-time calculation logic data Download PDFInfo
- Publication number
- CN113515285A CN113515285A CN202010279967.3A CN202010279967A CN113515285A CN 113515285 A CN113515285 A CN 113515285A CN 202010279967 A CN202010279967 A CN 202010279967A CN 113515285 A CN113515285 A CN 113515285A
- Authority
- CN
- China
- Prior art keywords
- task
- generate
- tasks
- syntax tree
- abstract syntax
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000004364 calculation method Methods 0.000 title claims abstract description 21
- 238000012545 processing Methods 0.000 claims abstract description 77
- 238000013507 mapping Methods 0.000 claims description 67
- 230000002776 aggregation Effects 0.000 claims description 31
- 238000004220 aggregation Methods 0.000 claims description 31
- 230000006870 function Effects 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 9
- 238000001914 filtration Methods 0.000 claims description 8
- 238000004806 packaging method and process Methods 0.000 claims description 7
- 230000001131 transforming effect Effects 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 9
- 230000007547 defect Effects 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 21
- 239000012634 fragment Substances 0.000 description 11
- 238000004891 communication Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 238000006243 chemical reaction Methods 0.000 description 5
- 238000013515 script Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
- G06F8/42—Syntactic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/51—Source to source
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a device for generating real-time computation logic data, and relates to the technical field of computers. One embodiment of the method comprises: converting the offline data processing logic into an abstract syntax tree by using a syntax analyzer; analyzing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic; converting a first task in the first task set to generate a second task set; and generating real-time calculation logic data according to the second task in the second task set. The embodiment solves the technical defects of high cost caused by the fact that the grammar used by an off-line engine and the compiling grammar used by a real-time calculation engine are different greatly and the processing of full scene data and the expansion of services cannot be realized in the prior art, and further achieves the technical effect of improving the efficiency of generating real-time calculation logic data.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a method and a device for generating real-time computation logic data.
Background
Big data processing technology includes offline data processing and real-time data processing, and generally, the offline data processing is based on an offline engine, while the real-time data processing is based on a real-time computing engine, but the offline engine and the real-time computing engine adopt different writing languages. In the prior art, the work of converting offline data into real-time data is usually handled by database management engineers.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:
1. for real-time data, it is difficult to achieve fast logic multiplexing and service expansion;
2. because the grammar used by the off-line engine and the compiling language used by the real-time computing engine have large difference and are difficult to be converted mutually, the prior art can not process the whole scene data.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for generating real-time computation logic data, which can solve the technical defect of high cost in the prior art that processing and service expansion of full scene data cannot be performed due to a large difference between a syntax used by an offline engine and a syntax compiled by a real-time computation engine, thereby achieving a technical effect of improving the efficiency of generating real-time computation logic data.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a method of generating real-time computation logic data, including:
converting the offline data processing logic into an abstract syntax tree by using a syntax analyzer;
analyzing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic;
converting a first task in the first task set to generate a second task set;
and generating real-time calculation logic data according to the second task in the second task set.
Optionally, before converting the offline data processing logic into the abstract syntax tree, using the parser, the method includes:
and utilizing an offline engine to perform grammar check on the offline data processing logic, and determining that the offline data processing logic meets a preset grammar specification.
Optionally, parsing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic includes:
judging whether keywords corresponding to set basic operation exist in the abstract syntax tree or not; wherein, the keywords corresponding to the set basic operation comprise: union, intersection or difference;
if yes, the abstract syntax tree is divided by the keywords, and the divided result is determined as a first task; determining a set of the first tasks as a first set of tasks.
Optionally, parsing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic includes:
judging whether the abstract syntax tree has associated nodes or not;
if the associated node exists, determining the child node of the associated node as a first task; determining a set of the first tasks as a first set of tasks.
Optionally, the determining whether there is an associated node in the abstract syntax tree includes:
judging whether a sub-query grammar exists in the abstract grammar tree or not;
if so, converting the abstract syntax tree corresponding to the sub-query syntax into the abstract syntax tree corresponding to the connection operation syntax containing the associated nodes according to the semantic rule.
Optionally, converting a first task in the first task set to generate a second task set, including:
mapping a first task in the first task set according to a language corresponding to the real-time computation logic data to generate a first intermediate task;
performing aggregation primitive mapping on the first intermediate task to generate a second intermediate task;
and performing filtering condition mapping on the second intermediate task to generate a second task set.
Optionally, mapping the first task in the first task set according to the language corresponding to the real-time computation logic data to generate a first intermediate task, including:
reading database mode definition language information in an off-line engine table;
reading a message body abstract in a body mode in a message queue;
establishing an entity relation mapping table according to the database mode definition language information and the message body abstract;
and mapping the first task in the first task set according to the language corresponding to the real-time computation logic data according to the entity relationship mapping table to generate a first intermediate task.
Optionally, performing aggregation primitive mapping on the first intermediate task to generate a second intermediate task, including:
identifying an aggregation node in the first intermediate task;
extracting leaf nodes corresponding to the identifier nodes from the aggregation nodes;
determining an aggregation operation type corresponding to the leaf node;
and performing aggregation primitive mapping on the first intermediate task according to the aggregation operation type to generate a second intermediate task.
Optionally, performing filtering condition mapping on the second intermediate task to generate a second task set, including:
identifying a query specification node in the second intermediate task;
identifying a logic expression corresponding to a query condition according to the Boolean expression in the query description node;
and mapping the logic expression in a primitive function of a filter to generate a second task set.
Optionally, before generating the real-time computation logic data according to the second task in the second task set, the method includes:
completing the frame code of the second task;
wherein the completion operation includes at least one of: creating a message data source environment, creating a data stream (DataStream), completing the Flink environment configuration, and completing the task execution function.
Optionally, generating real-time computation logic data according to a second task in the second task set includes:
performing task merging on a second task in the second task set;
and compiling and packaging the second task which is subjected to the task combination to generate real-time calculation logic data.
Optionally, task merging the second task in the second task set includes:
determining an incidence relation between second tasks in the second task set;
if the correlation relationship is a union set, an intersection set or a difference set, carrying out the operation of taking the union set, the intersection set or the difference set on the second task;
and if the association relationship is a mapping relationship, performing task merging on the second tasks in the second task set according to the mapping relationship.
Optionally, if the association relationship is a union, performing union on the second task includes:
if the incidence relation is a union, judging whether the keywords for reserving the repeated lines exist in the abstract syntax tree or not;
and if not, taking a union set for the second task, and removing repeated lines existing after the union.
According to still another aspect of the embodiments of the present invention, there is provided an apparatus for generating real-time computation logic data, including:
the abstract syntax tree generating module is used for converting the offline data processing logic into an abstract syntax tree by using the syntax analyzer;
a first task set generating module, configured to parse the abstract syntax tree and generate a first task set corresponding to the offline data processing logic;
the second task set generating module is used for converting the first tasks in the first task set to generate a second task set;
and the real-time calculation logic data generation module is used for generating real-time calculation logic data according to the second task in the second task set.
Optionally, parsing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic includes:
judging whether keywords corresponding to set basic operation exist in the abstract syntax tree or not; wherein, the keywords corresponding to the set basic operation comprise: union, intersection or difference;
if yes, the abstract syntax tree is divided by the keywords, and the divided result is determined as a first task; determining a set of the first tasks as a first set of tasks.
Optionally, parsing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic includes:
judging whether the abstract syntax tree has associated nodes or not;
if the associated node exists, determining the child node of the associated node as a first task; determining a set of the first tasks as a first set of tasks.
According to another aspect of the embodiments of the present invention, there is provided an electronic device for generating real-time computation logic data, including:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a method of generating real-time computational logic data as provided by the present invention.
According to a further aspect of embodiments of the present invention, there is provided a computer readable medium, on which a computer program is stored, which when executed by a processor implements a method of generating real-time computation logic data according to embodiments of the present invention.
One embodiment of the above invention has the following advantages or benefits:
according to the method and the device, the off-line data processing logic is conveniently divided into a plurality of tasks which can be executed independently through a technical means of generating the abstract syntax tree, the subsequent mapping is facilitated to be the tasks corresponding to the real-time calculation logic data, and the visual reflection of the mapping process is facilitated through the generation of the abstract syntax tree.
By means of the technical means of analyzing the abstract syntax tree to generate the first task set, when the service is expanded, only the expanded part needs to be analyzed to generate the first task, and the technical defect of high cost in the prior art of expanding the service is overcome.
According to the method and the device, the first tasks in the first task set are converted to generate the second task set, namely, the offline data processing logic is divided into small segments, and then mapping is carried out, so that the technical defects that in the prior art, the whole scene data processing cannot be realized due to the fact that grammar adopted by an offline engine and compiling language adopted by a real-time computing engine are different greatly and mutual conversion is difficult to realize are solved, and the technical effect of improving the efficiency of generating the real-time computing logic data is achieved.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of a main flow of a method of generating real-time computation logic data according to an embodiment of the invention;
FIG. 2 is a diagram of an abstract syntax tree in a specific embodiment of the present invention;
FIG. 3 is a diagram of a first embodiment of generating a first task set;
FIG. 4 is a diagram of a second embodiment of generating a first task set;
FIG. 5 is a schematic diagram illustrating a detailed flow of a method for generating real-time computation logic data according to an embodiment of the invention;
FIG. 6 is a schematic diagram of the major modules of an apparatus for generating real-time computation logic data according to an embodiment of the present invention;
FIG. 7 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 8 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a method for generating real-time computation logic data according to an embodiment of the present invention, as shown in fig. 1, including:
step S101, converting offline data processing logic into an Abstract Syntax Tree (AST) by using a syntax analyzer;
step S102, analyzing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic;
step S103, converting the first task in the first task set to generate a second task set;
and step S104, generating real-time calculation logic data according to the second task in the second task set.
And converting the offline data processing logic into an abstract syntax tree, namely performing abstract analysis on the offline data, and converting the structured format of SQL (structured query language) into the tree format of AST (AST).
Wherein the parser may be an ANTLR open source framework.
For example, some examples of SQL scripts are:
SELECT SKU,SUM(PRICE)FROM DEMO_TABLE WHERE PIN＝'1'AND STATUS＝0GROUP BY SKU
the conversion of the SQL script into an AST structure is shown in fig. 2.
Fig. 2 is a schematic diagram of an abstract syntax tree in an embodiment of the present invention. Fig. 2 is a diagram for showing the structure of an abstract syntax tree, and the concrete meaning of each node in the abstract syntax tree is not critical to the diagram.
By the technical means of generating the abstract syntax tree, the subsequent operations such as depth-first traversal, breadth-first traversal and the like can be conveniently performed on the syntax tree, the offline data processing logic can be conveniently divided into a plurality of tasks which can be executed independently, the subsequent mapping is favorably realized as the tasks corresponding to the real-time calculation logic data, and the mapping process can be favorably and visually reflected by generating the abstract syntax tree.
By means of the technical means of analyzing the abstract syntax tree to generate the first task set, when the service is expanded, only the expanded part needs to be analyzed to generate the first task, and the technical defect of high cost in the prior art of expanding the service is overcome.
According to the method and the device, the first tasks in the first task set are converted to generate the second task set, namely, the offline data processing logic is divided into small segments, and then mapping is carried out, so that the technical defects that in the prior art, the processing of full scene data cannot be realized due to the fact that grammar adopted by an offline engine and compiling language adopted by a real-time computing engine are different greatly and mutual conversion is difficult to realize are solved, and the technical effect of improving the efficiency of generating the real-time computing logic data is achieved.
Optionally, before converting the offline data processing logic into the abstract syntax tree, using the parser, the method includes:
and utilizing an off-line engine (such as a Hive engine) to carry out syntax check on the off-line data processing logic, and determining that the off-line data processing logic meets a preset syntax specification.
In an optional embodiment of the present application, by performing syntax checking on the offline data processing logic, for example, performing syntax checking on the SQL script, it is ensured that the SQL script can run and meet a standard syntax specification, specifically, the implementation method includes: calling an explain command of the Hive engine, judging whether a returned result is normal, and if the returned result is normal, determining that the SQL script meets a preset syntax specification.
Because the offline data processing logic is generally higher in complexity and cannot completely realize conversion through a single task, in the embodiment of the present application, the offline data processing logic to be converted is disassembled into a plurality of small fragments, each fragment can be operated independently as an independent task, that is, each small fragment corresponds to a first task, and a set formed by the first tasks is referred to as a first task set.
Specifically, parsing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic includes the following three embodiments:
the first embodiment is as follows:
parsing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic, including:
judging whether keywords corresponding to set basic operation exist in the abstract syntax tree or not; wherein, the keywords corresponding to the set basic operation comprise: UNION (UNION), Intersection (INTERSECT) or difference (EXCEPT);
if yes, the abstract syntax tree is divided by the keywords, and the divided result is determined as a first task; determining a set of the first tasks as a first set of tasks.
FIG. 3 is a diagram of a first embodiment of generating a first task set. As shown in fig. 3:
starting from the root node, keywords in the abstract syntax tree are traversed on a breadth first principle (queryTerm, which is a keyword defined by ANTLR and indicates that the sub-tree below this node is the query logic). If a UNION (UNION) key is found to exist, the left and right subtrees of the node are split into two first tasks.
If the UNION (UNION) key is followed by an ALL leaf node, i.e., the UNION node is followed by a setQuantifier, which is a leaf node of ALL (where setQuantifier is an ANTLR-defined key that represents the combined form of UNION ALL in the SQL syntax). The UNION set (UNION) keyword can be reserved following the information of ALL leaf nodes, so that the subsequent mapped second task can be conveniently merged, and further, whether the second task directly takes the intersection or takes the intersection after the duplication is removed can be distinguished.
The reason for traversing the abstract syntax tree according to the breadth-first principle is that the union operation usually occurs in the first layer below the root node, so that the candidate tasks can be quickly split.
Example two:
parsing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic, including:
judging whether a correlation node exists in the abstract syntax tree or not;
if the associated node exists, determining the child node of the associated node as a first task; determining a set of the first tasks as a first set of tasks.
FIG. 4 is a diagram of a second embodiment of generating a first task set. As shown in fig. 4:
searching for related (including relationship keywords) nodes according to a depth-first principle, in this embodiment, a relationship node, a relationship primary node, and a joinrelationship node;
the two subtrees relational Primary and join relationship are divided into two first tasks respectively, meanwhile, the type (including LEFT connection LEFT, RIGHT connection RIGHT, internal connection INNER, external connection OUTER and the like) of the connection relation (join type) and join criterion information (specific expression for positioning the association condition) are reserved, and the subtrees below the join criterion are used as the connection conditions for merging the mapped second tasks. By adopting the depth-first traversal method, the efficiency of identifying the query unit of the SQL can be improved.
Example three:
parsing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic, further comprising:
before determining whether there is an associated node in the abstract syntax tree, the method includes:
judging whether sub-query grammars (including IN and EXISTS) exist IN the abstract syntax tree;
if so, converting the abstract syntax tree corresponding to the sub-query syntax into the abstract syntax tree corresponding to the connection operation syntax containing the associated nodes according to the semantic rule. Then, the first task set is generated according to the steps in the second embodiment.
Specifically, the SQL code segments corresponding to the sub-query syntax are:
…WHERE A IN(SELECT A FROM AAAA)…
converting the abstract syntax tree into a code segment corresponding to the abstract syntax tree and corresponding to the connection operation syntax containing the associated nodes, wherein the code segment corresponding to the abstract syntax tree is as follows:
…AS X LEFT JOIN(SELECT A FROM AAAA)AS Y ON X.A＝Y.A WHERE Y.A IS NOT NULL…
and converting the first task in the first task set to generate a second task set, namely converting the first task in the SQL language into a second task corresponding to the second task set corresponding to the real-time computation logic data. By means of the technical means of mapping the first task into the second task, the technical effect that the generated second task is suitable for the target real-time computing system can be achieved.
Specifically, the method comprises the following steps:
mapping a first task in the first task set according to a language corresponding to the real-time computation logic data to generate a first intermediate task;
performing aggregation primitive mapping on the first intermediate task to generate a second intermediate task;
and performing filtering condition mapping on the second intermediate task to generate a second task set.
Optionally, mapping the first task in the first task set according to the language corresponding to the real-time computation logic data to generate a first intermediate task, including:
reading database schema definition language (DDL) information in an offline engine table;
reading a message body abstract (Schema) in a body Schema (Topic) in a Message Queue (MQ);
establishing an entity relation mapping table according to the database mode definition language information and the message body abstract;
and mapping the first task in the first task set according to the language corresponding to the real-time computation logic data according to the entity relationship mapping table to generate a first intermediate task.
Through the technical means for generating the first intermediate task, the technical effect of carrying out field alignment on the heterogeneous data source, namely the technical effect of completing the code of the first task can be realized.
For example, the data source of the offline data processing logic is the Hive table, while the data source of the real-time engine is the message queue (e.g., Kafka), so that it is necessary to field align the heterogeneous data sources. The specific implementation method comprises the steps of constructing an entity relation mapping table, reading the message body summary in the main body mode in the read message queue of the Hive table, and then establishing a one-to-one mapping relation between the data source of the offline data processing logic and the data source of the real-time engine. Upon parsing the query (select) field of the offline data processing logic, the corresponding componentization code is generated.
For example, the SQL code fragments are:
SELECT NAME,AGE,SEX……
a general message body analysis function is predefined to be parse (), a message text is converted into a JAVA MAP object, and the generated real-time engine corresponding code (JAVA code) fragments are as follows:
Map<String,Object>map＝parse(messageBody)；
String name＝map.get("NAME")；
String age＝map.get("AGE")；
String sex＝map.get("SEX")；
optionally, performing aggregation primitive mapping on the first intermediate task to generate a second intermediate task, including:
identifying an aggregation node in the first intermediate task;
extracting leaf nodes corresponding to identifier (identifier) nodes from the aggregation nodes;
determining an aggregation operation type corresponding to the leaf node;
and performing aggregation primitive mapping on the first intermediate task according to the aggregation operation type to generate a second intermediate task.
By identifying the aggregation node, extracting the leaf node name below the identifier node, and searching forward at the same time, a specific aggregation operation type (including SUM, minimum MIN, maximum MAX, etc.) can be obtained, and then mapped into a primitive function corresponding to the real-time computing platform Flink.
For example, the corresponding SQL code fragment is:
SELECT SKU,SUM(PRICE)FROM DEMO_TABLE GROUP BY SKU；
converting the current code into a chained combination form of a key () function and a sum () function of the Flink, and generating a real-time engine corresponding code (JAVA code) fragment as follows:
....dataStreamSource.keyBy("SKU").sum("PRICE")…
optionally, performing filtering condition mapping on the second intermediate task to generate a second task set, including:
identifying a query specification (querySpecification) node in the second intermediate task;
identifying a logic expression corresponding to a query condition according to a Boolean expression (bootean expression) positioned behind a WHERE leaf node in the query description node;
according to the comparison operator (compare operator) node in the Boolean expression as a division mark, identifying the logic expression of the query condition, splicing the leaf nodes together to generate a corresponding JAVA expression, and then nesting the JAVA expression in a filter primitive function to generate a second task set.
The offline data language automatically performs compatible processing of character String types, so that String types can be uniformly used when mapping into JAVA code fragments.
For example, the corresponding SQL code fragment:
SELECT SKU FROM DEMO_TABLE WHERE PIN＝'1'AND STATUS＝0
converting into a filter () function of Flink, and generating the JAVA code fragments as follows:
....
dataStreamSource.filter(new FilterFunction<String>(){
publicboolean filter(Map<String,String>values)throws Exception{
returnvalues.get("PIN").equals ("1")&&values.get("STATUS").equals("0")；}})....
in order to convert the second task into a JAVA program that becomes executable, before generating the real-time computation logic data according to the second task in the second task set, optionally, the method includes:
according to the context information of a second task, completing the frame code of the second task;
wherein the completion operation includes at least one of: creating a message data source environment, creating a data stream (DataStream), completing the Flink environment configuration, and completing the task execution function.
For example, the code for performing the code completion is as follows:
StreamExecutionEnvironment environment＝StreamExecutionEnvironment.getExecutionEnvironment()；
environment.enableCheckpointing(10000,CheckpointingMode.AT_LEAST_ONCE)；
environment.setRestartStrategy(RestartStrategies.fixedDelayRestart(params[0],params[1]))；
Properties properties＝FlinkJDQ3Auth.getJDQ3AuthClient().getConsumerAuthProperties(params[2],params[3],params[4],params[5],params[6])；
FlinkKafkaConsumer010<String>consumer＝new FlinkKafkaConsumer010<String>(params[7],new SimpleStringSchema(),properties)；
consumer.setStartFromEarliest()；
DataStreamSource<String>dataStreamSource＝environment.addSource(consumer)；
// code fragments generated as described above
//……
Configuration configuration＝new Configuration()；
configuration.setString("URL",params[2])；
environment.getConfig().setGlobalJobParameters(configuration)；
environment.execute("flink")；
To implement the offline data processing logic in the real-time computing logic platform, the second tasks in the second task set may be merged. Specifically, generating real-time computation logic data according to a second task in the second task set includes:
performing task merging on a second task in the second task set;
and compiling and packaging the second task which is subjected to the task combination to generate real-time calculation logic data.
Optionally, task merging the second task in the second task set includes:
determining an incidence relation between second tasks in the second task set;
if the correlation relationship is a union set, an intersection set or a difference set, executing a set operation, and specifically performing an operation of taking the union set, the intersection set or the difference set on the second task;
and if the association relationship is a mapping relationship, executing association operation, and specifically performing task merging on the second tasks in the second task set according to the mapping relationship.
Optionally, if the association relationship is a union, performing union on the second task includes:
if the incidence relation is a union, judging whether the keywords for reserving the repeated lines exist in the abstract syntax tree or not;
and if not, taking a union set for the second task, and removing repeated lines existing after the union.
Specifically, for the two second tasks a and B of the collective operation, the data after a and B processing is directly sent to the general aggregation task X in a summary manner. If the fields of A and B are consistent, the processing of X only needs to do intersection or union. And if the UNION node which is analyzed does not follow the ALL keyword when the abstract syntax tree is analyzed, the deduplication logic needs to be added.
For two second tasks C and D of the association operation, the association type and the association condition obtained by analyzing the abstract syntax tree may be input to the second task D by using the output of the second task C as a new message data source, and the association condition is automatically generated into the association code logic in the second task D by using a map () function.
And task merging is carried out on the second tasks through the relation among the second tasks, so that the generated real-time calculation logic data is more convenient to query.
Compiling and packaging the second task which completes the task combination to generate real-time computation logic data, wherein the real-time computation logic data comprises compiling the JAVA code which completes the task combination by using a Maven tool and packaging the JAVA code into a Fat Jar packet to generate a JAVA program which can directly run on a flight engine.
Optionally, two plugins, namely Maven-resources-plugin and Maven-assembly-plugin, of Maven are adopted, wherein the configuration of the assembly file is as follows:
FIG. 5 is a schematic diagram illustrating a detailed flow of a method for generating real-time computation logic data according to an embodiment of the invention;
step S501, SQL syntax check is carried out on the off-line data processing logic;
step S502, analyzing the off-line data processing logic to generate an abstract syntax tree;
step S503, dividing tasks according to the syntax tree to generate a first task set;
step S504, mapping the first task in the first task set according to the language corresponding to the real-time computation logic data to generate a first intermediate task;
step S505, performing aggregation primitive mapping on the first intermediate task to generate a second intermediate task;
step S506, performing filtering condition mapping on the second intermediate task to generate a second task set;
step S507, completing codes of the second task set;
step S508, merging the second tasks completing the code completion;
and step S509, compiling and packaging the merged second task to generate real-time computation logic data.
Fig. 6 is a schematic diagram of main modules of an apparatus 600 for generating real-time computation logic data according to an embodiment of the present invention, including:
an abstract syntax tree generating module 601, configured to convert the offline data processing logic into an abstract syntax tree by using a syntax analyzer;
a first task set generating module 602, configured to parse the abstract syntax tree to generate a first task set corresponding to the offline data processing logic;
a second task set generating module 603, configured to convert a first task in the first task set to generate a second task set;
and a real-time computation logic data generating module 604, configured to generate real-time computation logic data according to a second task in the second task set.
Optionally, before converting the offline data processing logic into the abstract syntax tree, using the parser, the method includes:
and utilizing an offline engine to perform grammar check on the offline data processing logic, and determining that the offline data processing logic meets a preset grammar specification.
Optionally, parsing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic includes:
judging whether keywords corresponding to set basic operation exist in the abstract syntax tree or not; wherein, the keywords corresponding to the set basic operation comprise: union, intersection or difference;
if yes, the abstract syntax tree is divided by the keywords, and the divided result is determined as a first task; determining a set of the first tasks as a first set of tasks.
Optionally, parsing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic includes:
judging whether the abstract syntax tree has associated nodes or not;
if the associated node exists, determining the child node of the associated node as a first task; determining a set of the first tasks as a first set of tasks.
Optionally, the determining whether there is an associated node in the abstract syntax tree includes:
judging whether a sub-query grammar exists in the abstract grammar tree or not;
if so, converting the abstract syntax tree corresponding to the sub-query syntax into the abstract syntax tree corresponding to the connection operation syntax containing the associated nodes according to the semantic rule.
Optionally, converting a first task in the first task set to generate a second task set, including:
mapping a first task in the first task set according to a language corresponding to the real-time computation logic data to generate a first intermediate task;
performing aggregation primitive mapping on the first intermediate task to generate a second intermediate task;
and performing filtering condition mapping on the second intermediate task to generate a second task set.
Optionally, mapping the first task in the first task set according to the language corresponding to the real-time computation logic data to generate a first intermediate task, including:
reading database mode definition language information in an off-line engine table;
reading a message body abstract in a body mode in a message queue;
establishing an entity relation mapping table according to the database mode definition language information and the message body abstract;
and mapping the first task in the first task set according to the language corresponding to the real-time computation logic data according to the entity relationship mapping table to generate a first intermediate task.
Optionally, performing aggregation primitive mapping on the first intermediate task to generate a second intermediate task, including:
identifying an aggregation node in the first intermediate task;
extracting leaf nodes corresponding to the identifier nodes from the aggregation nodes;
determining an aggregation operation type corresponding to the leaf node;
and performing aggregation primitive mapping on the first intermediate task according to the aggregation operation type to generate a second intermediate task.
Optionally, performing filtering condition mapping on the second intermediate task to generate a second task set, including:
identifying a query specification node in the second intermediate task;
identifying a logic expression corresponding to a query condition according to the Boolean expression in the query description node;
and mapping the logic expression in a primitive function of a filter to generate a second task set.
Optionally, before generating the real-time computation logic data according to the second task in the second task set, the method includes:
completing the frame code of the second task;
wherein the completion operation includes at least one of: creating a message data source environment, creating a data stream (DataStream), completing the Flink environment configuration, and completing the task execution function.
Optionally, generating real-time computation logic data according to a second task in the second task set includes:
performing task merging on a second task in the second task set;
and compiling and packaging the second task which is subjected to the task combination to generate real-time calculation logic data.
Optionally, task merging the second task in the second task set includes:
determining an incidence relation between second tasks in the second task set;
if the correlation relationship is a union set, an intersection set or a difference set, carrying out the operation of taking the union set, the intersection set or the difference set on the second task;
and if the association relationship is a mapping relationship, performing task merging on the second tasks in the second task set according to the mapping relationship.
Optionally, if the association relationship is a union, performing union on the second task includes:
if the incidence relation is a union, judging whether the keywords for reserving the repeated lines exist in the abstract syntax tree or not;
and if not, taking a union set for the second task, and removing repeated lines existing after the union.
FIG. 7 illustrates an exemplary system architecture 700 for a method or apparatus for generating real-time computing logic data to which embodiments of the present invention may be applied.
As shown in fig. 7, the system architecture 700 may include terminal devices 701, 702, 703, a network 704, and a server 705. The network 704 serves to provide a medium for communication links between the terminal devices 701, 702, 703 and the server 705. Network 704 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
A user may use the terminal devices 701, 702, 703 to interact with a server 705 over a network 704, to receive or send messages or the like. The terminal devices 701, 702, 703 may have installed thereon various communication client applications, such as a shopping-like application, a web browser application, a search-like application, an instant messaging tool, a mailbox client, social platform software, etc. (by way of example only).
The terminal devices 701, 702, 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 705 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the terminal devices 701, 702, 703. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.
It should be noted that the method for generating real-time computation logic data provided by the embodiment of the present invention is generally executed by the server 705, and accordingly, the device for generating real-time computation logic data is generally disposed in the server 705.
It should be understood that the number of terminal devices, networks, and servers in fig. 7 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 8, shown is a block diagram of a computer system 800 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 8, the computer system 800 includes a Central Processing Unit (CPU)801 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data necessary for the operation of the system 800 are also stored. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.
The following components are connected to the I/O interface 805: an input portion 806 including a keyboard, a mouse, and the like; an output section 807 including a signal such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. A drive 810 is also connected to the I/O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that a computer program read out therefrom is mounted on the storage section 808 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. The computer program executes the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 801.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a sending module, an obtaining module, a determining module, and a first processing module. The names of these modules do not form a limitation on the modules themselves in some cases, and for example, the sending module may also be described as a "module sending a picture acquisition request to a connected server".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise:
converting the offline data processing logic into an abstract syntax tree by using a syntax analyzer;
analyzing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic;
converting a first task in the first task set to generate a second task set;
and generating real-time calculation logic data according to the second task in the second task set.
According to the technical scheme of the embodiment of the invention, the following beneficial effects can be achieved:
according to the method and the device, the offline data processing logic is divided into a plurality of tasks which can be executed independently through a technical means of generating the abstract syntax tree, subsequent mapping is facilitated to be the tasks corresponding to the real-time calculation logic data, and the mapping process is facilitated to be reflected visually through the generation of the abstract syntax tree.
By means of the technical means of analyzing the abstract syntax tree to generate the first task set, when the service is expanded, only the expanded part needs to be analyzed to generate the first task, and the technical defect of high cost in the prior art of expanding the service is overcome.
According to the method and the device, the first tasks in the first task set are converted to generate the second task set, namely, the offline data processing logic is divided into small segments, and then mapping is carried out, so that the technical defects that in the prior art, the whole scene data processing cannot be realized due to the fact that grammar adopted by an offline engine and compiling language adopted by a real-time computing engine are different greatly and mutual conversion is difficult to realize are solved, and the technical effect of improving the efficiency of generating the real-time computing logic data is achieved.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (18)
1. A method of generating real-time computational logic data, comprising:
converting the offline data processing logic into an abstract syntax tree by using a syntax analyzer;
analyzing the abstract syntax tree to generate a first task set corresponding to the offline data processing logic;
converting a first task in the first task set to generate a second task set;
and generating real-time calculation logic data according to the second task in the second task set.
2. The method of claim 1, wherein prior to converting the offline data processing logic into an abstract syntax tree using the parser, comprising:
and utilizing an offline engine to perform grammar check on the offline data processing logic, and determining that the offline data processing logic meets a preset grammar specification.
3. The method of claim 1, wherein parsing the abstract syntax tree to generate a first set of tasks corresponding to the offline data processing logic comprises:
judging whether keywords corresponding to set basic operation exist in the abstract syntax tree or not; wherein, the keywords corresponding to the set basic operation comprise: union, intersection or difference;
if yes, the abstract syntax tree is divided by the keywords, and the divided result is determined as a first task; determining a set of the first tasks as a first set of tasks.
4. The method of claim 1, wherein parsing the abstract syntax tree to generate a first set of tasks corresponding to the offline data processing logic comprises:
judging whether the abstract syntax tree has associated nodes or not;
if the associated node exists, determining the child node of the associated node as a first task; determining a set of the first tasks as a first set of tasks.
5. The method of claim 4, wherein determining whether the associated node exists in the abstract syntax tree comprises:
judging whether a sub-query grammar exists in the abstract grammar tree or not;
if so, converting the abstract syntax tree corresponding to the sub-query syntax into an abstract syntax tree corresponding to a connection operation syntax containing the associated nodes according to the semantic rule.
6. The method of any of claims 1-5, wherein transforming a first task of the first set of tasks to generate a second set of tasks comprises:
mapping a first task in the first task set according to a language corresponding to the real-time computation logic data to generate a first intermediate task;
performing aggregation primitive mapping on the first intermediate task to generate a second intermediate task;
and performing filtering condition mapping on the second intermediate task to generate a second task set.
7. The method of claim 6, wherein mapping a first task in the first set of tasks according to a language corresponding to the real-time computation logic data to generate a first intermediate task comprises:
reading database mode definition language information in an off-line engine table;
reading a message body abstract in a body mode in a message queue;
establishing an entity relation mapping table according to the database mode definition language information and the message body abstract;
and mapping the first task in the first task set according to the language corresponding to the real-time computation logic data according to the entity relationship mapping table to generate a first intermediate task.
8. The method of claim 6, wherein performing an aggregate primitive mapping on the first intermediate task to generate a second intermediate task comprises:
identifying an aggregation node in the first intermediate task;
extracting leaf nodes corresponding to the identifier nodes from the aggregation nodes;
determining an aggregation operation type corresponding to the leaf node;
and performing aggregation primitive mapping on the first intermediate task according to the aggregation operation type to generate a second intermediate task.
9. The method of claim 8, wherein performing a filter condition mapping on the second intermediate task to generate a second set of tasks comprises:
identifying a query specification node in the second intermediate task;
identifying a logic expression corresponding to a query condition according to the Boolean expression in the query description node;
and mapping the logic expression in a primitive function of a filter to generate a second task set.
10. The method of claim 1, wherein generating real-time computation logic data from a second task of the second set of tasks comprises:
completing the frame code of the second task;
wherein the completion operation includes at least one of: creating a message data source environment, creating a data stream, completing the Flink environment configuration, and completing the task execution function.
11. The method of claim 3, wherein generating real-time computation logic data from a second task of the second set of tasks comprises:
performing task merging on a second task in the second task set;
and compiling and packaging the second task which is subjected to the task combination to generate real-time calculation logic data.
12. The method of claim 11, wherein task merging the second task of the second set of tasks comprises:
determining an incidence relation between second tasks in the second task set;
if the correlation relationship is a union set, an intersection set or a difference set, carrying out the operation of taking the union set, the intersection set or the difference set on the second task;
and if the association relationship is a mapping relationship, performing task merging on the second tasks in the second task set according to the mapping relationship.
13. The method of claim 12, wherein if the association relationship is a union, performing a union on the second task comprises:
if the incidence relation is a union, judging whether the keywords for reserving the repeated lines exist in the abstract syntax tree or not;
and if not, taking a union set for the second task, and removing repeated lines existing after the union.
14. An apparatus for generating real-time computational logic data, comprising:
the abstract syntax tree generating module is used for converting the offline data processing logic into an abstract syntax tree by using the syntax analyzer;
a first task set generating module, configured to parse the abstract syntax tree and generate a first task set corresponding to the offline data processing logic;
the second task set generating module is used for converting the first tasks in the first task set to generate a second task set;
and the real-time calculation logic data generation module is used for generating real-time calculation logic data according to the second task in the second task set.
15. The apparatus of claim 14, wherein parsing the abstract syntax tree to generate a first set of tasks corresponding to the offline data processing logic comprises:
judging whether keywords corresponding to set basic operation exist in the abstract syntax tree or not; wherein, the keywords corresponding to the set basic operation comprise: union, intersection or difference;
if yes, the abstract syntax tree is divided by the keywords, and the divided result is determined as a first task; determining a set of the first tasks as a first set of tasks.
16. The apparatus of claim 14, wherein parsing the abstract syntax tree to generate a first set of tasks corresponding to the offline data processing logic comprises:
judging whether the abstract syntax tree has associated nodes or not;
if the associated node exists, determining the child node of the associated node as a first task; determining a set of the first tasks as a first set of tasks.
17. An electronic device for generating real-time computational logic data, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-13.
18. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-13.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010279967.3A CN113515285A (en) | 2020-04-10 | 2020-04-10 | Method and device for generating real-time calculation logic data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010279967.3A CN113515285A (en) | 2020-04-10 | 2020-04-10 | Method and device for generating real-time calculation logic data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113515285A true CN113515285A (en) | 2021-10-19 |
Family
ID=78060592
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010279967.3A Pending CN113515285A (en) | 2020-04-10 | 2020-04-10 | Method and device for generating real-time calculation logic data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113515285A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116775276A (en) * | 2023-05-10 | 2023-09-19 | 阿里巴巴达摩院(杭州)科技有限公司 | Distributed computing method, device, equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080077598A1 (en) * | 2006-09-27 | 2008-03-27 | Wilmering Timothy J | Querying of distributed databases using neutral ontology model for query front end |
| CN106293892A (en) * | 2015-06-26 | 2017-01-04 | 阿里巴巴集团控股有限公司 | Distributed stream calculates system, method and apparatus |
| CN109189835A (en) * | 2018-08-21 | 2019-01-11 | 北京京东尚科信息技术有限公司 | The method and apparatus of the wide table of data are generated in real time |
| CN109271518A (en) * | 2012-04-28 | 2019-01-25 | 索尼公司 | Method and apparatus for carrying out classification display to micro-blog information |
| CN109684352A (en) * | 2018-12-29 | 2019-04-26 | 江苏满运软件科技有限公司 | Data analysis system, method, storage medium and electronic equipment |
| CN109885584A (en) * | 2019-01-28 | 2019-06-14 | 中科恒运股份有限公司 | The implementation method and terminal device of distributed data analyzing platform |
| US20190266271A1 (en) * | 2018-02-27 | 2019-08-29 | Elasticsearch B.V. | Systems and Methods for Converting and Resolving Structured Queries as Search Queries |
| CN110555032A (en) * | 2019-09-09 | 2019-12-10 | 北京搜狐新媒体信息技术有限公司 | Data blood relationship analysis method and system based on metadata |
-
2020
- 2020-04-10 CN CN202010279967.3A patent/CN113515285A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080077598A1 (en) * | 2006-09-27 | 2008-03-27 | Wilmering Timothy J | Querying of distributed databases using neutral ontology model for query front end |
| CN109271518A (en) * | 2012-04-28 | 2019-01-25 | 索尼公司 | Method and apparatus for carrying out classification display to micro-blog information |
| CN106293892A (en) * | 2015-06-26 | 2017-01-04 | 阿里巴巴集团控股有限公司 | Distributed stream calculates system, method and apparatus |
| US20190266271A1 (en) * | 2018-02-27 | 2019-08-29 | Elasticsearch B.V. | Systems and Methods for Converting and Resolving Structured Queries as Search Queries |
| CN109189835A (en) * | 2018-08-21 | 2019-01-11 | 北京京东尚科信息技术有限公司 | The method and apparatus of the wide table of data are generated in real time |
| CN109684352A (en) * | 2018-12-29 | 2019-04-26 | 江苏满运软件科技有限公司 | Data analysis system, method, storage medium and electronic equipment |
| CN109885584A (en) * | 2019-01-28 | 2019-06-14 | 中科恒运股份有限公司 | The implementation method and terminal device of distributed data analyzing platform |
| CN110555032A (en) * | 2019-09-09 | 2019-12-10 | 北京搜狐新媒体信息技术有限公司 | Data blood relationship analysis method and system based on metadata |
Non-Patent Citations (1)
| Title |
|---|
| 曹祺;: "基于领域语言的图书馆检索语言的优化", 中国高新科技, no. 06, 15 March 2020 (2020-03-15) * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116775276A (en) * | 2023-05-10 | 2023-09-19 | 阿里巴巴达摩院(杭州)科技有限公司 | Distributed computing method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110555030B (en) | SQL sentence processing method and device | |
| CN110704479A (en) | Task processing method and device, electronic equipment and storage medium | |
| US11983640B2 (en) | Generating question templates in a knowledge-graph based question and answer system | |
| WO2018228034A1 (en) | Method and device for generating elastic search request | |
| CN114357276A (en) | Data query method and device, electronic equipment and storage medium | |
| CN113238740B (en) | Code generation method, code generation device, storage medium and electronic device | |
| CN110858202A (en) | Method and device for generating where clause in database query statement | |
| CN110543297A (en) | method and apparatus for generating source code | |
| CN111666293A (en) | Database access method and device | |
| CN112860744A (en) | Business process processing method and device | |
| CN113419789A (en) | Method and device for generating data model script | |
| CN115509522A (en) | Interface arranging method and system for low-code scene and electronic equipment | |
| CN113962597A (en) | Data analysis method and device, electronic equipment and storage medium | |
| CN111178052A (en) | Method and device for constructing robot process automation application | |
| CN112818026A (en) | Data integration method and device | |
| US9081873B1 (en) | Method and system for information retrieval in response to a query | |
| CN112632082B (en) | Method and device for creating Flink job | |
| CN113760961B (en) | Data query method and device | |
| CN108959294B (en) | Method and device for accessing search engine | |
| CN110764769B (en) | Method and device for processing user request | |
| CN111221888A (en) | Big data analysis system and method | |
| CN113515285A (en) | Method and device for generating real-time calculation logic data | |
| CN112579151A (en) | Method and device for generating model file | |
| CN113760240B (en) | Method and device for generating data model | |
| CN111221860A (en) | Mixed query optimization method and device based on big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |