CN116955399A - Unified SQL query method, system and medium based on Calcite - Google Patents
Unified SQL query method, system and medium based on Calcite Download PDFInfo
- Publication number
- CN116955399A CN116955399A CN202310929918.3A CN202310929918A CN116955399A CN 116955399 A CN116955399 A CN 116955399A CN 202310929918 A CN202310929918 A CN 202310929918A CN 116955399 A CN116955399 A CN 116955399A
- Authority
- CN
- China
- Prior art keywords
- information
- unified
- sql
- calcite
- grammar
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 80
- 229910021532 Calcite Inorganic materials 0.000 title claims abstract description 52
- 230000004048 modification Effects 0.000 claims abstract description 13
- 238000012986 modification Methods 0.000 claims abstract description 13
- 238000012545 processing Methods 0.000 claims abstract description 10
- 238000004458 analytical method Methods 0.000 claims description 37
- 238000007781 pre-processing Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 7
- 239000003550 marker Substances 0.000 claims description 6
- 230000008859 change Effects 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 238000012423 maintenance Methods 0.000 abstract description 10
- 238000007726 management method Methods 0.000 description 10
- 238000005457 optimization Methods 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24573—Query processing with adaptation to user needs using data annotations, e.g. user-defined metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a unified SQL query method, a unified SQL query system and a unified SQL query medium based on Calcite; relates to the technical field of engine inquiry; registering different data sources into a unified database, maintaining and managing metadata information of the data sources, unifying maintenance table information and field information, and configuring query authority, modification authority and life cycle of the table information; table grammar compatible with each table information; after table grammar compatible processing is carried out, target data is queried in a unified database based on standard SQL, the scheme realizes a unified SQL standard query mode of different types of data sources, realizes federal query among different data sources, and simultaneously realizes unified management and maintenance of metadata information.
Description
Technical Field
The invention relates to the technical field of engine query, in particular to a unified SQL query method, a unified SQL query system and a unified SQL query medium based on Calcite.
Background
With the growing maturity of big data technology and the continuous development of business, under different business scenes and business demands, different database systems (such as Hive, elasticsearch, HBase, kafka) need to be used, so as to form various data sources, and result in data islands. Meanwhile, when data analysis personnel analyze data, the data analysis personnel need to inquire analysis data by using different grammar rules, unified standard entries are not available, federal inquiry among various data sources cannot be realized, metadata information is scattered in each database system, and unified management of the metadata information cannot be realized.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: aiming at a multi-type data source scene, the existing query method has the problems of data island and tedious query process; the invention aims to provide a unified SQL query method, a unified SQL query system and a unified SQL query medium based on Calcite, which are improved on the basis of the conventional engine query technology, register each data source into a unified database, maintain and manage metadata information, query target data in the unified database based on standard SQL after performing table grammar compatible processing, realize query analysis data according to the unified SQL standard, realize federal query between different data sources, and simultaneously realize unified management and maintenance of metadata information.
The invention is realized by the following technical scheme:
the invention provides a unified SQL query method based on Calcite, which comprises the following steps:
step one: acquiring data sources of different data types, and registering each data source into a unified database; the method mainly comprises a Hive data source, an Hbase data source, a Mysql data source, an elastic search data source and a Kafka data source at present;
step two: preprocessing metadata information of each data source in the unified database:
the metadata information includes: database information, table information, and field information;
maintaining and managing metadata information: uniformly maintaining table information and field information, and configuring query authority, modification authority and life cycle of the table information;
table grammar compatible with each table information;
step three: target data is queried in a unified database based on standard SQL.
Calcite in this scenario represents the Apache Calcite component.
The working principle of the scheme is as follows: aiming at a multi-type data source scene, the existing query method has the problems of data island and tedious query process; the invention aims to provide a unified SQL query method based on Calcite, which is an improvement on the prior engine query technology, registers each data source into a unified database, maintains and manages metadata information, and queries target data in the unified database based on standard SQL after table grammar compatible processing, thereby realizing a unified SQL standard query mode of different types of data sources, realizing federal query among different data sources, and simultaneously realizing unified management and maintenance of metadata information.
The Apache Calcite component currently only provides a standard SQL language, multiple query optimizations, and a basic framework for connecting various data sources, which is capable of accessing multiple data and implementing the use of SQL queries. However, the Apache Calcite component does not manage metadata information of various data sources, only supports analysis of conventional SQL and limited data sources, and can not provide standard JDBC service interfaces to the outside even if SQL analysis of different types of databases is wrong or analysis results are incomplete.
The unified SQL query method based on the Calcite not only realizes the query analysis data by adopting the unified SQL standard, but also realizes the federal query among different data sources, and simultaneously realizes the unified management and maintenance of metadata information.
The further optimization scheme is that the registering of the metadata information under each data source into the unified database comprises the following steps:
basic information of each data source is obtained, wherein the basic information comprises URL addresses, user names and passwords of the data sources;
and creating a JDBC interface for connecting the unified database with the metadata information of the data sources, and storing the metadata information of each data source into the unified database.
The further optimization scheme is that the maintenance table information and the field information comprise the following steps:
maintaining table information: name of table, remark name of table, owner of table, business meaning information of table, creation time of table and latest modification time of table
Maintaining field information: field type, field meaning and field remarks.
The further optimization scheme is that the maintaining and managing the metadata information further includes: a history of change record of the table information is stored.
The further optimization scheme is that the table grammar compatible with the table information comprises the steps of expanding incompatible table grammar of a current table:
newly adding keywords of incompatible table grammar in the config. Fmpp file;
creating an analysis class inheritance SqlCall statement, and rewriting a conversion method in the analysis class to convert an analysis mode of incompatible table grammar into SQL;
newly building a grammar analysis file in the codegen directory structure, and newly adding an SQLNODE array of incompatible table grammar;
and configuring a config. Fmpp file, generating a partner. Jj file according to the config. Fmpp file based on the FreeMarker template plug-in, and generating a final parser code based on the JavaCC compiling plug-in.
The further optimization scheme is that for Hbase data source, the table grammar compatible method of the table information comprises the following steps:
newly creating an HbaseTable analysis class, realizing an expansion interface QueryabableTable of the Apache Calcite component, and rewriting a getType method, a scan method, a project method and a getExpression method in the HbaseTable analysis class;
the getTowType method is used for defining a table mode and a column type, the scan method is used for realizing the scan method in the ScannableTable interface, the project method is used for realizing the project method in the QuerybleTable interface, and the getExpression method is used for realizing the getExpression method in the QuerybleTable interface.
According to the further optimization scheme, for target data of the Hbase data source, in the process of inquiring target data in a unified database based on standard SQL, a filterable interface is used for realizing condition filtering of the data source.
The further optimization scheme is that the method further comprises the following steps: constructing a standard JDBC external service interface;
a, a resident service is started, and is used for accessing a unified SQL engine through a JDBC external service interface: starting the resident service to realize providing service to the outside, and processing the client connection request and the SQL request; newly adding SqlSessionmanager class in resident service for managing session information of client; newly adding an SqlConf class for managing configuration information of a client connection session;
b, when the client uses the JDBC external service interface to connect to the unified SQL engine, the client sends out a connection request; the resident service monitors a connection request and forwards the connection request to an idle thread to read SQL sentences;
c, processing SQL sentences:
performing lexical analysis on the SQL sentence to convert the SQL sentence into a mark sequence, wherein each mark represents a keyword, an operator, a number or a character string;
the method comprises the steps of carrying out grammar analysis on SQL sentences, constructing a grammar tree by using a marker sequence by using an Apache Calcite component, and carrying out grammar analysis on the marker sequence based on SQL grammar rules to convert the marker sequence into a grammar tree;
the Apache Calcite component converts the syntax tree into a logical plan, which is a tree structure made up of a set of logical operators; the Apache Calcite component analyzes the grammar tree based on SQL semantic rules and converts the grammar tree into a Relnode tree, wherein each Relnode tree represents a logic operator;
the Apache Calcite component optimizes the logic plan, converts the optimized logic plan into a physical plan, and requests different services according to different data source types;
and D, finally, returning the query result to the client.
The solution also provides a unified SQL query system based on Calcite, which is used for implementing the unified SQL query method based on Calcite, and comprises the following steps:
the registration module is used for acquiring data sources with different data types and registering each data source into the unified database;
the preprocessing module is used for preprocessing the metadata information of each data source in the unified database:
the metadata information includes: database information, table information, and field information;
maintaining and managing metadata information: uniformly maintaining table information and field information, and configuring query authority, modification authority and life cycle of the table information; table grammar compatible with each table information;
and the query module is used for querying the target data in the unified database based on the standard SQL.
The present solution also provides a computer readable medium having stored thereon a computer program for execution by a processor to implement a Calcite-based unified SQL query method as described above.
Compared with the prior art, the invention has the following advantages and beneficial effects:
according to the unified SQL query method, system and medium based on Calcite, each data source is registered in the unified database, the metadata information is maintained and managed, and target data is queried in the unified database based on standard SQL after table grammar compatible processing, so that unified SQL standard query modes of different types of data sources are realized, federal query among different data sources is realized, and unified management and maintenance of the metadata information are also realized.
Drawings
In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the drawings that are needed in the examples will be briefly described below, it being understood that the following drawings only illustrate some examples of the present invention and therefore should not be considered as limiting the scope, and that other related drawings may be obtained from these drawings without inventive effort for a person skilled in the art. In the drawings:
FIG. 1 is a flow diagram of a unified SQL query method based on Calcite;
FIG. 2 is a functional schematic of a Calcite-based unified SQL query system;
FIG. 3 is a flowchart of the syntax parsing of the Apache Calcite component;
FIG. 4 is a flow chart of the unified SQL engine executing SQL.
Detailed Description
For the purpose of making apparent the objects, technical solutions and advantages of the present invention, the present invention will be further described in detail with reference to the following examples and the accompanying drawings, wherein the exemplary embodiments of the present invention and the descriptions thereof are for illustrating the present invention only and are not to be construed as limiting the present invention. In order to solve the defect of adopting unified SQL standard query and Apache Calcite components for different types of databases, the invention provides the following embodiments:
example 1
The embodiment provides a unified SQL query method based on Calcite, as shown in fig. 1, comprising the following steps:
step one: acquiring data sources of different data types, and registering each data source into a unified database; the method mainly comprises a Hive data source, an Hbase data source, a Mysql data source, an elastic search data source and a Kafka data source at present;
the method for registering the metadata information under each data source into the unified database comprises the following steps:
basic information of each data source is obtained, wherein the basic information comprises URL addresses, user names and passwords of the data sources;
and creating a JDBC interface for connecting the unified database with the metadata information of the data sources, and storing the metadata information of each data source into the unified database.
Taking Hive class data source registration as an example:
filling in the connection address, the user name and the password of the Hive database;
acquiring metadata information under the data source through Hive-JDBC interface connection information, wherein the metadata information comprises information such as database names, table remark names, field types, field meanings, field remarks and the like;
storing the acquired metadata information in a unified database Mysql;
step two: preprocessing metadata information of each data source in the unified database:
the metadata information includes: database information, table information, and field information;
maintaining and managing metadata information: uniformly maintaining table information and field information, and configuring query authority, modification authority and life cycle of the table information;
the maintenance table information and field information comprises the following steps:
maintaining table information: the name of the table, the remark name of the table, the owner of the table, the business meaning information of the table, the creation time of the table and the latest modification time of the table;
maintaining field information: field type, field meaning and field remarks;
table grammar compatible with each table information;
step three: target data is queried in a unified database based on standard SQL.
The user can search keywords to inquire the related table, and the data to be inquired can be completed through the authority flow, and the embodiment provides authority application, authority management, table modification and other authorities of the table; the administrator can set the life cycle of the table, when the life of the table exceeds the expiration limit, the table is automatically deleted, and a record 'exceeding the life cycle' is newly added in the metadata information history change record.
The maintenance and management of metadata information further includes: storing a history of changes to the table information; mainly comprises the operations of table name, field, table deletion, etc.
The metadata information is maintained and managed mainly comprising tables, fields, life cycle, change records and management of UDF functions, metadata can be conveniently and uniformly managed after all data sources in big data application are registered in the system, and powerful support is provided for data management and data quality analysis.
Example 2
Based on the above embodiments, this embodiment focuses on how, with the Apache Calcite component, specific syntax compatible with different types of data sources, table syntax compatible with each table information is provided, including expanding incompatible table syntax of the current table:
newly adding keywords of incompatible table grammar in the config. Fmpp file;
creating an analysis class inheritance SqlCall statement, and rewriting a conversion method in the analysis class to convert an analysis mode of incompatible table grammar into SQL;
newly building a grammar analysis file in the codegen directory structure, and newly adding an SQLNODE array of incompatible table grammar;
and configuring a config. Fmpp file, generating a partner. Jj file according to the config. Fmpp file based on the FreeMarker template plug-in, and generating a final parser code based on the JavaCC compiling plug-in.
To expand the forbidden table syntax in the Hbase table: disabled table_name
Newly adding keywords: newly adding a keyword disable in the config. Fmpp file;
newly creating an analysis class: creating a class of DisableSqlCall, inheriting SqlCall statement, realizing construction method and rewriting unperse (), wherein unperse () mode is a resolver of SqlNode and is responsible for converting SqlNode into Sql;
newly building a grammar analysis file, creating an Hbase. Ftl file in a codegen directory structure, and newly adding an SQLNODE array for disabling the table grammar disable;
configuring a config. Fmpp file, defining the package name and type of a parser, declaring a newly added keyword, a parsing method and the like;
and generating a parser.jj file according to the config.fmpp file by using a FreeMarker template plugin, and finally generating a final parser code by using a Java CC compiling plugin, wherein the detailed principle is that a Java is used as a grammar parser by using Java, a FreeMarker is used as a template, and the parser Impls.ftl file, the config.fmpp file and the Parser.jj file template are synthesized into a final grammar lexical file, and finally compiling into a custom parser source code by using Java, and the whole flow is shown in figure 3.
Example 2
Based on the above embodiment, the table syntax compatible method of the table information of the Hbase data source includes:
newly creating an HbaseTable analysis class, realizing an expansion interface QueryabableTable of the Apache Calcite component, and rewriting a getType method, a scan method, a project method and a getExpression method in the HbaseTable analysis class;
the getTowType method is used for defining a table mode and a column type, the scan method is used for realizing the scan method in the ScannableTable interface, the project method is used for realizing the project method in the QuerybleTable interface, and the getExpression method is used for realizing the getExpression method in the QuerybleTable interface.
And for target data of the Hbase data source, in the process of inquiring the target data in the unified database based on standard SQL, realizing condition filtering of the data source by using a filterable interface.
The user can then query the Hbase data using standard SQL, such as (select hbase_table_name limit 0).
Example 3
On the basis of embodiment 1, this embodiment further includes: constructing a standard JDBC external service interface; the flow is as shown in fig. 4:
a, a resident service (the thriftServer service in this embodiment) is started for accessing the unified SQL engine through JDBC: starting the resident service to realize providing service to the outside, and processing the client connection request and the SQL request; newly adding SqlSessionmanager class in resident service for managing session information of client; newly adding an SqlConf class for managing configuration information of a client connection session;
b, when the client uses the JDBC external service interface to connect to the unified SQL engine, the client sends out a connection request; the ThriftServer service monitors the connection request and forwards the connection request to the idle thread to read SQL sentences;
c, processing SQL sentences:
a) Lexical analysis: the SQL statement is lexically parsed and converted into a series of tokens (Token), each Token representing a key, operator, number or string, etc.
b) Syntax analysis: next, the Apache Calcite component constructs a syntax Tree (Parse Tree) using the tag sequence. In this process, it parses the markup sequence using the syntax rules of the SQL statement and converts it into a syntax tree.
c) Parsing into Relnode trees after going to the grammar tree, the Apache Calcite component will convert it into a Logical Plan (Logical Plan). A logical plan is a tree structure made up of a set of logical operators (e.g., projection, selection, join, etc.). In this process, the Apache Calcite component parses the syntax tree using the semantic rules of SQL, converting it into a RelNode tree, where each RelNode tree represents a logical operator.
d) Optimization logic plan resolution: the Apache Calcite component optimizes the logic plan using a series of optimization rules. And simultaneously converting the optimized logic plan into a physical plan, and requesting different services according to different data source types. And D, finally, returning the query result to the client.
Example 4
The present embodiment provides a Calcite-based unified SQL query system, configured to implement the Calcite-based unified SQL query method of the foregoing embodiment, as shown in fig. 2, including:
the registration module (data source registration) is used for acquiring data sources with different data types and registering each data source into the unified database;
the preprocessing module (metadata management) is used for preprocessing metadata information of each data source in the unified database:
the metadata information includes: database information, table information, and field information;
maintaining and managing metadata information: uniformly maintaining table information and field information, and configuring query authority, modification authority and life cycle of the table information; table grammar compatible with each table information;
a query module (SQL execution engine) for querying the unified database for the target data based on standard SQL.
Example 5
The present embodiment provides a computer readable medium having stored thereon a computer program for execution by a processor to implement the Calcite-based unified SQL query method of the above embodiments.
The invention realizes a unified query entry based on the Apache Calcite component, and uses standard SQL to query various types of data sources; the unified management and maintenance of metadata information are realized, the foundation is laid for data blood-margin analysis, data quality analysis and the like, and the data is queried by a standard JDBC interface provided externally. Compared with the prior art, the invention unifies all data sources together, provides unified query entry and JDBC standard query service, integrates metadata information together, and expands the functions of table life cycle, table history modification record, authority and the like in the metadata information.
The foregoing description of the embodiments has been provided for the purpose of illustrating the general principles of the invention, and is not meant to limit the scope of the invention, but to limit the invention to the particular embodiments, and any modifications, equivalents, improvements, etc. that fall within the spirit and principles of the invention are intended to be included within the scope of the invention.
Claims (10)
1. The unified SQL query method, system and medium based on Calcite are characterized by comprising the following steps:
step one: acquiring data sources of different data types, and registering each data source into a unified database;
step two: preprocessing metadata information of each data source in the unified database:
the metadata information includes: database information, table information, and field information;
maintaining and managing metadata information: uniformly maintaining table information and field information, and configuring query authority, modification authority and life cycle of the table information;
table grammar compatible with each table information;
step three: target data is queried in a unified database based on standard SQL.
2. The Calcite-based unified SQL query method according to claim 1, wherein registering metadata information under each data source into a unified database comprises the steps of:
basic information of each data source is obtained, wherein the basic information comprises URL addresses, user names and passwords of the data sources;
and creating a JDBC interface for connecting the unified database with the metadata information of the data sources, and storing the metadata information of each data source into the unified database.
3. The Calcite-based unified SQL query method according to claim 1, wherein the maintaining table information and field information comprises the steps of:
maintaining table information: name of table, remark name of table, owner of table, business meaning information of table, creation time of table and latest modification time of table
Maintaining field information: field type, field meaning and field remarks.
4. The Calcite-based unified SQL query method according to claim 3, wherein said maintaining and managing metadata information further comprises: a history of change record of the table information is stored.
5. The Calcite-based unified SQL query method according to claim 1, wherein said table syntax compatible with each table information comprises expanding an incompatible table syntax of a current table:
newly adding keywords of incompatible table grammar in the config. Fmpp file;
creating an analysis class inheritance SqlCall statement, and rewriting a conversion method in the analysis class to convert an analysis mode of incompatible table grammar into SQL;
newly building a grammar analysis file in the codegen directory structure, and newly adding an SQLNODE array of incompatible table grammar;
and configuring a config. Fmpp file, generating a partner. Jj file according to the config. Fmpp file based on the FreeMarker template plug-in, and generating a final parser code based on the JavaCC compiling plug-in.
6. The Calcite-based unified SQL query method according to claim 5, wherein for Hbase data sources, the table syntax compatible method of table information thereof comprises:
newly creating an HbaseTable analysis class, realizing an expansion interface QueryabableTable of the Apache Calcite component, and rewriting a getType method, a scan method, a project method and a getExpression method in the HbaseTable analysis class;
the getTowType method is used for defining a table mode and a column type, the scan method is used for realizing the scan method in the ScannableTable interface, the project method is used for realizing the project method in the QuerybleTable interface, and the getExpression method is used for realizing the getExpression method in the QuerybleTable interface.
7. The Calcite-based unified SQL query method according to claim 6, wherein for target data of Hbase data sources, in the process of querying target data in a unified database based on standard SQL, a filterable interface is used for realizing conditional filtering of the data sources.
8. The Calcite-based unified SQL query method according to claim 1, further comprising: constructing a standard JDBC external service interface;
a, a resident service is started for accessing a unified SQL engine through JDBC: starting the resident service to realize providing service to the outside, and processing the client connection request and the SQL request; newly adding SqlSessionmanager class in resident service for managing session information of client; newly adding an SqlConf class for managing configuration information of a client connection session;
b, when the client uses the JDBC external service interface to connect to the unified SQL engine, the client sends out a connection request; the resident service monitors a connection request and forwards the connection request to an idle thread to read SQL sentences;
c, processing SQL sentences:
performing lexical analysis on the SQL sentence to convert the SQL sentence into a mark sequence, wherein each mark represents a keyword, an operator, a number or a character string;
the method comprises the steps of carrying out grammar analysis on SQL sentences, constructing a grammar tree by using a marker sequence by using an Apache Calcite component, and carrying out grammar analysis on the marker sequence based on SQL grammar rules to convert the marker sequence into a grammar tree;
the Apache Calcite component converts the syntax tree into a logical plan, which is a tree structure made up of a set of logical operators; the Apache Calcite component analyzes the grammar tree based on SQL semantic rules and converts the grammar tree into a Relnode tree, wherein each Relnode represents a logic operator;
the Apache Calcite component optimizes the logic plan, converts the optimized logic plan into a physical plan, and requests different services according to different data source types;
and D, finally, returning the query result to the client.
9. The Calcite-based unified SQL query system, which is configured to implement the Calcite-based unified SQL query method according to any one of claims 1 to 8, and includes:
the registration module is used for acquiring data sources with different data types and registering each data source into the unified database;
the preprocessing module is used for preprocessing the metadata information of each data source in the unified database:
the metadata information includes: database information, table information, and field information;
maintaining and managing metadata information: uniformly maintaining table information and field information, and configuring query authority, modification authority and life cycle of the table information; table grammar compatible with each table information;
and the query module is used for querying the target data in the unified database based on the standard SQL.
10. A computer readable medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the Calcite-based unified SQL query method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310929918.3A CN116955399A (en) | 2023-07-26 | 2023-07-26 | Unified SQL query method, system and medium based on Calcite |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310929918.3A CN116955399A (en) | 2023-07-26 | 2023-07-26 | Unified SQL query method, system and medium based on Calcite |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116955399A true CN116955399A (en) | 2023-10-27 |
Family
ID=88444219
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310929918.3A Pending CN116955399A (en) | 2023-07-26 | 2023-07-26 | Unified SQL query method, system and medium based on Calcite |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116955399A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117349332A (en) * | 2023-12-06 | 2024-01-05 | 宁波港信息通信有限公司 | Method and device for generating application programming interface API and electronic equipment |
| CN117556452A (en) * | 2024-01-10 | 2024-02-13 | 支付宝(杭州)信息技术有限公司 | Access control method for database and related equipment |
| CN117743452A (en) * | 2023-12-25 | 2024-03-22 | 湖北省珍岛数字智能科技有限公司 | Multi-source data management system based on Datart technology |
-
2023
- 2023-07-26 CN CN202310929918.3A patent/CN116955399A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117349332A (en) * | 2023-12-06 | 2024-01-05 | 宁波港信息通信有限公司 | Method and device for generating application programming interface API and electronic equipment |
| CN117349332B (en) * | 2023-12-06 | 2024-03-01 | 宁波港信息通信有限公司 | Method and device for generating application programming interface API and electronic equipment |
| CN117743452A (en) * | 2023-12-25 | 2024-03-22 | 湖北省珍岛数字智能科技有限公司 | Multi-source data management system based on Datart technology |
| CN117556452A (en) * | 2024-01-10 | 2024-02-13 | 支付宝(杭州)信息技术有限公司 | Access control method for database and related equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106934062B (en) | Implementation method and system for querying elastic search | |
| CN116955399A (en) | Unified SQL query method, system and medium based on Calcite | |
| US9305057B2 (en) | Extensible indexing framework using data cartridges | |
| US6748374B1 (en) | Method for generating a relational database query statement using one or more templates corresponding to search conditions in an expression tree | |
| US8412746B2 (en) | Method and system for federated querying of data sources | |
| US20030105745A1 (en) | Text-file based relational database | |
| CN112434059B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN111177788A (en) | Hive dynamic desensitization method and dynamic desensitization system | |
| JP5600736B2 (en) | Database management method and system | |
| US20080120322A1 (en) | Techniques of efficient query over text, image, audio, video and other domain specific data in XML using XML table index with integration of text index and other domain specific indexes | |
| CN107291471B (en) | Meta-model framework system supporting customizable data acquisition | |
| US20100293161A1 (en) | Automatically avoiding unconstrained cartesian product joins | |
| US20100049694A1 (en) | Method and system for extending a relational schema | |
| CN112579610A (en) | Multi-data source structure analysis method, system, terminal device and storage medium | |
| EP4155965A1 (en) | System and method for facilitating metadata identification and import | |
| CN114756569A (en) | Multi-layer parsing method of structured query statement, computer device and storage medium | |
| CN115905740A (en) | Multi-data source service engine interface interconnection method | |
| US20030093770A1 (en) | Generic persistence engine | |
| KR100417569B1 (en) | Search method of distributed/heterogeneous gis databases using metadata interchange standard | |
| JP4724177B2 (en) | Index for accessing XML data | |
| Mark et al. | Information interchange between self-describing databases | |
| US11093492B1 (en) | System and method of fetching data from an external program | |
| CN114003583A (en) | Method, device, medium and equipment for constructing target format data request body | |
| CN116795859A (en) | Data analysis method, device, computer equipment and storage medium | |
| JP2976891B2 (en) | Remote database access method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |