WO2023029854A1

WO2023029854A1 - Data query method and apparatus, storage medium, and electronic device

Info

Publication number: WO2023029854A1
Application number: PCT/CN2022/109468
Authority: WO
Inventors: 孙科; 韩帅; 郭俊
Original assignee: 北京火山引擎科技有限公司
Priority date: 2021-09-03
Filing date: 2022-08-01
Publication date: 2023-03-09
Also published as: CN113704291A

Abstract

The present disclosure relates to a data query method and apparatus, a storage medium, and an electronic device, for use in reducing the use cost during data query of a user, and improving the data query efficiency. The method comprises: obtaining a structured query language statement determined on the basis of a unified structured query language standard; determining a query feature corresponding to the structured query language statement, the query feature being used for representing the query semantics of the structured query language statement; determining, from a plurality of calculation engines, a target calculation engine according to the query feature of the structured query language statement, and converting the structured query language statement into a target data query statement which can be executed by the target calculation engine; and executing the target data query statement by means of the target calculation engine.

Description

Data query method, device, storage medium and electronic equipment

Cross References to Related Applications

This application is based on the Chinese patent application with the application number 202111032755.6, the filing date is September 03, 2021, and the name is "data query method, device, storage medium and electronic equipment", and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated by reference into this application.

technical field

The present disclosure relates to the field of data technology, and in particular, to a data query method, device, storage medium and electronic equipment.

Background technique

OLAP (Online Analytical Processing, Online Analytical Processing) is online data access and analysis for specific problems. The goal of OLAP is to meet the specific query and report requirements of decision support or multi-dimensional environment.

In related technologies, in the OLAP process, data may first be obtained from a data source through a computing engine for analysis. However, the architectures of various computing engines are different. When using multiple heterogeneous computing engines, users are required to master different engine usage skills, including SQL (Structured Query Language, Structured Query Language) syntax, function definition, and parameter tuning. For example, when users use the Presto engine, they need to write SQL according to the syntax specification of the Presto engine, while when using the Spark engine, they need to write SQL according to the syntax specification of the Spark engine, which greatly increases the user's cost in the data query process , thus affecting OLAP efficiency.

Contents of the invention

This Summary is provided to introduce a simplified form of concepts that are described in detail later in the Detailed Description. This summary of the invention is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution.

In a first aspect, the present disclosure provides a data query method, the method comprising:

Obtain the structured query language statement determined based on the unified structured query language standard;

Determine the query feature corresponding to the structured query language statement, the query feature is used to characterize the query semantics of the structured query language statement;

According to the query feature of the structured query language statement, determine a target computing engine among multiple computing engines, and convert the structured query language statement into a target data query statement executable by the target computing engine;

The target data query statement is executed by the target computing engine.

In a second aspect, the present disclosure provides a data query device, the device comprising:

An acquisition module, configured to acquire the structured query language statement determined based on the unified structured query language standard;

A first determining module, configured to determine a query feature corresponding to the structured query language statement, where the query feature is used to characterize the query semantics of the structured query language statement;

The second determining module is configured to determine a target computing engine among a plurality of computing engines according to the query characteristics of the structured query language statement, and convert the structured query language statement into a target computing engine executable The target data query statement;

A query module, configured to execute the target data query statement through the target computing engine.

In a third aspect, the present disclosure provides a non-transitory computer-readable storage medium on which a computer program is stored, and when the program is executed by a processing device, the steps of the method described in the first aspect are implemented.

In a fourth aspect, the present disclosure provides an electronic device, including:

a storage device on which a computer program is stored;

A processing device configured to execute the computer program in the storage device to implement the steps of the method in the first aspect.

Through the above technical solution, users do not need to write corresponding SQL statements for computing engines of different architectures, but can write structured query language statements based on the unified structured query language standard, and then automatically adapt to the SQL statement based on the query characteristics of the structured query language statement. Configure the target computing engine to perform corresponding data query operations. Thereby, the usage cost of the user in the data query process can be reduced, the efficiency of data query can be improved, and the efficiency of OLAP can be improved in the scene of online analytical processing (OLAP).

Other features and advantages of the present disclosure will be described in detail in the detailed description that follows.

Description of drawings

The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale. In the attached picture:

Fig. 1 is a flow chart showing a data query method according to an exemplary embodiment of the present disclosure;

Fig. 2 is a schematic diagram of a processing procedure of a data query method shown according to an exemplary embodiment of the present disclosure;

Fig. 3 is a block diagram of a data query device according to an exemplary embodiment of the present disclosure;

Fig. 4 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein; A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the protection scope of the present disclosure.

It should be understood that the various steps described in the method implementations of the present disclosure may be executed in different orders, and/or executed in parallel. Additionally, method embodiments may include additional steps and/or omit performing illustrated steps. The scope of the present disclosure is not limited in this regard.

As used herein, the term "comprise" and its variations are open-ended, ie "including but not limited to". The term "based on" is "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one further embodiment"; the term "some embodiments" means "at least some embodiments." Relevant definitions of other terms will be given in the description below.

It should be noted that concepts such as "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the sequence of functions performed by these devices, modules or units or interdependence. In addition, it should be noted that the modifications of "a" and "plurality" mentioned in the present disclosure are illustrative and not restrictive. Those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "a or more".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are used for illustrative purposes only, and are not used to limit the scope of these messages or information.

As mentioned in the background art, in the OLAP process of related technologies, data can be obtained from data sources through a big data computing engine for analysis. However, the architectures of various computing engines are different. When using multiple heterogeneous computing engines, users are required to master different engine usage skills, including SQL (Structured Query Language, Structured Query Language) syntax, function definition, and parameter tuning. For example, when users use the Presto engine, they need to write SQL according to the syntax specification of the Presto engine, while when using the Spark engine, they need to write SQL according to the syntax specification of the Spark engine, which greatly increases the user's cost in the data query process , thus affecting OLAP efficiency.

In view of this, the present disclosure provides a data query method to automatically adapt to various computing engines based on a unified SQL syntax, thereby reducing user costs in the data query process.

Fig. 1 is a flowchart showing a data query method according to an exemplary embodiment of the present disclosure. Referring to Fig. 1, the data query method includes:

Step 101, acquiring a structured query language statement written based on a unified structured query language standard.

Step 102, determine the query feature corresponding to the structured query language statement, and the query feature is used to characterize the query semantics of the structured query language statement.

Step 103, according to the query features of the structured query language statement, determine a target computing engine among multiple computing engines, and convert the structured query language statement into a target data query statement executable by the target computing engine.

Step 104, execute the target data query statement through the target computing engine.

Through the above method, users do not need to write corresponding SQL statements for computing engines of different architectures, but can write structured query language statements based on the unified structured query language standard, and then automatically adapt according to the query characteristics of the structured query language statement The target computing engine is used to perform corresponding data query operations. Thereby, the usage cost of the user in the data query process can be reduced, the efficiency of data query can be improved, and the efficiency of OLAP can be improved in the scene of online analytical processing (OLAP).

In order to make those skilled in the art better understand the data query method provided by the present disclosure, the above steps are described in detail below with examples.

Exemplarily, SQL ANSI-2011 standard can be selected as the basis, supplemented with part of Hive-style DDL (Data Definition Language, database data schema definition language) and Flink-style stream syntax as the unified structured query in the embodiment of the present disclosure language standards. Of course, in other possible manners, any SQL syntax supported by any computing engine may also be selected as the unified structured query language standard, which is not limited in this embodiment of the present disclosure.

The embodiment of the present disclosure pre-sets the unified structured query language standard, so the user can write SQL statements based on the unified structured query language standard during the data query process, without learning the use skills of each computing engine and writing the SQL corresponding to the computing engine statement. In addition, the embodiment of the present disclosure can also set a unified calling interface, and then obtain the structured query language statement written based on the unified structured query language standard through the unified calling interface. That is to say, the embodiments of the present disclosure provide a unified SQL entry and SQL standard. In this way, the usage cost of the user in the process of data query can be reduced, and the efficiency of data query can be improved.

After obtaining the structured query language statement written based on the unified structured query language standard, in order to accurately and automatically adapt the computing engine to execute data query and improve the efficiency of data query, you can first determine the query characteristics of the structured query language statement, and then According to the query characteristics of the structured query language statement, a target computing engine that is more suitable for data query can be determined among multiple computing engines.

Exemplarily, the query feature may represent the query semantics of the structured query language statement, and may include the complexity feature of the structured query language statement and/or the data source feature of the data to be queried in the structured query language statement. Multiple computing engines may include pure computing big data computing engines: Spark, Hive, Presto, Flink, or may also include computing and storage integrated big data computing engines: ClickHouse, Druid, ElasticSearch, which is not limited in the embodiments of the present disclosure.

In a possible manner, determining the query feature corresponding to the structured query language statement may be: determining the complexity feature corresponding to the structured query language statement and/or the data source feature of the data to be queried in the structured query language statement. Correspondingly, according to the query characteristics of the structured query language statement, determining the target computing engine among the multiple computing engines may be: according to the complexity characteristics and/or data source characteristics of the structured query language statement, determining among the multiple computing engines target computing engine.

Exemplarily, a pre-configured engine adaptation rule can be obtained, and the engine adaptation rule is used to characterize the corresponding relationship between the query features of the structured query language statement and multiple computing engines, so that the query can be based on the structured query language statement Feature and engine adaptation rules determine the target computing engine among multiple computing engines.

For example, pre-configured first engine adaptation rules and second engine adaptation rules can be obtained. The first engine adaptation rules are used to characterize the correspondence between the complexity of structured query language statements and multiple computing engines. The second engine adaptation rule is used to characterize the corresponding relationship between the data source and multiple computing engines, so that according to the complexity characteristics of the structured query language statement and the first engine adaptation rule, and the structure query language statement The characteristics of the data source and the adaptation rules of the second engine determine the target computing engine among the multiple computing engines.

For example, the pre-configured first engine adaptation rule is: engine A is used to execute simple SQL statements, and engine B is used to execute complex SQL statements. Therefore, according to the complexity characteristics of the SQL statement, determining the target computing engine can be: if it is determined that the complexity of the SQL statement is greater than the preset complexity, then it can be determined that the target computing engine is the B engine; if it is determined that the complexity of the SQL statement is less than or equal to If the complexity is preset, it can be determined that the target computing engine is the A engine.

For another example, the pre-configured adaptation rule of the second engine is: computing engine A is used for data query of data source a, and computing engine B is used for data query of data source b. Therefore, if it is determined that the data source of the data to be queried in the SQL statement is data source a, then the target computing engine is determined to be the A computing engine; if it is determined that the data source of the data to be queried in the SQL statement is the b data source, then the target computing engine is determined to be BCompute engine.

For another example, the pre-configured first engine adaptation rule is: A1 computing engine executes simple SQL statements, A2 engine executes complex SQL statements, and the second engine adaptation rule is pre-configured: A1 computing engine and A2 computing engine are used for a Data source for data query. In this case, if it is determined that the data source of the data to be queried in the SQL statement is data source a, then the target computing engine can be determined to be A1 computing engine and A2 computing engine based on the characteristics of the data source and the second engine adaptation rule. Further, a target computing engine may be determined on the A1 computing engine and the A2 computing engine according to the complexity characteristics of the SQL statement and the first engine adaptation rule.

It should be understood that the above examples are only possible ways to determine the target computing engine according to the query characteristics of the SQL statement, and are not intended to limit the present disclosure. In specific applications, the target computing engine can be determined according to the query characteristics of the SQL statement in other ways. The embodiment of the present disclosure does not limit this.

In a possible way, in order to improve the execution efficiency of the SQL statement, the structured query language statement can be optimized first according to the query characteristics of the structured query language statement and the preset statement optimization strategy to obtain the optimized query statement, and then according to the optimized A query statement to determine the target computing engine among multiple computing engines.

It should be understood that in related technologies, when users use heterogeneous computing engines, they not only need to write corresponding SQL statements for different computing engines, but also need to set corresponding SQL statements for different computing engines because of different execution characteristics of different computing engines. Statement optimization strategies increase user costs and affect data query efficiency. However, in the embodiments of the present disclosure, after the SQL statement is written based on the unified SQL standard, the SQL statement can be uniformly optimized, thereby reducing the user's use cost and improving data query efficiency.

Exemplarily, the preset statement optimization strategy may include general optimization methods in related technologies, such as materialized view selection strategy, expression merging strategy, advanced constant deduction strategy, and built-in function optimization strategy (that is, a strategy for converting inefficient functions into efficient functions) etc., which are not limited in the embodiments of the present disclosure.

For example, if the SQL statement is used to query the data of tables A, B, and C, including the expression "A join B join C", then according to the query characteristics of the SQL statement, it can be determined that an operation needs to be performed: A join B join C. In this case, whether to perform operation A join B and then perform operation join C, or perform operation B join C and then perform operation join A, or perform operation A join C and then perform operation join B, which can improve the performance of SQL statements. Execution efficiency is the goal, determined according to the preset statement optimization strategy, that is, the SQL statement can be optimized through the expression combination strategy to obtain the optimized query statement. Afterwards, the target computing engine may be determined among multiple computing engines according to the optimized query statement.

In this way, the SQL statements can be uniformly optimized, thereby reducing the user's usage cost in the data query process, and further improving the data query efficiency.

After the target computing engine is determined, the acquired SQL statement can be converted into a target data query statement that the target computing engine can execute. For example, the UDF (User Defined Function, user-defined function) and UDAF (User Defined Aggregation Function, user-defined aggregation function) corresponding to the original SQL statement can be converted, so that the target computing engine can execute the corresponding query statement according to the converted target data. data query operations.

In a possible way, one of the multiple computing engines can be selected as the standard engine first, and based on the data format of the structured query language statement that the standard engine can process, the structured query language statement is first converted into an intermediate query statement. Correspondingly, converting the structured query language statement into the target data query statement that the target computing engine can execute may be: determining whether the target computing engine is a standard engine, and if the target computing engine is not a standard engine, converting the intermediate query statement into the target computing engine The target data query statement that the engine can execute.

In other possible manners, if the target computing engine is a standard engine, the intermediate query statement may be directly executed through the target computing engine.

Exemplarily, the standard engine may be a computing engine with universal SQL statement rules and relatively low conversion costs for converting to SQL statements executable by other engines. In this way, the conversion efficiency of the SQL statement can be improved, thereby improving the data query efficiency. For example, considering the versatility of the Calcite engine and the low cost of SQL statement conversion between the Calcite engine, the Spark engine, and the Presto engine, the Calcite engine can be used as the standard engine. Of course, in other possible manners, any other computing engine may also be selected as the standard engine, for example, the Spark engine or the Presto engine may be selected as the standard engine, which is not limited in this embodiment of the present disclosure.

As a result, data query efficiency can be further improved while automatically adapting to various computing engines.

In a possible manner, if the standard engine is the Calcite engine and the target computing engine is the Spark engine, then based on the data format of the structured query language statement that the standard engine can process, converting the structured query language statement into an intermediate query statement can be: Based on the format of the structured query language statement that the standard engine can process, the structured query language statement is converted into a RelNode statement. Correspondingly, converting the intermediate query statement into a target data query statement executable by the target computing engine may be: converting the RelNode statement into a DataFrame statement executable by the target computing engine.

That is to say, in the embodiment of the present disclosure, in order to automatically adapt the computing engine, the input SQL statement can be converted into a RelNode statement first, and then if the target computing engine is determined to be a Spark engine, the RelNode statement can be further converted into a DataFrame statement, and then execute the DataFrame statement through the Spark engine to realize data query. If it is determined that the target calculation engine is the Calcite engine, the RelNode statement can be directly executed through the Calcite engine.

It should be understood that, compared to converting a RelNode statement into a LogicalPlan statement or a PhysicalPlan statement that can be processed by the Spark engine, the API of the DataFrame statement is relatively stable, which can ensure the stability of the corresponding statement through the API call, thereby ensuring the normal execution of the data query . In addition, compared to reconverting RelNode statements into corresponding SQL statements, two sets of SQL parsing need to be maintained. However, due to the large difference between the SQL parser of the Spark engine and the SQL parser of the Calcite engine, in complex scenarios, the parsing time Both overhead and resource overhead are large, which will affect the efficiency of data query. Therefore, converting the RelNode statement into a DataFrame statement in the embodiment of the present disclosure can further improve the efficiency of data query in the scenario of adapting to multiple engines.

In a possible manner, if the standard engine is the Calcite engine and the target computing engine is the Presto engine, then based on the format of the structured query language statement that the standard engine can handle, converting the structured query language statement into an intermediate query statement can be: based on The standard engine can process the data format of the structured query language statement, and convert the structured query language statement into a RelNode statement. Correspondingly, converting the intermediate query statement into a target data query statement executable by the target computing engine may be: converting the RelNode statement into a structured query language statement executable by the target computing engine.

It should be understood that the Calcite engine does not have native support for the Presto engine, and the Presto engine does not have a stable API interface. A possible implementation is to first convert the SQL statement into a RelNode statement that the Calcite engine can process, and then convert the RelNode statement into a Node structure that the Presto engine can process. However, because the Presto engine lacks an efficient API, the docking cost of this method is very high and it is difficult to apply it quickly.

The inventor's research found that, compared with the difference between the data query statement that the Spark engine can execute and the unified structured query language standard, the difference between the data query statement that the Presto engine and the Calcite engine can execute and the unified structured query language standard is even greater. Small, so in order to adapt the Calcite engine and the Presto engine in the multi-engine scenario, the embodiment of the present disclosure can first convert the SQL statement into a RelNode statement that the Calcite engine can process, and then convert the RelNode statement into a SQL statement that the Presto engine can execute. statement. Therefore, although two sets of SQL parsing need to be maintained, since the SQL parsing of the Presto engine and the Calcite engine are relatively similar, the time and resource overhead of parsing can be reduced and the efficiency of data query can be guaranteed.

Through the above method, the automatic adaptation of multiple engines can be realized based on the unified SQL standard, and the cost of using the user in the data query process can be reduced, thereby improving the efficiency of data query.

In practical applications, if the data to be queried spans multiple data sources, it is necessary to first synchronize the data of other data sources to the target data source, and then perform data query on the target data source after data synchronization. Wherein, the target data source is any data source in the multiple data sources, and other data sources are the remaining data sources in the multiple data sources except the target data source. According to this method, data synchronization tasks need to be performed when performing joint queries across data sources. If there is a lot of data to be synchronized, the efficiency of data query will be greatly affected.

In the embodiment of the present disclosure, in order to realize automatic adaptation of multiple data sources, support cross-data source joint query, and save additional data synchronization tasks, the data source of the data to be queried in the structured query language statement may include at least two different In the case of the data source, according to the data source characteristics corresponding to the structured query language statement, determine the target computing engine corresponding to each data source among multiple computing engines, and based on the data source characteristics, the structured query language statement It is converted into a target data query statement that the corresponding target computing engine can execute, and then after executing the corresponding target data query statement through the target computing engine, any engine among multiple computing engines is determined as a joint processing engine, and each target The calculation engine sends the target data queried from the corresponding data source to the joint processing engine by executing the target data query statement, and finally performs joint processing on each target data through the joint processing engine.

Exemplarily, the joint processing engine may be any one of the target computing engines, or any other engine among the multiple computing engines except the target computing engine, and may be configured according to actual conditions. It should be understood that determining any one of the target computing engines as a joint processing engine can reduce Data transmission, which can improve the efficiency of data query.

For example, referring to Figure 2, the user triggers a data query operation through a BI (Business Intelligence, business intelligence) tool, and generates an SQL statement corresponding to the data query operation based on a unified structured query language standard. After that, the SQL statement can be sent to the federated query layer of the database through JDBC (Java Database Connectivity, Java database connection) or REST interface. Then, the engine adaptation module of the federated query layer can determine the target computing engine corresponding to each data source among the multiple computing engines preset in the engine layer according to the characteristics of the data source corresponding to the SQL statement, and based on the characteristics of the data source, the The SQL statement is converted into a target data query statement that the corresponding target computing engine can execute. Afterwards, the target data query statement can be sent to the engine layer, and data query can be performed on the corresponding data source of the data source layer through the target computing engine in the engine layer. Finally, the target computing engine can send the target data queried from the data source to any computing engine in the engine layer (that is, the joint processing engine), so that the cross-data source joint processing can be performed through the computing engine, that is, through a certain computing The engine associates the target data queried by each target computing engine and returns it to the user. As a result, cross-data source joint query can be supported, and additional data synchronization tasks can be omitted, thereby improving data query efficiency.

In addition, referring to FIG. 2 , the federated query layer may also include an optimization module, a management module, a permission module and a metadata module. Wherein, the optimization module can uniformly optimize the structured query language statement according to the query characteristics of the structured query language statement and a preset statement optimization strategy. The management module can manage the process of submitting target data query statements to the target computing engine during the data query process, or can also perform processes such as log collection and result saving. After obtaining the SQL statement determined based on the unified structured query language standard, the authority module can first determine whether the originating user of the SQL statement has the authority to execute the data operation authority corresponding to the SQL statement or can verify the correctness of the SQL statement. The metadata module is used to store the data permission information of each user and the metadata corresponding to each data source, so as to better determine the target computing engine. Among them, the specific implementation methods of the management module, the authority module and the metadata module in the federated query layer are similar to related technologies, and will not be repeated here.

Continuing to refer to Figure 2, the multiple computing engines preset at the engine layer include Spark, Hive, Presto, Flink, ClickHouse, and ElasticSearch. The data source layer includes HDFS, RDS, Kafka, ClickHouse, and ElasticSearch. It should be understood that ClickHouse and ElasticSearch are computing and storage integrated engines, so they can be included in the engine layer and data source layer.

For example, according to the architecture shown in FIG. 2 , in an online analytical processing (OLAP) scenario, the data query process may be: obtaining an SQL statement determined based on a unified structured query language standard through a JDBC or REST interface. Then, verify the correctness of the SQL statement through the metadata module of the federated query layer, and determine whether the user has the authority to query the data to be queried in the SQL statement through the authority module. Afterwards, the SQL statement can be uniformly optimized through the optimization module of the federated query layer to obtain an optimized query statement. Then, the engine adaptation module of the federated query layer can determine the target computing engine according to the query characteristics of the SQL statement, and convert the SQL statement into a target data query statement that the target computing engine can execute and send it to the target computing engine in the engine layer. Afterwards, the target computing engine can determine the execution strategy for the target data query statement according to its own optimization capability of the physical layer, and execute the target data query statement according to the execution strategy to query data from the corresponding data source. If there are multiple data sources, after the data is queried, the queried data can be returned to each target calculation, and finally the federated query layer performs joint processing.

Through the above method, it is possible to realize not only multi-engine data query, but also multi-data source joint query, which can greatly reduce the user's use cost in the data query process, thereby improving the efficiency of data query.

Based on the same inventive concept, the present disclosure also provides a data query device, which can become a part or all of the electronic equipment through software, hardware or a combination of both. Referring to Fig. 3, the data query device 300 may include:

An acquisition module 301, configured to acquire a structured query language statement determined based on a unified structured query language standard;

The first determination module 302 is configured to determine query features corresponding to the structured query language statement, where the query feature is used to characterize the query semantics of the structured query language statement;

The second determining module 303 is configured to determine a target computing engine among multiple computing engines according to the query features of the structured query language statement, and convert the structured query language statement into a target computing engine capable of The executed target data query statement;

The query module 304 is configured to execute the target data query statement through the target calculation engine.

Optionally, the first determining module 302 is configured to:

Determine the complexity characteristics corresponding to the structured query language statement and/or the data source characteristics of the data to be queried in the structured query language statement;

The second determination module 303 is used for:

According to the complexity feature of the structured query language statement and/or the data source feature, a target computing engine is determined among multiple computing engines.

Optionally, the data source of the data to be queried in the structured query language statement includes at least two different data sources, and the second determining module 303 is used for:

According to the data source characteristics corresponding to the structured query language statement, determine a target computing engine corresponding to each of the data sources among multiple computing engines, and convert the structured query language statement based on the data source characteristics A target data query statement that can be executed by the corresponding target computing engine;

The device 300 also includes:

A joint module, configured to determine any one of the multiple computing engines as a joint processing engine after executing the target data query statement through the target computing engine, and execute each of the target computing engines by executing the The target data query statement from the corresponding data source sends the target data to the joint processing engine, and each target data is jointly processed by the joint processing engine.

Optionally, the second determining module 303 is configured to:

Optimizing the structured query language statement according to the query characteristics of the structured query language statement and a preset statement optimization strategy to obtain an optimized query statement;

According to the optimized query statement, a target computing engine is determined among multiple computing engines.

Optionally, the device 300 also includes:

An intermediate conversion module, configured to select an engine among the plurality of computing engines as a standard engine, and convert the structured query language statement into an intermediate query based on the format of the structured query language statement that the standard engine can process statement;

The second determination module 303 is used for:

determining whether the target computing engine is the standard engine;

If the target computing engine is not the standard engine, converting the intermediate query statement into a target data query statement executable by the target computing engine.

Optionally, the standard engine is a Calcite engine, the target computing engine is a Spark engine, and the intermediate conversion module is used to convert the structured Query language statements are converted to RelNode statements;

The second determining module 303 is used for converting the RelNode statement into a DataFrame statement executable by the target computing engine.

Optionally, the standard engine is a Calcite engine, the target calculation engine is a Presto engine, and the intermediate conversion module is used to convert the structured query language statement based on the data format of the standard engine to process Query language statements are converted to RelNode statements;

The second determining module 303 is used to convert the RelNode statement into a structured query language statement executable by the target computing engine.

Regarding the apparatus in the foregoing embodiments, the specific manner in which each module executes operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

Based on the same idea, the present disclosure also provides a non-transitory computer-readable storage medium on which a computer program is stored, and when the program is executed by a processing device, the steps of any one of the above data query methods are implemented.

Based on the same idea, the present disclosure also provides an electronic device, including:

a storage device on which a computer program is stored;

A processing device, configured to execute the computer program in the storage device, so as to realize the steps of any data query method described above.

Referring now to FIG. 4 , it shows a schematic structural diagram of an electronic device 400 suitable for implementing an embodiment of the present disclosure. The terminal equipment in the embodiment of the present disclosure may include but not limited to such as mobile phone, notebook computer, digital broadcast receiver, PDA (personal digital assistant), PAD (tablet computer), PMP (portable multimedia player), vehicle terminal (such as mobile terminals such as car navigation terminals) and fixed terminals such as digital TVs, desktop computers and the like. The electronic device shown in FIG. 4 is only an example, and should not limit the functions and scope of use of the embodiments of the present disclosure.

As shown in FIG. 4, an electronic device 400 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 401, which may be randomly accessed according to a program stored in a read-only memory (ROM) 402 or loaded from a storage device 408. Various appropriate actions and processes are executed by programs in the memory (RAM) 403 . In the RAM 403, various programs and data necessary for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input/output (I/O) interface 405 is also connected to bus 404 .

Typically, the following devices can be connected to the I/O interface 405: input devices 406 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speaker, vibration an output device 407 such as a computer; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication means 409 may allow the electronic device 400 to perform wireless or wired communication with other devices to exchange data. While FIG. 4 shows electronic device 400 having various means, it should be understood that implementing or having all of the means shown is not a requirement. More or fewer means may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product, which includes a computer program carried on a non-transitory computer readable medium, where the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 409, or from storage means 408, or from ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.

It should be noted that the above-mentioned computer-readable medium in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted by any appropriate medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol) can be used to communicate, and can communicate with digital data in any form or medium (for example, communication network) interconnection. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network of.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device, or may exist independently without being incorporated into the electronic device.

The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains the structured query language statement determined based on the unified structured query language standard; determines the The query feature corresponding to the structured query language statement, the query feature is used to characterize the query semantics of the structured query language statement; according to the query feature of the structured query language statement, it is determined in multiple computing engines a target computing engine, and convert the structured query language statement into a target data query statement executable by the target computing engine; execute the target data query statement through the target computing engine.

Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, or combinations thereof, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages - such as "C" or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, using an Internet service provider to connected via the Internet).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The modules involved in the embodiments described in the present disclosure may be implemented by software or by hardware. Wherein, the name of the module does not constitute a limitation on the module itself under certain circumstances.

The functions described herein above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System on Chips (SOCs), Complex Programmable Logical device (CPLD) and so on.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

According to one or more embodiments of the present disclosure, Example 1 provides a data query method, the method comprising:

The target data query statement is executed by the target computing engine.

According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, and the determining the query features corresponding to the structured query language statement includes:

According to the query feature of the structured query language statement, determining a target computing engine among multiple computing engines includes:

According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 1, the data source of the data to be queried in the structured query language statement includes at least two different data sources, and the structured Querying the query features of a language statement, determining a target computing engine among multiple computing engines, and converting the structured query language statement into a target data query statement that can be executed by the target computing engine, including:

After the target data query statement is executed by the target calculation engine, it also includes:

determining any one of the plurality of computing engines as a joint processing engine;

sending the target data queried by each target computing engine from the corresponding data source to the joint processing engine by executing the target data query statement, and performing each target data through the joint processing engine joint processing.

According to one or more embodiments of the present disclosure, Example 4 provides the method described in any one of Examples 1-3, wherein the target is determined in multiple computing engines according to the query characteristics of the structured query language statement Calculation engine, including:

According to one or more embodiments of the present disclosure, Example 5 provides the method described in any one of Examples 1-3, the method further comprising:

Selecting an engine among the plurality of computing engines as a standard engine, and converting the structured query language statement into an intermediate query statement based on the format of the structured query language statement that the standard engine can handle;

The converting the structured query language statement into a target data query statement executable by the target computing engine includes:

determining whether the target computing engine is the standard engine;

According to one or more embodiments of the present disclosure, Example 6 provides the method described in Example 5, the standard engine is a Calcite engine, the target calculation engine is a Spark engine, and the structure that can be processed based on the standard engine is the format of the structured query language statement, and convert the structured query language statement into an intermediate query statement, including:

Converting the structured query language statement into a RelNode statement based on the data format of the structured query language statement capable of being processed by the standard engine;

The converting the intermediate query statement into a target data query statement executable by the target computing engine includes:

Converting the RelNode statement into a DataFrame statement executable by the target computing engine.

According to one or more embodiments of the present disclosure, Example 7 provides the method described in Example 5, the standard engine is a Calcite engine, the target calculation engine is a Presto engine, and the structure that can be processed based on the standard engine the format of the structured query language statement, and convert the structured query language statement into an intermediate query statement, including:

Converting the RelNode statement into a structured query language statement executable by the target computing engine.

According to one or more embodiments of the present disclosure, Example 8 provides a data query device, the device comprising:

According to one or more embodiments of the present disclosure, Example 9 provides a non-transitory computer-readable storage medium on which a computer program is stored, and when the program is executed by a processing device, any one of Examples 1-7 is implemented. steps of the method described above.

According to one or more embodiments of the present disclosure, Example 10 provides an electronic device, comprising:

a storage device on which a computer program is stored;

A processing device configured to execute the computer program in the storage device to implement the steps of any one of the methods in Examples 1-7.

The above description is only a preferred embodiment of the present disclosure and an illustration of the applied technical principle. Those skilled in the art should understand that the disclosure scope involved in this disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, but also covers the technical solutions formed by the above-mentioned technical features or Other technical solutions formed by any combination of equivalent features. For example, a technical solution formed by replacing the above-mentioned features with technical features disclosed in this disclosure (but not limited to) having similar functions.

In addition, while operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or performed in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while the above discussion contains several specific implementation details, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the foregoing embodiments, the specific manner in which each module executes operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

Claims

A data query method, characterized in that the method comprises:

Obtain the structured query language statement determined based on the unified structured query language standard;

Determine the query feature corresponding to the structured query language statement, the query feature is used to characterize the query semantics of the structured query language statement;

According to the query feature of the structured query language statement, determine a target computing engine among multiple computing engines, and convert the structured query language statement into a target data query statement executable by the target computing engine;

The target data query statement is executed by the target computing engine.
The method according to claim 1, wherein the determining the query features corresponding to the structured query language statement comprises:

Determine the complexity characteristics corresponding to the structured query language statement and/or the data source characteristics of the data to be queried in the structured query language statement;

According to the query feature of the structured query language statement, determining a target computing engine among multiple computing engines includes:

According to the complexity feature of the structured query language statement and/or the data source feature, a target computing engine is determined among multiple computing engines.
The method according to claim 1, wherein the data source of the data to be queried in the structured query language statement includes at least two different data sources, and the query according to the structured query language statement The feature is to determine a target computing engine among multiple computing engines, and convert the structured query language statement into a target data query statement that can be executed by the target computing engine, including:

According to the data source characteristics corresponding to the structured query language statement, determine a target computing engine corresponding to each of the data sources among multiple computing engines, and convert the structured query language statement based on the data source characteristics A target data query statement that can be executed by the corresponding target computing engine;

After the target data query statement is executed by the target calculation engine, it also includes:

determining any one of the plurality of computing engines as a joint processing engine;

sending the target data queried by each target computing engine from the corresponding data source to the joint processing engine by executing the target data query statement, and performing each target data through the joint processing engine joint processing.
The method according to any one of claims 1-3, wherein, according to the query feature of the structured query language statement, determining a target computing engine among multiple computing engines includes:

Optimizing the structured query language statement according to the query characteristics of the structured query language statement and a preset statement optimization strategy to obtain an optimized query statement;

According to the optimized query statement, a target computing engine is determined among multiple computing engines.
The method according to any one of claims 1-3, wherein the method further comprises:

Selecting an engine among the plurality of computing engines as a standard engine, and converting the structured query language statement into an intermediate query statement based on the format of the structured query language statement that the standard engine can handle;

The converting the structured query language statement into a target data query statement executable by the target computing engine includes:

determining whether the target computing engine is the standard engine;

If the target computing engine is not the standard engine, converting the intermediate query statement into a target data query statement executable by the target computing engine.
The method according to claim 5, wherein the standard engine is a Calcite engine, and the target computing engine is a Spark engine, and the format of the structured query language statement that can be processed by the standard engine is converted to The above structured query language statement is converted into an intermediate query statement, including:

Converting the structured query language statement into a RelNode statement based on the data format of the structured query language statement capable of being processed by the standard engine;

The converting the intermediate query statement into a target data query statement executable by the target computing engine includes:

Converting the RelNode statement into a DataFrame statement executable by the target computing engine.
The method according to claim 5, wherein the standard engine is a Calcite engine, and the target calculation engine is a Presto engine, and the format of the structured query language statement that can be processed by the standard engine is converted to The above structured query language statement is converted into an intermediate query statement, including:

Converting the structured query language statement into a RelNode statement based on the data format of the structured query language statement capable of being processed by the standard engine;

The converting the intermediate query statement into a target data query statement executable by the target computing engine includes:

Converting the RelNode statement into a structured query language statement executable by the target computing engine.
A data query device, characterized in that said device comprises:

An acquisition module, configured to acquire the structured query language statement determined based on the unified structured query language standard;

A first determining module, configured to determine a query feature corresponding to the structured query language statement, where the query feature is used to characterize the query semantics of the structured query language statement;

The second determining module is configured to determine a target computing engine among a plurality of computing engines according to the query characteristics of the structured query language statement, and convert the structured query language statement into a target computing engine executable The target data query statement;

A query module, configured to execute the target data query statement through the target computing engine.
A non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that, when the program is executed by a processing device, the steps of the method in any one of claims 1-7 are implemented.
An electronic device, characterized in that it comprises:

a storage device on which a computer program is stored;

A processing device configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1-7.