CN115470240A - Data query method, data query device, electronic equipment and storage medium - Google Patents

Data query method, data query device, electronic equipment and storage medium Download PDF

Info

Publication number
CN115470240A
CN115470240A CN202211136908.6A CN202211136908A CN115470240A CN 115470240 A CN115470240 A CN 115470240A CN 202211136908 A CN202211136908 A CN 202211136908A CN 115470240 A CN115470240 A CN 115470240A
Authority
CN
China
Prior art keywords
query
data
data sources
results
middleware
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211136908.6A
Other languages
Chinese (zh)
Inventor
王杰斌
林文辉
解军伟
杨虎军
王煜
何腾翔
王斌斌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Aisino Corp
Original Assignee
Aisino Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Aisino Corp filed Critical Aisino Corp
Priority to CN202211136908.6A priority Critical patent/CN115470240A/en
Publication of CN115470240A publication Critical patent/CN115470240A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2453—Query optimisation
    • G06F16/24532—Query optimisation of parallel queries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2453—Query optimisation
    • G06F16/24534—Query rewriting; Transformation
    • G06F16/24542—Plan optimisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24553—Query execution of query operations
    • G06F16/24554—Unary operations; Data partitioning operations
    • G06F16/24556—Aggregation; Duplicate elimination

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Operations Research (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the invention provides a data query method, a data query device, electronic equipment and a storage medium. And generating a plurality of query execution plans of the plurality of query statement analysis results, wherein the plurality of query execution plans respectively correspond to the middleware of the plurality of data sources in the data asset management. Obtaining a plurality of query results of the plurality of query execution plans through the middleware of the plurality of data sources, respectively. And aggregating the plurality of query results based on the hierarchical relationship among the plurality of query execution plans to obtain the current query result. The scheme of the invention improves the efficiency of data query in data asset management.

Description

Data query method, data query device, electronic equipment and storage medium
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a data processing method, a data query device, electronic equipment and a storage medium.
Background
The data asset management is focused on constructing a data asset management system, the data standard management and the data processing can be organically fused through the data asset management, the metadata description of specific resource data is realized, the standardized data interface and the form-rich chart display tool are supported to be utilized, various data asset applications can be rapidly customized, the comprehensive evaluation of the data assets is matched, and the full life cycle management, the full flow management and the panoramic management of the data assets are realized.
Data averaging in a data asset management system for each enterprise involves at least two to three different types of data sources that are connected to relational databases such as mysql, postgres, oracle, etc., and possibly non-relational databases such as elasticserver, hive, mongoDB, etc. The differences between the databases to which these data sources are connected make sharing of data between systems difficult, and data asset management is affected as a result. Especially, when multi-table query is performed under the condition of multiple data sources, different databases connected with the data sources need to be accessed one by one to obtain all required data, and the data query process is time-consuming and low in query efficiency.
Therefore, how to improve the efficiency of data query based on multiple data sources in data asset management becomes an urgent technical problem to be solved.
Disclosure of Invention
Embodiments of the present invention provide a data query method, a data query apparatus, an electronic device, and a storage medium, so as to at least partially solve the above problems.
According to a first aspect of the embodiments of the present invention, a data query method is provided, which includes parsing a query statement to obtain a plurality of query statement parsing results. And generating a plurality of query execution plans of the plurality of query statement analysis results, wherein the plurality of query execution plans respectively correspond to the middleware of the plurality of data sources in the data asset management. Obtaining a plurality of query results of the plurality of query execution plans through the middleware of the plurality of data sources, respectively. And aggregating the plurality of query results based on the hierarchical relationship among the plurality of query execution plans to obtain the current query result.
In one implementation, parsing the query statement to obtain a plurality of query statement parsing results includes parsing the query statement by using a preset multi-data source query engine to obtain types of a plurality of data sources. And respectively determining a plurality of query statement analysis results according to the types of the plurality of data sources.
In another implementation, obtaining a plurality of query results for a plurality of query execution plans respectively through middleware of a plurality of data sources includes dividing the plurality of query execution plans into a plurality of subtasks based on a hierarchical relationship between the plurality of query plans. And pairing the plurality of subtasks and the plurality of data sources through the middleware of the plurality of data sources to obtain a plurality of query results.
In another implementation, aggregating the plurality of query results based on a hierarchical relationship between the plurality of query execution plans, the obtaining the current query result includes returning the plurality of query results to the multiple data source query engine. And aggregating the plurality of query results according to the hierarchy relation among the plurality of query execution plans through a multi-data source query engine to obtain the current query result, and returning the current query result to the client side which inputs the query statement.
In another implementation, the data query method further includes determining the plurality. The plurality of data sources are managed by middleware of the plurality of data sources according to a directory state of the management directory.
In another implementation, managing the plurality of data sources through the middleware of the plurality of data sources according to the directory state of the management directory includes acquiring data information of at least one data source into the middleware when it is monitored that the directory state of the management directory indicates that at least one data source is added.
According to a second aspect of the embodiments of the present invention, there is provided a data query apparatus, including a parsing unit, configured to parse a query statement to obtain a plurality of query statement parsing results. And the generating unit is used for generating a plurality of query execution plans of a plurality of query statement analysis results, and the query execution plans correspond to the middleware of the plurality of data sources respectively. And the execution unit is used for obtaining a plurality of query results of the plurality of query execution plans through the middleware of the plurality of data sources respectively. And the aggregation unit is used for aggregating the plurality of query results based on the hierarchical relationship among the plurality of query execution plans to obtain the query result.
In one implementation manner, the data query device further includes a data asset data source management unit, configured to determine a management directory of the multiple data sources, and manage the multiple data sources through middleware of the multiple data sources according to a directory state of the management directory.
According to a third aspect of embodiments of the present invention, there is provided an electronic device including a processor, a memory storing a program. Wherein the program comprises instructions which, when executed by a processor, cause the processor to perform the method of the first aspect.
According to a fourth aspect of embodiments of the present invention, there is provided a computer storage medium having stored thereon a computer program which, when executed by a processor, performs the method of the first aspect.
In the scheme of the embodiment of the invention, the plurality of query execution plans correspond to the middleware of the plurality of data sources, and the plurality of query execution plans realize parallel query of data from the plurality of data sources through the middleware, so that the efficiency of data query based on the plurality of data sources in data asset management is improved. Meanwhile, a hierarchical relationship exists among the multiple query execution plans, multiple query results are obtained from the data source through the middleware, and the obtained multiple query results also have the hierarchical relationship. The multiple query results are directly aggregated based on the hierarchical relationship to obtain the current query result, so that the data asset management and integration capability is improved, and the data query efficiency based on multiple data sources in the data asset management is further improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the embodiments of the present invention, and it is also possible for a person skilled in the art to obtain other drawings based on the drawings.
Fig. 1 is a flowchart illustrating steps of a data query method according to an embodiment of the present invention.
Fig. 2 is a block diagram of a data query device corresponding to the embodiment of fig. 1.
Fig. 3 is a schematic structural diagram of an electronic device according to another embodiment of the invention.
Description of reference numerals:
210. an analysis unit; 220. a generating unit; 230. a query unit; 240. a polymerization unit; 300. an electronic device; 302. a processor; 304. a communication interface; 306. a memory; 308. a bus; 310. and (5) programming.
Detailed Description
In order to more clearly understand the technical features, objects and effects of the embodiments of the present application, specific embodiments of the present application will be described with reference to the accompanying drawings.
"exemplary" means "serving as an example, instance, or illustration" herein, and any illustration, embodiment, or steps described as "exemplary" herein should not be construed as a preferred or advantageous alternative.
For the sake of simplicity, the drawings only schematically show the parts relevant to the present application, and they do not represent the actual structure of the product. In addition, for simplicity and clarity of understanding, elements having the same structure or function in some of the figures may be shown only schematically or only schematically.
In order to make those skilled in the art better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all the embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments of the present invention shall fall within the scope of the protection of the embodiments of the present invention.
The following further describes concrete implementation of the embodiment of the invention by combining the drawings of the embodiment of the invention.
According to a first aspect of the embodiments of the present invention, a data query method is provided. Referring to fig. 1, fig. 1 is a flowchart illustrating steps of a data query method according to an embodiment of the present invention.
As shown in the figure, the present embodiment mainly includes the following steps:
step S110, parsing the query statement to obtain a plurality of query statement parsing results.
It should be understood that the query statement is sent by the client through an API, which is an application programming interface, which is a predefined function that aims to provide the application and developer the ability to access a set of routines based on certain software or hardware, without having to access source code or understand the details of the internal working mechanisms. One of the primary functions of an API is to provide a common set of functions.
It should also be understood that APIs are generally used for data exchange between clients and servers. The conventional data exchange needs to perform an open file reading operation after receiving data at one end, and if the file and data size is small, the operation is efficient and has a small time delay, but if the data to be transmitted is scattered and has a large data size, the time delay of the operation is large. The API is used for hanging data on the WEB through the HTTP through a program, and when the data is needed, the data is obtained from the WEB, so that the operations of downloading, opening, reading and the like of the data are reduced, and the running efficiency of the program is improved.
It should also be understood that, before the query statement is analyzed, it is determined whether the query statement is a multi-table query statement, and if the query statement is not a multi-table query statement but a single-table query statement, a single table is directly queried from a corresponding database for multiple times.
It should also be understood that, when the query statement is a multi-table query statement, the query statement is parsed to obtain a plurality of query statement parsing results. The method is essentially used for decoupling the query sentences, reducing the complexity of the query sentences, obtaining a plurality of query sentence analysis results with lower complexity, facilitating the subsequent data query based on the plurality of query sentence analysis results with lower complexity, and being beneficial to improving the efficiency of data query based on multiple data sources in data asset management.
Step S120 is to generate a plurality of query execution plans of the plurality of query statement analysis results, the plurality of query execution plans corresponding to the middleware of the plurality of data sources in the data asset management, respectively.
It should be understood that the Query Execution Plan, i.e., execution Plan or Query Plan, is a specific step and process of executing a Query statement for a database in data asset management. The query execution plan mainly comprises a table access mode, a database access mode, a data query mode and a data query mode, wherein the data query mode comprises the table access mode, the database access mode comprises the table access mode, the table access mode comprises the table access mode, and the data query mode comprises the table access mode, the database access mode comprises the table access mode, the table access mode comprises the table access mode, the data query mode comprises the table access mode, the mode of index or full table scanning and the like, and the data in the table access mode is realized by the database; the connection mode of the multiple tables, what connection algorithm is used by the database to realize the connection of the tables, including the sequential access sequence of the multiple tables; packet aggregation and ordering, etc. The manner of accessing the table, the manner of connecting the multiple tables, and the implementation manner of operations such as grouping, aggregating, and sorting in the embodiments of the present invention are only used for illustrating the embodiments of the present application, and are not limited to the embodiments of the present application.
It should also be understood that in data asset management, the data source is the data source, which provides the location of the data described by the application, the data source ensures the specification and protocol of the interaction between the application and the target data, and the data source may also be a database, a file system, etc., which defines the location information, user authentication information, and the configuration of some characteristics required for the interaction, and which encapsulates how to establish a connection with the database, exposing the interface for obtaining the connection to the outside. The application connection database need not be concerned with how its underlying layers are built, i.e., the application business logic is loosely coupled to the connection database operations.
It should also be understood that the middleware of the data source includes a plurality of data source connectors for accessing a different plurality of data sources to implement the interface.
In the embodiment of the invention, the data source provides relevant information connected to the database, the underlying database is shielded, and based on a plurality of query statement analysis results, operations such as data pulling, aggregation, connection and the like from a plurality of data sources can be realized across different data sources in data asset management, so that the steps of data moving, data migration and the like are reduced, the calculation time is reduced, namely, the data query speed is accelerated, and the data query efficiency based on a plurality of data sources in data asset management is improved.
Step S130, obtaining a plurality of query results of the plurality of query execution plans through the middleware of the plurality of data sources, respectively.
It should be understood that the middleware, i.e. the plurality of data source connectors, contains information for connecting the data sources, the middleware is correspondingly connected with the data sources, and the plurality of query plans can acquire data from the plurality of data sources through the middleware, i.e. obtain a plurality of query results from the data sources through the middleware, thereby realizing query and sharing of data of the plurality of data sources, shielding the underlying database, simplifying data query operation steps and reducing data query time.
Step S140, aggregating the plurality of query results based on the hierarchical relationship among the plurality of query execution plans to obtain the current query result.
It should be understood that a hierarchical relationship exists between the multiple query execution plans, and a hierarchical relationship also exists between the multiple query results obtained by the middleware from the data source. The hierarchical relationship is determined according to the semantic abundance of a plurality of query statement analysis results generated by analyzing the query statement, the semantic abundance is the amount of data to be queried in one data source, and the data amount is large, namely the semantic abundance is rich. The hierarchy requiring a small amount of data to be queried is located at the front, and the hierarchy requiring a large amount of data to be queried is located at the back. The sequence among a plurality of query results in the current query result also corresponds to the hierarchical relationship among a plurality of query execution plans, when data loss occurs, the data loss position can be quickly located according to the hierarchical relationship, and the data source of the lost data is traced, so that the efficiency of data query based on multiple data sources in data asset management is improved.
In summary, in the embodiment of the present invention, the plurality of query execution plans correspond to the middleware of the plurality of data sources, and the plurality of query execution plans implement parallel query of data from the plurality of data sources through the middleware, so that efficiency of data query based on the plurality of data sources in data asset management is improved. Meanwhile, a hierarchical relationship exists among the multiple query execution plans, multiple query results are obtained from the data source through the middleware, and the obtained multiple query results also have the hierarchical relationship. The multiple query results are directly aggregated based on the hierarchical relationship to obtain the current query result, so that the data asset management and integration capability is improved, and the data query efficiency based on multiple data sources in the data asset management is further improved.
In one implementation, the analyzing the query statement to obtain a plurality of query statement analysis results includes analyzing the query statement by a preset multi-data source query engine to obtain types of a plurality of data sources. And respectively determining a plurality of query statement analysis results according to the types of the plurality of data sources.
Specifically, the preset multi-data source query engine is a Trino query engine, and the Trino is a distributed SQL multi-data source query engine for big data analysis, is a pure computation engine, and can decouple bottom storage, and is named prestossql. The Trino supports a plurality of common data sources and can perform mixed calculation, and meanwhile, the Trino is memory calculation, is high in calculation speed and provides interactive query experience. The memory calculation can carry out real-time analysis and operation on large-scale mass data without the need of data preprocessing and data modeling in advance.
Further, in the embodiment of the present invention, the query statement is analyzed by the preset Trino query engine, which is equivalent to performing semantic analysis on the query statement, decoupling the complex query statement, and converting the semantic of the query statement into the executable query statement corresponding to the multiple data sources, so as to obtain multiple query statement analysis results, where the multiple query statement analysis results have lower complexity compared with the query statement, so as to facilitate subsequent data query based on the multiple query statement analysis results with lower complexity, and improve the efficiency of data query based on multiple data sources in data asset management.
In another implementation, obtaining a plurality of query results for a plurality of query execution plans respectively through middleware of a plurality of data sources includes dividing the plurality of query execution plans into a plurality of subtasks based on a hierarchical relationship between the plurality of query plans. And pairing the plurality of subtasks and the plurality of data sources through the middleware of the plurality of data sources to obtain a plurality of query results.
It should be appreciated that the middleware includes a plurality of data source connectors for accessing a different plurality of data sources to implement the interface. In the embodiment of the invention, the data source Connector is a Connector, the plurality of query plans are divided into a plurality of subtasks, the plurality of query plans interact with the plurality of data sources through the Connector, and required data are acquired from the data sources, so that a plurality of query results are obtained. The method and the system realize the query of the data of multiple data sources, shield the underlying database, further simplify the data query operation steps and improve the efficiency of the data query based on the multiple data sources in the data asset management.
It should also be understood that the plurality of subtasks include information indicating respective corresponding data sources, and the plurality of subtasks are paired with the plurality of data sources corresponding to the plurality of subtasks through the plurality of data source connectors, and data required by the plurality of subtasks are obtained from the plurality of data sources, so as to obtain a plurality of query results. The data source can be quickly searched and the plurality of query results can be obtained aiming at the plurality of subtasks through the plurality of data source connectors, so that the problem that the query speed is low due to traversal query is avoided, unnecessary query time is reduced, and the efficiency of data query based on the plurality of data sources in data asset management is improved.
Illustratively, in data asset management, a plurality of query execution plans are represented by stages, i.e., a plurality of stages, such as { Stage1, stage2, stage3, … … }, each Stage corresponding to a different data source. Dividing a plurality of query execution plans into a plurality of subtasks, namely dividing each Stage into a plurality of tasks, for example, dividing Stage1 into { task1, task2 and task3}; stage2 is divided into { task2, task3, task6, task7 and task8}; stage3 is divided into { task2, task10, task11, task12, task13} and the like.
The hierarchical relationship among a plurality of stages is determined according to the data volume to be inquired in a data source, the data volume of the data volume is the same, the lower the hierarchy of the data volume-less division is, the number of the subtasks corresponds to the data volume to be inquired in the data source, wherein the data volume of Stage2 and Stage3 is the same, and the data volume of Stage1 is less than that of Stage2 and Stage3 in the same hierarchy, so that Stage1 is in the hierarchy before Stage2 and Stage 3. When data is queried, tasks in each Stage are calculated in parallel, and the data query efficiency in data asset management is improved.
When multiple stages are related, namely when data needing to be queried in different data sources of stages 1, 2 and 3 have the same part, the divided tasks have the same part, according to the hierarchical relationship, stage1 can be executed first, and a query result obtained by executing Stage1 first is used as a task-t to replace { task2 and task3} in stages 2 and 3 for data query, so that multiple query results are obtained, the workload of data query is reduced, and the efficiency of data query based on multiple data sources in data asset management is improved.
In another implementation, aggregating the plurality of query results based on a hierarchical relationship between the plurality of query execution plans, obtaining the current query result includes returning the plurality of query results to the multiple data source query engine. And aggregating the plurality of query results according to the hierarchy relation among the plurality of query execution plans through a multi-data source query engine to obtain the current query result, and returning the current query result to the client side which inputs the query statement.
It should be appreciated that the plurality of query results are aggregated based on a hierarchical relationship between the plurality of query execution plans, resulting in a current query result. The sequence of the plurality of query results in the current query result also corresponds to the hierarchical relationship among the plurality of query execution plans, and once data is lost in the data query process, the position of the lost data can be quickly found according to the hierarchical relationship, so that the data source of the lost data is quickly positioned, the data is searched again aiming at the data source, and the efficiency of searching the data is favorably improved.
In another implementation, the data query method further includes determining an administrative directory for the plurality of data sources. The plurality of data sources are managed by middleware of the plurality of data sources according to a directory state of the management directory.
It should be understood that the management directory of the plurality of data sources is used to dynamically manage the data information of the plurality of data sources, so as to update the data information of the plurality of data sources.
In another implementation, managing the plurality of data sources through the middleware of the plurality of data sources according to the directory state of the management directory includes acquiring data information of at least one data source into the middleware when it is monitored that the directory state of the management directory indicates that at least one data source is added.
It should be understood that when the directory state of the managed directory indicates that at least one data source is added, indicating that an added data source is generated, the data source is used to indicate a database, indicating that a new database is present. At the moment, the data information of the added data source is obtained, and the data information is added into the data source connector so as to deal with more complicated query statements in data asset management, and the success rate of data query under multiple data sources is improved.
According to a second aspect of the embodiment of the present invention, a data query device is provided, referring to fig. 2, and fig. 2 is a block diagram of a data query device corresponding to the embodiment of fig. 1. The data query device of the embodiment includes:
the parsing unit 210 is configured to parse the query statement to obtain a plurality of query statement parsing results.
The generating unit 220 is configured to generate a plurality of query execution plans of a plurality of query statement parsing results, where the plurality of query execution plans correspond to the middleware of the plurality of data sources, respectively.
The execution unit 230 is configured to obtain a plurality of query results of the plurality of query execution plans through middleware of the plurality of data sources, respectively.
And the aggregation unit 240 is configured to aggregate the plurality of query results based on the hierarchical relationship between the plurality of query execution plans to obtain the query result.
In one implementation, the data query apparatus further includes a data asset data source management unit, configured to determine a management directory of the multiple data sources, and manage the multiple data sources through middleware of the multiple data sources according to a directory state of the management directory.
It should be understood that the data asset data source management unit is configured to dynamically manage a plurality of data sources, and when the directory state of the management directory is an addition state, which represents that there is an additional data source, add information of the additional data source to the middleware, and manage the plurality of data sources through the middleware.
In the scheme of the embodiment of the invention, the plurality of query execution plans correspond to the middleware of the plurality of data sources, and the plurality of query execution plans realize parallel query of data from the plurality of data sources through the middleware, so that the efficiency of data query based on the plurality of data sources in data asset management is improved. Meanwhile, hierarchical relationships exist among the multiple query execution plans, multiple query results are obtained from the data source through the middleware, and the obtained multiple query results also have the hierarchical relationships. The multiple query results are directly aggregated based on the hierarchical relationship to obtain the current query result, so that the data asset management and integration capability is improved, and the data query efficiency based on multiple data sources in the data asset management is further improved.
The apparatus of this embodiment is used to implement the corresponding method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described herein again. In addition, the functional implementation of each module in the apparatus of this embodiment can refer to the description of the corresponding part in the foregoing method embodiment, and is not described herein again.
According to a third aspect of the embodiments of the present invention, there is provided an electronic device, and referring to fig. 3, a block diagram of a structure of an electronic device 300 that may be a server or a client of the present application, which is an example of a hardware device that may be applied to aspects of the present application, will now be described. Electronic device is intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
The electronic device 300 may include: a processor (processor) 302, a communication Interface 304, a memory 306, and a communication bus 308.
The processor 302, communication interface 304, and memory 306 communicate with each other via a communication bus 308. A communication interface 304 for communicating with other electronic devices or servers.
The processor 302 is configured to execute the program 310, and may specifically perform the relevant steps in the above method embodiments.
In particular, program 310 may include program code comprising computer operating instructions.
The processor 302 may be a processor CPU, or an Application Specific Integrated Circuit ASIC (Application Specific Integrated Circuit), or one or more Integrated circuits configured to implement embodiments of the present invention. The intelligent device comprises one or more processors which can be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
And a memory 306 for storing a program 310. Memory 306 may comprise high-speed RAM memory and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The program 310 may specifically be configured to cause the processor 302 to perform the following operations: and analyzing the query statement to obtain a plurality of query statement analysis results. And generating a plurality of query execution plans of the plurality of query statement analysis results, wherein the plurality of query execution plans respectively correspond to the middleware of the plurality of data sources in the data asset management. Obtaining a plurality of query results of the plurality of query execution plans through the middleware of the plurality of data sources, respectively. And aggregating the plurality of query results based on the hierarchical relationship among the plurality of query execution plans to obtain the current query result.
In addition, for specific implementation of each step in the program 310, reference may be made to corresponding steps and corresponding descriptions in units in the foregoing method embodiments, which are not described herein again. It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described devices and modules may refer to the corresponding process descriptions in the foregoing method embodiments, and are not described herein again.
It should be noted that, according to the implementation requirement, each component/step described in the embodiment of the present invention may be divided into more components/steps, and two or more components/steps or partial operations of the components/steps may also be combined into a new component/step to achieve the purpose of the embodiment of the present invention.
The above-described methods according to the embodiments of the present invention may be implemented in hardware, firmware, or as software or computer code that may be stored in a recording medium such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk, or as computer code downloaded through a network, originally stored in a remote recording medium or a non-transitory machine-readable medium, and to be stored in a local recording medium, so that the methods described herein may be stored in such software processes on a recording medium using a general purpose computer, a dedicated processor, or programmable or dedicated hardware such as an ASIC or FPGA. It will be appreciated that a computer, processor, microprocessor controller, or programmable hardware includes memory components (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code that, when accessed and executed by a computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code for implementing the methods illustrated herein, execution of the code transforms the general-purpose computer into a special-purpose computer for performing the methods illustrated herein.
It should be understood that although the present description has been described in terms of various embodiments, not every embodiment includes only a single embodiment, and such description is for clarity purposes only, and those skilled in the art will recognize that the embodiments described herein may be combined as suitable to form other embodiments, as will be appreciated by those skilled in the art.
The above embodiments are only used for illustrating the embodiments of the present application, and not for limiting the embodiments of the present application, and those skilled in the relevant art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application, so that all equivalent technical solutions also belong to the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims (10)

1. A method of querying data, comprising:
analyzing the query statement to obtain a plurality of query statement analysis results;
generating a plurality of query execution plans of the plurality of query statement parsing results, the plurality of query execution plans corresponding to middleware of a plurality of data sources in data asset management, respectively;
obtaining a plurality of query results of the plurality of query execution plans through middleware of the plurality of data sources, respectively;
and aggregating the plurality of query results based on the hierarchical relationship among the plurality of query execution plans to obtain the current query result.
2. The method of claim 1, wherein parsing the query statement to obtain a plurality of query statement parsing results comprises:
analyzing the query statement through a preset multi-data source query engine to obtain the types of the multiple data sources;
and respectively determining the plurality of query statement analysis results according to the types of the plurality of data sources.
3. The method of claim 1, wherein obtaining, by the middleware of the plurality of data sources, a plurality of query results for the plurality of query execution plans, respectively, comprises:
dividing the plurality of query execution plans into a plurality of subtasks based on a hierarchical relationship between the plurality of query plans;
and pairing the plurality of subtasks and the plurality of data sources through the middleware of the plurality of data sources to obtain the plurality of query results.
4. The method of claim 2, wherein aggregating the plurality of query results based on a hierarchical relationship between the plurality of query execution plans resulting in a current query result comprises:
returning the plurality of query results to the multiple data source query engine;
and aggregating the plurality of query results according to the hierarchy relation among the plurality of query execution plans through the multi-data-source query engine to obtain the current query result, and returning the current query result to the client side which inputs the query statement.
5. The method of claim 1, wherein the data query method further comprises:
determining an administrative directory for the plurality of data sources;
managing the plurality of data sources through middleware of the plurality of data sources according to the directory state of the management directory.
6. The method of claim 5, wherein managing the plurality of data sources through middleware of the plurality of data sources according to the directory state of the management directory comprises:
and when monitoring that the directory state of the management directory indicates that at least one data source is added, acquiring data information of the at least one data source into the middleware.
7. A data query apparatus, comprising:
the analysis unit is used for analyzing the query statement to obtain a plurality of query statement analysis results;
a generating unit, configured to generate a plurality of query execution plans of the plurality of query statement analysis results, where the plurality of query execution plans correspond to the middleware of the plurality of data sources, respectively;
the execution unit is used for obtaining a plurality of query results of the plurality of query execution plans through the middleware of the plurality of data sources respectively;
and the aggregation unit is used for aggregating the plurality of query results based on the hierarchical relation among the plurality of query execution plans to obtain the query results.
8. The method of claim 7, wherein the data querying device further comprises:
and the data asset data source management unit is used for determining the management directories of the data sources and managing the data sources through the middleware of the data sources according to the directory states of the management directories.
9. An electronic device, comprising:
a processor;
a memory storing a program;
wherein the program comprises instructions which, when executed by the processor, cause the processor to carry out the method according to any one of claims 1-6.
10. A computer storage medium, characterized in that a computer program is stored thereon, which program, when being executed by a processor, carries out the method according to any one of claims 1-6.
CN202211136908.6A 2022-09-19 2022-09-19 Data query method, data query device, electronic equipment and storage medium Pending CN115470240A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211136908.6A CN115470240A (en) 2022-09-19 2022-09-19 Data query method, data query device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211136908.6A CN115470240A (en) 2022-09-19 2022-09-19 Data query method, data query device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN115470240A true CN115470240A (en) 2022-12-13

Family

ID=84371408

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211136908.6A Pending CN115470240A (en) 2022-09-19 2022-09-19 Data query method, data query device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN115470240A (en)

Similar Documents

Publication Publication Date Title
US11775501B2 (en) Trace and span sampling and analysis for instrumented software
US11379482B2 (en) Methods, systems, and computer readable mediums for performing an aggregated free-form query
US11907246B2 (en) Methods, systems, and computer readable mediums for performing a free-form query
US9930113B2 (en) Data retrieval via a telecommunication network
CN110990420A (en) Data query method and device
US11954133B2 (en) Method and apparatus for managing and controlling resource, device and storage medium
CN112860730A (en) SQL statement processing method and device, electronic equipment and readable storage medium
US10901811B2 (en) Creating alerts associated with a data storage system based on natural language requests
CN115335821B (en) Offloading statistics collection
CA3094727C (en) Transaction processing method and system, and server
CN109947736B (en) Method and system for real-time computing
CN108319604B (en) Optimization method for association of large and small tables in hive
CN111125226B (en) Configuration data acquisition method and device
CN112163948A (en) Method, system, equipment and storage medium for separately-moistening calculation
CN111159213A (en) Data query method, device, system and storage medium
CN115470240A (en) Data query method, data query device, electronic equipment and storage medium
WO2022001626A1 (en) Time series data injection method, time series data query method and database system
CN115168358A (en) Database access method and device, electronic equipment and storage medium
CN113064914A (en) Data extraction method and device
EP2990960A1 (en) Data retrieval via a telecommunication network
CN107368477B (en) HBase coprocessor-based SQL-like query method and system
CN110928938B (en) Interface middleware system
CN117520447A (en) Data processing system, method, electronic device, and computer-readable storage medium
CN115017392A (en) Access path analysis method, device, equipment and computer storage medium
CN116414917A (en) Data transmission method, device, equipment and storage medium based on Myhouse database

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination