CN114036184A - Federal distributed query method, system and computer readable storage medium - Google Patents
Federal distributed query method, system and computer readable storage medium Download PDFInfo
- Publication number
- CN114036184A CN114036184A CN202210024739.0A CN202210024739A CN114036184A CN 114036184 A CN114036184 A CN 114036184A CN 202210024739 A CN202210024739 A CN 202210024739A CN 114036184 A CN114036184 A CN 114036184A
- Authority
- CN
- China
- Prior art keywords
- data
- layer
- data acquisition
- distributed
- statement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a federal distributed query method, a system and a computer readable storage medium, wherein the federal distributed query method is applied to a federal distributed query system, and the federal distributed query system comprises a gateway layer, a distributed service layer, a control layer and a data acquisition layer; the federal distributed query method comprises the following steps: receiving a data acquisition request sent by an application layer through a gateway layer; generating at least one type of data acquisition statement based on the data acquisition request through a distributed service layer; and acquiring target data from the data acquisition layer through the management and control layer based on at least one type of data acquisition statement, and returning the target data to the application layer, wherein the target data comprises at least one type of data corresponding to the at least one type of data acquisition statement. Based on the mode, the convenience of data query can be effectively improved.
Description
Technical Field
The present application relates to the field of data query technologies, and in particular, to a federated distributed query method, system, and computer-readable storage medium.
Background
The 21 st century is the century of information explosion, and with the rapid development of IT technology, more and more applications are continuously producing hundreds of millions of data. The state strongly supports the development of advanced technologies such as internet of things, 5G, artificial intelligence, block chains, cloud computing and big data, and further enables data which can be acquired by people to be exponentially increased.
In the prior art, because of the wide variety of data and the need of various types of data in a storage system, databases with various characteristics are invented to respectively and specially store and manage the data of corresponding types.
The prior art has the defects that when a user needs to obtain a batch of data, the batch of data may relate to data in different databases, the user needs to use different data grammars to respectively query and extract corresponding data from the different databases, and the steps are complicated, so that the convenience of data query is poor.
Disclosure of Invention
The technical problem that this application mainly solved is how to improve the convenience of data query.
In order to solve the above technical problem, the first technical solution adopted by the present application is: a federated distributed query method is applied to a federated distributed query system, and the federated distributed query system comprises a gateway layer, a distributed service layer, a control layer and a data acquisition layer; the federal distributed query method comprises the following steps: receiving a data acquisition request sent by an application layer through a gateway layer; generating at least one type of data acquisition statement based on the data acquisition request through a distributed service layer; and acquiring target data from the data acquisition layer through the management and control layer based on at least one type of data acquisition statement, and returning the target data to the application layer, wherein the target data comprises at least one type of data corresponding to the at least one type of data acquisition statement.
In order to solve the above technical problem, the second technical solution adopted by the present application is: a federated distributed query system, comprising: the gateway layer is connected with the application layer and used for receiving a data acquisition request sent by the application layer; the distributed service layer is connected with the gateway layer and used for generating at least one type of data acquisition statement based on the data acquisition request; a data acquisition layer; and the management and control layer is respectively connected with the distributed service layer and the data acquisition layer, acquires target data from the data acquisition layer based on at least one type of data acquisition statement, and returns the target data to the application layer, wherein the target data comprises at least one type of data corresponding to the at least one type of data acquisition statement.
In order to solve the above technical problem, a third technical solution adopted by the present application is: a computer readable storage medium having stored therein program instructions that are executed to implement the above-described federated distributed query method.
The beneficial effect of this application lies in: different from the prior art, the method and the device are characterized in that on the basis of a federal distributed query system comprising a gateway layer, a distributed service layer, a control layer and a data acquisition layer, a data acquisition request sent by an application layer is received through the gateway layer, at least one type of data acquisition statement is generated through the distributed service layer based on the data acquisition request, target data is acquired from the data acquisition layer through the control layer based on the at least one type of data acquisition statement, and the target data is returned to the application layer, so that a user only needs to send a data acquisition request, the at least one type of data acquisition statement can be generated through the federal distributed query system, the at least one type of data which needs to be queried is acquired based on the data acquisition statement, the user does not need to send query instructions of different statements to query different types of data, the steps of data query are simplified, and the convenience of data query is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic structural diagram of an embodiment of a federated distributed query system of the present application;
FIG. 2 is a schematic flow chart diagram illustrating one embodiment of a federated distributed query method of the present application;
FIG. 3 is a schematic structural diagram of another embodiment of the federated distributed query system of the present application;
FIG. 4 is a schematic structural diagram of yet another embodiment of the federated distributed query system of the present application;
FIG. 5 is a schematic structural diagram of a federated distributed query system in an application scenario of the present application;
FIG. 6 is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terms "first" and "second" in this application are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless explicitly specifically limited otherwise. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
The present application first proposes a federated distributed query method, as shown in fig. 1, fig. 1 is a schematic structural diagram of an embodiment of a federated distributed query system according to the present application, where the federated distributed query method is applied to a federated distributed query system, and the federated distributed query system includes a gateway layer 11, a distributed service layer 12, a management and control layer 13, and a data acquisition layer 14.
As shown in fig. 2, fig. 2 is a schematic flow chart of an embodiment of a federated distributed query method according to the present application, where the federated distributed query method includes:
step S11: and receiving a data acquisition request sent by the application layer through the gateway layer.
The gateway layer 11 may be in communication connection with an application layer used by a user, the application layer may specifically be any one of a mobile terminal, a data analysis device, and other application devices, the user may perform any operation on the application layer, and if some data needs to be acquired in the process of performing any operation, a corresponding data acquisition request may be sent to the gateway layer 11, so as to obtain the required data based on the federal distributed query system in the subsequent steps.
Specifically, the gateway layer 11 may convert and send the data acquisition request in different Protocol forms according to different data to be acquired in the data acquisition request, where the Protocol that the gateway layer 11 may convert may include jdbc (Java Database Connectivity, Java Database connection) Protocol and http (Hyper Text Transfer Protocol), and may also include other types of Database connection protocols, which is not limited herein. Based on the gateway layer 11, policies such as load balancing, service distribution high availability, fusing degradation and the like can also be realized.
Step S12: and generating at least one type of data acquisition statement based on the data acquisition request through a distributed service layer.
The distributed service layer 12 may be in communication connection with the gateway layer 11, and after receiving the data acquisition request converted by the gateway layer 11, the distributed service layer 12 may learn, based on the data acquisition request, each type of data that the user needs to obtain, and may generate, based on each type of data, a data acquisition statement of each type, so as to obtain, in a subsequent step, data of a corresponding type from a corresponding database or a data storage location based on the data acquisition statement of the corresponding type.
Optionally, the data obtaining request is used to request to obtain at least one type of data, and the distributed service layer includes at least one data obtaining statement conversion module.
The step of generating at least one type of data acquisition statement based on the data acquisition request in step S12 may specifically include:
and respectively converting the sentences corresponding to the data acquisition requests into corresponding sentences based on at least one data acquisition sentence conversion module so as to generate at least one type of data acquisition sentences.
Specifically, the data obtaining statement converting module may be an SQL (Structured Query Language) service module, and the at least one type of data requested to be obtained by the data obtaining request may be data in at least one type of database, so that the at least one SQL service module may convert and generate at least one type of SQL statement (for example, SQL2003 statement) that may be respectively used for requesting data from the at least one type of database based on the statement corresponding to the data obtaining statement for requesting data. No matter what type of statement the statement corresponding to the data acquisition request is, the corresponding statement is converted into the data request statement required by the corresponding database, so that the convenience and the efficiency of data query are improved.
In addition, the data obtaining statement conversion module may also be another language service module, which is used for performing conversion operation of another data query language, and the data obtaining statement conversion module may be specifically determined according to actual requirements, and is not limited herein.
Step S13: and acquiring target data from the data acquisition layer through the management and control layer based on at least one type of data acquisition statement, and returning the target data to the application layer.
The target data comprises at least one type of data corresponding to at least one type of data acquisition statement.
The management and control layer 13 may be in communication connection with the distributed service layer 12 and the data acquisition layer 14, and is configured to receive at least one type of data acquisition statement sent by the distributed service layer 12, and acquire at least one type of corresponding data from the data acquisition layer 14 based on the at least one type of data acquisition statement.
Optionally, as shown in fig. 3, fig. 3 is a schematic structural diagram of another embodiment of the federal distributed query system of the present application, and the management layer 13 includes a metadata management module 131.
The federated distributed query method further comprises:
at least one type of data is subjected to data management through the metadata management and control module 131.
Specifically, through the metadata management and control module 131, effective management and control on hundred million level data sources, data tables, multidimensional retrieval, database schema and the like can be realized, quality monitoring, high-performance concurrent reading and writing, sharing and other processing on data can also be realized, and strong consistency of data can be ensured.
Optionally, as shown in fig. 3, the management layer 13 includes a rights management module 132.
The federated distributed query method further comprises:
permission setting is performed on at least one type of data through the permission management module 132.
Specifically, the authority control module 132 can realize encryption and decryption control, column-level data authority control, data range authority control, data desensitization and other processing on data. The right management module 132 can set a readable user for each data, and if a user is not a readable user for the corresponding data, the user cannot read the data even if the user obtains the data.
Optionally, as shown in fig. 3, the governing layer 13 includes a policy engine 133.
The federated distributed query method further comprises:
the federated distributed query system is controlled by policy engine 133 based on preset policies.
Specifically, the policy engine 133 supports diversified data storage, and can implement flexible data management based on a regularized and strategic policy, and the functions of the policy engine 133 may include data hierarchical storage, copy management, TLL (Time To Live) validity management, data encryption and decryption, column-level data storage management, database schema management, metadata management, data lifecycle management, data security management and control, data dynamic loading, data synchronization, data sharing, and the like. Based on the policy engine 133, data can be flexibly stored in a data block manner through a streaming memory-level cache, so that data calculation performance, throughput performance, security and stability can be improved.
Different from the prior art, the method and the device are characterized in that on the basis of a federal distributed query system comprising a gateway layer, a distributed service layer, a control layer and a data acquisition layer, a data acquisition request sent by an application layer is received through the gateway layer, at least one type of data acquisition statement is generated through the distributed service layer based on the data acquisition request, target data is acquired from the data acquisition layer through the control layer based on the at least one type of data acquisition statement, and the target data is returned to the application layer, so that a user only needs to send a data acquisition request, the at least one type of data acquisition statement can be generated through the federal distributed query system, the at least one type of data which needs to be queried is acquired based on the data acquisition statement, the user does not need to send query instructions of different statements to query different types of data, the steps of data query are simplified, and the convenience of data query is improved.
The present application further provides a federated distributed query method, as shown in fig. 4, fig. 4 is a schematic structural diagram of another embodiment of the federated distributed query system according to the present application, where the federated distributed query method is applied to a federated distributed query system, the federated distributed query system includes a gateway layer 11, a distributed service layer 12, a management and control layer 13, and a data acquisition layer 14, the federated distributed query method includes steps S11-S13 mentioned in the foregoing embodiment, and details are not repeated here.
As shown in fig. 4, the data acquisition layer 14 includes a distributed engine layer 141, a local data layer 142, and a multi-source data connector layer 143, where the multi-source data connector layer 143 is respectively connected to at least one database, and the distributed engine layer includes at least one data query engine.
In step S13, the step of acquiring the target data from the data acquisition layer based on the at least one type of data acquisition statement may specifically include:
and responding to the data corresponding to the data acquisition statement as the data in the local data layer, and acquiring corresponding data from the local data layer based on at least one data query engine.
And responding to the data corresponding to the data acquisition statement as data in a target database, and acquiring corresponding data in the target database based on at least one data query engine and the multi-source data connector layer, wherein the target database is one of the at least one database.
Specifically, the distributed engine layer 141 may include a presto, a trino, an openlookeng, and other federal query engines, which may be determined according to actual needs, and is not limited herein.
The distributed engine layer 141 may determine that data needs to be obtained from the target database according to the received data obtaining statement, and may perform query on the database by using one or more of the above federal query engines.
The at least one database to which the multi-source data connector layer 143 is connected may include a conventional relational database, such as: oracle, MySQL, RDS (Relational Database Service), microsoft sqlserver, PostgreSQL, etc., and may also include big data common databases such as: hive, Impala, Presto, ClickHouse, TiDB, DorisDB, ElasticSearch, etc., and may also include common non-relational databases: HBase, Cassandra, MongoDB, CouchDB, etc.
Optionally, as shown in fig. 4, the local data layer 142 includes a cache layer 1421 and a storage layer 1422.
Responding to the data corresponding to the data acquisition statement as the data in the local data layer, and acquiring corresponding data from the local data layer based on at least one data query engine comprises the following steps:
and responding to the data corresponding to the data acquisition statement as the data in the cache layer, and acquiring corresponding data from the cache layer based on at least one data query engine.
And responding to the data corresponding to the data acquisition statement as the data in the storage layer, and acquiring corresponding data from the storage layer based on at least one data query engine.
Specifically, the cache layer 1421 may be a cache space constructed by a local cache device, and the storage layer 1422 may be a storage space constructed by a local storage device.
If the data to be acquired corresponding to a data acquisition statement is data that has been stored in the cache layer 1421 or the storage layer 1422 and has not been deleted yet, the distributed engine layer 141 can directly acquire the data from the cache layer 1421 or the storage layer 1422 without acquiring the data from a corresponding database, thereby speeding up data query.
Further, when the target data flows through the multi-source data connector layer 143, and/or flows through the storage layer 1422, and/or flows through the gateway layer 11, the target data may be encrypted once, and the user may decrypt the received target data through a decryption tool preset in the application layer. The encryption method may be one or more of dynamic encryption, KMS (Key Management Service) encryption, static encryption, and other encryption methods, which are not limited herein.
Different from the prior art, the method and the device are characterized in that on the basis of a federal distributed query system comprising a gateway layer, a distributed service layer, a control layer and a data acquisition layer, a data acquisition request sent by an application layer is received through the gateway layer, at least one type of data acquisition statement is generated through the distributed service layer based on the data acquisition request, target data is acquired from the data acquisition layer through the control layer based on the at least one type of data acquisition statement, and the target data is returned to the application layer, so that a user only needs to send a data acquisition request, the at least one type of data acquisition statement can be generated through the federal distributed query system, the at least one type of data which needs to be queried is acquired based on the data acquisition statement, the user does not need to send query instructions of different statements to query different types of data, the steps of data query are simplified, and the convenience of data query is improved.
The present application further provides a federated distributed query system, as shown in fig. 1, fig. 1 is a schematic structural diagram of an embodiment of the federated distributed query system of the present application, where the federated distributed query system includes: gateway layer 11, distributed service layer 12, management and control layer 13, and data acquisition layer 14.
The gateway layer 11 is connected to the application layer, and is configured to receive a data acquisition request sent by the application layer. The distributed service layer 12 is connected to the gateway layer 11, and is configured to generate at least one type of data obtaining statement based on the data obtaining request. The management and control layer 13 is connected to the distributed service layer 12 and the data acquisition layer 14, respectively, and acquires target data from the data acquisition layer 14 based on at least one type of data acquisition statement, and returns the target data to the application layer, where the target data includes at least one type of data corresponding to the at least one type of data acquisition statement.
Optionally, the data obtaining layer 14 includes a distributed engine layer, a local data layer, and a multi-source data connector layer, the multi-source data connector layer is respectively connected to at least one database, and the distributed engine layer includes at least one data query engine.
Management and control layer 13 is specifically configured to:
and responding to the data corresponding to the data acquisition statement as the data in the local data layer, and acquiring corresponding data from the local data layer based on at least one data query engine.
And responding to the data corresponding to the data acquisition statement as data in a target database, and acquiring corresponding data in the target database based on at least one data query engine and the multi-source data connector layer, wherein the target database is one of the at least one database.
Further, the local data layer includes a cache layer and a storage layer, and the management and control layer 13 is specifically configured to:
and responding to the data corresponding to the data acquisition statement as the data in the cache layer, and acquiring corresponding data from the cache layer based on at least one data query engine.
And responding to the data corresponding to the data acquisition statement as the data in the storage layer, and acquiring corresponding data from the storage layer based on at least one data query engine.
Specifically, based on the federated distributed query system, the executable instructions include:
first, data sources to which the multi-source data connector layer is connected are added.
In practice, specific code examples are as follows:
create connector mysql_connector_1 ENGINE=mysql
PROPERTIES
("host" = "mysql_server_host",
"port" = "mysql_server_port",
"user" = "your_user_name",
"password" = "your_password");
second, add a designated database to which the multi-source data connector layer connects.
In practice, specific code examples are as follows:
create connector mysql_connector_1 ENGINE=mysql
PROPERTIES
("host" = "mysql_server_host",
"port" = "mysql_server_port",
"user" = "your_user_name",
"password" = "your_password",
"database" = "database_name");
third, add a specified data table in a specified database to which the multi-source data connector layer is connected.
In practice, specific code examples are as follows:
create connector mysql_connector_1 ENGINE=mysql
PROPERTIES
("host" = "mysql_server_host",
"port" = "mysql_server_port",
"user" = "your_user_name",
"password" = "your_password",
"database" = "database_name",
"table" = "table_name");
fourth, add the specified partial data in the specified data table in the specified database to which the multi-source data connector layer is connected.
In practice, specific code examples are as follows:
create connector mysql_connector_1 ENGINE=mysql
PROPERTIES
("host" = "mysql_server_host",
"port" = "mysql_server_port",
"user" = "your_user_name",
"password" = "your_password",
"database" = "database_name",
"table" = "table_name" ) AS
（select id, name, age from table_name where age > 10）;
and fifthly, adding data sources connected with the multi-source data connector layer, and allocating spaces of a cache layer and a storage layer for the added data sources.
In practice, specific code examples are as follows:
create connector mysql_connector_1 ENGINE=mysql
PROPERTIES
("host" = "mysql_server_host",
"port" = "mysql_server_port",
"user" = "your_user_name",
"password" = "your_password",
"database" = "database_name",
"table" = "table_name"，
cache_TTL="3600",
max_memory="10G",
sync_time="10s",encrype_fileds={"phone|id_card","AES"});
sixth, the specified database is queried.
In practice, specific code examples are as follows:
select * from mysql_connector_1.database_name.table_name
seventh, a specified number of databases are queried.
In practice, specific code examples are as follows:
select * from mysql_connector_1.database_name.table_name t1 left join hive_connector_1.database_name.table_name t2 on t1.id = t2.id where t1.age >10;
and eighth, setting the reading authority of the target data to determine a readable user of the target data.
In practice, specific code examples are as follows:
grantselect, insert, update, delete on
mysql_connector_1.database_name.table_name to test_user@’%’;
the executable instructions may also include other types of instructions, which may be specific to the actual needs and are not limited herein.
Different from the prior art, the method and the device are characterized in that on the basis of a federal distributed query system comprising a gateway layer, a distributed service layer, a control layer and a data acquisition layer, a data acquisition request sent by an application layer is received through the gateway layer, at least one type of data acquisition statement is generated through the distributed service layer based on the data acquisition request, target data is acquired from the data acquisition layer through the control layer based on the at least one type of data acquisition statement, and the target data is returned to the application layer, so that a user only needs to send a data acquisition request, the at least one type of data acquisition statement can be generated through the federal distributed query system, the at least one type of data which needs to be queried is acquired based on the data acquisition statement, the user does not need to send query instructions of different statements to query different types of data, the steps of data query are simplified, and the convenience of data query is improved.
In an application scenario, as shown in fig. 5, fig. 5 is a schematic structural diagram of a federal distributed query system in an application scenario of the present application, where the federal distributed query system 50 includes: an application layer 10, a gateway layer 11, a distributed service layer 12, a management layer 13, a distributed engine layer 141, a cache layer 1421, a storage layer 1422, and a multi-source data connector layer 143.
The application layer 10 includes at least one of a large screen device, a reporting device, a data analysis device, an application device, and other application end devices, which may be determined according to actual requirements and is not limited herein.
The gateway layer 11 may convert the data obtaining request to be sent out in different protocols according to different data to be obtained in the data obtaining request.
The distributed service layer 12 includes at least one data acquisition statement conversion module for generating at least one type of data acquisition statement based on the data acquisition request.
The management and control layer 13 includes a metadata management and control module, a rights management and control module, and a policy engine.
The distributed engine layer 141 may include a presto, a trino, an openlookeng, and other federal query engines, which may be specific to actual needs, and is not limited herein.
The databases to which the multi-source data connector layer 143 connects may include conventional relational databases such as: oracle, MySQL, RDS (Relational Database Service), Microsoft SQL Server, PostgreSQL, etc., and may also include big data common databases such as: hive, Impala, Presto, ClickHouse, TiDB, DorisDB, ElasticSearch, etc., and may also include common non-relational databases: HBase, Cassandra, MongoDB, CouchDB, etc.
The present application further provides a computer-readable storage medium, as shown in fig. 6, fig. 6 is a schematic structural diagram of an embodiment of the computer-readable storage medium of the present application, and the computer-readable storage medium 60 has program instructions 61 stored thereon, and when the program instructions 61 are executed by a processor (not shown), the federated distributed query method in the above-described embodiment is implemented.
The computer readable storage medium 60 of the embodiment can be, but is not limited to, a usb disk, an SD card, a PD optical drive, a removable hard disk, a high-capacity floppy drive, a flash memory, a multimedia memory card, a server, etc.
Different from the prior art, the method and the device are characterized in that on the basis of a federal distributed query system comprising a gateway layer, a distributed service layer, a control layer and a data acquisition layer, a data acquisition request sent by an application layer is received through the gateway layer, at least one type of data acquisition statement is generated through the distributed service layer based on the data acquisition request, target data is acquired from the data acquisition layer through the control layer based on the at least one type of data acquisition statement, and the target data is returned to the application layer, so that a user only needs to send a data acquisition request, the at least one type of data acquisition statement can be generated through the federal distributed query system, the at least one type of data which needs to be queried is acquired based on the data acquisition statement, the user does not need to send query instructions of different statements to query different types of data, the steps of data query are simplified, and the convenience of data query is improved.
The above description is only for the purpose of illustrating embodiments of the present application and is not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application or are directly or indirectly applied to other related technical fields, are also included in the scope of the present application.
Claims (10)
1. A federated distributed query method is characterized in that the federated distributed query method is applied to a federated distributed query system, and the federated distributed query system comprises a gateway layer, a distributed service layer, a management and control layer and a data acquisition layer;
the federal distributed query method comprises the following steps:
receiving a data acquisition request sent by an application layer through the gateway layer;
generating at least one type of data acquisition statement based on the data acquisition request through the distributed service layer;
and acquiring target data from a data acquisition layer through the management and control layer based on the at least one type of data acquisition statement, and returning the target data to the application layer, wherein the target data comprises at least one type of data respectively corresponding to the at least one type of data acquisition statement.
2. The federated distributed query method of claim 1, wherein the data acquisition layer comprises a distributed engine layer, a local data layer, and a multi-source data connector layer, the multi-source data connector layer is respectively connected with at least one database, the distributed engine layer comprises at least one data query engine;
the step of acquiring target data from the data acquisition layer based on the at least one type of data acquisition statement includes:
responding to the data corresponding to the data acquisition statement as the data in the local data layer, and acquiring corresponding data from the local data layer based on the at least one data query engine;
and responding to the data corresponding to the data acquisition statement as data in a target database, and acquiring corresponding data in the target database based on the at least one data query engine and the multi-source data connector layer, wherein the target database is one of the at least one database.
3. The federated distributed query method of claim 2, wherein the local data layer comprises a caching layer and a storage layer;
the step of acquiring, based on the at least one data query engine, corresponding data from the local data layer in response to the data corresponding to the data acquisition statement being data in the local data layer includes:
responding to the data corresponding to the data acquisition statement as the data in the cache layer, and acquiring corresponding data from the cache layer based on the at least one data query engine;
and responding to the data corresponding to the data acquisition statement as the data in the storage layer, and acquiring corresponding data from the storage layer based on the at least one data query engine.
4. A federated distributed query method as claimed in any one of claims 1-3, wherein the data acquisition request is for requesting acquisition of at least one type of data, and the distributed service layer includes at least one data acquisition statement conversion module;
the step of generating at least one type of data acquisition statement based on the data acquisition request comprises:
and respectively converting the sentences corresponding to the data acquisition requests into corresponding sentences based on the at least one data acquisition sentence conversion module so as to generate the at least one type of data acquisition sentences.
5. A federated distributed query method as claimed in any one of claims 1-3, wherein the administration layer includes a metadata administration module;
the federated distributed query method further comprises:
and performing data management on the at least one type of data through the metadata management and control module.
6. A federated distributed query method as claimed in any one of claims 1-3, wherein the management layer includes a rights management module;
the federated distributed query method further comprises:
and setting the authority of the at least one type of data through the authority control module.
7. A federated distributed query method as claimed in any one of claims 1-3, wherein the management layer includes a policy engine;
the federated distributed query method further comprises:
and controlling the federated distributed query system based on a preset strategy through the strategy engine.
8. A federated distributed query system, comprising:
the gateway layer is connected with the application layer and used for receiving the data acquisition request sent by the application layer;
the distributed service layer is connected with the gateway layer and used for generating at least one type of data acquisition statement based on the data acquisition request;
a data acquisition layer;
and the management and control layer is respectively connected with the distributed service layer and the data acquisition layer, acquires target data from the data acquisition layer based on the at least one type of data acquisition statement, and returns the target data to the application layer, wherein the target data comprises at least one type of data respectively corresponding to the at least one type of data acquisition statement.
9. The federated distributed query system of claim 8, wherein the data acquisition layer comprises a distributed engine layer, a local data layer, and a multi-source data connector layer, the multi-source data connector layer being respectively connected to at least one database, the distributed engine layer comprising at least one data query engine;
the management and control layer is specifically configured to:
responding to the data corresponding to the data acquisition statement as the data in the local data layer, and acquiring corresponding data from the local data layer based on the at least one data query engine;
and responding to the data corresponding to the data acquisition statement as data in a target database, and acquiring corresponding data in the target database based on the at least one data query engine and the multi-source data connector layer, wherein the target database is one of the at least one database.
10. A computer readable storage medium having stored therein program instructions that are executed to implement the federated distributed query method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210024739.0A CN114036184A (en) | 2022-01-11 | 2022-01-11 | Federal distributed query method, system and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210024739.0A CN114036184A (en) | 2022-01-11 | 2022-01-11 | Federal distributed query method, system and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114036184A true CN114036184A (en) | 2022-02-11 |
Family
ID=80141605
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210024739.0A Pending CN114036184A (en) | 2022-01-11 | 2022-01-11 | Federal distributed query method, system and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114036184A (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104331477A (en) * | 2014-11-04 | 2015-02-04 | 哈尔滨工业大学 | Method for testing concurrency property of cloud platform based on federated research |
| CN105335403A (en) * | 2014-07-23 | 2016-02-17 | 华为技术有限公司 | Database access method and device, and database system |
| CN106649630A (en) * | 2016-12-07 | 2017-05-10 | 乐视控股（北京）有限公司 | Data query method and device |
| CN109144982A (en) * | 2018-09-29 | 2019-01-04 | 北京友友天宇系统技术有限公司 | Multidimensional holographic Database Dynamic constructing technology system |
| CN110443059A (en) * | 2018-05-02 | 2019-11-12 | 中兴通讯股份有限公司 | Data guard method and device |
| CN112351099A (en) * | 2020-11-06 | 2021-02-09 | 北京金山云网络技术有限公司 | Data access method and device and server |
| CN112527876A (en) * | 2020-12-08 | 2021-03-19 | 国网四川省电力公司信息通信公司 | Unified database access system based on multi-source heterogeneous data analysis |
| CN112905595A (en) * | 2021-03-05 | 2021-06-04 | 腾讯科技（深圳）有限公司 | Data query method and device and computer readable storage medium |
| CN113064914A (en) * | 2021-04-22 | 2021-07-02 | 中国工商银行股份有限公司 | Data extraction method and device |
| CN113094387A (en) * | 2021-04-08 | 2021-07-09 | 杭州数梦工场科技有限公司 | Data query method and device, electronic equipment and machine-readable storage medium |
-
2022
- 2022-01-11 CN CN202210024739.0A patent/CN114036184A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105335403A (en) * | 2014-07-23 | 2016-02-17 | 华为技术有限公司 | Database access method and device, and database system |
| CN104331477A (en) * | 2014-11-04 | 2015-02-04 | 哈尔滨工业大学 | Method for testing concurrency property of cloud platform based on federated research |
| CN106649630A (en) * | 2016-12-07 | 2017-05-10 | 乐视控股（北京）有限公司 | Data query method and device |
| CN110443059A (en) * | 2018-05-02 | 2019-11-12 | 中兴通讯股份有限公司 | Data guard method and device |
| CN109144982A (en) * | 2018-09-29 | 2019-01-04 | 北京友友天宇系统技术有限公司 | Multidimensional holographic Database Dynamic constructing technology system |
| CN112351099A (en) * | 2020-11-06 | 2021-02-09 | 北京金山云网络技术有限公司 | Data access method and device and server |
| CN112527876A (en) * | 2020-12-08 | 2021-03-19 | 国网四川省电力公司信息通信公司 | Unified database access system based on multi-source heterogeneous data analysis |
| CN112905595A (en) * | 2021-03-05 | 2021-06-04 | 腾讯科技（深圳）有限公司 | Data query method and device and computer readable storage medium |
| CN113094387A (en) * | 2021-04-08 | 2021-07-09 | 杭州数梦工场科技有限公司 | Data query method and device, electronic equipment and machine-readable storage medium |
| CN113064914A (en) * | 2021-04-22 | 2021-07-02 | 中国工商银行股份有限公司 | Data extraction method and device |
Non-Patent Citations (3)
| Title |
|---|
| 傅海成 王才志 刘英明: "《新一代测井软件CIFLog技术手册》", 28 February 2015, 科学技术文献出版社 * |
| 娄岩: "《大数据应用基础》", 31 October 2018, 中国铁道出版社 * |
| 徐敏奎: "《管理信息系统》", 31 August 2003 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11521176B2 (en) | Service flow system and service data processing method and apparatus | |
| US10474835B2 (en) | Zero-knowledge databases | |
| US11249953B2 (en) | Method and apparatus for sharing big data using block chain | |
| CN109936571B (en) | Mass data sharing method, open sharing platform and electronic equipment | |
| CN112511599B (en) | Civil air defense data sharing system and method based on block chain | |
| WO2018201887A1 (en) | Data response method, apparatus, terminal device, and medium | |
| US11153071B2 (en) | Citation and attribution management methods and systems | |
| WO2016011239A1 (en) | Method and server of remote information query | |
| US20220209945A1 (en) | Method and device for storing encrypted data | |
| CN111737720A (en) | Data processing method and device and electronic equipment | |
| WO2024001028A1 (en) | Method and apparatus for maintaining blockchain data, and electronic device and storage medium | |
| CN110990877A (en) | Medical image file segmentation encryption and decryption system and method based on greenplus | |
| US20230325521A1 (en) | Data processing method and apparatus based on blockchain network, device, and storage medium | |
| CN114036184A (en) | Federal distributed query method, system and computer readable storage medium | |
| US10621207B2 (en) | Execution of queries in relational databases | |
| US10296760B2 (en) | System of shared secure data storage and management | |
| US10114864B1 (en) | List element query support and processing | |
| Karakasidis et al. | More sparking soundex-based privacy-preserving record linkage | |
| Amaechi et al. | Data Storage Management in Cloud Computing Using Deduplication Technique | |
| CN116305288B (en) | Method, device, equipment and storage medium for isolating database resources | |
| US10708253B2 (en) | Identity information including a schemaless portion | |
| CN114611137B (en) | Data access method, data access device and electronic equipment | |
| US11829498B2 (en) | Real-time dynamic blockchain securitization platform | |
| EP4227820A1 (en) | System for managing data | |
| AU2015101745A4 (en) | System of shared secure data storage and management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220211 |