CN114064684B - Unified access method for heterogeneous data sources - Google Patents
Unified access method for heterogeneous data sources Download PDFInfo
- Publication number
- CN114064684B CN114064684B CN202111192740.6A CN202111192740A CN114064684B CN 114064684 B CN114064684 B CN 114064684B CN 202111192740 A CN202111192740 A CN 202111192740A CN 114064684 B CN114064684 B CN 114064684B
- Authority
- CN
- China
- Prior art keywords
- access
- data source
- data
- heterogeneous
- access request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000004806 packaging method and process Methods 0.000 claims abstract description 3
- 230000004044 response Effects 0.000 claims description 11
- 238000004891 communication Methods 0.000 claims description 9
- 238000001914 filtration Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 8
- 150000001875 compounds Chemical class 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 3
- 238000013461 design Methods 0.000 claims description 3
- 238000011161 development Methods 0.000 description 7
- 238000012360 testing method Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 238000005192 partition Methods 0.000 description 2
- 241000543704 Pistacia Species 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a unified access method for heterogeneous data sources, which comprises the following steps: receiving a data access request sent by a client, wherein the access request is used for requesting to access any one of heterogeneous data sources; according to the access request, assembling a corresponding access statement, analyzing a data source corresponding to the access request, and loading a driver of the data source to acquire the data source connection; and connecting the data source to execute the access statement, receiving a result returned by the data source, packaging the result into a general entity, and returning the general entity to the client. The method performs service abstraction based on the difference of heterogeneous data source access on the premise of ensuring the self characteristics of each database, and realizes the integrated unified management and access of the heterogeneous data sources; a set of heterogeneous database generalized access interfaces and access rules capable of crossing languages and platforms are designed by adopting a Restful mode, and a unified data access mode is provided for upper-layer applications.
Description
Technical Field
The invention relates to the technical field of database middleware, in particular to a unified access method of heterogeneous data sources.
Background
A database is a major repository in software systems for organizing, storing, and managing data, and is typically used as a data source. For domestic databases (such as dream, large-man Jin Cang), java database connection (Java Database Connectivity, JDBC) and other modes are generally adopted for access, and for different database accesses, a business system needs to integrate a plurality of database access drivers and methods in software. The limitation is that:
When multiple databases are mixed and coexist, namely heterogeneous data sources, each business system needs to develop corresponding database access codes and services, while the traditional development mode is a typical 'chimney' architecture, namely all software services are developed and accessed through an MVC (Model-View-Controller) mode, and a JDBC or open database connection (Open Database Connectivity, ODBC) interface is called to realize database access, so that a lightweight unified access interface cannot be realized, and multiple data sources are integrated and accessed in software.
Disclosure of Invention
In order to solve the technical problems described above or at least partially solve the technical problems described above, the present invention provides a unified access method for heterogeneous data sources.
In a first aspect, the present invention provides a heterogeneous data source access method, where the heterogeneous data source access method includes: receiving an access request transmitted by a client, wherein the access request is used for requesting to access any one of heterogeneous data sources; solving and separating out a data source corresponding to the access request according to the access request, and generating a driver corresponding to the access request; and accessing corresponding data sources in the heterogeneous data sources through the driver.
Compared with the prior art, the technical scheme provided by the embodiment of the invention has the following advantages:
According to the method provided by the embodiment of the invention, the data access request sent by the client is received, and the access request is used for requesting to access any one of heterogeneous data sources; according to the access request, assembling a corresponding access statement, analyzing a data source corresponding to the access request, and loading a driver of the data source to acquire the data source connection; the data source is connected to execute the access statement, a result returned by the data source is received, the result is packaged into a general entity and returned to the client, service abstraction is carried out based on the access difference of each database on the premise of guaranteeing the self characteristics of each database aiming at main flow databases such as domestic dream, human macro Jin Cang, non-domestic Oracle, mySQL and the like, the access difference of the heterogeneous data sources is effectively shielded, and unified management and sharing of the heterogeneous data sources are realized; a set of heterogeneous database generalized access interfaces capable of crossing languages and platforms are designed by Restful Web Service, in addition, typical SQL query parameters are designed, operations such as association tables, compound primary keys, paging, sequencing, field filtering, record filtering, counting, record returning and the like are supported, consistent data access support is provided for upper-layer application, the problems of efficiency and universality of the current military information system in database use are effectively solved, repetitive work of developers is greatly reduced, and system research and development efficiency and data resource sharing rate are improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, and it will be obvious to a person skilled in the art that other drawings can be obtained from these drawings without inventive effort.
FIG. 1 is a schematic diagram of a basic architecture of a conventional development mode according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a development architecture of a heterogeneous data source access method according to an embodiment of the present invention;
FIG. 3 is a basic flow chart of a unified access method for heterogeneous data sources according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an implementation basic principle of analyzing a data source according to an access request, loading a database driver, and obtaining a data source connection according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of an implementation basic principle of accessing a data source according to an embodiment of the present invention;
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, the conventional development modes are all typical "chimney" architectures, when development and access are performed on different data sources (Jin Cang database and dream database), development and access are required to be performed according to the different data sources, that is, all software services are developed and accessed through MVC, and a JDBC or ODBC interface is called to realize database access, so that a lightweight unified access interface cannot be realized, and a plurality of data sources are integrated and accessed in software; in order to solve or partially solve the above-mentioned problems, the present embodiment proposes a heterogeneous data source access method based on the idea of fig. 2, and fig. 3 is a flow chart of a heterogeneous data source access method according to an embodiment of the present invention, which includes but is not limited to:
S101, receiving a data access request sent by a client, wherein the access request is used for requesting to access any one of heterogeneous data sources;
It is to be understood that the method adopts a Restful mode to design a general data access interface, the client sends the access request through the access interface, so that cross-language access to data in heterogeneous data sources is realized, and the access request is sent through the general access interface. RESTful is a style of design and development of web applications, and based on HTTP, XML format definitions or JSON format definitions may be used. The RESTful is suitable for a scene that a mobile internet manufacturer serves as a service interface, the function of calling mobile network resources by a third party OTT is realized, the action types are newly added, changed and deleted, wherein different RESTful clients can access data of heterogeneous data sources by each language, for example, java language can access data by using RESTTEMPLATE, OKHTTP and other HTTP clients; the C++ language can use CPP REST SDK, pistache and other HTTP clients to access data; the Python language may use Requests, curl or other HTTP clients for data access or the like.
The user sends a data access request through the provided Restful generalized access interface, so that the user can access the data in the heterogeneous data source.
S102, assembling a corresponding access statement according to the access request, analyzing a data source corresponding to the access request, and loading a driver of the data source to acquire the data source connection;
it should be understood that by parsing the access request data, the sql statement to be executed by the access data is assembled, and the corresponding driver is loaded according to the accessed data source, so as to obtain the data source connection.
S103, connecting the data source to execute the access statement, receiving a result returned by the data source, packaging the result into a general entity, and returning the general entity to the client;
Specifically, in some examples, according to the data source connection, by creating DSLContext instance in JOOQ toolkit, the sql statement is executed, and the execution result is packaged into a custom corresponding class entity, and is returned to the user, where the result is the result set of access data, and the process of accessing the data ends.
It should be understood that a plurality of data sources can exist in the heterogeneous data sources, and the data sources in the heterogeneous data sources can be issued or deleted according to the issuing instruction; in some examples of this embodiment, before receiving the data access request sent by the client, the method further includes: receiving a data source issuing instruction, and creating and issuing the heterogeneous data source, wherein the issuing instruction comprises the required data source basic information filled in through a front-end interface; designing a heterogeneous data source generalized access interface capable of crossing languages and platforms in a Restful mode, setting access rules and typical query parameters, and sending the access request by the client through the access interface; for example, a user sends a request for adding data source information through a front-end interface, and when the request for adding the data source information is received, the data source information can be released, so that the released data source information is added in heterogeneous data sources.
The typical query parameters are set, the access request is composed of the typical query parameters, typical query parameters support data source operations such as association table, compound primary key, paging, sorting, field filtering, record filtering, query single record, counting, exist query, grouping, inserting, updating and deleting, and the specific typical query parameters use methods are shown in the following table 1, wherein GET and POST respectively refer to types corresponding to each typical query parameter:
TABLE 1
In some examples of the invention, the method further comprises: the comparison operator and the logic operator are designed to realize filtering according to the result set of the recorded content, wherein the comparison operator and the logic operator can be used as an access rule, and the comparison operator is shown in the following table 2:
| Comparison operator | Comparison operator meaning | Usage method |
| eq | =, Equal to | (colName,eq,colValue) |
| ne | ! =, Not equal to | (colName,ne,colValue) |
| gt | Greater than | (colName,gt,colValue) |
| gte | =, Greater than or equal to | (colName,gte,colValue) |
| lt | < Is smaller than | (colName,lt,colValue) |
| lte | < = Less than or equal to | (colName,lte,colValue) |
| is | Single value matching | (colName,is,true/false/null) |
| in | Multi-value matching | (colName,in,val1,val2,val3,val4) |
| bw | Value range matching | (colName,bw,val1,val2) |
| like | Wild card matching | (ColName, like, -, name), note use-substitution% |
| nlike | Wild card matching | (ColName, nlike, -, name), note use of-replace% |
Table 2 wherein the logical operators are shown in table 3 below:
| Logical operators | Meaning of logical operators |
| ～and | And (3) with |
| ～or | Or (b) |
| ～xor | Exclusive OR |
TABLE 3 Table 3
The access request is constructed by typical query parameters, and the data sources to be accessed are different according to the different typical query parameters constructing the access request.
In some examples of this embodiment, after creating and publishing the heterogeneous data source, the method further comprises: and storing the published heterogeneous data sources, and storing the heterogeneous data sources in a local disk in a configuration file mode. It should be understood that, the local disk refers to a disk (hard disk) that is installed on the same computer motherboard and cannot be arbitrarily inserted and removed, and generally includes a partition where a computer operating system is located and other partitions.
In some examples of this embodiment, before loading the driver of the data source to obtain the data source connection to access corresponding data source data in the heterogeneous data source through the driver data source connection, the method further includes: loading a corresponding driver of the data source; judging whether the existing connection pool has the matched connection pool information of the data source corresponding to the access request or not, if so, acquiring the data source connection to access the data source corresponding to the access request; if the connection pool does not exist, a new connection pool of the data sources is created according to a preset connection pool configuration rule, and the data source connection is acquired, so that it is required to understand that in order to better manage the connection of each data source in the heterogeneous data sources, druid connection pools are introduced, so that the data source connection is better distributed, managed and released, and a client is allowed to repeatedly use an existing data source connection, so that response time of a data request and performance of a server are improved. In some examples of this embodiment, before accessing, by the driver, a corresponding one of the heterogeneous data sources, the method further comprises: judging whether a matched connection pool exists in the data source corresponding to the access request, if not, establishing a corresponding connection pool for the data source corresponding to the access request;
Specifically, druid connection pools will create initialSize connections to put into the pool at initialization, and the connection pool will always guarantee to have at least so many connection ports, regardless of whether the connection ports of the data source are used or not. When an application (client) requests a database operation from a connection pool, a connection is fetched from the pool; if the number of connections being used in the current pool is equal to maxActive, the requests are added into a waiting queue to wait for releasing one connection, and if the waiting time exceeds maxWait, an error is reported; if the number of the currently used connections does not reach maxActive, judging whether the connection port is idle currently, if yes, directly using the idle connection port, and if not, newly establishing a connection. After the connection port is used, the physical connection is not closed, but is put into a pool for waiting for other operations to be reused.
In some examples of this embodiment, after accessing the corresponding data source, a data response entity is generated by a custom generic response class and returned to the client. The defined generic response class (HttpResponse) attributes include "status" (status code, such as 200 represents request success, 400 represents request failure), error "represents error information of request failure," message "represents information of request success," data "represents returned data, and is an array type, in which Map structures in the form of key value pairs are stored, and each Map represents a data record. Examples are as follows:
In order to better understand the present invention, this embodiment provides a more specific example to illustrate the present invention, where the driver accesses a corresponding data source in heterogeneous data sources, accesses data in the data source according to a general RESTful access interface access request, and returns data to the RESTful access interface according to a general return class format. Specifically, as shown in fig. 4, the basic principles of the implementation of analyzing a data source according to an access request, loading a database driver, and acquiring a data source connection include:
s201, loading a database driver;
It should be understood that loading the database driver further includes receiving an access request, where the database driver is the database driver corresponding to the access request.
S202, judging whether the existing connection pool is free or not, if not, turning to S203, if yes, turning to S205;
It should be understood that, when there is an idle in the connection pool, the process goes to step S205 to acquire a database connection to realize database access, and if there is no idle in the connection pool, the process goes to step S203 to determine whether the number of active connections reaches maxActivive.
S203, judging whether the number of active connections has reached maxActive, if not, turning to S204; if so, go to S206;
It should be understood that, when the determination is made as to whether the number of active connections has reached maxActive, if not, go to S204; if so, go to S206.
S204, creating a new connection;
It should be appreciated that when it is determined that there is no free space in the existing connection pool and the number of active connections does not reach maxActive, a new database connection is created and then connected to the database.
S205, acquiring database connection;
It is to be understood that when there is an idle connection in the connection pool, the database connection is directly obtained, so as to achieve the effect of repeated use.
S206, judging whether timeout exists, if yes, going to S207, otherwise, going to S208.
It should be understood that, if the judgment is timeout, it is judged whether the timeout exists in the existing database connection, if yes, S207, if not, S208 is proceeded to.
S207, removing the overtime connection.
When the timeout connection is removed, the process goes to step S204 to create a new connection.
S208, putting into a waiting queue to wait for available connection.
It should be appreciated that when it is determined that there is no connection port that is timed out, the connection is made after waiting until there is an available connection port and waiting until the available connection port.
For better understanding of the present invention, this embodiment describes the basic principle of implementing access to a data source according to the present embodiment, as shown in fig. 5, where the basic principle of implementing access to a data source includes:
S301 { data topic publication service hostname } { data topic publication service port number }/api/access/{ data source name }/{ table name }
It should be understood that step S301 is essentially that data needed at the parsing place according to the access request, wherein the needed data includes, but is not limited to: { data topic publication service hostname } { data topic publication service port number }/api/access/{ data source name }/{ table name }.
S302, assembling sql statements of operation data to be executed according to the data source name, the table name and the set field data;
S303, loading a database driver;
s304, whether the existing database connection pool has url information of the data source or not;
s305, using an existing Druid connection pool;
S306, creating a new Druid connection pool for the data source connection;
s307, acquiring database connection, generating DSLContext objects and executing the configured and ready-to-execute sql statement;
S308, generating a data response entity return through a custom response class.
It is to be appreciated that in some examples, the data source connection port may be obtained by referencing the behavior of the org.jooq.configuration configuration jOOQ, creating a DSLContext instance in the JOOQ toolkit, executing the already prepared sql statement using DSLContext, and returning response data to the client according to the generic return class format.
For a better illustration of the present invention, a more specific example is provided below to illustrate the present invention when an access request performs an insert operation for a data source in a heterogeneous data source, where the access request is as follows:
POST http://192.168.24.25:8080/api/access/{ data Source name }/test
Request body:
method＝POST
id＝6
name＝zhaosi
age＝24
after generating a driver according to the access request, accessing a corresponding data source in the corresponding heterogeneous data to execute a corresponding operation, and returning a result to the client; specifically, the above access request of the insert operation example corresponds to executing an SQL statement: "INSERT INTO test (id, name, age) VALUES (6, zhaosi, 24)".
It should be appreciated that when the access request is to delete data in any one of the heterogeneous data sources, the access request is as follows:
POST http://192.168.24.25:8080/api/access/{ data Source name }/test/1
Request body:
method＝DELETE
deleteReally＝true
After the driver is generated according to the access request, accessing the corresponding data source in the corresponding heterogeneous data to execute the corresponding operation, and specifically, the access request of the deletion example corresponds to executing the SQL statement: "DELETE FROM TEST WHERE ID = '1'", with the id field being the primary key of table test. It should be appreciated that specifying a_where parameter by the comparison operator specifies deleting a certain data record, such as_where= (NAME, eq, "c 1") corresponds to an sql statement: WHERE NAME = "c1", i.e., data whose NAME field is c1 is deleted. In addition, it should be noted that the database publishing service supports both real delete and logical delete, which requires that the data table must contain a "del_flag" field (case not limited), and that the logical delete operation actually sets the "del_flag" field to "1".
It should be appreciated that when an access request is to update data in any one of the heterogeneous data sources, the access request is as follows:
POST http://192.168.24.25:8080/api/access/{ data Source name }/test
Request body:
method＝PUT
where＝(id,eq,5)
name＝zhangsan
age＝18
After generating a driver according to the access request, accessing a corresponding data source in the corresponding heterogeneous data to execute a corresponding operation, and returning a result to the client; specifically, the above update example access request corresponds to executing an SQL statement: "UPDATE TEST SET AGE = '25', name= 'zhangsan', age= '18' where id = 5". In addition, the value of the data record can be updated by specifying the primary key value, when the updating operation is performed in this way, whether the primary key value exists or not needs to be judged before the updating operation, if so, normal updating operation is performed, and if not, the primary key column is wrongly reported to be not exist. (i.e., update operation), without this field, the logical delete operation will not be performed.
It should be understood that when operating on data of any one of heterogeneous data sources, it may be necessary to obtain information such as a primary key and an external key of the data record, but because different data sources (databases) have differences in sql statements for these complex operations, and the sql statements of these operations themselves are also complex. Thus, the present invention employs the general approach provided by the DatabaseMetaData interface in JDBC to simplify such operations, for example: the foreign key set of the specified data table is obtained by a method ResultSet getImportedKeys (String category, STRING SCHEMA, string table).
It should be understood that, in the heterogeneous data source access method provided in this embodiment, the access request may also be table information of any one of the heterogeneous data sources, by operating a metadata object of the data source, using the getTables method provided by DatabaseMetaData to obtain a result set, traversing the result set, obtaining data of all tables and returning with a custom generic entity. The access request may also be an association table query operation. In some examples, the access request may also be an association table query operation for a data record in a foreign key access sub-table for a data record. Firstly, obtaining an external key in a sub-table by operating a metadata object of the data source by using a getImportedKeys method provided by DatabaseMetaData; and secondly, loading a database driver to acquire connection, and creating DSLContext an sql statement for executing the association table query by the instance, so as to return a result set to the client by the self-defined general class entity.
According to the method provided by the embodiment of the invention, the access request transmitted by the client is received, and the access request is used for requesting to access any one of heterogeneous data sources; solving and separating out a data source corresponding to the access request according to the access request, and generating a driver corresponding to the access request; the driver is used for accessing the corresponding data sources in the heterogeneous data sources, so that the access according to the access request is realized, a lightweight unified access interface is provided through the client, a plurality of data sources are integrated and accessed in software, the situation that when a plurality of databases exist, namely the heterogeneous data sources, each business system needs to develop corresponding database access codes and services is avoided, and further the user experience is improved.
As shown in fig. 6, an embodiment of the present invention provides an electronic device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 perform communication with each other through the communication bus 114,
A memory 113 for storing a computer program;
In one embodiment of the present invention, the processor 111 is configured to implement the steps of the heterogeneous data source unified access method provided in any one of the foregoing method embodiments when executing the program stored in the memory 113.
The embodiment of the invention also provides a storage medium, on which a computer program is stored, which when executed by a processor implements the steps of the heterogeneous data source unified access method provided by any one of the method embodiments described above.
It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing is only a specific embodiment of the invention to enable those skilled in the art to understand or practice the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (5)
1. The unified access method for the heterogeneous data sources is characterized by comprising the following steps of:
Receiving a data access request sent by a client, wherein the access request is used for requesting to access any one of heterogeneous data sources;
According to the access request, assembling a corresponding access statement, analyzing a data source corresponding to the access request, and loading a driver of the data source to acquire the data source connection;
Connecting the data source to execute the access statement, receiving a result returned by the data source, packaging the result into a general entity, and returning the general entity to the client;
before receiving the data access request sent by the client, the method further comprises:
Receiving a data source issuing instruction, and creating and issuing the heterogeneous data source, wherein the issuing instruction comprises the required data source basic information filled in through a front-end interface;
Designing a heterogeneous data source generalized access interface capable of crossing languages and platforms in a Restful mode, setting access rules and typical query parameters, and sending the access request by the client through the access interface;
The Restful mode is adopted to design a universal access interface of heterogeneous data sources which can cross languages and cross platforms, comprising,
Designing a generalized access interface for accessing heterogeneous data sources based on a RESTful method, wherein the access interface supports cross-programming language access;
In the process of accessing data, each programming language can realize that different RESTful clients access data of heterogeneous data sources, and access the data in the published data sources by calling the access interface;
access rules and typical query parameters are set, including,
Typical query parameters are designed, wherein the query parameters support association tables, compound primary keys, paging, sorting, field filtering, record filtering, querying single records, counting, exist querying, grouping, inserting, updating and deleting data source operations;
designing a comparison operator and a logic operator to realize filtering of a result set according to recorded contents;
The custom general response class generates a data response entity, and the response access request is returned to the client in a unified format.
2. The method of claim 1, wherein after receiving a data source issue instruction, creating and issuing the heterogeneous data source, the method further comprises:
and storing the published heterogeneous data sources, and storing the heterogeneous data sources in a local disk in a configuration file mode.
3. The method of claim 2, wherein after receiving a data source issue instruction, creating and issuing the heterogeneous data source, the method further comprises: and creating a corresponding connection pool according to different data sources, wherein the connection pool is used for distributing, managing and releasing the data source connection, and the connection pool allows an application program to repeatedly use one existing data source connection, so that the response time to the data request and the server performance are improved.
4. A method according to claim 3, wherein loading the driver of the data source to obtain the data source connection comprises:
loading a corresponding driver of the data source;
Judging whether a data source corresponding to the access request exists in the existing connection pool, if so, acquiring data source connection to access the data source corresponding to the access request;
If the data source connection request does not exist, a new connection pool of the data source is created according to a preset connection pool configuration rule, the data source connection is obtained, and then the data source corresponding to the access request is accessed.
5. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus,
A memory for storing a computer program;
a processor for implementing the steps of the heterogeneous data source unified access method according to any one of claims 1 to 4 when executing a program stored on a memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111192740.6A CN114064684B (en) | 2021-10-13 | 2021-10-13 | Unified access method for heterogeneous data sources |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111192740.6A CN114064684B (en) | 2021-10-13 | 2021-10-13 | Unified access method for heterogeneous data sources |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114064684A CN114064684A (en) | 2022-02-18 |
| CN114064684B true CN114064684B (en) | 2024-07-19 |
Family
ID=80234370
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111192740.6A Active CN114064684B (en) | 2021-10-13 | 2021-10-13 | Unified access method for heterogeneous data sources |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114064684B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116303233A (en) * | 2022-12-19 | 2023-06-23 | 广州市玄武无线科技股份有限公司 | Data storage management method, device, equipment and computer storage medium |
| CN115809102A (en) * | 2022-12-22 | 2023-03-17 | 北京航星永志科技有限公司 | Data source configuration device, method and equipment |
| CN116303581B (en) * | 2023-05-19 | 2023-08-04 | 山东浪潮数字商业科技有限公司 | Method, system, equipment and medium for adapting split-flow query load among heterogeneous databases |
| CN117539912B (en) * | 2023-11-23 | 2024-06-14 | 武汉软件工程职业学院(武汉开放大学) | System, method and equipment for accelerating heterogeneous data source access |
| CN117453731A (en) * | 2023-12-22 | 2024-01-26 | 北京宇信科技集团股份有限公司 | Multi-source data query system and multi-source data query method |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101930469A (en) * | 2010-09-02 | 2010-12-29 | 江西省电力公司超高压分公司 | Universal data access method of cross-platform multi-heterogeneous database in scheduling data service platform |
| CN107122365A (en) * | 2016-02-25 | 2017-09-01 | 阿里巴巴集团控股有限公司 | The access method and device of heterogeneous database |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB0206810D0 (en) * | 2002-03-22 | 2002-05-01 | Isocra Ltd | Database system |
| CN108959496A (en) * | 2018-06-26 | 2018-12-07 | 南京汇银迅信息技术有限公司 | Integration across database access method and abstract data access method based on dynamic proxy |
| CN109992617B (en) * | 2019-03-21 | 2022-08-26 | 国电南瑞科技股份有限公司 | Data access method and system supporting multiple programming languages and multiple data sources |
-
2021
- 2021-10-13 CN CN202111192740.6A patent/CN114064684B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101930469A (en) * | 2010-09-02 | 2010-12-29 | 江西省电力公司超高压分公司 | Universal data access method of cross-platform multi-heterogeneous database in scheduling data service platform |
| CN107122365A (en) * | 2016-02-25 | 2017-09-01 | 阿里巴巴集团控股有限公司 | The access method and device of heterogeneous database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114064684A (en) | 2022-02-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114064684B (en) | Unified access method for heterogeneous data sources | |
| US7526769B2 (en) | Message translation and parsing of data structures in a distributed component architecture | |
| US11574070B2 (en) | Application specific schema extensions for a hierarchical data structure | |
| US7895226B2 (en) | System and method for translating and executing update requests | |
| US7290049B2 (en) | Management information to object mapping and correlator | |
| US7383552B2 (en) | Object manager for common information model | |
| US8438190B2 (en) | Generating web services from business intelligence queries | |
| US7702725B2 (en) | Digital object repositories, models, protocol, apparatus, methods and software and data structures, relating thereto | |
| US8275775B2 (en) | Providing web services from business intelligence queries | |
| US6401083B1 (en) | Method and mechanism for associating properties with objects and instances | |
| WO2002019652A2 (en) | System and method for transmitting and retrieving data via a distributed persistence framework | |
| US8694500B2 (en) | Method and apparatus for document matching | |
| CN108038213A (en) | A kind of method of data processing, client, server and system | |
| CN113254470B (en) | Data modification method, device, computer equipment and storage medium | |
| CN113254519B (en) | Access method, device, equipment and storage medium of multi-source heterogeneous database | |
| US20110231889A1 (en) | Security policy as query predicate | |
| US20040025142A1 (en) | Method and apparatus for managing objects in a CIM environment | |
| Kim et al. | Comprehensive Knowledge Archive Network harvester improvement for efficient open‐data collection and management | |
| CN112835638A (en) | Configuration information management method and device based on embedded application program | |
| US20050080759A1 (en) | Transparent interface to a messaging system from a database engine | |
| CN113779068B (en) | Data query method, device, equipment and storage medium | |
| US11860679B2 (en) | Detecting security vulnerabilities in queries via an information flow graph | |
| JP2006146615A (en) | Object-related information management program, management method and management apparatus | |
| CN115905274A (en) | Data processing method and device, electronic equipment and medium | |
| CN114003583A (en) | Method, device, medium and equipment for constructing target format data request body |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |