CN112612809A - Data access assembly and data access method - Google Patents

Abstract

The invention discloses a data access assembly, comprising: the system comprises a data access interface, a model mapping component, a model adapter and a query and command arranging component; the model mapping component is used for connecting the data access interface and each model adapter; the model adapter is used for directly operating the database connected with the model adapter, and the model adapter corresponds to various databases one by one; and the query and command arrangement component is used for analyzing the query statement and automatically routing the query statement to the database with the highest efficiency for query operation. The data access assembly of the invention can ensure the multivariate storage and the multivariate access of data, and the automatic routing can ensure certain access performance.

Description

Data access assembly and data access method

Technical Field

The invention belongs to the technical field of DAO data access, and particularly relates to a data access assembly and a data access method of the data access assembly.

Background

In a software system, there may be multiple data query requirements, such as cache query, structured data query, unstructured data query, fuzzy query, etc., which seek high performance query, and certain query performance needs to be ensured. However, the existing data access component in the industry has the following problems:

1) in a traditional data system, the same data is often stored in the same medium, and IO operation aims at the same storage medium, so that IO performance bottleneck and deadlock are easily caused; in addition, some systems adopt a means of data copying and distributed storage, and in actual operation, a means for maintaining data consistency is often lacked or a data copying program is unreliable;

2) the traditional data access assembly needs to adopt different data access modes for different storage media, even if some data frames use ORM technology, the data in different storage media are difficult to access by using a uniform model, and particularly, a relationship in a logic model has no uniform description mode;

3) in the aspect of improving the query performance, a layer of memory database is added between the application and the database as cache, so that the method is heavy; during development, the data of the cache library and the database needs to be added, deleted, modified and checked and the data consistency needs to be considered, and the workload of developers is large.

Disclosure of Invention

The invention aims to overcome the defects in the prior art and provides a data access assembly and a data access method.

To solve the above technical problem, the present invention provides a data access component, including:

the system comprises a data access interface, a model mapping component, a model adapter and a query and command arranging component;

the data access interface is used for providing a uniform data access interface for the outside;

the model mapping component is used for connecting the data access interface and each model adapter;

the model adapter is used for directly operating the database connected with the model adapter, and the model adapter corresponds to various databases one by one;

and the query and command arrangement component is used for analyzing the query statement and automatically routing the query statement to the database with the highest efficiency for query operation.

Further, the database includes: one or more of Oracle, MySQL, MongoDB, HBASE, Elastic Search and Gemfire.

Further, the MongoDB database is used for storing hotspot data.

Further, the Elastic Search database is used to store data that requires fuzzy queries.

Correspondingly, based on the data access component, the data access method of the data access component of the invention is characterized by comprising the following processes:

acquiring a uniform query statement through a uniform data access interface;

analyzing the unified query statement;

and automatically routing to the most efficient database for query operation.

Further, the automatic routing to the most efficient database for query operation includes:

the automatic inquiry routing is realized by sequentially judging according to the following rules:

rule 1: a target data source to be queried specified in a unified object query statement;

rule 2: whether a certain database supports all clauses contained in the uniform object query statement or not;

rule 3: whether the attribute of the inquired object is stored in the target database;

rule 4: the data access component defines default query priorities for the different databases, and the default priorities are configurable.

Compared with the prior art, the invention has the following beneficial effects: the data access assembly of the invention can ensure the multivariate storage and the multivariate access of data, and the automatic routing can ensure certain access performance.

Drawings

FIG. 1 is a diagram of a unified data model and access method;

FIG. 2 shows a data assembly with write-once-many and read-write separation;

FIG. 3 is a diagram illustrating database priorities.

Detailed Description

The invention is further described below with reference to the accompanying drawings. The following examples are only for illustrating the technical solutions of the present invention more clearly, and the protection scope of the present invention is not limited thereby.

Example 1

The invention conception of the invention is as follows: firstly, supporting multi-element storage, namely supporting the access of various relational databases and nosql databases; then, when the content is queried, the query sentence is automatically routed to the database with the highest efficiency for query operation, so that the query performance is improved. Therefore, not only can diversified query requirements be met, but also higher query efficiency can be ensured.

Based on the technical scheme, the data access component design method based on data multivariate storage and automatic query routing comprises the following steps:

in the first step, the multi-element storage is supported, and the diversified storage of data is met

The invention adopts various database combinations to store data, realizes data multivariate storage, and the databases adopted in the embodiment comprise: one or more combinations of Oracle, MySQL, MongoDB, HBASE, Elastic Search, and Gemfire databases. The different databases can store suitable data according to different characteristics of the databases to meet different query requirements. For example, hot spot attributes of an object may be stored in a memory library (MongoDB), and attributes that require fuzzy query may be stored in a full-text index library (Elastic Search). And through the query and command arrangement component, automatic query arrangement optimization, IO threshold control, write-once-many and read-write separation of data can be realized, and the data read-write performance is further improved.

These databases support access to and manipulation of a variety of data, such as model mapping, adapter schema, unified query language completion query operations.

And step two, automatically inquiring the route, shielding the routing process of the data and improving the software development efficiency.

When the query statement is executed, the query statement is automatically routed to the database with the highest efficiency through a routing algorithm for query operation. The routing algorithm considers factors such as database priority, whether a database is formulated, query statement composition analysis, whether object data is stored in a target database and the like.

Example 2

(1) Implementation supporting multiple storage

The multi-storage specific implementation architecture is as shown in fig. 1, a uniform data access interface is provided for the outside, and the data access interface is connected to the model adapters corresponding to the databases one to one through the model mapping component. By adopting model mapping, adapter mode and uniform data access interface (namely uniform query language), the method supports access and operation of various data: oracle, MySQL, MongoDB, HBASE, Elastic Search, Gemfire.

One or more data storage modes can be selected for the software system for different service scenarios. For example, a memory bank is selected for hot data, a column-type bank is adopted for massive data, and full-text index is adopted for data needing fuzzy query. Through the query and command arrangement component, automatic query arrangement optimization, IO threshold control, write-once-many, read-write separation of data can be realized, as shown in fig. 2.

(2) Implementation of automated query routing algorithm

The data access component designed in the invention uses the uniform object query language matched with the data access component to execute the query operation. The data access component supports query clauses (e.g., where, having, join, distinting, groupby, orderby, etc.) of a unified query language, in part or in whole, depending on the own characteristics of each database. For example, MongoDB and HBase do not realize the support of the join clause; only Elastic Search implements support for the Search clause. When the data access component executes query, the query and command arrangement component analyzes the unified query statement and then sequentially judges according to the following rules to realize automatic query routing, and further determines which database to execute query operation on:

rule 1: a target database to query specified in a unified object query statement.

Rule 2: whether a specific database supports all the clauses contained in the uniform object query statement.

Rule 3: whether the attributes of the queried object are stored in the target database.

Rule 4: the data access component defines default query priorities for the different databases, and the default priorities are configurable. For example, the priority order may be as in FIG. 3:

①Gemfire

②MongoDB

③HBase

④Oracle、MySQL

⑤Elastic Search

the routing algorithm is judged by the rules 1 to 4 in sequence. Wherein, for rule 1: the query target database may be specified in a unified object query statement. It should be noted that the data access component in the present invention does not support cross-database association queries, so that more than one database may not be specified in a unified object query statement. For rule 2: after the unified query statement to be executed is analyzed by the query and command arrangement component, a specific database is selected by the corresponding query clause support capability so as to achieve the purpose of data routing.

The data access component in the invention designs the clause supporting capability of different databases as follows according to the characteristics of different databases (Y':

TABLE 1 clause support capabilities for disparate databases

In addition, in the case where objects are stored in multiple databases, the data access component in the present invention determines at which database the query is executed by default by querying database capability match (rule 2), database priority (rule 4).

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.

Claims (6)

1. A data access component, comprising:

the system comprises a data access interface, a model mapping component, a model adapter and a query and command arranging component;

the data access interface is used for providing a uniform data access interface for the outside;

the model mapping component is used for connecting the data access interface and each model adapter;

the model adapter is used for directly operating the database connected with the model adapter, and the model adapter corresponds to various databases one by one;

and the query and command arrangement component is used for analyzing the query statement and automatically routing the query statement to the database with the highest efficiency for query operation.

2. A data access component according to claim 1, wherein the database comprises: one or more of Oracle, MySQL, MongoDB, HBASE, Elastic Search and Gemfire.

3. A data access component as claimed in claim 2 wherein the montgodb database is used to store hotspot data.

4. A data access component according to claim 2, wherein the Elastic Search database is arranged to store data requiring fuzzy queries.

5. A data access method for a data access component, comprising the steps of:

acquiring a uniform query statement through a uniform data access interface;

analyzing the unified query statement;

and automatically routing to the most efficient database for query operation.

6. The data access method of claim 5, wherein said automatically routing to a most efficient database for query operations comprises:

the automatic inquiry routing is realized by sequentially judging according to the following rules:

rule 1: a target data source to be queried specified in a unified object query statement;

rule 2: whether a certain database supports all clauses contained in the uniform object query statement or not;

rule 3: whether the attribute of the inquired object is stored in the target database;

rule 4: the data access component defines default query priorities for the different databases, and the default priorities are configurable.

Images