CN116541444A

CN116541444A - Heterogeneous database integration system and database access method

Info

Publication number: CN116541444A
Application number: CN202210095845.8A
Authority: CN
Inventors: 刘亮; 张晟; 顾强; 孙小娟
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Group Jiangsu Co Ltd
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Group Jiangsu Co Ltd
Priority date: 2022-01-26
Filing date: 2022-01-26
Publication date: 2023-08-04

Abstract

The application provides a heterogeneous database integration system and a database access method, and belongs to the technical field of computers. The system comprises: a unified access interface layer, a data routing layer, a unified data operation layer and a data storage layer; the unified access interface layer is used for analyzing the user access request and acquiring unified data model object parameters corresponding to the user access request; the data routing layer is used for determining a target database in the data storage layer based on the unified data model object parameters and the routing algorithm; the unified data operation layer is used for accessing the target database through the unified database operation interface based on the unified data model object parameters and the types of the target database, and obtaining a target data result set corresponding to the user access request. The heterogeneous database integration system provided by the embodiment of the application can realize dynamic switching of databases based on application parameter transmission and routing algorithms, and realize heterogeneous data access mode unification in a unified data operation layer.

Description

Heterogeneous database integration system and database access method

Technical Field

The application relates to the technical field of computers, in particular to a heterogeneous database integration system and a database access method.

Background

The advent of cloud computing has created an impact and challenge to traditional internet technology (Internet Technology, IT). The container technology brings a breakthrough solution to a series of challenges such as explosive growth of data, increase of service types, increase of application complexity, and increase of software iteration speed.

The container has the characteristics of agile development, rapid delivery, portability and the like, can provide an excellent running environment for the Internet application, and subverts the mode of enterprise-level application delivery. With cloud-native technology, the upper layer applications become more flexible, but the development of the underlying data layer remains in the rest. Because the adjustment and the faults of the bottom database have great influence on the upper application, the common unified data access layer only solves the management of database links, and how to realize unified data access to various databases and dynamic switching of the databases becomes a problem to be solved.

Disclosure of Invention

Aiming at the problems existing in the prior art, the embodiment of the application provides a heterogeneous database integration system and a database access method.

In a first aspect, an embodiment of the present application provides a heterogeneous database integration system, including:

a unified access interface layer, a data routing layer, a unified data operation layer and a data storage layer;

The unified access interface layer is used for analyzing a user access request and acquiring unified data model object parameters corresponding to the user access request;

the data routing layer is used for determining a target database in the data storage layer based on the unified data model object parameters and a routing algorithm; the routing algorithm is determined based on target influencing factors influencing routing;

the unified data operation layer comprises a unified database operation interface, and is used for accessing the target database through the unified database operation interface based on the unified data model object parameters and the type of the target database to obtain a target data result set corresponding to the user access request.

In one embodiment, the data routing layer is further configured to:

determining the target influencing factors;

determining a routing expression based on the target influencing factors;

the routing algorithm is determined based on the routing expression.

In one embodiment, the data routing layer is specifically configured to:

and combining the target influencing factors through a logic operator to determine the routing expression.

In one embodiment, the routing algorithm has the expression:

routing expression x preset weight & routing state & instance number;

and the instance numbers are used for representing database instances corresponding to any one database in the data storage layer, and belong to different instance numbers corresponding to different database instances in the same database respectively.

In one embodiment, the heterogeneous database integration system further comprises: a data caching layer;

the unified data manipulation layer is further configured to: converting the data in the target data result set into a unified data model, and inputting the unified data model into the data caching layer;

the data caching layer is used for caching the unified data model.

In one embodiment, the heterogeneous database integration system further comprises: a business application layer;

the data caching layer is further used for:

judging whether the data caching layer caches the target data result set or not based on the unified data model object parameters;

and under the condition that the data caching layer caches the target data result set, transmitting a unified data model corresponding to the target data result set to the service application layer through the unified access interface layer.

In one embodiment, the heterogeneous database integration system further comprises: an insert layer;

and the plug-in layer is used for performing function expansion on the heterogeneous database integrated system.

In one embodiment, the interposer layer includes:

and the unified transaction management and control unit is used for carrying out distributed transaction management on the database operation sequence under the condition that the unified data operation layer accesses the target database through the unified database operation interface.

In a second aspect, an embodiment of the present application provides a database access method based on any one of the heterogeneous database integration systems in the first aspect, including:

analyzing a user access request and acquiring unified data model object parameters corresponding to the user access request;

determining a target database based on the unified data model object parameters and a routing algorithm; the routing algorithm is determined based on target influencing factors influencing routing;

and accessing the target database through the unified database operation interface based on the unified data model object parameters and the type of the target database, and obtaining a target data result set corresponding to the user access request.

In one embodiment, the determining the target database based on the unified data model object parameters and the routing algorithm includes:

determining the target influencing factors;

determining a routing expression based on the target influencing factors;

the routing algorithm is determined based on the routing expression.

In one embodiment, the determining a routing expression based on the target influencing factor includes:

In one embodiment, the routing algorithm has the expression:

routing expression x preset weight & routing state & instance number;

the instance numbers are used for representing database instances corresponding to any one database, and different database instances in the same database correspond to different instance numbers respectively.

In one embodiment, after the accessing the target database through the unified database operation interface based on the unified data model object parameters and the type of the target database, the method further comprises, after obtaining a target data result set corresponding to the user access request:

Converting the data in the target data result set into a unified data model;

and caching the unified data model.

In one embodiment, before said determining a target database based on said unified data model object parameters and a routing algorithm, said method further comprises:

judging whether the target data result set is cached or not based on the unified data model object parameters;

and under the condition that the target data result set is determined to be cached, sending a unified data model corresponding to the target data result set to the service application layer.

In one embodiment, the method further comprises:

and under the condition that the target database is accessed through the unified database operation interface, carrying out distributed transaction management on a database operation sequence.

In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory storing a computer program, where the processor implements the steps of the database access method according to the second aspect when executing the program.

In a fourth aspect, embodiments of the present application provide a computer program product comprising a computer program which, when executed by a processor, implements the steps of the database access method of the second aspect.

According to the heterogeneous database integration system and the database access method, a user access request is analyzed, unified data model object parameters corresponding to the user access request are obtained, a target database is determined based on the unified data model object parameters and a routing algorithm, the target database is accessed through a unified database operation interface based on the unified data model object parameters and the type of the target database, and a target data result set corresponding to the user access request is obtained; the method is based on a cloud native technology architecture, dynamic switching of databases is achieved through application of a parameter transmission and routing algorithm, and heterogeneous data access mode unification is achieved through a unified database operation interface.

Drawings

For a clearer description of the present application or of the prior art, the drawings that are used in the description of the embodiments or of the prior art will be briefly described, it being apparent that the drawings in the description below are some embodiments of the present application, and that other drawings may be obtained from these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a heterogeneous database integration system according to an embodiment of the present disclosure;

fig. 2 is a schematic flow chart of determining a target database example by the data routing layer according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a unified data manipulation layer converting unified data model provided in an embodiment of the present application;

fig. 4 is a schematic flow chart of data caching of the data caching layer according to the embodiment of the present application;

FIG. 5 is a second schematic diagram of a heterogeneous database integration system according to an embodiment of the present disclosure;

FIG. 6 is a schematic flow chart of a database access method according to an embodiment of the present disclosure;

FIG. 7 is a second flowchart of a database access method according to an embodiment of the present disclosure;

fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

In order to facilitate a clearer understanding of the various embodiments of the present application, some relevant background knowledge is first presented below.

The existing database access scheme can not realize related technologies such as bottom instance, library switching, unified transaction management, data buffering, cold and hot data separation and the like, and mainly has the following 3-point defects:

(1) Only the master and slave take-over functions in the database can be supported, and the dynamic switching among multiple databases can not be realized: for example, the service multi-instance switching of Oracle solves the problem of instance single point under the same library, and the internal driving layer at the application side is automatically reconnected when the instance under the same library is switched, so that the influence of the instance switching under the single library on the application can be solved to a certain extent;

(2) Only homogeneous data link layer or link pool management can be implemented: the relatively good data middleware which is commonly used in various database link pool tools can only solve the function of the bottom layer route, is limited to a Mysql database, and cannot be applied to a KV (Key-Value) database or a distributed memory database. After heterogeneous data switching is encountered, the problem of service interruption caused by application restarting cannot be avoided;

(3) The operation modes are not uniform, and the development threshold is higher: after the application of the multimode data types is clouded, the relational database can shield code redundancy caused by the bottom layer difference through a common solution of the common relational database and service object mapping (for example, object relational mapping (Object Relational Mapping, ORM), so that development mainly focuses on object-oriented programming, but the solution must depend on java database connection (Java DataBase Connectivity, JDBC) interfaces, so that the supportable database must implement JDBC related interfaces, and as the technical architecture of big data evolves, a part of non-relational databases focusing on big data processing do not support JDBC interfaces well, so that the solution cannot meet the use of the novel database, even if a part of non-relational databases realize JDBC interfaces, but due to the reason of the bottom architecture, JDBC interface performance is not optimal solution, even some performances do not meet the actual production requirement performance, so that the JDBC interfaces must be taken into consideration when the technical architecture is selected, and application program interfaces (Application Programming Interface, API) with higher performance are directly abandoned for interactive use.

In order to overcome the defects in the prior art, the embodiment of the application converges the management and access of the database resources through the unified data access layer, so that the application access database is just like an application after cloud loading, the application does not need to care about the bottom environment, only needs to ask for the proposal, the proposal provided by the embodiment of the application is uniformly arranged and uniformly managed, and meanwhile, the application also comprises the characteristics of data buffering, cold and hot data separation, dynamic routing, distributed transaction and the like, and the data bottom layer is more 'cloud-primary'. The whole scheme of the embodiment of the application takes the cloud original design concept as a standard to realize clouding of the data access layer.

The heterogeneous database integration system and the database access method provided in the embodiments of the present application are described below with reference to fig. 1 to 8.

Fig. 1 is a schematic structural diagram of a heterogeneous database integration system according to an embodiment of the present application. Referring to fig. 1, an embodiment of the present application provides a heterogeneous database integration system, which may include:

a unified access interface layer 110, a data routing layer 120, a unified data manipulation layer 130, and a data storage layer 140;

the unified access interface layer 110 is configured to parse a user access request and obtain unified data model object parameters corresponding to the user access request;

The data routing layer 120 is configured to determine a target database in the data storage layer 140 based on the unified data model object parameters and a routing algorithm; the routing algorithm is determined based on target influencing factors influencing routing;

the unified data operation layer 130 includes a unified database operation interface, and the unified data operation layer is configured to access the target database through the unified database operation interface based on the unified data model object parameter and the type of the target database, and obtain a target data result set corresponding to the user access request.

Specifically, in order to overcome the defect that the conventional database access technology is difficult to realize unified data access of multiple databases and dynamic switching of the databases, the embodiment of the application provides a heterogeneous database integrated system which comprises a unified access interface layer, a data routing layer, a unified data operation layer and a data storage layer. Firstly, analyzing a user access request through a unified access interface layer, and acquiring unified data model object parameters corresponding to the user access request; further, the data routing layer determines a target database based on the unified data model object parameters and the routing algorithm; then, the unified data operation layer accesses the target database through the unified database operation interface based on the unified data model object parameters and the type of the target database, and acquires a target data result set corresponding to the user access request; the embodiment of the application is based on a cloud native technology architecture, realizes dynamic switching of databases by applying a parameter transmission and routing algorithm, and realizes heterogeneous data access mode unification by unifying database operation interfaces.

Optionally, an embodiment of the present application provides a heterogeneous database integrated system, which may include: a unified access interface layer 110, a data routing layer 120, a unified data manipulation layer 130, and a data storage layer 140.

Alternatively, the unified access interface layer 110 may be configured to parse the user access request and obtain unified data model object parameters corresponding to the user access request.

Optionally, the unified access interface layer 110 may decouple the service code from the underlying data access, and apply coding to implement all actions of operating the database to be intercepted by the layer, and convert the request message into a unified data model and a unified operation method, where the request message may include database operation method add-Delete (CRUD) and related request parameters.

Specifically, in the case that the user initiates the data access request, the unified access interface layer 110 may first intercept the user request, and analyze the manner (including the manner of database operation method CRUD) of the database accessed by the application and other relevant request parameters; the business data is then parsed by the model based on the request parameters, converted into a unified data model, and unified data model object parameters are generated for transmission to the data routing layer 120 and the unified data manipulation layer 130 for processing.

Alternatively, the data routing layer 120 may be used to determine a target database in the data storage layer based on the unified data model object parameters and routing algorithms.

Alternatively, the routing algorithm may be determined based on target influencing factors that influence the routing.

Alternatively, the data routing layer 120 may be used to implement routing capabilities for a variety of databases, through routing algorithms, to dynamically determine databases or database instances in conjunction with application request parameters at program runtime.

Specifically, after receiving the request parameters sent by the unified access interface layer 110, the data routing layer 120 may perform real-time calculation based on the request parameters and the routing algorithm, and select the database to be accessed and the corresponding database instance.

Alternatively, the unified data manipulation layer 130 may include a unified database manipulation interface.

It will be appreciated that to implement the underlying data operation automation model, heterogeneous data access model unification must be implemented at the unified data operation layer 130, and this goal is primarily dependent on the need for a unified data operation interface.

Alternatively, two different solutions may be employed for the characteristics of relational and non-relational databases: the relational database can be improved by the Mybatis framework; the non-relational database may be encapsulated secondarily by the API. Because the interfaces of each non-relational database are not consistent, the embodiment of the application can realize the bottom layer operation of different databases based on the interface by abstracting a set of data interfaces similar to the relational database, namely a unified data operation interface layer, and shield the bottom layer difference.

Optionally, for the non-relational database or the memory cache which does not support JDBC, according to the definition of the unified data operation interface layer method, the method logic implementation is completed on the basis of each database API, so that the service application development is not aware of the difference operation of the underlying heterogeneous database, the development is smoother, the service code implementation is emphasized more, the application can experience the convenience of the relational database, and the timeliness of the massive data query.

Alternatively, embodiments of the present application may provide for application development using an expression style of a structured query language (Structured Query Language, SQL), the expression format may refer to table 1.

Table 1 class SQL expression examples

When an in-memory database or a Nosql database is used in the statistical application code, the type of a database native interface and interface parameter definition which are frequently called are used for extracting several elements required in the expression: table name (or cache name, expression KEY: cacheName), a primary KEY (or KEY of KV data, expression KEY), a query condition (expression KEY: filter, o.user_id= =1||o.username < 'jessa 20003' is conditional content.

Optionally, the embodiment of the application can map the database operation method into a relatively simple SQL expression (for example, query a record, access through a cahce.get () API interface is needed when the non-relational database is realized, the scheme is directly converted into a mode of selecting from cahce where key =and the application does not need to call through codes, analyze SQL provided by the application and automatically generate API calls), so that hard coding realization is reduced, and more attention is paid to how the service realization is realized by using a more concise expression description when the application is developed.

It can be understood that the distributed computing is a common component of the non-relational database, in a big data scenario, the traditional data extraction to the computing node and the operation mode are not applicable any more, and the data transmission process is far longer than the computing processing time, so that a large number of non-relational databases are only in an EntryProcess mode (also called a remote micro-service mode), the common micro-computing service application is sent to a server where the data is located, the remote data extraction process is changed into a local processing mode, and finally the processed data is summarized again, thereby greatly reducing the application processing time. Aiming at the EP (Expression Param) expression, the application can configure an interface needing to call the remote service through the EP key, and the scheme provided by the embodiment of the application can automatically generate a result of calling codes to extract remote micro-service calculation in a remote call mode.

It can be understood that the SQL parsing engine is powerful but complex and is not suitable for a non-relational database, and the scheme adopts a custom EL (Expression Language) expression to realize the operation of the non-relational database, wherein the main elements in the expression are as follows: the table structure names, operation methods, conditions, EP and the like are aimed at the operation interface description commonly used by the non-relational database, and SQL-like operation can be realized by analyzing the custom EL expression.

Alternatively, any type of database in data storage layer 140 may be accessed through a unified database operating interface.

Alternatively, the unified data operation layer 130 may be configured to access the target database through the unified database operation interface based on the unified data model object parameters and the type of the target database, and obtain the target data result set corresponding to the user access request.

Optionally, the unified data operation layer 130 may be configured to unify the operation methods of the underlying databases, automatically adapt the underlying databases according to the types of the operation methods, select implementation interfaces of different databases to access the target database, and feed back the accessed target data result set to the service application.

Alternatively, the data storage layer 140 may include different types of databases, which may include relational, non-relational, KV, in-memory databases, column-oriented databases, and the like, for example.

In particular, the data storage layer 140 may include an Oracle database, a Reids database, an Hbase database, and the like, which is not particularly limited in the embodiments of the present application.

According to the heterogeneous database integration system provided by the embodiment of the application, the unified data model object parameters corresponding to the user access request are obtained through analyzing the user access request, the target database is determined based on the unified data model object parameters and the routing algorithm, the target database is accessed through the unified database operation interface based on the unified data model object parameters and the type of the target database, and the target data result set corresponding to the user access request is obtained; the method is based on a cloud native technology architecture, dynamic switching of databases is achieved through application of a parameter transmission and routing algorithm, and heterogeneous data access mode unification is achieved through a unified database operation interface.

Optionally, the data routing layer 120 is further configured to:

determining the target influencing factors;

determining a routing expression based on the target influencing factors;

the routing algorithm is determined based on the routing expression.

Specifically, the data routing layer 120 may first determine, based on the traffic data, a target influencing factor that influences the routing; determining a routing expression further based on the target influencing factor; finally, a routing algorithm is determined based on the determined routing expression.

Specifically, the data routing layer 120 may have dynamic routing access capability of different databases and different tables, and in the encoding stage, a routing expression may be written based on variable conditions; in the operation stage, the method can automatically calculate according to the method parameters, the system parameters and the context parameters, calculate the target database to be accessed, and dynamically adjust and access the main database, the standby database or the emergency according to the condition of the target data.

Optionally, during service development, target influencing factors affecting the routing, such as factor a= =100, factor B <1000, etc., may be abstracted according to the switching requirement of the actual service data, and a routing expression is determined based on the abstracted factors, so as to determine a specific routing algorithm based on the routing expression.

According to the method and the device, the routing expression is determined based on the target influence factors, and then the routing algorithm is determined based on the routing expression, so that the dynamic switching of the database based on the application parameter transmission and the determined routing algorithm can be realized.

Optionally, the data routing layer 120 is specifically configured to:

Alternatively, the data routing layer 120 may be specifically configured to: the target influencing factors are combined through a logic operator, and a routing expression is determined.

For example, the determined target influencing factors a= =100 and target influencing factors B <1000 etc. may be combined by logical operators of |or & etc. to form one complete route description, i.e. a route expression.

It will be appreciated that different factor values may produce different results of operations based on routing expressions, and thus the data routing layer 120 may have multiple access capabilities.

Alternatively, the routing expression may be in a custom multi-level K-V (Key-Value) description manner, for example, the first layer mainly declares the type name of the data route (e.g., relational database route, hbse route, coherence type route), the second layer declares the second attribute of the object (e.g., dn-rdb indicates that this is a full amount relational data declaration), the third layer declares the third attribute of the object (including authority, database type, database state and database instance packet name), and so on.

Fig. 2 is a schematic flow chart of determining a target database instance by the data routing layer according to the embodiment of the present application, and as shown in fig. 2, the data routing layer 120 may first determine a data routing group, and then further determine a database instance group. Wherein the routing configuration rules may refer to table 2.

Table 2 route configuration rule example

For example, in table 2, the routing expression description includes:

the expression=context_city= 11|context_city= 12|context_city= 13 optionally weights (weights) may be preset for the routing expressions based on the specific application scenario.

Alternatively, in the case where a plurality of routing expressions are simultaneously satisfied, the target route may be selected according to a preset weight level.

For example, expression a (in-memory database):

(context_city= 11||context_city= 12& mole_num < 6) |coh_flag= 1 expression B (relational database):

(context_city＝＝11||context_city＝＝12&month_num<6)|total<1000

expression C (NOSQL database):

(context_city＝＝11||context_city＝＝12&month_num<6)|&timeout<5

under the condition that all the three expressions are established, in the accounting query system, the result set is medium, the timeliness is medium, the relational database needs to be accessed preferentially, and the relational database can be selected preferentially by increasing the weight of the expression B at the moment; however, in the recharging system, timeliness is preferential, the memory database is preferentially accessed, the weight of the memory database expression A can be increased, and the memory database is preferentially selected by the route; in the bill inquiry system, the result set is larger, the timeliness is relatively lower, the NoSQL access is better, and the NOSQL is preferentially selected by increasing the weight of the expression C.

Optionally, the routing algorithm has the expression:

routing expression x preset weight & routing state & instance number;

Alternatively, the expression of the routing algorithm determined by the data routing layer 120 may be:

routing expression x preset weight & routing state & instance number.

Alternatively, the instance number may be dynamically controlled by dynamically modifying the state of the runtime memory variable.

Optionally, the design of the instance number is convenient for meeting the instance switching of sudden faults such as main faults, standby faults, disaster recovery faults, emergency faults and the like, and the routing change is conveniently and directly triggered under the condition that the application is not required to be restarted, so that the application is quickly brought into an emergency state.

The embodiment of the application determines the routing algorithm based on the routing expression, and can realize dynamic switching of the database or database instance based on the application of the routing algorithm determined by the parameter transmission.

Optionally, the heterogeneous database integration system further includes: a data caching layer;

the unified data manipulation layer 130 is further configured to: converting the data in the target data result set into a unified data model, and inputting the unified data model into the data caching layer;

the data caching layer is used for caching the unified data model.

Optionally, the heterogeneous database integration system provided in the embodiments of the present application may further include a data caching layer.

Optionally, the unified data manipulation layer 130 may also be configured to: and converting the data in the target data result set into a unified data model, and inputting the unified data model into the data caching layer.

Optionally, the unified data manipulation layer 130 may also be configured to: and converting the data in the target data result set into a unified data model, and inputting the unified data model into an application layer.

It will be appreciated that to achieve a fully automated model of underlying data operations, heterogeneous data access patterns must be unified at the unified data manipulation layer 130, which is also dependent on a unified data model.

It can be understood that each database has a respective data model, and the existing solution cannot unify all databases, and the implementation degree of each database driver on the JDBC specification is greatly different, so that the application layer cannot uniformly develop based on JDBC completely, wherein the problem of data model difference is most prominent. The embodiment of the application can adopt a custom data serialization mode and a reverse serialization mode. Fig. 3 is a schematic diagram of a unified data operation layer conversion unified data model provided in an embodiment of the present application, as shown in fig. 3, a framework self-defines corresponding data serialization and anti-serialization protocols for different database types, unifies data models of all databases into a standard data model, shields application development from sensing a bottom data structure, and reduces development cost and learning cost of developing heterogeneous databases.

Optionally, a data caching layer may be used to cache the unified data model.

Optionally, the data caching layer may be used to cache data with a higher access frequency, so as to reduce access to the underlying database and speed up access of the application.

Optionally, the heterogeneous database integration system further includes: a business application layer;

the data caching layer is further used for:

Optionally, the heterogeneous database integration system provided in the embodiments of the present application may further include a service application layer.

Optionally, the service application layer may refer to a service code application implementation layer, for example, service logic implementation such as data query, rating, etc., where the layer may be implemented by a service developer according to actual requirements by normal coding.

Optionally, the data caching layer may further be configured to: and judging whether the data caching layer caches the target data result set or not based on the unified data model object parameters.

Optionally, the data caching layer may further be configured to: and under the condition that the data caching layer is determined to cache the target data result set, transmitting a unified data model corresponding to the target data result set to the service application layer through the unified access interface layer.

According to the embodiment of the application, the data caching layer provides the data caching capability for the bottom data access, so that the service data access speed in a specific scene is accelerated, and the application response speed is improved.

Alternatively, a separate data buffer may be created for each database instance, determining how to acquire data each time based on the type of operation that was entered at the time of application access.

The traditional data cache can only achieve the cache of a single database, but can not achieve the cache of multiple databases, particularly multiple heterogeneous databases, so that the performance is relatively low when the operation of multiple databases is involved, even if the performance is improved after the operation is achieved through a micro-service architecture, the network interaction is increased, and the overall performance improvement is not obvious. And embodiments of the present application implement this functionality based on above the data manipulation layer.

Fig. 4 is a schematic flow chart of data caching by the data caching layer according to the embodiment of the present application, as shown in fig. 4: the data caching layer can judge whether the data state in the cache is valid or not according to the query type parameters transmitted during application access, if so, the data in the cache is directly returned, and if not, the query bottom database is penetrated by invalidation, and the cache is updated by the query result.

Optionally, the data caching layer may determine whether the data of the operation is stored in the cache according to the transaction parameters that are input during the application access.

Optionally, if the data of the operation is stored in the cache, based on the data in the cache, updating the data according to the request, then continuing to put the data into the cache, generating a Write-Ahead log (WAL) file, returning to the operation successfully under the condition that the file and the cache are updated successfully, and ending the service processing; when an exception occurs in the processing process, different processing schemes are automatically selected according to the exception types (for example, the exception is divided into a system exception, a database exception, a business exception and the like, the system exception refers to a network instantaneous break, a remote service exception and the like, the retry strategy is automatically determined by application aiming at the exception, the database exception refers to the exception generated when the database is operated, retry or discard is selected according to error codes returned by the database, for example, the data manipulation language (Data Manipulation Language, DML) error directly fails, the database is dormant and retried when the database is busy, the business exception refers to the fact that business code writing is not strict, the exception is generated when the data is processed, the transaction is directly and quickly failed and rolled back aiming at the exception, and the operation is defined as realizing, and the transaction is directly rolled back. The background maintains the data consistency of the cache data and the database at regular time, reads WAL logs, rewrites the operation process into the database, and ensures the consistency of the cache data and the entity data.

Optionally, the maintenance of the cache can be based on LRU (Least Recently Used) algorithm, and the limited memory space is fully utilized to clear the cold data from the cache, so that enough space is reserved for the frequently accessed data, and the application is convenient for quick access.

The embodiment of the application realizes centralized management of the data cache based on the data cache layer, has strong cache data consistency of the multi-data source data, and is suitable for trace data query scenes of high-frequency access in a short time.

Optionally, the heterogeneous database integration system further includes: an insert layer;

Optionally, the heterogeneous database integration system provided in the embodiments of the present application may further include a plug-in layer.

Alternatively, a plug-in layer may be used to extend the functionality of the heterogeneous database integration system.

For example, the plug-in layer may implement application-side data filtering, data processing, data desensitization, and the like.

Optionally, the interposer layer includes:

and the unified transaction management and control unit is used for carrying out distributed transaction management on a database operation sequence under the condition that the unified data operation layer 130 accesses the target database through the unified database operation interface.

Alternatively, the plug-in layer may include a unified transaction management unit.

Alternatively, the unified transaction management unit may be configured to perform distributed transaction management on the database operation sequence in a case where the unified data operation layer 130 accesses the target database through the unified database operation interface.

Optionally, by combining the cloud native characteristics and fusing the access capability of the application database bottom layer, the 'two-state' (running state and development state) high expansion capability can be realized by precisely cutting in between the application layer and the data bottom layer, wherein the embodiment of the application focuses on expanding the distributed transaction of the service scene.

Optionally, the unified transaction control unit may implement unified transaction control, which converges the transaction control to the framework layer, and the framework provides selectable options for the application side to use, so that the application does not need to perform any operation on the transaction, and the unified transaction control is controlled by the framework layer, thereby avoiding related investigation work of data link, transaction, and the like caused by the development capability difference.

Alternatively, the framework may provide the following transaction types:

(1) Local transaction: each time the database is operated, a new link is created if there is no link in the context, released immediately after use, and the transaction is directly used if there is a partially uncommitted transaction. The type is suitable for query operation, does not need independent transaction, and has fast operation and short response time;

(2) Local transaction: binding a part of database operations requested once into a transaction, and integrally submitting the transaction after all the parts of operations are successful;

(3) Global transaction: binding all database operations of one request into one large transaction, and submitting the transaction only after the request is completely successful;

(4) The distributed transaction solves the problem of using the transaction in the micro-service mode.

In the embodiment of the application, by means of fusing the distributed transaction and the common transaction, the application is promoted to concentrate on the development of the service, and the unified delivery of the resources of the underlying infrastructure is operated and maintained by the heterogeneous database integrated system provided by the embodiment of the application, so that the heterogeneous database integrated system is more fit with the cloud protogenesis and is more beneficial to cloud migration.

Fig. 5 is a second schematic structural diagram of a heterogeneous database integration system according to an embodiment of the present application, where, as shown in fig. 5, the heterogeneous database integration system includes: the system comprises a business application layer, a unified access interface layer, a data cache layer, a data routing layer, a plug-in layer, a unified data operation layer and a data storage layer.

The unified access interface layer, the data cache layer, the data routing layer, the plug-in layer and the unified data operation layer in fig. 5 can be regarded as a unified data access layer, the unified data access layer acts between the data storage layer and the service application layer, plays a role of a bridge, provides unified database access interface capability for an application, and can realize that the data cache can accelerate the application access efficiency in the cache layer; the data routing layer can be responsible for maintaining access conversion capability of different data types and different libraries, so that the application can easily realize management of the data life cycle; the plug-in layer may be used for capability extensions to unified interface services (e.g., table name population, log logging, and very long link processing, etc.).

Optionally, in order to conform to the cloud native concept, a cloud native technology architecture is applied, so that fast switching and fast recovery of the database can be realized, but the bottom layer data is difficult to acquire relative to the upper layer. The embodiment of the application provides a unified data access layer, when an application needs to be switched, the bottom heterogeneous database is rapidly switched through intelligent routing, so that the application can meet the cloud original concept from top to bottom, and the influence of the bottom environment on the application is shielded. The method is particularly suitable for gray level release processes, data layer service degradation, emergency switching scenes and the like. The cross-table and cross-library calling is realized from the bottom layer service, on the basis, the application can directly and easily realize the functions of database and table division, and can also realize the management and control of the whole life cycle of the data, thereby improving the expansion capability of the whole service system.

According to the embodiment of the application, the cloud primary design concept is taken as a main, the unified access capability is realized for application development by unified packaging and expansion on the basis of the relational database and the non-relational efficient access interface, meanwhile, the data routing capability is increased on the basis, the application does not need to consider what storage medium the data itself is located on when accessing the data, the database type required to be accessed by development is automatically calculated according to the application parameter and routing rules, the application code is suitable for the application of a heterogeneous database system for accessing the cold and hot data separation, the application code does not need to develop two parts or increase logic judgment branches, and the development code is simpler and the expansion capability is stronger. The database and the table can be easily realized without adjusting the database or the migration application code, and the expansion capability of the system can be greatly improved from the architecture layer.

According to the embodiment of the application, the cloud primary concept is attached, the bottom database resources are transparently presented to the application in a standardized and automatic use mode, the running state high expansion capacity is achieved, and meanwhile the high expansion capacity is provided functionally.

Specifically, the heterogeneous database integration system provided by the embodiment of the application can achieve the following two beneficial effects: (1) Unified data access mode, shielding heterogeneous database difference: in the prior art, only a relational database can be processed, but a non-relational database cannot be processed, data access can be realized only in a customized coding mode in a non-relational database scene, and the application can be realized only by code reconstruction when other heterogeneous databases are migrated; (2) Unified data route management and control, realizing cross-base and cross-table access: the conventional database link pool tool only can support a single-library single-instance mode, a database instance is generally required to be selected according to a specific rule, the scheme cannot realize the data session sharing capability among the cross libraries, the application must be realized by itself, and the heterogeneous database integrated system provided by the embodiment of the application realizes the session maintenance among the cross libraries through a built-in data unified access layer, so that the application development is not required to be concerned with the underlying data storage mode.

The database access method provided in the embodiments of the present application is described below, and the database access method described below and the heterogeneous database integration system described above may be referred to correspondingly.

Fig. 6 is a schematic flow chart of a database access method according to an embodiment of the present application. As shown in fig. 6, the database access method provided in the embodiment of the present application may include:

step 600: analyzing a user access request and acquiring unified data model object parameters corresponding to the user access request;

Step 610: determining a target database based on the unified data model object parameters and a routing algorithm; the routing algorithm is determined based on target influencing factors influencing routing;

step 620: and accessing the target database through the unified database operation interface based on the unified data model object parameters and the type of the target database, and obtaining a target data result set corresponding to the user access request.

Alternatively, the database access method provided in this embodiment may be implemented based on the heterogeneous database integration system provided in the foregoing embodiment.

Specifically, the heterogeneous database integration system may first parse the user access request to obtain a unified data model object parameter corresponding to the user access request; further, determining a target database based on the unified data model object parameters and the routing algorithm; then, based on the unified data model object parameters and the types of the target databases, the target databases are accessed through the unified database operation interface, and a target data result set corresponding to the user access request is obtained.

Optionally, the user access request is parsed, and unified data model object parameters corresponding to the user access request are obtained.

Alternatively, the target database may be determined based on the unified data model object parameters and the routing algorithm.

Alternatively, the target database may be accessed through the unified database operation interface based on the unified data model object parameters and the type of the target database, to obtain a target data result set corresponding to the user access request.

According to the database access method, the user access request is analyzed, unified data model object parameters corresponding to the user access request are obtained, the target database is determined based on the unified data model object parameters and the routing algorithm, the target database is accessed through the unified database operation interface based on the unified data model object parameters and the types of the target database, and the target data result set corresponding to the user access request is obtained; according to the method and the device, dynamic switching of the database is realized through application of the parameter transmission and routing algorithm, and uniform heterogeneous data access mode is realized through a uniform database operation interface.

Optionally, the determining the target database based on the unified data model object parameters and the routing algorithm includes:

Determining the target influencing factors;

determining a routing expression based on the target influencing factors;

the routing algorithm is determined based on the routing expression.

Specifically, based on the unified data model object parameters and the routing algorithm, the specific step of determining the target database may include the following steps (1) -step (3):

(1) Determining the target influencing factors;

(2) Determining a routing expression based on the target influencing factors;

(3) Based on the routing expression, a routing algorithm is determined.

Optionally, the determining a routing expression based on the target influencing factor includes:

Alternatively, the target influencing factors may be combined by logical operators to determine the routing expression.

Optionally, the routing algorithm has the expression:

routing expression x preset weight & routing state & instance number;

Alternatively, the expression of the routing algorithm may be:

routing expression x preset weight & routing state & instance number;

Optionally, after the accessing the target database through the unified database operation interface based on the unified data model object parameter and the type of the target database, the method further includes:

converting the data in the target data result set into a unified data model;

and caching the unified data model.

Alternatively, after accessing the target database through the unified database operation interface based on the unified data model object parameters and the type of the target database, the data in the target data result set corresponding to the user access request may be converted into the unified data model.

Optionally, the transformed unified data model may be cached in order to reduce access to the underlying database and speed up application access.

Optionally, before the determining the target database based on the unified data model object parameters and the routing algorithm, the method further comprises:

Alternatively, it may be determined whether the heterogeneous database integration system has cached the target data result set based on the unified data model object parameters.

Optionally, under the condition that the target data result set is determined to be cached, sending the unified data model corresponding to the target data result set to the service application layer.

Optionally, the method further comprises:

Alternatively, in the case of accessing the target database through a unified database operation interface, distributed transaction management may be performed on the sequence of database operations.

Fig. 7 is a second flowchart of the database access method according to the embodiment of the present application, as shown in fig. 7, mainly including the following steps (1) - (11):

(1) A user initiates a data access request, the user request is intercepted at a unified access interface layer, and the mode and the request parameters of a database accessed by an application are analyzed;

(2) According to the request parameters, analyzing the service data through a model, converting the service data into a unified data model, generating unified data model object parameters, and transferring the unified data model object parameters to lower-layer processing;

(3) Calling a data caching layer interface by taking the database operation mode and the query condition analyzed in the step (1) as request parameters, and judging whether the operation and the query condition exist in storage or not;

(4) If there is already a direct return of the corresponding normalized data result set. For example, the information of the user is queried, and if the information exists in the cache, the information of the user is directly returned to the application layer. When the request parameter does not exist, the request parameter is transmitted to a data routing layer;

(5) After receiving the request parameters, the data routing layer carries out real-time calculation according to the request parameters and combining with a pre-defined routing strategy, selects a database to be accessed and a corresponding instance, and continuously transmits the request parameters into the plug-in layer;

(6) The function expansion, such as unified data transaction control, is realized in the plug-in layer, and the transaction is started before the bottom layer data is operated, so that the data consistency of different libraries and different examples is realized;

(7) In the unified data operation layer, selecting a related database driving type and a database connection pool according to the incoming operation mode and the database instance type determined by the routing decision; the request parameters can be provided with data life cycle marks, so that a cold and hot data partition storage mode is realized; for example, data of nearly three months need to be queried, at this time, incoming bill_Month is 9 months, at this time, according to bill_Month <3 in preset conditions and current month is 10 months, routing is directed to an Oracle database;

(8) According to the database operation mode in the request, combining with specific database types, selecting the operation mode (such as SQL, NOSQL or SQL-like) of a corresponding database, and applying the operation convenience of the traditional relational database to a novel database through a custom database access mode in a unified data operation layer, so that development is not directly faced with an API any more, and other novel databases which cannot be realized by using the traditional database operation thinking operation are operated; for example, the user query is a get data action, the implementation in Oraloce is select from user_info, the implementation in Coherence is get user_info, and the routing rule calculates that the source to be accessed is the database, and the selection from user_info is adopted to execute the operation;

(9) When the database server side executes the corresponding SQL or method, a data result set of the database type is returned to the unified data operation layer, and the unified data operation layer processes the data into standard data and transmits the standard data to the data cache layer;

(10) The data caching layer updates the internal cache data by taking the request operation mode and the request parameters as query conditions so as to facilitate subsequent access;

(11) And returning the standard data result set returned by the cache layer to the service code service application layer through the unified access interface layer. And notifying the distributed management of the plug-in layer to commit the operation transaction and close the connection, and ending the process.

According to the database access method, the type of the bottom database is not required to be known, standard data objects can be directly operated, and operation redundant codes of the bottom database are greatly reduced.

Optionally, the database access method provided by the embodiment of the application is suitable for accessing various types of databases, and when a new database is added, the implemented service codes almost do not need to be adjusted, so that the rapid migration of the databases can be realized.

Fig. 8 is a schematic structural diagram of an electronic device provided in an embodiment of the present application, and as shown in fig. 8, the electronic device may include: processor 810, communication interface (Communication Interface) 820, memory 830, and communication bus 840, wherein processor 810, communication interface 820, memory 830 accomplish communication with each other through communication bus 840. The processor 810 may call a computer program in the memory 830 to perform the steps of a database access method, for example, including:

Further, the logic instructions in the memory 830 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

In another aspect, embodiments of the present application further provide a computer program product, where the computer program product includes a computer program, where the computer program may be stored on a non-transitory computer readable storage medium, where the computer program when executed by a processor is capable of executing the steps of the database access method provided in the foregoing embodiments, for example, including:

In yet another aspect, embodiments of the present application further provide a processor-readable storage medium storing a computer program for causing a processor to execute the steps of the database access method provided in the foregoing embodiments, for example, including:

The processor-readable storage medium may be any available medium or data storage device that can be accessed by a processor, including, but not limited to, magnetic storage (e.g., floppy disks, hard disks, magnetic tape, magneto-optical disks (MOs), etc.), optical storage (e.g., CD, DVD, BD, HVD, etc.), semiconductor storage (e.g., ROM, EPROM, EEPROM, nonvolatile storage (NAND FLASH), solid State Disk (SSD)), and the like.

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and are not limiting thereof; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the corresponding technical solutions.

Claims

1. A heterogeneous database integration system, comprising:

2. The heterogeneous database integration system of claim 1, wherein the data routing layer is further configured to:

determining the target influencing factors;

determining a routing expression based on the target influencing factors;

the routing algorithm is determined based on the routing expression.

3. The heterogeneous database integration system of claim 2, wherein the data routing layer is specifically configured to:

4. A heterogeneous database integration system according to claim 2 or 3, wherein the routing algorithm is expressed as:

routing expression x preset weight & routing state & instance number;

5. The heterogeneous database integration system of claim 1, further comprising: a data caching layer;

the data caching layer is used for caching the unified data model.

6. The heterogeneous database integration system of claim 5, further comprising: a business application layer;

The data caching layer is further used for:

7. The heterogeneous database integration system of claim 1, further comprising: an insert layer;

8. The heterogeneous database integration system of claim 7, wherein the plug-in layer comprises:

9. A database access method applied to the heterogeneous database integration system of any one of claims 1-8, comprising:

10. The database access method according to claim 9, wherein the determining a target database based on the unified data model object parameters and a routing algorithm comprises:

determining the target influencing factors;

determining a routing expression based on the target influencing factors;

the routing algorithm is determined based on the routing expression.

11. The database access method of claim 10, wherein the determining a routing expression based on the target influencing factor comprises:

12. The database access method according to claim 10 or 11, wherein the routing algorithm has the expression:

Routing expression x preset weight & routing state & instance number;

13. The database access method according to claim 9, wherein after said accessing said target database through said unified database operation interface based on said unified data model object parameters and the type of said target database, said method further comprises, after obtaining a target data result set corresponding to said user access request:

converting the data in the target data result set into a unified data model;

and caching the unified data model.

14. The database access method of claim 13, wherein prior to said determining a target database based on said unified data model object parameters and routing algorithm, said method further comprises:

15. The database access method of claim 9, wherein the method further comprises:

16. An electronic device comprising a processor and a memory storing a computer program, characterized in that the processor implements the steps of the database access method of any of claims 9 to 15 when executing the computer program.

17. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the database access method of any of claims 9 to 15.