CN115454975A - Orm frame supporting cloud native architecture - Google Patents

Abstract

The invention provides a orm framework supporting a cloud native architecture, which is characterized by comprising a step of realizing unification of all data models through unified data model definition statements and avoiding diversification; completing the escape from the data model to the database model through a unified data arrangement service; translating the corresponding orm query into various database query statements adapted through an abstract syntax tree to uniformly realize the query of all data models; the frame overall architecture adopts a cqrs architecture in a field-driven model to separate the query and the operation instruction of data, so that the effects of flexibly expanding, simplifying the query and improving the performance are achieved; the invention also divides the micro-service from the data layer, reduces the loading of the data by the service, reduces the resource consumption of the system, increases the performance, ensures the final consistency of the data, reduces the complexity of the development system of the developer and solves the problem of no shutdown of the micro-service in the frequent upgrading process.

Description

Orm frame supporting cloud native architecture

Technical Field

The invention relates to the field of databases, in particular to a orm framework supporting a cloud native architecture.

Background

In the existing data architecture, as the scale of the system increases, the complexity of a data model of the system increases exponentially, and the coupling degree of each data table is high, so that the change and the expansion are difficult, and the maintenance difficulty of the system is high. There is a need for a data architecture that supports the micro-service architecture.

The traditional data architecture needs to adopt different data access modes for different storage media, the same data is often stored in the same medium, and IO operation aims at the same storage medium, so that IO performance bottleneck and deadlock are easily caused.

The traditional data architecture usually lacks a uniform data change interception mode, and a large amount of data copying and monitoring programs access an original database in a polling mode. The high-frequency polling can obviously increase the IO pressure of the system and reduce the operation efficiency of the original system. The low-frequency polling causes the mirror image data to have obvious time delay compared with the original data, and the overall data consistency of the multiple systems cannot be ensured.

Currently, most mature orm frames are essentially stand-alone, single instance, operational. With the popularization of the cloud market in recent years and the large trend of cloud in enterprises, a orm framework capable of supporting the cloud architecture and ensuring compatibility with the existing software system is required.

With the increase of the traffic and the progression of the whole function iteration, some problems (insufficient test, service interruption and the like) inevitably occur, the problems can cause a series of losses, and with the continuous increase of the requirements of enterprises on the sla grade, the whole software continuously optimizes the platform thereof. A complete solution at present is to continuously deploy (as in fig. 1) a cloud-native architecture, and upgrade a software platform. However, currently, the cloud native architecture can be implemented only by functional services, and accurate routing control can be invoked between services (as shown in fig. 2), but the functional services have limitations for data operation, cannot implement accurate routing of data traffic, and are rarely compatible with the cloud native architecture.

Disclosure of Invention

The invention aims to provide a orm framework supporting a cloud native architecture, so as to solve the problems in the background art.

In order to achieve the purpose, the invention provides the following technical scheme:

a orm framework supporting a cloud native architecture comprises the steps of realizing unification of all data models through unified data model definition statements and avoiding diversification; completing the escape from the data model to the database model through unified data arrangement service; translating the corresponding orm query into various database query statements adapted through an abstract syntax tree to uniformly realize the query of all data models; the framework overall architecture adopts a cqrs architecture in a domain-driven model to separate the query and the operation instruction of data, so that the following effects are achieved:

(1) flexible expansion-read-write data storage can be independently expanded according to requirements;

(2) simple queries-since the data models for reading and writing are separate, the read data model can be designed in a way that avoids complex queries;

(3) performance-by optimizing read operations and placing data storage copies on different storage media, read heavy operations can significantly improve performance;

the method is characterized in that a read-write separation support for a relational database is built in, a plurality of data sources (an oracle relational database, a mongo database, a hbase column database, a redis memory database, a mysql relational database, a gemfire memory database and an elastic search full-text index library) can be configured for the service, and a write library and a read-only library are designated; the query and command orchestration component automatically forwards the data update operation to the writable library and the query operation to the read-only library.

The orm framework is divided into a client and a server, and the corresponding main process is divided into a common step, a client process and a server process.

The common step specifically comprises:

(1) the micro service defines all related data aiming at the service range of the micro service;

(2) defining specific fields and types of the data, and specifically writing to and reading from which databases;

(3) binary packets corresponding everywhere;

if multiple storage media (oracle, mongo, redis, gemfire) are defined for the same object; when an ORM command is called to update an object, data can be simultaneously stored in different storage media according to corresponding mapping definitions; when the object is queried, the arranging component automatically selects the storage medium with the highest efficiency to execute the query operation according to the priority order defined by the query content in the definition stage (if none is defined, the priority order is determined according to the query efficiency of each storage medium, and the query efficiency is Gemfire > redis > full-text index > relational database).

The client process specifically includes:

(1) loading a binary package corresponding to the data;

(2) business operation, specifically, data addition, deletion, modification and check;

(3) pushing the corresponding adding, deleting, modifying and checking to a message queue;

data consistency and clustering services: the orm framework is designed based on BASE architecture (BASE refers to basic Available, soft State, and final Consistency), and the core idea is to improve system Availability (Availability) and adopt a distributed transaction implementation mode to achieve final Consistency (Consistency) even though Strong Consistency cannot be achieved;

assuming all data operations are traceable, it places the data update operation into the command buffer queue for sequential processing. A plurality of server instances can be started in the cluster to write data into a plurality of storage media in parallel, and finally data consistency in the plurality of storage media is realized; if the system fails, the server may back-track the command cache to recover the data.

The server process specifically includes:

(1) and loading the binary package corresponding to the data.

(2) For each type of data in the data definition, the corresponding field is initialized, the database executor class (processor of oracle, processor of mysql, processor of redis, etc.) that needs to be operated on

(3) Monitoring the message in the message bus in real time according to the tag (data label) of the message, and correspondingly calling the processor in the step (2) to process after the message is taken;

(4) the processing logic of the processor correspondingly writes the data into a database in charge of the processor;

and for the data types configured with the consistency check, inquiring the state of the data in each database.

In the single instance, through a first-in-first-out queue, other elements in the array do not need to be moved when data is taken away, and under the model of a single consumer single producer, locking is not needed, and data can be accessed more quickly, namely, the ringbuffer. By using a data structure such as ringbuff, one-to-many scenarios are realized. Aiming at different database types, due to the difference of self storage characteristics and the difference of provided database drive packages, a program needs to compile different codes adapting to the database drive to finish the storage of a plurality of databases of the same data; by processors implementing different databases; for the same data, a specific inserted database needs to be determined according to a flow label, and the determination is realized through a tag mechanism of a message bus.

Compared with the prior art, the invention has the following beneficial effects:

the data among the micro services of the orm framework of the invention are not interfered, and the division of the micro services is also carried out on the data layer, so that the separation of duties is clear, the loading of the services to the data is greatly reduced, and the resource consumption of the system is reduced; the reading and the writing are separated, so that the isolation of data operation and access is ensured, and the access bottleneck of the system is increased; the write-in once is added, the operation of multiple database types is adapted, convenience is provided for subsequent big data expansion, data statistics is facilitated, and the final consistency of data can be ensured while the performance is increased; the service grid is adapted, and the problem that the micro-service does not stop in the frequent upgrading process is solved; the unified data model can complete the operation of various databases, reduces the complexity of development systems of developers, can greatly improve the throughput of data processing by expanding the message bus cluster, and does not need to modify any service codes.

Drawings

FIG. 1 is a principal concept of cloud-origin;

FIG. 2 is a schematic diagram of precise routing of data traffic;

FIG. 3 is a flow chart of the invention showing the main flow of the orm frame;

FIG. 4 is an overall frame diagram of the orm frame of the present invention;

FIG. 5 is a schematic diagram of the present invention configuring multiple data sources for a service and specifying a write library and a read-only library;

FIG. 6 is a schematic diagram illustrating an operation of automatically selecting a storage medium with the highest efficiency to perform a query according to the present invention;

FIG. 7 is a schematic diagram of a data recovery method for a backtracking command cache by a server according to the present invention;

FIG. 8 is a schematic diagram of a database implemented by a tag mechanism of a message bus to determine specific insertion according to a traffic tag according to the present invention;

FIG. 9 illustrates the data model definition implementation code of the present invention;

FIG. 10 is a block diagram of the present invention embodying code for registering data in a container;

FIG. 11 is code for a specific implementation of update and query operations;

FIG. 12 shows the server side listening for message queue data and forwarding to the processor specific implementation code;

FIG. 13 is a diagram illustrating processor processing logic code corresponding to an oracle database type;

FIG. 14 is a second diagram illustrating the logical code of processor processing corresponding to the oracle database type.

Detailed Description

The present invention will be described in further detail with reference to the following examples in order to clarify technical problems, technical solutions, implementation processes and performance displays. It should be understood that the specific embodiments described herein are for illustrative purposes only. The present invention is not limited to the above embodiments. Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the present disclosure.

Example 1

A orm frame supporting a cloud native architecture is characterized in that an overall architecture diagram comprising a orm frame is shown in FIG. 4, unification of all data models is realized through unified data model definition statements, and diversification is avoided; completing the escape from the data model to the database model through a unified data arrangement service; translating the corresponding orm query into various database query statements through an abstract syntax tree to uniformly realize the query of all data models; the framework overall architecture adopts a cqrs architecture in a domain-driven model to separate the query and the operation instruction of data, so that the following effects are achieved:

(1) flexible expansion-read-write data storage can be independently expanded according to requirements;

(2) simple queries-since the data models for reading and writing are separate, the read data model can be designed in a way that avoids complex queries;

(3) performance-by optimizing read operations and placing data storage copies on different storage media, read heavy operations can significantly improve performance;

as shown in fig. 5, a built-in read-write separation support for a relational database may configure a plurality of data sources (oracle relational database, mongo database, hbase column-based database, redis memory database, mysql relational database, gemfire memory database, elastic search full-text index database) for a service and specify a write library and a read-only library; the query and command arrangement component automatically forwards the data update operation to the writable library and the query operation to the read-only library.

The orm framework is divided into a client and a server, and the corresponding main process is divided into a common step, a client process and a server process.

The common step specifically comprises:

(1) the micro service defines all related data aiming at the service range of the micro service;

(2) defining specific fields and types of the data, and specifically writing to and reading from which databases;

(3) binary packages corresponding everywhere;

as shown in fig. 6, for the same object, if a plurality of storage media (oracle, mongo, redis, gemfire) are defined; when an ORM command is called to update an object, data can be simultaneously stored in different storage media according to corresponding mapping definitions; when the object is queried, the editing component automatically selects the storage medium with the highest efficiency to execute the query operation according to the priority order defined by the query content in the definition stage (if none is defined, the priority order is determined according to the query efficiency of each storage medium, and the query efficiency is determined by gemfire > redis > full-text index > relational database).

The client process specifically includes:

(1) loading a binary package corresponding to the data;

(2) business operation, specifically to the data add, delete, modify and check;

(3) pushing the corresponding adding, deleting, modifying and checking to a message queue;

data consistency and clustering services: the orm framework is designed based on BASE architecture (BASE refers to basic Available, soft State, and final Consistency), and the core idea is to improve system Availability (Availability) and adopt a distributed transaction implementation mode to achieve final Consistency (Consistency) even though Strong Consistency cannot be achieved;

as shown in FIG. 7, assuming all data operations are traceable, it places the data update operation into the command buffer queue for sequential processing. A plurality of server instances can be started in the cluster to write data into a plurality of storage media in parallel, and finally data consistency in the plurality of storage media is realized; if the system fails, the server may back-track the command cache to recover the data.

The server process specifically includes:

(1) and loading the binary package corresponding to the data.

(2) For each type of data in the data definition, the corresponding fields are initialized, the database executor class (processor of oracle, processor of mysql, processor of redis, etc.) that needs to be operated on

(3) Monitoring the message in the message bus in real time according to the tag (data label) of the message, and correspondingly calling the processor in the step (2) to process after the message is taken;

(4) the processing logic of the processor correspondingly writes the data into a database which is in charge of the processor;

and for the data types configured with consistency check, querying the state of the data in each database.

As shown in fig. 8, in a single instance, through a first-in-first-out queue, the data can be accessed faster without requiring locks under the model of single consumer single producer, without requiring other elements in the array to be moved when the data is taken, i.e., ringbuffer. By using a data structure such as ringbuff, one-to-many scenarios are realized. Aiming at different database types, due to the difference of self storage characteristics and the difference of provided database drive packages, a program needs to compile different codes adapting to the database drive to finish the storage of a plurality of databases of the same data; by processors implementing different databases; for the same data, a specific inserted database needs to be determined according to a flow label, and the determination is realized through a tag mechanism of a message bus.

The specific implementation can be realized by 4 steps;

s1. Data definition

S2, the client refers to the code and executes specific loud operation to send data to the message bus

S3, the server side monitors message queue data and distributes the message queue data to a specific processor

S4. The processor is specifically responsible for processing data

Wherein the step S1 specifically includes: defining data structures and corresponding data stores

Data model definition: corresponding to a database table water and a field id name color; the specific implementation code is shown in fig. 9;

registering the data in the container; the specific implementation code is shown in fig. 10;

the testMysql is a database corresponding to the data model, and of course, here, the testMysql is a streaming expression, and may correspond to a plurality of databases.

The autoConfig is a default configuration, and can also be a customized configuration, the mapping relation of data fields is configured, the query priority of each database and the like.

The step S2 specifically includes: client side executing concrete database operation

The update and query operation specific implementation code is shown in FIG. 11; and performing an updating operation by the ormcomands, wherein the data type Water is the data type defined in the step 1, and the name field of the updating Water is 1.

Data with the primary key of TEST001 in the water table is queried.

The query operation is realized by self-defining a syntax tree, and the syntax tree structure is that the alias # in # of the from # table corresponds to the entity object # of the database

Where # Condition #

Step S3 specifically includes: the server monitors the message queue data and forwards the message queue data to the processor, and the specific implementation code is shown in fig. 12; eventData is the data taken from the message bus according to tag.

An ormserverdispatcher is a stand-alone high-performance inter-thread message (using the ringbuffer architecture) delivery library that takes messages out of the message bus and delivers them to the dispatcher.

The dispatcher creates a corresponding processor for processing according to the database type configured by each data.

If the processing type is the batch processing type, the batch processing type is handed over to the BatchCommand processor for processing.

Step S4 specifically includes: a specific processor implementation, which only lists the processor processing logic corresponding to the oracle database type, and the specific code is shown in fig. 13 and fig. 14; after receiving the data distributed by the dispatcher, acquiring orm configuration and the type of a database;

according to the specific operation type of the data, the method comprises the steps of inserting or updating, updating or inserting, adding the association relationship, deleting the association relationship and submitting the association relationship to an execute function for processing.

The execute function integrates orm configuration, database configuration and specific operation type configuration, obtains an sql statement for a specific database type, and calls a corresponding database tool to execute the sql statement.

The present invention solves the following problems:

(1) on the large structure, data are unified: all data in the micro-service cluster form a logically uniform data set, and intersection should be avoided among data of different services;

(2) diversification of storage modes: each microservice is responsible for maintaining its own data. The different service data are not physically associated. Data of different services can be stored in different positions respectively;

(3) isolation between microservices: the data generated by each microservice should remain private, and the data can only be modified by one microservice. The micro-services call the acquired data through a service interface, and the private data of other services are strictly forbidden to be directly accessed;

(4) isolation between deployment environments: the isolation of data operation among each environment is ensured, the data operation stability in the service upgrading and releasing process is ensured, and the sla grade of the whole system is improved.

The foregoing shows and describes the general principles, essential features, and advantages of the invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention, and that various changes and modifications may be made without departing from the spirit and scope of the present invention, which fall within the scope of the claimed invention. The scope of the invention is defined by the appended claims and equivalents thereof.

Claims (5)

1. A orm framework supporting a cloud native architecture is characterized by comprising a data model definition statement which is unified to realize unification of all data models and avoid diversification; completing the escape from the data model to the database model through a unified data arrangement service; translating the corresponding orm query into various database query statements adapted through an abstract syntax tree to uniformly realize the query of all data models; the framework overall architecture adopts a cqrs architecture in a domain-driven model to separate the query and the operation instruction of data, so that the following effects are achieved:

(1) flexible expansion-read-write data storage can be independently expanded according to requirements;

(2) simple queries-since the data models for reading and writing are separate, the read data model can be designed in a way that avoids complex queries;

(3) performance-by optimizing read operations and placing data storage copies on different storage media, read heavy operations can significantly improve performance;

the method is characterized in that a read-write separation support for a relational database is built in, a plurality of data sources (an oracle relational database, a mongo database, a hbase column database, a redis memory database, a mysql relational database, a gemfire memory database and an elastic search full-text index library) can be configured for the service, and a write library and a read-only library are designated; the query and command arrangement component automatically forwards the data update operation to the writable library and the query operation to the read-only library.

2. The orm framework supporting cloud-based architecture as claimed in claim 1, wherein the orm framework is divided into a client and a server, and the corresponding main flow is divided into a common step, a client flow and a server flow.

3. The cloud-native architecture enabled orm framework of claim 2, wherein the common steps specifically comprise:

(1) the micro service defines all related data aiming at the service range of the micro service;

(2) defining specific fields and types of the data, and specifically writing to and reading from which databases;

(3) deriving the data corresponding to the attribute defined in the step (2), and generating a binary package for the subsequent steps;

if multiple storage media (oracle, mongo, redis, gemfire) are defined for the same object; when an ORM command is called to update an object, data can be simultaneously stored in different storage media according to corresponding mapping definitions; when the object is queried, the arranging component automatically selects the storage medium with the highest efficiency to execute the query operation according to the priority order defined by the query content in the definition stage (if none is defined, the priority order is determined according to the query efficiency of each storage medium, and the query efficiency is Gemfire > redis > full-text index > relational database).

4. The cloud-native-architecture-enabled orm framework of claim 2, wherein the client process specifically comprises:

(1) loading a binary package corresponding to the data;

(2) business operation, specifically, data addition, deletion, modification and check;

(3) pushing the corresponding adding, deleting, modifying and checking to a message queue;

at this time, one problem to be solved urgently is that, when multiple types of database storage are defined, it is necessary to ensure that data of each database can be written successfully at the same time, or one database fails, and the other databases also fail, at this time, data consistency and clustering service are required: the orm framework is designed based on BASE architecture (BASE refers to basic Available, soft State, and final Consistency), and the core idea is to improve system Availability (Availability) and adopt a distributed transaction implementation mode to achieve final Consistency (Eventual Consistency) even if Strong Consistency cannot be achieved;

assuming all data operations are traceable, it places the data update operation into the command buffer queue for sequential processing. A plurality of server instances can be started in the cluster to write data into a plurality of storage media in parallel, and finally data consistency in the plurality of storage media is realized; if the system fails, the server may back-track the command cache to recover the data.

5. The orm framework supporting the cloud-native architecture of claim 2, wherein the server process specifically comprises:

(1) and loading the binary package corresponding to the data.

(2) For each type of data in the data definition, the corresponding fields are initialized, the database executor class (processor of oracle, processor of mysql, processor of redis, etc.) that needs to be operated on

(3) Monitoring the message in the message bus in real time according to the tag (data label) of the message, and correspondingly calling the processor in the step (2) to process after the message is taken;

(4) the processing logic of the processor correspondingly writes the data into a database which is in charge of the processor;

and for the data types configured with the consistency check, inquiring the state of the data in each database.

In a single instance, through a first-in-first-out queue, there is no need to move other elements in the array when taking data, and in a single consumer-single producer model, there is no need to lock, and data can be accessed faster, such a data structure is ringbuffer. By using a data structure such as ringbuff, one-to-many scenarios are realized. Aiming at different database types, due to the difference of self storage characteristics and the difference of provided database drive packages, a program needs to compile different codes adapting to the database drive to finish the storage of a plurality of databases of the same data; by processors implementing different databases; for the same data, a specific inserted database needs to be determined according to a flow label, and the determination is realized through a tag mechanism of a message bus.

Images