CN116303766A - Data synchronization method, device, storage medium and computer equipment - Google Patents

Abstract

The invention discloses a data synchronization method, a data synchronization device, a storage medium and computer equipment, and relates to the technical field of data synchronization. According to the invention, through related technologies such as data synchronization, different copying schemes are selected from clusters to cross-domain data centers according to different service characteristics, and the operation mode of the multi-activity data center is a premise and a basis for researching remote data synchronization among the multi-data centers, various multi-data center construction modes are provided by analyzing the construction mode of the existing multi-data center, and the multi-data center operation mode of a bank financial service information system is constructed, so that the synchronization of heterogeneous databases cannot be well supported aiming at the existing remote data synchronization efficiency, and the corresponding SQL is captured and sent to the heterogeneous databases for execution through setting a capturer in a driver.

Description

Data synchronization method, device, storage medium and computer equipment

Technical Field

The present invention relates to the field of data synchronization technologies, and in particular, to a data synchronization method, a data synchronization device, a storage medium, and a computer device.

Background

The data synchronization means that the palm computer can rapidly realize data synchronization and information sharing with computer equipment such as a desktop computer, a notebook computer and the like, so that the data can keep integrity and uniformity, in recent years, along with rapid development of cloud computing and big data, the traditional single data center cannot cope with ever-increasing data, the pressure born by a system is larger and larger, and especially the challenges faced by the national important information system with requirements of high availability, high reliability, high safety, real-time high efficiency and the like are huge, the theoretically best data synchronization method is that after a user updates data in a cloud data center in a certain area, data synchronization is carried out on all cloud data centers in other areas, so that more and more enterprises build a plurality of data centers, the computing and big data are used in a bank financial information service system, and the bank financial information service system has the characteristics of high instantaneity, very complex service, multiple subsystems, mutual independent and mutual coordination of all subsystems and the like;

therefore, by researching the operation modes of the multiple data centers, the operation modes of the multiple data centers are analyzed and classified, so that the operation modes are not only the preconditions and the basis for the research of the remote data synchronization mechanism of the multiple data centers, but also the reference is provided for the construction of the multiple data centers of the information system in the whole information industry, and the operation modes have very important reference significance for the construction of the multiple data centers of the information system in the whole information industry.

Disclosure of Invention

The invention provides a data synchronization method, a data synchronization device, a storage medium and computer equipment, and solves the problems.

In order to solve the technical problems, the invention is realized by the following technical scheme:

the data synchronization method of the invention comprises the following steps:

s1, copying one part of data from one machine to another machine through data copying, recording the mirror image of the database at a certain time point by the database snapshot according to the snapshot method, the trigger method and the log method, storing the mirror image, recovering by the mirror image, and recovering the rest of the database by the mirror image so as to completely copy;

s2, a data replication consistency algorithm is used for carrying out communication and mutual operation through a message transmission system, wherein the distribution of the data replication consistency algorithm can be different computers in geographic positions or different virtual nodes in the same physical machine;

and S3, in the master-slave replication, data of one node is replicated to other nodes, the node is called a master node, the other nodes are called slave nodes, an application program can read the data of the slave nodes, so that the access pressure of the master node is reduced, the master-slave replication can improve the fault capability of the reading operation, and the peer-to-peer replication is also called master-master replication, namely all nodes are identical in position and can replicate each other without dividing the master-slave nodes.

Further, the data replication has a synchronous replication mode and an asynchronous replication mode, the synchronous replication means that a replication initiator actively waits for the replication result, and the asynchronous replication means that the replication initiator does not need to wait for whether the replication is successful or not, and directly performs the next action.

The data synchronization device comprises a data synchronization related technology research, a multi-data center operation mode analysis and construction and a remote database synchronization mechanism based on a driver SQL capture;

the related technical research of the data synchronization comprises a data replication module, a data replication consistency algorithm module and a master-slave replication and peer-to-peer replication module;

the multi-data center operation module comprises a typical multi-data center operation mode, a multi-data center operation mode analysis module and a bank financial data center operation mode construction module;

the remote database synchronization mechanism based on the SQL capture of the driver comprises a remote data synchronization mechanism overall design module, a remote data synchronization mechanism communication mode design module, a SQL capture method module in the driver, a data consistency check module, a remote data synchronization experiment design module and a remote data synchronization experiment result analysis module.

Furthermore, the data replication consistency algorithm is a distributed system, and the system is communicated and interoperated through message transmission, wherein the distribution can be different computers in geographic positions or different virtual nodes in the same physical machine.

Further, the multi-data center operation module comprises an internet company data center operation mode and a transport company ticketing system data center operation mode.

Further, the multi-data center operation mode analysis module comprises a same-city main-standby mode, a two-place three-center mode, a different-place double-activity mode and a different-place multi-activity mode.

Further, the bank financial data center operation mode construction module comprises a single-write multi-read remote double-activity mode of the bank financial service data center and a multi-write multi-read remote multi-activity mode of the bank financial service data center.

Further, the remote data synchronization mechanism overall design module comprises a master-slave synchronization mechanism design and a peer-to-peer synchronization mechanism design.

A data synchronization storage medium having stored thereon a computer program which, when executed by a processor, implements a method corresponding to steps S1-S3 described above.

The data synchronization device comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the method corresponding to the steps S1-S3.

Compared with the prior art, the invention has the following beneficial effects:

(1) According to the invention, through related technologies such as data synchronization, different copying schemes and synchronization strategies are selected from the cluster to the cross-domain data center according to different service characteristics.

(2) The operation mode of the multi-active data center is a premise and a basis for researching remote data synchronization among the multi-data centers, various multi-data center construction modes are provided by analyzing the construction modes of the existing multi-data centers, and a multi-data center operation mode of a banking financial service information system is constructed, so that a new data synchronization mechanism is provided aiming at the problems that the existing remote data synchronization is low in efficiency and synchronization among heterogeneous databases cannot be well supported.

(3) By setting the capturer in the driver program, capturing the corresponding SQL and sending to the remote database for execution, the mechanism optimizes the existing data synchronization flow, reduces the performance influence caused by network delay, and further effectively supports the synchronization between heterogeneous databases.

Of course, it is not necessary for any one product to practice the invention to achieve all of the advantages set forth above at the same time.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a design of a data master-slave synchronization mechanism captured by SQL based on a driver;

FIG. 2 is a flow chart of a design of a data peer-to-peer synchronization mechanism for SQL capture based on a driver of the present invention;

FIG. 3 is a flow chart of the SQL capture principle in connection driven JDBC of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The data synchronization method comprises the steps of copying one part of data from one machine to another machine through data copying, recording the mirror image of a database at a certain time point by a database snapshot according to the snapshot method, the trigger method and the log method, storing the mirror image, recovering by the mirror image, and recovering the rest of the database by the mirror image so as to completely copy;

the data replication consistency algorithm is a system for communication and mutual operation through message transmission, and the distribution of the data replication consistency algorithm can be different computers in geographic positions or different virtual nodes on the same physical machine;

in master-slave replication, data of one node is replicated to other nodes, the node is called a master node, the other nodes are called slave nodes, an application program can read the data of the slave nodes, so that the access pressure of the master node is relieved, the master-slave replication can improve the fault capability of reading operation, and peer-to-peer replication is also called master-master replication, namely all nodes are identical in position without distinguishing the master-slave nodes and can replicate each other.

1-3, the data synchronization device, the storage medium and the computer equipment of the invention comprise a data synchronization related technology research, a multi-data center operation mode analysis and construction and a remote database synchronization mechanism captured based on a driver SQL;

each data synchronization computer is divided into three data storage areas, specifically, a computer module A comprises a local data storage area of the computer module A, a mapping data storage area of the computer module B and a mapping data storage area of the computer module C; the computer module B comprises a local data storage area of the computer module B, a mapping data storage area of the computer module A and a mapping data storage area of the computer module C; the computer module C comprises a local data storage area of the computer module C, a mapping data storage area of the computer module A and a mapping data storage area of the computer module B;

the related technical researches of data synchronization comprise data replication, a data replication consistency algorithm, master-slave replication and peer-to-peer replication;

the multi-data center operation mode comprises a typical multi-data center operation mode, multi-data center operation mode analysis and bank financial data center operation mode construction;

the remote database synchronization mechanism based on the SQL capture of the driver comprises a remote data synchronization mechanism overall design, a remote data synchronization mechanism communication mode design, an SQL capture method in the driver, data consistency verification, a remote data synchronization experiment design and remote data synchronization experiment result analysis;

the remote data synchronization system controlled by the driver comprises a client side, a desktop client side or a mobile phone app side which are positioned in a browser, and is connected with a main database system through a connection driver to realize unified interface connection.

The data replication can be divided into database replication, file system replication and disk storage replication;

the database replication mainly comprises a snapshot method, a trigger method and a log method, wherein the database snapshot records the mirror image of the database at a certain time point and stores the mirror image, the mirror image can be used for recovering the database, other databases can be recovered by the mirror image so as to be completely replicated, the replication method is simplest, but the database is completely replicated, the network transmission quantity is large, the efficiency is very low, the method is generally adopted when the database is initialized and replicated, the database trigger means that when a defined certain condition is met, the corresponding event execution is triggered, but the influence of excessive triggers on the performance of the database is greatly defined, the database log method is a method which uses more at present, all operation behaviors of the database are saved by a series of log files, other databases can be orderly replicated by the aid of the log files, but the log formats, the generation modes, the operation interfaces and the like of different databases are different, so that the method is unfavorable for replication among heterogeneous databases;

the synchronization is to coordinate events occurring in a system, the phenomenon of consistency and unification occurs in time, when the system sends a calling request, if the result of the calling request is not obtained, the system waits until the result of the request operation is reached, and the caller actively waits for the result of the calling;

the data replication has synchronous replication and asynchronous replication modes, synchronous replication means that a replication initiator actively waits for the replication result, and asynchronous replication means that the replication initiator does not need to wait for whether the replication is successful or not, and directly performs the next action.

The data replication consistency algorithm is a distributed system, namely hardware or software is distributed on different computers, and the system is communicated and interoperated through message transmission, wherein the distribution is different in different computers in geographic positions or different virtual nodes on the same physical machine;

the data replication can make the data among all distributed nodes identical, which is helpful for improving the availability of the distributed system, but the data replication also easily causes consistency problem, the distributed consistency refers to the fact that after each node adopts a data replication mode in the distributed system, the data among different nodes can be different, the consistency scheme of all the distributed systems is not satisfied, the consistency of the data is ensured, the running performance of the system is not affected, and each distributed system has to be considered and weighted with the following characteristics: distribution, peer-to-peer, concurrency, lack of global clocks, failure always occur, and problems such as communication abnormality, network partition, tri-state (success, failure, timeout), node failure and the like occur in a distributed environment;

consistency category:

the consistency can be divided into strong consistency, weak consistency and final consistency, the data read from the system is required to be completely identical to the data just written in the system, which is the most ideal state but generally has larger influence on the performance of the system, the weak consistency requires that the value read immediately after the system is successfully written in is not guaranteed to be identical to the value just written in, but after a certain period of time, the read value is completely identical to the written value, the weak consistency can be further divided into session consistency and user consistency according to specific engineering implementation, the session consistency, the weak consistency level of the property only ensures that the read value is completely identical to the written value under the same client, the read value of the property is not guaranteed to be identical to the written value of the other user, the final consistency is a special form of the weak consistency, and the data can be read out from the system within a certain period of time, but the data can be read from the same;

CAP theorem:

the CAP theorem refers to that a normal distributed system cannot simultaneously satisfy three characteristics, namely consistency, availability and partition fault tolerance, in a distributed environment, consistency refers to whether data consistency can be maintained among nodes, if a system performs writing operation in a data consistency state, the read value of the system is ensured to be the same as the written value, and availability refers to that the system can return corresponding operation results in a limited time for each operation request, and uninterrupted service is provided. Partition fault tolerance refers to the fact that under the condition that any part of nodes of a system fail, consistency and availability of services can be guaranteed, unless the whole network environment or all nodes fail;

BASE theorem:

BASE refers to basic availability, soft state and final consistency, basic availability refers to that a system allows partial availability loss under the condition of faults, such as response time can be longer, partial functions can not work, weak state is also called soft state, time for allowing the system to synchronize data among different nodes can be delayed, final consistency promises to be within a certain time, the same system can achieve the same data value read and write, it emphasizes that the same data value can be achieved finally, BASE is consistency and availability in the balance CAP theorem, and the core of the BASE is that even if strong consistency is not achieved, each system can adopt a mode suitable for itself according to own service characteristics, such as sacrifice of response time, sacrifice of partial functions and the like, so that the whole system achieves the final consistency;

2PC and 3PC:

2PC is two-stage commit, in order to ensure that all nodes in the distributed environment can keep atomicity and consistency in the process of conducting transaction processing, the two-stage commit protocol is divided into two stages, and the specific implementation flow is as follows:

stage one: submitting a transaction request, inquiring the transaction, sending a request for whether the transaction can be submitted or not to all participants by the coordinator, waiting for feedback of the request, executing the transaction by each participant, logging relevant information, feeding back the relevant information, and sending feedback information to the coordinator by each participant;

stage two: executing transaction commit, wherein 3PC is three-stage commit, splitting the process of 'commit transaction request' in a two-stage commit protocol to form a transaction protocol which can be composed of commit, pre-commit and confirm commit;

stage one: a transaction query may be submitted, each participant feeding back to the coordinator;

stage two: the pre-commit phase, the condition that each participant feeds back to the coordinator determines whether they can then do pre-commit of the transaction;

stage three: and in the formal commit stage, if a participant feeds back a response of no to the coordinator or the transmission time is too long, the coordinator does not wait any more, the coordinator interrupts the transaction, otherwise, the commit is performed, and the three-stage commit protocol can reduce blocking and handle abnormal situations such as single-point faults and the like, but can solve the problem of inconsistent data.

Paxos algorithm:

the Paxos algorithm is a message transmission consistency algorithm with high fault tolerance of Leslie report, is one of the most effective algorithms for solving the message transmission consistency in the distributed environment at present, and when a certain node fault occurs, the whole node cluster needs to rapidly agree on certain submitted data, which is exactly the problem that the Paxos algorithm can solve;

the Paxos algorithm utilizes a few rules obeying majority, supports the conversion between the roles of the distributed nodes, and well avoids the occurrence of distributed single points. The Paxos algorithm solves the problem of infinite latency.

The master-slave replication and the peer-to-peer replication are divided into master-slave replication and peer-to-peer replication according to node roles of replication.

The multi-data center operation mode comprises an Internet company data center operation mode and a transport company ticketing system data center operation mode.

The multi-data center operation mode analysis comprises a same-city main-standby mode, a two-place three-center mode, a different-place double-activity mode, a different-place multi-activity mode and mode comparison.

The operation mode of the bank financial data center is constructed to comprise a single-write multi-read remote double-activity mode of the bank financial service data center and a multi-write multi-read remote multi-activity mode of the bank financial service data center.

The remote data synchronization mechanism overall design comprises a master-slave synchronization mechanism design and a peer-to-peer synchronization mechanism design;

the application program operates the database through a connection driver, the application program can be a desktop client or a browser client, the connection driver is a unified interface for the communication between the application program and the database system, the application program has a changeable application program structure due to different business logics, the same SQL capturer is difficult to set in the application program, and part of SQL sentences cannot be captured, so that the SQL capturer can be set in the connection driver.

As shown in FIG. 1, a SQL capturer is arranged at a connection driving layer between an application program and a database, and only captures SQL sentences which can change data records of a main database system, such as operation types of insertion, update and the like. For the operation types such as query and the like, sentences which do not change database data are directly filtered, the captured SQL sentences are sent by adding some identifications (such as time stamps and self-increasing IDs) and are analyzed by using a message queue server, related operations are carried out on a system by the aid of a standby database, if the requirements on data consistency are higher, a strong consistency strategy can be adopted, two operations are packaged into a transaction, the transaction can be submitted after the main standby database system successfully executes the same operations, otherwise, any operation failure rolls back the transaction, so that strong consistency can be effectively ensured, if the usability requirements on the system are higher, a final consistency strategy can be changed, namely, the result of standby database operation is not required to be waited, the main database can be directly submitted, the condition of operation failure can be checked by a check database, the check database is a database for the condition that the data of the main standby database is possibly inconsistent in the synchronization process, the check database generally adopts a NodeB of key value type, the actual production algorithm can be checked at intervals, the actual production period is set according to the concrete description of the actual production period;

in addition, because SQL is a standard general structured query language for relational databases and does not depend on a certain database, the mechanism can well support synchronization among heterogeneous databases, and transactions, scheduling methods, conflict detection and the like in the existing synchronization technology are mature, so that the mechanism is not described in detail herein.

As shown in fig. 2, similar to the master-slave synchronization mechanism, the peer-to-peer synchronization mechanism sets an SQL capturer in the local and remote connection drivers respectively, and sets the SQL capturer in the connection driver layer between the application program a and the local database a, where the SQL capturer captures only the SQL statements that can change the data record of the master database system, such as the operation types of insertion, update, etc., and directly filters out the statements that do not change the database data for the operation types of query, etc. The captured SQL sentence is added with some identifications, sent by using a message queue server, analyzed after the message is received, and then relevant operation is carried out on the remote database system A, and an SQL capturer is also arranged on a connection driving layer between the remote application program B and the remote database system B, the message queue server is used for sending the encapsulated captured SQL sentence message, analyzed after the message is received, and then relevant operation is carried out on the local database system B, wherein the diagram is too complex, so that the feedback condition of successful and failed operation is omitted;

the communication mode of the remote data synchronization mechanism is designed to be a common communication mode of a distributed system, wherein the common communication mode comprises remote procedure call, remote object call and message-oriented communication.

The SQL capturing method application program in the driver program is communicated with the database system through a connection driver;

as shown in fig. 3, the application communicates with the database system through a connection driver. In all connection drivers, JDBC is used more because of its cross-platform property, and thus, as an example, a method of capturing SQL in connection drivers is studied.

The beneficial effects are that:

according to related technologies such as data synchronization, different copying schemes are selected from clusters to cross-domain data centers according to different service characteristics, a synchronization strategy is selected, a remote data synchronization mechanism captured by SQL (structured query language) in a driver is based, a new data synchronization mechanism is provided for synchronization among heterogeneous databases which is low in efficiency and cannot be well supported in the existing remote data synchronization, a capturer is arranged in the driver, corresponding SQL is captured and sent to the heterogeneous databases for execution, a multi-data center construction mode is analyzed and constructed, an operation mode of the multi-active data center is a premise and a basis for researching remote data synchronization among the multi-data centers, and an operation mode of the multi-data center of a banking financial service information system is constructed by analyzing the construction mode of the existing multi-data center, so that the mechanism optimizes the existing data synchronization flow, reduces performance influence caused by network delay, and further effectively supports synchronization among the heterogeneous databases.

The preferred embodiments of the invention disclosed above are intended only to assist in the explanation of the invention. The preferred embodiments are not exhaustive or to limit the invention to the precise form disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best understand and utilize the invention. The invention is limited only by the claims and the full scope and equivalents thereof.

Claims (10)

1. The data synchronization method is characterized by comprising the following steps:

s1, copying one part of data from one machine to another machine through data copying, recording the mirror image of the database at a certain time point by the database snapshot according to the snapshot method, the trigger method and the log method, storing the mirror image, recovering by the mirror image, and recovering the rest of the database by the mirror image so as to completely copy;

s2, a data replication consistency algorithm is used for carrying out communication and mutual operation through a message transmission system, wherein the distribution of the data replication consistency algorithm can be different computers in geographic positions or different virtual nodes in the same physical machine;

and S3, in the master-slave replication, data of one node is replicated to other nodes, the node is called a master node, the other nodes are called slave nodes, an application program can read the data of the slave nodes, so that the access pressure of the master node is reduced, the master-slave replication can improve the fault capability of the reading operation, and the peer-to-peer replication is also called master-master replication, namely all nodes are identical in position and can replicate each other without dividing the master-slave nodes.

2. The method according to claim 1, wherein the data is copied in a synchronous copying manner and an asynchronous copying manner, the synchronous copying means that the copying initiator actively waits for the result of the copying, and the asynchronous copying means that the copying initiator does not have to wait for whether the copying is successful or not, and the next action is directly performed.

3. The data synchronization device is characterized by comprising a data synchronization related technology research, a multi-data center operation mode analysis and construction and a remote database synchronization mechanism captured based on a driver SQL;

the related technical research of the data synchronization comprises a data replication module, a data replication consistency algorithm module and a master-slave replication and peer-to-peer replication module;

the multi-data center operation module comprises a typical multi-data center operation mode, a multi-data center operation mode analysis module and a bank financial data center operation mode construction module;

the remote database synchronization mechanism based on the SQL capture of the driver comprises a remote data synchronization mechanism overall design module, a remote data synchronization mechanism communication mode design module, a SQL capture method module in the driver, a data consistency check module, a remote data synchronization experiment design module and a remote data synchronization experiment result analysis module.

4. A data synchronization device according to claim 3, wherein the data replication consistency algorithm is a distributed system, the system communicating and interoperating via messaging, the distribution being different computers in geographic locations or different virtual nodes on the same physical machine.

5. The data synchronization device of claim 3, wherein the multiple data center operation module comprises an internet company data center operation mode and a carrier ticketing system data center operation mode.

6. The data synchronization device of claim 3, wherein the multi-data center operation mode analysis module comprises a co-city primary-standby mode, a two-place three-center mode, a two-place two-active mode, and a two-place multi-active mode.

7. The data synchronization device of claim 3, wherein the bank financial data center operation mode construction module includes a single write multiple read off-site dual activity mode of the bank financial service data center and a multiple write multiple read off-site multiple activity mode of the bank financial service data center.

8. A data synchronization device according to claim 3, wherein the remote data synchronization mechanism overall design module comprises a master-slave synchronization mechanism design, a peer-to-peer synchronization mechanism design.

9. A data synchronization storage medium having stored thereon a computer program, which when executed by a processor performs the steps of the method according to any of claims 1-2.

10. Data synchronization device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor executing the program is the steps of implementing the method according to any of claims 1-2.

Images