CN115080664A - Data synchronization method, system and computer readable storage medium - Google Patents

Data synchronization method, system and computer readable storage medium Download PDF

Info

Publication number
CN115080664A
CN115080664A CN202210751777.6A CN202210751777A CN115080664A CN 115080664 A CN115080664 A CN 115080664A CN 202210751777 A CN202210751777 A CN 202210751777A CN 115080664 A CN115080664 A CN 115080664A
Authority
CN
China
Prior art keywords
data
standby
synchronization device
operation instruction
synchronization
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210751777.6A
Other languages
Chinese (zh)
Inventor
冯泽盛
张达摩
陶燕燕
杨申
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Industrial and Commercial Bank of China Ltd ICBC
Original Assignee
Industrial and Commercial Bank of China Ltd ICBC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Industrial and Commercial Bank of China Ltd ICBC filed Critical Industrial and Commercial Bank of China Ltd ICBC
Priority to CN202210751777.6A priority Critical patent/CN115080664A/en
Publication of CN115080664A publication Critical patent/CN115080664A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1448—Management of the data involved in backup or backup restore
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1458—Management of the backup or restore process
    • G06F11/1464—Management of the backup or restore process for networked environments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2379—Updates performed during online database operations; commit processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/80—Database-specific techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/54—Indexing scheme relating to G06F9/54
    • G06F2209/548—Queue

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Quality & Reliability (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Hardware Redundancy (AREA)

Abstract

The application discloses a data synchronization method, a data synchronization system and a computer readable storage medium. Relating to the field of cloud computing, the method comprises the following steps: the master synchronization device acquires a first operation instruction and updates data in the master database according to the first operation instruction; and the standby synchronization device acquires the first operation instruction and updates the data in the standby database according to the first operation instruction, wherein the main synchronization device and the standby synchronization device asynchronously update the data based on the first operation instruction. Through the data synchronization method and device, the problems of poor timeliness and poor stability of data synchronization in the related technology are solved.

Description

Data synchronization method, system and computer readable storage medium
Technical Field
The present application relates to the field of cloud computing, and in particular, to a data synchronization method, system, and computer-readable storage medium.
Background
In the related art, in the field where data needs to be updated and synchronized in time, data synchronization is usually performed by using a distributed database MySQL or a centralized database Oracle. Although the foregoing method in the related art can achieve data synchronization, the following drawbacks exist: when the MySQL is adopted to realize data synchronization, modification of contents such as indexes and SQL (Structured Query Language) can be involved, and the requirement of rapid data synchronization cannot be met due to the large investment required for migrating data; when data synchronization is carried out by adopting Oracle, data synchronization is realized by the main and standby databases built by the Oracle self-synchronizing technology, and the timeliness and stability of the data synchronization are poor. That is, the related art has a problem that the timeliness and stability of data synchronization are poor.
Aiming at the problem of poor timeliness and stability of data synchronization in the related technology, an effective solution is not provided at present.
Disclosure of Invention
The present application mainly aims to provide a data synchronization method, system and computer readable storage medium, so as to solve the problem of poor timeliness and stability of data synchronization in the related art.
In order to achieve the above object, according to one aspect of the present application, there is provided a data synchronization method. The method comprises the following steps: the method comprises the steps that a primary synchronization device obtains a first operation instruction and updates data in a primary database according to the first operation instruction; and the standby synchronization device acquires the first operation instruction and updates data in a standby database according to the first operation instruction, wherein the main synchronization device and the standby synchronization device asynchronously update the data based on the first operation instruction.
Optionally, the primary synchronization device and the standby synchronization device obtain the first operation instruction through a preset message queue, where the preset message queue is used to store an instruction for updating data in the primary database and/or an instruction for updating data in the standby database.
Optionally, the method further comprises: the instruction stored in the message queue is a target instruction, wherein the increment of the updated data corresponding to the target instruction exceeds a preset increment threshold.
Optionally, the method further comprises: under the condition that the increment of the data to be updated does not exceed the preset increment threshold, the primary synchronization device updates the data in the primary database based on the first operation instruction to obtain a data updating part, and sends the data updating part to the secondary synchronization device; and the standby synchronization device updates the data in the standby database based on the data updating part.
Optionally, the instruction in the preset message queue is generated based on the received operation object, the update value corresponding to the operation object, and an instruction code in the JAR file format.
Optionally, the obtaining, by the standby synchronization apparatus, the first operation instruction and updating data in the standby database according to the first operation instruction includes: under the condition that the standby synchronization device acquires the first operation instruction from the main synchronization device, the standby synchronization device detects whether the execution of updating the data in the standby database according to the first operation instruction is abnormal; the standby synchronization device sends an updating state of updating according to the first operation instruction to the main synchronization device under the condition that the standby synchronization device detects that the data in the standby database is updated and executed abnormally according to the first operation instruction; and the master synchronization device sends the first operation instruction to the standby synchronization device, and the standby synchronization device updates the data in the standby database based on the first operation instruction sent by the master synchronization device.
Optionally, the primary synchronization device and the secondary synchronization device are primary and secondary.
To achieve the above object, according to another aspect of the present application, there is provided a data synchronization system. The system comprises: the master synchronization device is used for acquiring a first operation instruction and updating data in the master database according to the first operation instruction; and the standby synchronization device is used for acquiring the first operation instruction and updating the data in the standby database according to the first operation instruction, wherein the main synchronization device and the standby synchronization device asynchronously update the data based on the first operation instruction.
Optionally, the system further comprises: and the preset message queue is used for storing an instruction for updating data in the main database and/or an instruction for updating data in the standby database.
Optionally, the synchronization apparatus includes: a first synchronization state recording module, configured to receive and store an update state sent by the slave synchronization apparatus, where the update state includes: the standby synchronization device sends the state of abnormal execution when detecting that the execution of the data in the standby database is abnormal; and the first timing task service module is used for acquiring a first operation instruction corresponding to the execution abnormal state from the first synchronization state recording module within a preset time interval, sending the first operation instruction to the standby synchronization device and updating the data in the standby database by the standby synchronization device according to the first operation instruction.
To achieve the above object, according to another aspect of the present application, there is provided a computer-readable storage medium having a program stored thereon, the program being executed by a processor to perform the method of any one of the above
To achieve the above object, according to another aspect of the present application, there is provided an electronic device comprising one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any one of the above.
Through the application, the following steps are adopted: the method comprises the steps that a primary synchronization device obtains a first operation instruction and updates data in a primary database according to the first operation instruction; and the standby synchronization device acquires the first operation instruction and updates the data in the standby database according to the first operation instruction, wherein the main synchronization device and the standby synchronization device asynchronously update the data based on the first operation instruction, and the problem of poor timeliness and stability of data synchronization in the related technology is solved. And then the effect of timely and stably synchronizing the data in the main database and the standby database is achieved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application. In the drawings:
FIG. 1 is a flow chart of a data synchronization method provided according to an embodiment of the present application;
FIG. 2 is a block diagram of a data synchronization system provided according to an embodiment of the present application;
FIG. 3 is a schematic structural diagram of a data synchronization system provided according to an embodiment of the present application;
FIG. 4 is a flowchart of a data update instruction processing method according to an embodiment of the present application;
fig. 5 is a block diagram of an electronic device for implementing a data synchronization method according to an embodiment of the present application.
Detailed Description
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances in order to facilitate the description of the embodiments of the application herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For convenience of description, some terms or expressions referred to in the embodiments of the present application are explained below:
MySQL, a relational database management system.
Oracle, a relational database management system.
DG (Data Guard), which implements Data protection by means of redundant Data, in particular, ensures synchronization between redundant Data and primary Data by means of a log synchronization mechanism. The method is mainly used for remote disaster recovery and high-availability data synchronization schemes of small enterprises.
CDC (Change Data Capture), a method of Data incremental extraction in Oracle. Updated data in the database after the previous incremental extraction of data can be obtained through CDC. Capturing of changed data is achieved by extracting changed data while performing operations such as insertion, upgrade, or deletion on a source table according to the CDC and saving the changed data to a database. After the changed data is captured, it can be transmitted to the target system in a controlled way using the database view.
MyBatis, a persistent layer framework, supports the execution, storage, and advanced mapping of customized SQL statements.
And the interceptor plug-in is used for modifying the SQL instruction.
And the message queue is a container for storing data in the data transmission process.
Kafka, a message queue.
The single-point service runs risks, only one component providing the same function in the system is needed, and if the component fails, the normal operation of the whole system is affected.
It should be noted that relevant information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for presentation, analyzed data, etc.) referred to in the present disclosure are information and data that are authorized by the user or sufficiently authorized by various parties. For example, an interface is provided between the system and the relevant user or institution, and before obtaining the relevant information, an obtaining request needs to be sent to the user or institution through the interface, and after receiving the consent information fed back by the user or institution, the relevant information needs to be obtained.
The present invention is described below with reference to preferred implementation steps, and fig. 1 is a flowchart of a data synchronization method provided in an embodiment of the present application, and as shown in fig. 1, the method includes the following steps:
step S101, the master synchronization device obtains a first operation instruction, and updates data in the master database according to the first operation instruction.
And S102, the standby synchronization device acquires a first operation instruction and updates data in the standby database according to the first operation instruction, wherein the main synchronization device and the standby synchronization device asynchronously update the data based on the first operation instruction.
According to the data synchronization method provided by the embodiment of the application, the primary synchronization device and the standby synchronization device respectively update the data in the corresponding primary database and the standby database based on the first operation instruction, so that the data in different databases are synchronized with high timeliness and high stability. The problems of poor data synchronization performance, poor data synchronization timeliness and poor stability when a centralized database is used for data synchronization in the related technology are solved, the running risk of single-point service of the centralized database is reduced, and in addition, the problem that a stock system is high in migration cost to a distributed database when a distributed database is used for data synchronization is solved.
In an alternative embodiment, there are multiple sets of the master synchronizer and the slave synchronizer, and each master synchronizer and each slave synchronizer are respectively disposed in different areas. The master synchronization device obtains the first operation instruction and sends the first operation instruction to the standby synchronization device. And the master synchronization device and the standby synchronization device respectively update the data in the corresponding databases according to the first operation instruction, so that asynchronous double writing of the data is realized. Therefore, the synchronization of the updated data of different areas is realized, and the timeliness and the stability of the data synchronization are good.
In an alternative embodiment, there are multiple sets of the master synchronizer and the slave synchronizer, and each master synchronizer and each slave synchronizer are respectively disposed in different areas. The standby synchronization device acquires the first operation instruction and sends the first operation instruction to the main synchronization device. And the master synchronization device and the standby synchronization device respectively update the data in the corresponding databases according to the first operation instruction, so that asynchronous double writing of the data is realized. Therefore, the synchronization of the updated data of different areas is realized, and the timeliness and the stability of the data synchronization are good.
In some optional embodiments, the primary synchronization apparatus and the standby synchronization apparatus obtain the first operation instruction through a preset message queue, where the preset message queue is used to store an instruction for updating data in the primary database and/or an instruction for updating data in the standby database. In an alternative embodiment, there are multiple categories of message queues. For example, message queues include ActiveMQ, RabbitMQ, Kafka, RockketMQ, and so forth.
In this optional embodiment, an instruction for updating data in the database is stored through a preset message queue, and the primary synchronization device and the secondary synchronization device acquire the first operation instruction through the preset message queue and update the data in the database according to the acquired first operation instruction. Compared with the instruction sending method in the related art, the instruction is directly sent to the standby synchronization device by the main synchronization device or is directly sent to the main synchronization device by the standby synchronization device, the corresponding instruction is stored by adopting the preset message queue, and the main synchronization device and the standby synchronization device obtain the instruction through the preset message queue, so that the memory queue, the multithreading management and the decoupling of the message queue and the main thread can be realized, and the stability of data synchronization is improved.
In some optional embodiments, the instruction stored in the message queue is a target instruction, where an increment of the updated data corresponding to the target instruction exceeds a preset increment threshold.
It should be understood that the increment is a unit of measurement of a time period in which the data is changed from the data in the previous unit of measurement. The increment of the data is large, which indicates that the variation of the data in the preset time period is large, and the data with large variation in the preset time period is the hot data.
In one embodiment, whether the data is hot data is determined based on a preset increment threshold, specifically, the data with the increment exceeding the preset increment threshold is determined to be the hot data, and the data with the increment not exceeding the preset increment threshold is determined to be non-hot data. The hot spot data updating method comprises the steps that an instruction used for updating hot spot data is a target instruction, the target instruction is stored in a message queue, and a main synchronization device and a standby synchronization device update the hot spot data in a corresponding database according to the target instruction stored in the message queue. Thereby. The data synchronization can be carried out on the hot spot data with high timeliness and high stability.
In some optional embodiments, the primary synchronization device sends an instruction for updating the hot spot data to the message queue, and the standby synchronization device obtains the instruction for updating the hot spot data through the message queue and updates the corresponding hot spot data in the standby database according to the instruction; the main synchronization device updates the non-hot data of which the increment does not exceed the preset increment threshold value in the main database, and sends the update part of the non-hot data to the standby synchronization device, and the standby synchronization device updates the data in the standby database based on the received data update part. The method comprises the steps of synchronizing hot spot data through an asynchronous double-write method, and directly sending a data updating part of non-hot spot data in a main database to a standby synchronization device so as to update the corresponding non-hot spot data in a standby database. Therefore, on the basis of ensuring the timeliness and the stability of data synchronization, the efficiency of data synchronization is improved.
In some optional embodiments, the instructions in the preset message queue are generated based on the received operation objects, the updated values corresponding to the operation objects, and the instruction codes in the JAR file format.
In an optional embodiment, at the application side of the primary synchronization device and the standby synchronization device, the judgment instruction for judging whether the data is a hotspot or a non-hotspot is packaged into a JAR packet, and the JAR packet is externally output by the plug-in of the framework interceptor of the persistent layer. Therefore, the hot spot data can be accurately and efficiently determined, and the hot spot data can be synchronized by an asynchronous double-writing method.
In some optional embodiments, the method for the standby synchronization apparatus to obtain the first operation instruction and update the data in the standby database according to the first operation instruction may include the following steps: under the condition that the standby synchronization device acquires the first operation instruction from the main synchronization device, the standby synchronization device detects whether the execution of updating the data in the standby database is abnormal according to the first operation instruction; the method comprises the steps that a standby synchronization device sends an updating state for updating according to a first operation instruction to a main synchronization device under the condition that the standby synchronization device detects that data in a standby database is updated and abnormally executed according to the first instruction; and the master synchronization device sends the first operation instruction to the standby synchronization device, and the standby synchronization device updates the data in the standby database based on the sent first operation instruction. There are various cases of execution exception, for example, the case of execution exception includes a case where data should be updated but not updated.
In this optional embodiment, the standby synchronization device sends the update state of the execution exception to the primary synchronization device, the primary synchronization device sends the corresponding first operation instruction to the standby synchronization device according to the update state of the execution exception, and the standby synchronization device updates the data in the standby database according to the received first operation instruction. The master synchronization device can send the first operation instruction corresponding to the abnormal update state to the standby synchronization device through the message queue. Therefore, the problem of poor data synchronization effect caused by abnormal execution of data updating is solved, and the stability and the synchronization effect of data synchronization are improved.
In the foregoing embodiment and optional embodiments, the primary synchronization device and the standby synchronization device are primary and standby each other. Namely: when the main synchronization device is used as the main equipment, the standby synchronization device is standby equipment of the main synchronization device, and when the standby synchronization device is used as the main equipment, the main synchronization device is standby equipment of the standby synchronization device. In the process of data synchronization, the main synchronization device and the standby synchronization device are hot standby each other.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
The embodiment of the present application further provides a data synchronization system, and it should be noted that the data synchronization system of the embodiment of the present application may be used to execute the method for data synchronization provided in the embodiment of the present application. The following describes a data synchronization system provided in an embodiment of the present application.
Fig. 2 is a block diagram of a data synchronization system according to an embodiment of the present application. As shown in fig. 2, the system includes: the master synchronizer 201 and the slave synchronizer 202 are explained below.
The master synchronization device 201 is configured to obtain a first operation instruction, and update data in the master database according to the first operation instruction; and the standby synchronization device 202 is connected to the main synchronization device 201, and is configured to obtain the first operation instruction and update the data in the standby database according to the first operation instruction, where the main synchronization device and the standby synchronization device asynchronously update the data based on the first operation instruction.
According to the data synchronization system, the main synchronization device and the standby synchronization device respectively update the data in the corresponding main database and the standby database based on the first operation instruction, and the data in different databases are synchronized with high timeliness and high stability. The problems of poor data synchronization performance, poor data synchronization timeliness and poor stability when a centralized database is used for data synchronization in the related technology are solved, the running risk of single-point service of the centralized database is reduced, and in addition, the problem that a stock system is high in migration cost to a distributed database when a distributed database is used for data synchronization is solved.
In some optional embodiments, the system further includes a preset message queue, configured to enable the primary synchronization apparatus and the standby synchronization apparatus to obtain the first operation instruction, where the preset message queue is configured to store an instruction for updating data in the primary database and/or an instruction for updating data in the standby database.
In some optional embodiments, the master synchronization apparatus comprises: the first synchronization state recording module is used for receiving and storing the update state sent by the synchronization device, and the update state comprises: the standby synchronization device sends the state of abnormal execution when detecting that the data in the standby database is abnormal; and the first timing task service module is used for acquiring a first operation instruction corresponding to the execution abnormal state from the first synchronization state recording module within a preset time interval, sending the first operation instruction to the standby synchronization device, and updating the data in the standby database by the standby synchronization device according to the first operation instruction.
Based on the foregoing embodiment and the optional embodiment, an optional implementation manner is provided, and taking the bank field as an example, taking the example that the master synchronization apparatus and the standby synchronization apparatus are respectively provided with one group, and the master synchronization apparatus and the standby synchronization apparatus are respectively located in the area a and the area B, taking the example that the permanent layer framework is MyBatis, the example that the instruction is generated based on the Java language, and the example that the message queue is Kafka, the detailed description is specifically given.
When data synchronization is carried out in the bank field, the situation that single-point service exists in an Oracle centralized library exists, and with the rapid increase of system business, the single-point service has the problems of high operation risk and high operation pressure, and the problems cause that the data synchronization method in the related technology is difficult to meet the requirements of capacity increase and high availability of the system. In order to ensure high availability of the database, in the related technology, data synchronization is realized by mostly adopting a routing middleware and a MySQL distributed database, or data synchronization is realized by adopting a main and standby database built by the routing middleware and an Oracle self-synchronization technology. These two methods in the related art are specifically described below.
In the related art, data synchronization is usually realized by adopting a routing middleware and a MySQL distributed database method. When the method is adopted for data synchronization, the change of objects such as views, materialized views, data storage processes, functions and instruction encapsulation packages can exist, the modification of contents such as indexes and SQL can be involved, large investment is needed for migrating data during data synchronization, and the timeliness of the data synchronization is poor when the method is adopted for data synchronization.
In the related technology, a main and standby database can be built by adopting a route middleware and the synchronization technology of the Oracle, and data synchronization is realized based on the main and standby database. However, in this method, data synchronization needs to be realized by DG and CDC synchronization technologies based on the Oracle database. The disadvantages of this synchronization method are: based on DG and CDC synchronization technologies, the synchronization performance of hot spot data synchronization cannot be guaranteed, the timeliness and stability of data synchronization between the main database and the standby database are poor, the synchronization method cannot meet the data synchronization of the hot spot data, and in addition, the data synchronization method cannot meet the high timeliness requirement of the data synchronization between the main database and the standby database.
In view of this, in the embodiments of the present disclosure, a data synchronization system is provided to solve the problems that in the related art, when data synchronization is performed in the bank field, there is a risk of operation of an Oracle centralized repository single-point service, and the data synchronization has poor timeliness and stability.
Fig. 3 is a schematic structural diagram of a data synchronization system provided according to an embodiment of the present application. Referring to fig. 3, the system includes a master synchronizer, a slave synchronizer, and Kafka, wherein the master synchronizer and the slave synchronizer are respectively disposed in zone a and zone B.
The master synchronization device includes: a main application server, a main synchronization server, a main database, a main new flow increasing water meter (equivalent to the first synchronization state recording module), and a main timing task server (equivalent to the first timing task service module); the standby synchronization device comprises: the system comprises a standby application server, a standby synchronous server, a standby database, a standby new flow increasing water meter and a standby timing task server.
The master synchronization device and the slave synchronization device are each a master device and a slave device, and the following takes the transmission direction of data from the master synchronization device to the slave synchronization device as an example to specifically describe each device in the data synchronization system.
And storing a data interception instruction sealed into a JAR packet at the application side of the system, and intercepting the data updating instruction based on the data interception instruction. Fig. 4 is a flowchart of a data update instruction processing method according to an embodiment of the present application. The following describes a method for processing a data update instruction with reference to fig. 4: writing SQL sentences in a Mapper (Mapper) by combining Java components, then pre-compiling the SQL sentences in a persistence layer framework MyBatis, executing the SQL sentences according to data interception instructions and setting parameters sealed into JAR packets, packaging execution results, and transmitting the packaging results to database Records (DB Records). The SQL statement is processed through the data interception instruction, the intercepted operation instruction is obtained, and the intercepted target instruction is transmitted to the message queue Kafka. In the optional real-time mode, the SQL statements corresponding to the target instructions include SQL statements corresponding to the operations of deleting, inserting, and updating the hotspot data.
Intercepting the update, insertion and deletion statements executed by the synchronization target database, writing the intercepted SQL statements into Kafka, and updating the data in the standby database by the standby synchronization device according to the SQL statements in the Kafka. The method is positioned at the synchronization of SQL statement levels, data synchronization codes are separated from main thread original service codes in the data synchronization process, the standby synchronization device has low logic invasion to the main thread original services, the data synchronization function and the original service function have clear boundaries, and the data synchronization stability is good.
Continuing to refer to fig. 3, the main application server sends the target instruction to Kafka, and puts the target instruction into storage in the main new flow-increasing water meter; the standby synchronous server acquires a target instruction from the Kafka, executes the target instruction, puts an execution result into a standby database, sends a changed data state to the main new flow increasing water meter, and can determine an instruction with abnormal execution by combining the target instruction stored in the main new flow increasing water meter and the changed data state sent by the standby synchronous server; the timing task server acquires an execution abnormal instruction according to a preset time interval and sends the execution abnormal instruction to the Kafka; the standby synchronization server fetches the instruction sent to the execution exception in Kafka and re-executes the instruction to synchronize the data in the standby database.
The updating data corresponding to the target instruction stored in the Kafka is deletion, insertion and updating of the hot spot data, and the hot spot data are synchronized according to the method. For non-hotspot data, synchronization is performed according to the following method: the main synchronous server updates data according to the instruction corresponding to the non-hotspot data and stores the data updating part into the main database, and the main database directly transmits the data updating part to the standby database through a CDC (data control and data recovery) synchronization method, so that the synchronization of the non-hotspot data in the main database and the standby database is realized.
It should be understood that the message queue that can be used in the embodiments of the present disclosure is not limited to Kafka, but can also be a message queue ActiveMQ, RabbitMQ, or rocktmmq, etc.; the persistent layer framework that can be used in the embodiments of the present disclosure is not limited to MyBatis, but may also be Hibernate, EJB, or other persistent layer framework.
In the embodiment of the present disclosure, synchronization of non-hotspot data between the primary database and the backup database is achieved by a CDC synchronization method, and synchronization of hotspot data between the primary database and the backup database is achieved by an asynchronous double-write synchronization method. The method is a simple and relatively efficient CDC synchronization method, non-hot data which are not easy to have the problems of synchronous timeliness and stability are synchronized, the timeliness and the stability of the data synchronization system for synchronizing the data can be guaranteed, and the synchronization efficiency of the data synchronization system for synchronizing the data can be guaranteed.
In the embodiment of the disclosure, Kafka is introduced into a system as a message queue, so that bidirectional synchronization of data in a main database and a standby database is realized; recording a data synchronization state by introducing a newly added flow table in the system; by introducing the timing task server into the system, the timing task server can periodically complement the data to be synchronized in the database according to the data synchronization state data recorded by the newly added flow meter; the Kafka message is extracted and executed by introducing the synchronization server into the system, the data in the database is updated and synchronized according to the execution result, and the execution result is sent to the new flow increasing water meter, so that the stability and the synchronization effect of data synchronization are improved.
The data synchronization system provided by the embodiment of the disclosure does not need to completely migrate from the architecture to the distributed database during data synchronization, so that development investment of object changes such as views, materialized views, storage processes, functions and packages, investment of content modification of indexes and SQL statements, and investment of migration data can be avoided, thereby reducing workload of development teams and meeting the requirement of rapid online of the system. And the synchronization of hot spot data is completed by adopting the program asynchronous double-writing main and standby databases, so that the logic invasion to the original service is small.
An embodiment of the present invention provides a computer-readable storage medium on which a program is stored, the program implementing a data synchronization method when executed by a processor.
The embodiment of the invention provides a processor, which is used for running a program, wherein a data synchronization method is executed when the program runs.
As shown in fig. 5, an embodiment of the present invention provides an electronic device, where the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor executes the program to implement the following steps: the master synchronization device acquires a first operation instruction and updates data in the master database according to the first operation instruction; and the standby synchronization device acquires the first operation instruction and updates the data in the standby database according to the first operation instruction, wherein the main synchronization device and the standby synchronization device asynchronously update the data based on the first operation instruction.
Optionally, the processor may further implement the following steps when executing the program: the main synchronization device and the standby synchronization device acquire a first operation instruction through a preset message queue, and the preset message queue is used for storing an instruction for updating data in the main database and/or an instruction for updating data in the standby database.
Optionally, the processor may further implement the following steps when executing the program: and the instruction stored in the message queue is a target instruction, wherein the increment of the updated data corresponding to the target instruction exceeds a preset increment threshold value.
Optionally, the processor may further implement the following steps when executing the program: under the condition that the increment of the data to be updated does not exceed the preset increment threshold, the main synchronization device updates the data in the main database based on the first operation instruction to obtain a data updating part and sends the data updating part to the standby synchronization device; the backup synchronization device updates the data in the backup database based on the data updating portion.
Optionally, the processor may further implement the following steps when executing the program: and generating an instruction in the preset message queue based on the received operation object, the update value corresponding to the operation object and the instruction code in the JAR file format.
Optionally, the processor may further implement the following steps when executing the program: the method for acquiring the first operation instruction by the standby synchronization device and updating the data in the standby database according to the first operation instruction comprises the following steps: under the condition that the standby synchronization device acquires the first operation instruction from the main synchronization device, the standby synchronization device detects whether the execution of updating the data in the standby database is abnormal according to the first operation instruction; the method comprises the steps that a standby synchronization device sends an updating state for updating according to a first operation instruction to a main synchronization device under the condition that the standby synchronization device detects that data in a standby database is updated and abnormally executed according to the first instruction; and the master synchronization device sends the first operation instruction to the standby synchronization device, and the standby synchronization device updates the data in the standby database based on the first operation instruction sent by the master synchronization device.
Optionally, the processor may further implement the following steps when executing the program: the main synchronizer and the standby synchronizer are mutually main and standby.
The device herein may be a server, a PC, a PAD, a handset, etc.
The present application further provides a computer program product adapted to perform a program for initializing the following method steps when executed on a data processing device: the master synchronization device acquires a first operation instruction and updates data in the master database according to the first operation instruction; and the standby synchronization device acquires the first operation instruction and updates the data in the standby database according to the first operation instruction, wherein the main synchronization device and the standby synchronization device asynchronously update the data based on the first operation instruction.
Optionally, the computer program product described above, when being executed on a data processing device, is further adapted to perform a procedure for initializing the following method steps: the main synchronization device and the standby synchronization device acquire the first operation instruction through a preset message queue, and the preset message queue is used for storing an instruction for updating data in the main database and/or an instruction for updating data in the standby database.
Optionally, the computer program product described above, when being executed on a data processing device, is further adapted to perform a procedure for initializing the following method steps: and the instruction stored in the message queue is a target instruction, wherein the increment of the updated data corresponding to the target instruction exceeds a preset increment threshold value.
Optionally, the computer program product described above, when being executed on a data processing device, is further adapted to perform a procedure for initializing the following method steps: under the condition that the increment of the data to be updated does not exceed the preset increment threshold, the main synchronization device updates the data in the main database based on the first operation instruction to obtain a data updating part and sends the data updating part to the standby synchronization device; the backup synchronization device updates the data in the backup database based on the data updating portion.
Optionally, the computer program product described above, when being executed on a data processing device, is further adapted to perform a procedure for initializing the following method steps: and generating an instruction in the preset message queue based on the received operation object, the update value corresponding to the operation object and the instruction code in the JAR file format.
Optionally, the computer program product described above, when being executed on a data processing device, is further adapted to perform a procedure for initializing the following method steps: the method for acquiring the first operation instruction by the standby synchronization device and updating the data in the standby database according to the first operation instruction comprises the following steps: under the condition that the standby synchronization device acquires the first operation instruction from the main synchronization device, the standby synchronization device detects whether the execution of updating the data in the standby database is abnormal according to the first operation instruction; the method comprises the steps that a standby synchronization device sends an updating state for updating according to a first operation instruction to a main synchronization device under the condition that the standby synchronization device detects that data in a standby database is updated and abnormally executed according to the first instruction; the master synchronization device sends the first operation instruction to the standby synchronization device, and the standby synchronization device updates the data in the standby database based on the first operation instruction sent by the master synchronization device.
Optionally, the computer program product described above, when being executed on a data processing device, is further adapted to perform a procedure for initializing the following method steps: the main synchronizer and the standby synchronizer are mutually main and standby.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the use of the verb "comprise a" to define an element does not exclude the presence of another, identical element in a process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims (10)

1. A method of data synchronization, comprising:
the method comprises the steps that a primary synchronization device obtains a first operation instruction and updates data in a primary database according to the first operation instruction;
and the standby synchronization device acquires the first operation instruction and updates data in a standby database according to the first operation instruction, wherein the main synchronization device and the standby synchronization device asynchronously update the data based on the first operation instruction.
2. The method according to claim 1, wherein the primary synchronization apparatus and the secondary synchronization apparatus obtain the first operation instruction through a preset message queue, and the preset message queue is used for storing an instruction for updating data in the primary database and/or an instruction for updating data in the secondary database.
3. The method of claim 2, further comprising:
the instruction stored in the message queue is a target instruction, wherein the increment of the updated data corresponding to the target instruction exceeds a preset increment threshold.
4. The method of claim 3, further comprising:
under the condition that the increment of the data to be updated does not exceed the preset increment threshold, the primary synchronization device updates the data in the primary database based on the first operation instruction to obtain a data updating part, and sends the data updating part to the secondary synchronization device;
the standby synchronization device updates the data in the standby database based on the data updating part.
5. The method according to claim 2, wherein the instructions in the preset message queue are generated based on the received operation object, the updated value corresponding to the operation object, and an instruction code in a JAR file format.
6. The method of claim 1, wherein the retrieving the first operation instruction and updating the data in the standby database according to the first operation instruction by the standby synchronization apparatus comprises:
under the condition that the standby synchronization device acquires the first operation instruction from the main synchronization device, the standby synchronization device detects whether the execution of updating the data in the standby database according to the first operation instruction is abnormal;
the standby synchronization device sends an updating state of updating according to the first operation instruction to the main synchronization device under the condition that the standby synchronization device detects that the updating execution of the data in the standby database is abnormal according to the first operation instruction;
and the master synchronization device sends the first operation instruction to the standby synchronization device, and the standby synchronization device updates the data in the standby database based on the first operation instruction sent by the master synchronization device.
7. The method according to any one of claims 1 to 6, wherein the primary synchronization device and the backup synchronization device are primary and backup to each other.
8. A data synchronization system, comprising:
the master synchronization device is used for acquiring a first operation instruction and updating data in the master database according to the first operation instruction;
and the standby synchronization device is used for acquiring the first operation instruction and updating the data in the standby database according to the first operation instruction, wherein the main synchronization device and the standby synchronization device asynchronously update the data based on the first operation instruction.
9. A computer-readable storage medium, characterized in that a program is stored thereon, which program is executed by a processor for performing the method of any one of claims 1 to 7.
10. An electronic device comprising one or more processors and memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-7.
CN202210751777.6A 2022-06-29 2022-06-29 Data synchronization method, system and computer readable storage medium Pending CN115080664A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210751777.6A CN115080664A (en) 2022-06-29 2022-06-29 Data synchronization method, system and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210751777.6A CN115080664A (en) 2022-06-29 2022-06-29 Data synchronization method, system and computer readable storage medium

Publications (1)

Publication Number Publication Date
CN115080664A true CN115080664A (en) 2022-09-20

Family

ID=83254744

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210751777.6A Pending CN115080664A (en) 2022-06-29 2022-06-29 Data synchronization method, system and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN115080664A (en)

Similar Documents

Publication Publication Date Title
CN110209726B (en) Distributed database cluster system, data synchronization method and storage medium
US10949447B2 (en) Blockchain-based data synchronizing and data block parsing method and device
CN109101627B (en) Heterogeneous database synchronization method and device
CN104809202B (en) A kind of method and apparatus of database synchronization
CN102081611B (en) Method and device for synchronizing databases of master network management system and standby network management system
US20120297249A1 (en) Platform for Continuous Mobile-Cloud Services
CN107515874B (en) Method and equipment for synchronizing incremental data in distributed non-relational database
CN110019514B (en) Data synchronization method and device and electronic equipment
JP6987340B2 (en) Database data change request processing method and equipment
CN106599061A (en) SQLite-based embedded database synchronization method
CN111581285B (en) Data information synchronization method, device, electronic equipment and medium
CN112015795A (en) System and method for large-data-volume ad hoc query
CN115080664A (en) Data synchronization method, system and computer readable storage medium
CN116414914A (en) Data synchronization method and device, processor and electronic equipment
CN115757642A (en) Data synchronization method and device based on filing log file
CN111241044B (en) Method, device, equipment and readable storage medium for constructing heterogeneous database
CN113641760A (en) Data synchronization method and device
CN112217881B (en) File synchronization method and equipment based on operation and maintenance audit system
CN116567007B (en) Task segmentation-based micro-service water conservancy data sharing and exchanging method
US20230409535A1 (en) Techniques for resource utilization in replication pipeline processing
CN116303760A (en) Data synchronization method and device of bank core system and electronic equipment
US20240086417A1 (en) Techniques for replication-aware resource management and task management of file systems
Ji et al. Design of Heterogeneous Data Synchronization Framework for Power System
CN116204120A (en) Multi-cluster data searching method, device and computer readable storage medium
CN117725130A (en) Data synchronization method and device, storage medium and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination