CN116860881A

CN116860881A - Data synchronization method, data synchronization device and electronic device

Info

Publication number: CN116860881A
Application number: CN202310923055.9A
Authority: CN
Inventors: 张莹; 董卉; 王元元; 杨�嘉; 陈铎
Original assignee: Postal Savings Bank of China Ltd
Current assignee: Postal Savings Bank of China Ltd
Priority date: 2023-07-25
Filing date: 2023-07-25
Publication date: 2023-10-10

Abstract

The application provides a data synchronization method, a data synchronization device and an electronic device, wherein the data synchronization method comprises the following steps: controlling a database to be synchronized to receive service operation data, and screening the service operation data according to a preset interception rule to obtain the data to be synchronized, wherein the service operation data is data for performing service operation on the database, and the service operation at least comprises an adding operation, a deleting operation and a modifying operation; controlling a database to be synchronized to send data to be synchronized to a message middleware, and storing the data to be synchronized to a message queue of the message middleware; and controlling the target database to receive the message queue of the message middleware so as to write the data to be synchronized stored in the message queue into the target database. The method solves the problem of large resource consumption in the process of data synchronization of the database in the prior art.

Description

Data synchronization method, data synchronization device and electronic device

Technical Field

The present application relates to the field of databases, and more particularly, to a data synchronization method, a data synchronization device, a computer-readable storage medium, and an electronic device.

Background

With the continuous development of information technology, more and more systems or project designs adopt a micro-service architecture and a technical architecture of a database. With the continuous increase of service demands and transaction amounts, the high concurrency performance of the system can be improved by adopting a mode of continuous optimization of codes and transverse expansion of service nodes, or the consistency of data across centers can be ensured on the basis of a database self-flow replication mode, so that the bottleneck of the system is gradually focused on the database. The current informatization system is constructed by adopting a multi-center multi-activity and different-place disaster recovery deployment mode or a distributed deployment mode, so that the increasingly-increased performance requirement of the system and the disaster resistance capability of flexibly coping with natural disasters and emergency conditions are improved. In a multi-center multi-activity or application distributed deployment mode, data fall into different databases according to a certain routing rule, service operation continuity requires that the data among the different databases are consistent, and only key core service data among the different databases are consistent, and the flow log data can not be synchronously processed.

For example, when a business performs (new adding, modifying, deleting) operations on a database, the generated data only falls on an a database, but the next request (inquiring, adding, modifying, deleting, exporting, etc.) operation may fall on a B database or a C database, if the data of the a database cannot be written to the B database or the C database in near real time when the data of the a database changes, the phenomenon of data disorder and inconsistency occurs during the operation before and after the business, and how to realize the synchronization of the a database change to the B database and the C database data is needed to be solved. However, in the prior art, the data synchronization is realized by using the self-stream replication of the database, the system performance is affected, all data are not required to be synchronized, a certain resource waste exists, and the data synchronization is not flexible enough.

Therefore, a method is needed to solve the problem of large resource consumption in the process of data synchronization of the database.

Disclosure of Invention

The application aims to provide a data synchronization method, a data synchronization device, a computer readable storage medium and an electronic device, which are used for at least solving the problem of high resource consumption in the process of data synchronization of a database in the prior art.

According to an aspect of the present application, there is provided a data synchronization method, the method comprising: controlling a database to be synchronized to receive service operation data, and screening the service operation data according to a preset interception rule to obtain the data to be synchronized, wherein the service operation data is data for performing service operation on the database, and the service operation at least comprises an adding operation, a deleting operation and a modifying operation; controlling the database to be synchronized to send the data to be synchronized to a message middleware, and storing the data to be synchronized into a message queue of the message middleware; and controlling a target database to receive the message queue of the message middleware so as to write the data to be synchronized stored in the message queue into the target database.

Optionally, screening the service operation data according to a preset interception rule to obtain data to be synchronized, including: and under the condition that the business operation data meets the preset interception rule, determining that the business operation data is the reject data, rejecting the reject data from the business operation data, and the rest of the business operation data is the data to be synchronized.

Optionally, controlling the database to be synchronized to send the data to be synchronized to a message middleware includes: controlling the database to be synchronized to start transactions, and storing the data to be synchronized to a system context; and controlling the system context to send the data to be synchronized to the message middleware.

Optionally, after obtaining the data to be synchronized, before controlling the database to be synchronized to send the data to be synchronized to the message middleware, the method further includes: performing corresponding operation on the data of the database to be synchronized corresponding to the data to be synchronized according to the service operation corresponding to the data to be synchronized, receiving return information responding to the operation, and determining whether the operation is successful or not according to the return information; under the condition that the operation of the database to be synchronized is successful, controlling the database to be synchronized to carry out transaction submission, and outputting a success signal, wherein the success signal is used for representing that the operation of the database to be synchronized is successful according to the data to be synchronized; and under the condition that the operation of the database to be synchronized is unsuccessful, controlling the database to be synchronized to perform transaction rollback, and outputting a first failure signal, wherein the first failure signal is used for representing that the operation of the database to be synchronized according to the data to be synchronized is unsuccessful.

Optionally, the database to be synchronized includes a parameter service unit, a first synchronization service unit, and a first data storage unit, and the controlling the database to be synchronized to send the data to be synchronized to a message middleware includes: controlling the parameter service unit to register as a message producer of the message middleware; controlling the first synchronization service unit to register as a message consumer of the message middleware.

Optionally, the target database includes a second synchronization service unit and a second data storage unit, and the controlling the target database to receive the message queue of the message middleware to write the data to be synchronized stored in the message queue into the target database includes: controlling the parameter service unit to acquire the data to be synchronized stored in the first data storage unit and sending the data to be synchronized to the first synchronization service unit; controlling the first synchronous service unit to forward the message queue to the second synchronous service unit; and controlling the second synchronous service unit to write the data to be synchronized stored in the message queue into the second data storage unit.

Optionally, after controlling the first synchronization service unit to forward the message queue to the second synchronization service unit, the method further includes: determining whether the first synchronous service unit is successfully forwarded; under the condition that the first synchronous service unit is successful in forwarding, the first synchronous service unit is controlled to stop forwarding; and under the condition that the forwarding of the first synchronous service unit is unsuccessful, outputting a second failure signal and controlling the first synchronous service unit to repeatedly forward the message queue to the second synchronous service unit, wherein the second failure signal is used for representing that the forwarding of the first synchronous service unit is unsuccessful.

According to another aspect of the present application, there is provided a data synchronization apparatus including: the first control unit is used for controlling the database to be synchronized to receive service operation data, screening the service operation data according to a preset interception rule to obtain the data to be synchronized, wherein the service operation data is data for performing service operation on the database, and the service operation at least comprises an adding operation, a deleting operation and a modifying operation; the second control unit is used for controlling the database to be synchronized to send the data to be synchronized to the message middleware and storing the data to be synchronized into a message queue of the message middleware; and the third control unit is used for controlling a target database to receive the message queue of the message middleware so as to write the data to be synchronized stored in the message queue into the target database.

According to still another aspect of the present application, there is provided a computer readable storage medium including a stored program, wherein the program when run controls a device in which the computer readable storage medium is located to perform any one of the methods.

According to a further aspect of the application there is provided an electronic device comprising a memory having a computer program stored therein and a processor arranged to perform any one of the methods by means of the computer program.

By applying the technical scheme of the application, firstly, a database to be synchronized is controlled to receive service operation data, and the service operation data is screened according to a preset interception rule to obtain the data to be synchronized, wherein the service operation data is data for performing service operation on the database, and the service operation at least comprises an adding operation, a deleting operation and a modifying operation; controlling the database to be synchronized to send the data to be synchronized to the message middleware, and storing the data to be synchronized into a message queue of the message middleware; and finally, controlling the target database to receive the message queue of the message middleware so as to write the data to be synchronized stored in the message queue into the target database. The data to be synchronized is obtained by intercepting the service operation data, the service application performance of the database is not affected differently, the intercepted data to be synchronized is synchronized to the target database through the message middleware asynchronously, the cross-database data synchronization is realized, the self resources of the database are not consumed, and the problem of large resource consumption in the process of data synchronization of the database in the prior art is solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

fig. 1 is a block diagram showing a hardware structure of a mobile terminal performing a data synchronization method according to an embodiment of the present application;

FIG. 2 is a flow chart of a method for synchronizing data according to an embodiment of the present application;

FIG. 3 is a schematic overall flow chart of a data synchronization method according to an embodiment of the present application;

fig. 4 shows a block diagram of a data synchronization device according to an embodiment of the present application.

Wherein the above figures include the following reference numerals:

102. a processor; 104. a memory; 106. a transmission device; 108. and an input/output device.

Detailed Description

It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other. The application will be described in detail below with reference to the drawings in connection with embodiments.

In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the application herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

As described in the background art, in order to solve the above problem, embodiments of the present application provide a data synchronization method, a data synchronization device, a computer readable storage medium, and an electronic device.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

The method embodiments provided in the embodiments of the present application may be performed in a mobile terminal, a computer terminal or similar computing device. Taking the mobile terminal as an example, fig. 1 is a block diagram of a hardware structure of a mobile terminal of a data synchronization method according to an embodiment of the present application. As shown in fig. 1, a mobile terminal may include one or more (only one is shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a microprocessor MCU or a processing device such as a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 for communication functions and an input-output device 108. It will be appreciated by those skilled in the art that the structure shown in fig. 1 is merely illustrative and not limiting of the structure of the mobile terminal described above. For example, the mobile terminal may also include more or fewer components than shown in fig. 1, or have a different configuration than shown in fig. 1.

The memory 104 may be used to store a computer program, for example, a software program of application software and a module, such as a computer program corresponding to a data synchronization method in an embodiment of the present application, and the processor 102 executes the computer program stored in the memory 104 to perform various functional applications and data processing, that is, implement the above-mentioned method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, which may be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, simply referred to as NIC) that can connect to other network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is configured to communicate with the internet wirelessly.

In this embodiment, a data synchronization method operating on a mobile terminal, a computer terminal, or a similar computing device is provided, and it should be noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different from that herein.

Fig. 2 is a flowchart of a data synchronization method according to an embodiment of the present application. As shown in fig. 2, the method comprises the steps of:

step S201, controlling a database to be synchronized to receive service operation data, and screening the service operation data according to a preset interception rule to obtain the data to be synchronized, wherein the service operation data is data for performing service operation on the database, and the service operation at least comprises an adding operation, a deleting operation and a modifying operation;

specifically, the service operation data and the data to be synchronized can be transferred in the form of SQL sentences. The above-mentioned screening of the above-mentioned business operation data to obtain the data to be synchronized can be implemented by means of interceptor, which is a component used for intercepting and processing the request in software. It may perform certain operations before and after the request is processed. Interceptors are typically used to pre-process a request before it reaches a target component or post-process a target component after it returns a result. Interceptors are commonly used with frameworks and middleware, for example in Java, which can be used with Spring frameworks for implementing various functions such as rights verification, transaction management, etc. The interceptor may be a mybatis interceptor. The preset interception rules can be flexibly configured, synchronous consumption of some non-critical data is avoided, and the interception rules can be configured according to library instance names, table names and operation types.

Step S202, the database to be synchronized is controlled to send the data to be synchronized to a message middleware, and the data to be synchronized is stored in a message queue of the message middleware;

in particular, message middleware is used to communicate and process messages in a distributed system to enable decoupling and asynchronous communications. The message middleware may enable decoupling and asynchronous communication, wherein by sending messages to the message middleware, the degree of coupling between the message producer and the message consumer is reduced, and the message producer need only send messages to the message middleware without knowing which consumers the message will be processed by. Also, the message consumer need only receive messages from the message middleware, and need not know which producer the message was sent by. In addition, message middleware provides a mechanism for asynchronous communication, i.e., message transmission and reception are independent. The message producer can send the message immediately without waiting for the message consumer's response. Such an asynchronous communication model may improve the response performance and throughput of the system. Message middleware also provides a retry mechanism and an error handling mechanism for messages to handle error conditions that may occur when sending or receiving messages. Common message middleware includes Apache Kafka, rabbitMQ, activeMQ, etc. They provide various functions and features in different scenarios and requirements, and appropriate message middleware can be selected according to specific business requirements.

Step S203, the control target database receives the message queue of the message middleware, so as to write the data to be synchronized stored in the message queue into the target database;

specifically, the target database can be multiple, and a multi-center multi-activity deployment mechanism for synchronizing data across multiple databases is realized through a message broadcasting mode of a message middleware. Asynchronous operation processing can be performed through the message middleware, the current application performance is not affected, and meanwhile timeliness of data synchronization can be guaranteed.

Through the embodiment, firstly, a database to be synchronized is controlled to receive service operation data, and the service operation data is screened according to a preset interception rule to obtain the data to be synchronized, wherein the service operation data is data for performing service operation on the database, and the service operation at least comprises an adding operation, a deleting operation and a modifying operation; controlling the database to be synchronized to send the data to be synchronized to the message middleware, and storing the data to be synchronized into a message queue of the message middleware; and finally, controlling the target database to receive the message queue of the message middleware so as to write the data to be synchronized stored in the message queue into the target database. The data to be synchronized is obtained by intercepting the service operation data, the service application performance of the database is not affected differently, the intercepted data to be synchronized is synchronized to the target database through the message middleware asynchronously, the cross-database data synchronization is realized, the self resources of the database are not consumed, and the problem of large resource consumption in the process of data synchronization of the database in the prior art is solved.

In a specific implementation process, the step S201 may be implemented by the following steps: and under the condition that the service operation data meets the preset interception rule, determining that the service operation data is the reject data, rejecting the reject data from the service operation data, and the rest of the service operation data is the data to be synchronized. The method is used for interception, the non-invasive processing mode does not consume database resources, and the decoupling of the core application module and the synchronous module can be realized, so that the system resources and the bandwidth of the database are further saved, and the maintenance and coding cost is reduced.

Specifically, the preset interception rules can be set according to the name of the library instance, the name of the table and the operation type, for example, in the process of transferring accounts by a user, the records of the account transfer records of the user, the specific operation trace of the account and the like are recorded, so that interception is realized, and the intercepted data are analyzed. The process of setting the preset interception rules can be implemented in an apollo configuration center.

In order to further avoid the occurrence of abnormal situations, the above step S202 of the present application may be implemented by the following steps: step S2021, controlling the database to be synchronized to open the transaction, and storing the data to be synchronized to the system context; step S2022 controls the system context to send the data to be synchronized to the message middleware. The method firstly puts the intercepted data to be synchronized into a system context, and then sends the data to be synchronized to a message queue when the transaction is submitted successfully, and further avoids the abnormal scene that the data synchronization action is executed first and the transaction rollback cannot be cancelled by utilizing the system context.

In particular, the system context refers to the environment and conditions in a computer system that are required in order for a program to function properly. The system context includes an operating system, hardware devices, system configuration, and the like. During program operation, the system context directly affects the behavior and results of the program. For example, the operating system's operating state, allocation and scheduling of system resources, the state of hardware devices, etc. are all part of the system context. The program needs to be correspondingly processed and adjusted according to the change of the system context in the running process, so that the program can be ensured to run normally and reach the expected result.

After the step S201, the step S202 further includes: step S204, according to the business operation corresponding to the data to be synchronized, performing corresponding operation on the data corresponding to the data to be synchronized of the database to be synchronized, receiving the return information responding to the operation, and determining whether the operation is successful according to the return information; step S205, under the condition that the operation of the database to be synchronized is successful, controlling the database to be synchronized to carry out transaction submission, and outputting a success signal, wherein the success signal is used for representing that the operation of the database to be synchronized is successful according to the data to be synchronized; step S206, under the condition that the operation of the database to be synchronized is unsuccessful, controlling the database to be synchronized to perform transaction rollback, and outputting a first failure signal, wherein the first failure signal is used for representing that the operation of the database to be synchronized according to the data to be synchronized is unsuccessful. The method can further ensure the consistency of the database, and can restore to the original state even if errors occur.

Specifically, the start transaction refers to starting a transaction in a database operation, and treating a series of database operations as a whole, in order to ensure that the series of operations all succeed or all fail. Transaction commit refers to committing a previously opened transaction after database operations are completed, and permanently saving all operations to the database. The transaction commit characterizes that all operations in the transaction were successfully performed and have been validated. Transaction rollback refers to the state before the transaction begins, in which all previous operations may be undone during database operations if an error occurs or some operations fail. Transaction rollback can ensure consistency of the database and can restore to the original state even if errors occur.

The step S202 may also be implemented in other manners, where the database to be synchronized includes a parameter service unit, a first synchronization service unit, and a first data storage unit, for example: step S2023, controlling the parameter service unit to register as a message producer of the message middleware; step S2024 controls the first synchronization service unit to register as a message consumer of the message middleware. According to the method, the message middleware is built in the database to be synchronized, so that the coupling degree between the databases can be further reduced compared with the message middleware built between the database to be synchronized and the target database.

Specifically, the message middleware is composed of three components, a message producer and a message consumer. Wherein the message producer is responsible for generating and sending messages into the message middleware, which can be any application, service or device. Message middleware is responsible for receiving and storing messages and ensuring reliable delivery of messages for storing and distributing messages according to specific rules. The message consumer is responsible for receiving and processing messages from the message middleware. The message consumer may be any application, service or device.

In some embodiments, the step S203 may be specifically implemented by the following steps: the target database includes a second synchronization service unit and a second data storage unit, and in step S2031, the parameter service unit is controlled to obtain the data to be synchronized stored in the first data storage unit, and send the data to be synchronized to the first synchronization service unit; step S2032, controlling the first synchronization service unit to forward the message queue to the second synchronization service unit; step S2033, controlling the second synchronization service unit to write the data to be synchronized stored in the message queue into the second data storage unit. After the message middleware is established in the database to be synchronized, the method can enable the target database to further and rapidly receive the message queue in a forwarding mode.

Specifically, a corresponding connection manner is selected according to the type of the message middleware, for example, the MQTT protocol is used to connect to the MQTT message middleware. The first synchronization service unit uses a subscription function to subscribe to message topics or queues that need to be forwarded. When a message arrives, a message is received from the message middleware and sent to the second synchronization service unit, which may be another message middleware, a back-end service, a database, etc. It should be noted that, according to the forwarding requirement, a proper manner may be selected to forward the message.

In some embodiments, after the step S2032, the method further includes: step S2034, determining whether the forwarding of the first synchronization service unit is successful; step S2035, controlling the first synchronization service unit to stop forwarding if the forwarding of the first synchronization service unit is successful; step S2036, when the forwarding of the first synchronization service unit is unsuccessful, outputting a second failure signal and controlling the first synchronization service unit to repeatedly forward the message queue to the second synchronization service unit, where the second failure signal is used to indicate that the forwarding of the first synchronization service unit is unsuccessful. The method can further realize the monitoring of the abnormal operation of the database by outputting the second failure signal.

Specifically, depending on the destination address of the forwarding, the manner of confirming whether the forwarding is successful is also different. For example, if forwarded to another message middleware, it may check whether it was successfully sent to the target topic or queue; if forwarded to the backend service, it may be checked whether the message was successfully received and processed. If forwarding fails, an error log may be recorded and an attempt may be made to resend the message or take other processing action to ensure that the message is not lost.

It should be noted that different message middleware and forwarding targets may have different implementations and interfaces, and the specific implementation method needs to be adjusted according to the message middleware and forwarding targets used. In addition, the requirements on reliability, performance and the like of the message are considered, and proper message middleware and forwarding modes are selected.

In order to enable those skilled in the art to more clearly understand the technical solutions of the present application, the implementation process of the data synchronization method of the present application will be described in detail below with reference to specific embodiments.

The embodiment relates to a specific data synchronization method, as shown in fig. 3, including the following steps:

step S1: the configuration unit of the database to be synchronized configures interception rules according to the library, the table and the operation statement;

Step S2: the database parameter service unit to be synchronized initializes and loads the synchronization rule and loads the rule into the memory;

step S3: the message middleware of the database to be synchronized configures the message into a broadcasting mode, and establishes a message queue of a target database;

step S4: the database parameter service unit to be synchronized is registered as a message producer of the message middleware and establishes a message queue link with the message middleware;

step S5: the first synchronization service unit of the database to be synchronized is registered as a message consumer of the message middleware and establishes message queue connection with the message middleware;

step S6: the database parameter service unit to be synchronized receives service operation data;

step S7: judging whether the current operation accords with the interception rule, if not, directly updating the operation to a parameter library of a database to be synchronized without interception, and ending the flow process; if yes, intercepting and starting the transaction, and continuing the subsequent processing;

step S8: updating the operation to a parameter library of the database to be synchronized;

step S9: judging whether the operation of the database to be synchronized is successful, if yes, submitting the transaction, finishing the processing flow of the database to be synchronized, returning a successful response to the management end, and simultaneously carrying out subsequent data synchronization operation processing; if not, performing transaction rollback, sending abnormal information to monitor and early warning, and simultaneously returning a first failure signal to the management end, and ending the flow process;

Step S10: the intercepted data to be synchronized is sent to a message queue;

step S11: the message middleware of the database to be synchronized broadcasts a notification message consumer (a first synchronization service unit of the database to be synchronized) has a target database queue message to be processed, and the first synchronization service unit of the database to be synchronized forwards the message to a second synchronization service unit of the target database;

step S12: judging whether forwarding is successful or not according to the response of the target database, if forwarding is failed, sending abnormal information to be synchronized to perform data cry monitoring and early warning, and ending the flow process; if the forwarding is successful, the sending processing of the target database is finished, and the target database is used for receiving data subsequently;

step S12: the second synchronous service unit of the target database matches the target database according to the message;

step S13: the target database performs data insertion target database processing, judges whether the writing is successful, and if the writing is successful, the cross-database synchronous whole-flow processing is finished; and if the writing fails, outputting a second failure signal, and ending the flow process.

The embodiment of the application also provides a data synchronization device, and the data synchronization device of the embodiment of the application can be used for executing the data synchronization method provided by the embodiment of the application. The device is used for realizing the above embodiments and preferred embodiments, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

The following describes a data synchronization device provided by an embodiment of the present application.

Fig. 4 is a schematic diagram of a data synchronization device according to an embodiment of the present application. As shown in fig. 4, the apparatus includes:

the first control unit 10 is configured to control the database to be synchronized to receive service operation data, and screen the service operation data according to a preset interception rule to obtain the data to be synchronized, where the service operation data is data for performing service operation on the database, and the service operation at least includes an adding operation, a deleting operation and a modifying operation;

A second control unit 20, configured to control the database to be synchronized to send the data to be synchronized to a message middleware, and store the data to be synchronized in a message queue of the message middleware;

A third control unit 30, configured to control a target database to receive the message queue of the message middleware, so as to write the data to be synchronized stored in the message queue into the target database;

According to the embodiment, a first control unit controls a database to be synchronized to receive service operation data, and screens the service operation data according to preset interception rules to obtain the data to be synchronized, wherein the service operation data is data for performing service operation on the database, and the service operation at least comprises an adding operation, a deleting operation and a modifying operation; the second control unit controls the to-be-synchronized database to send to-be-synchronized data to the message middleware and stores the to-be-synchronized data to a message queue of the message middleware; the third control unit controls the target database to receive the message queue of the message middleware so as to write the data to be synchronized stored in the message queue into the target database. The data to be synchronized is obtained by intercepting the service operation data, the service application performance of the database is not affected differently, the intercepted data to be synchronized is synchronized to the target database through the message middleware asynchronously, the cross-database data synchronization is realized, the self resources of the database are not consumed, and the problem of large resource consumption in the process of data synchronization of the database in the prior art is solved.

As an optional solution, the first control unit is further configured to determine that the service operation data is reject data if the service operation data meets the preset interception rule, reject the reject data from the service operation data, and the remaining service operation data is the data to be synchronized. The device is used for intercepting, the non-invasive processing mode does not consume database resources, the decoupling of the core application module and the synchronous module can be realized, the database system resources and the bandwidth are further saved, and the maintenance and coding cost is reduced.

In order to further avoid abnormal situations, the second control unit of the present application includes a first control module and a second control module, where the first control module is configured to control the database to be synchronized to open a transaction, and store the data to be synchronized to a system context; the second control module is used for controlling the system context to send the data to be synchronized to the message middleware. The device firstly puts the intercepted data to be synchronized into the system context, and then sends the data to be synchronized to the message queue when the transaction is submitted successfully, and further avoids the abnormal scene that the data synchronization action is firstly executed and the transaction rollback cannot be cancelled by utilizing the system context.

The device of the application further comprises a receiving unit, a first output unit and a second output unit, wherein the receiving unit is used for carrying out corresponding operation on the data of the database to be synchronized corresponding to the data to be synchronized according to the service operation corresponding to the data to be synchronized, receiving return information responding to the operation, and determining whether the operation is successful or not according to the return information; the first output unit is used for controlling the database to be synchronized to carry out transaction submission under the condition that the operation of the database to be synchronized is successful, and outputting a success signal, wherein the success signal is used for representing that the operation of the database to be synchronized is successful according to the data to be synchronized; the second output unit is used for controlling the database to be synchronized to perform transaction rollback under the condition that the operation of the database to be synchronized is unsuccessful, and outputting a first failure signal, wherein the first failure signal is used for representing that the operation of the database to be synchronized according to the data to be synchronized is unsuccessful. The device can further ensure the consistency of the database, and can restore to the original state even if errors occur.

The second control unit further comprises a third control module and a fourth control module, wherein the third control module is used for controlling the parameter service unit to register as a message producer of the message middleware; the fourth control module is configured to control the first synchronization service unit to register as a message consumer of the message middleware. By establishing the message middleware in the database to be synchronized, the device can further reduce the coupling degree between the databases compared with the message middleware established between the database to be synchronized and the target database.

In some embodiments, the third control unit includes a fifth control module, a sixth control module, and a seventh control module, where the fifth control module is configured to control the parameter service unit to obtain the data to be synchronized stored in the first data storage unit, and send the data to be synchronized to the first synchronization service unit; the sixth control module is configured to control the first synchronization service unit to forward the message queue to the second synchronization service unit; and a seventh control module controls the second synchronization service unit to write the data to be synchronized stored in the message queue into the second data storage unit. After the message middleware is established in the database to be synchronized, the device can enable the target database to further and rapidly receive the message queue in a forwarding mode.

Optionally, the device further includes a determining unit, a fourth control unit, and a third output unit, where the determining unit is configured to determine whether forwarding of the first synchronization service unit is successful; the fourth control unit is configured to control the first synchronization service unit to stop forwarding when the forwarding of the first synchronization service unit is successful; and the third output unit is used for outputting a second failure signal and controlling the first synchronous service unit to repeatedly forward the message queue to the second synchronous service unit under the condition that the forwarding of the first synchronous service unit is unsuccessful, wherein the second failure signal is used for representing that the forwarding of the first synchronous service unit is unsuccessful. The device can further monitor the abnormal operation of the database by outputting the second failure signal.

The data synchronization device comprises a processor and a memory, wherein the first control unit, the second control unit, the third control unit and the like are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions. The modules are all located in the same processor; alternatively, the above modules may be located in different processors in any combination.

The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. The kernel may be provided with one or more kernel parameters for data synchronization.

The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip.

The embodiment of the invention provides a computer readable storage medium, which comprises a stored program, wherein the program is controlled to control a device where the computer readable storage medium is located to execute the data synchronization method.

Specifically, the data synchronization method includes:

Optionally, the screening the service operation data according to a preset interception rule to obtain data to be synchronized includes: and under the condition that the service operation data meets the preset interception rule, determining that the service operation data is the reject data, rejecting the reject data from the service operation data, and the rest of the service operation data is the data to be synchronized.

Optionally, controlling the database to be synchronized to send the data to be synchronized to a message middleware includes: controlling the database to be synchronized to open the transaction, and storing the data to be synchronized to the system context; and controlling the system context to send the data to be synchronized to the message middleware.

Optionally, after obtaining the data to be synchronized, before controlling the database to be synchronized to send the data to be synchronized to the message middleware, the method further includes: performing corresponding operation on the data of the database to be synchronized corresponding to the data to be synchronized according to the service operation corresponding to the data to be synchronized, receiving return information responding to the operation, and determining whether the operation is successful or not according to the return information; under the condition that the operation of the database to be synchronized is successful, controlling the database to be synchronized to carry out transaction submission, and outputting a success signal, wherein the success signal is used for representing that the operation of the database to be synchronized is successful according to the data to be synchronized; and under the condition that the operation of the database to be synchronized is unsuccessful, controlling the database to be synchronized to perform transaction rollback, and outputting a first failure signal, wherein the first failure signal is used for representing that the operation of the database to be synchronized is unsuccessful according to the data to be synchronized.

Optionally, the database to be synchronized includes a parameter service unit, a first synchronization service unit, and a first data storage unit, and the controlling the database to be synchronized to send the data to be synchronized to the message middleware includes: a message producer controlling the parameter service unit to register as the message middleware; controlling the first synchronization service unit to register as a message consumer of the message middleware.

Optionally, the target database includes a second synchronization service unit and a second data storage unit, and controls the target database to receive the message queue of the message middleware, so as to write the data to be synchronized stored in the message queue into the target database, including: controlling the parameter service unit to acquire the data to be synchronized stored in the first data storage unit, and sending the data to be synchronized to the first synchronization service unit; controlling the first synchronous service unit to forward the message queue to the second synchronous service unit; and controlling the second synchronous service unit to write the data to be synchronized stored in the message queue into the second data storage unit.

Optionally, after controlling the first synchronization service unit to forward the message queue to the second synchronization service unit, the method further includes: determining whether the first synchronous service unit is successfully forwarded; under the condition that the forwarding of the first synchronous service unit is successful, controlling the first synchronous service unit to stop forwarding; and under the condition that the forwarding of the first synchronous service unit is unsuccessful, outputting a second failure signal and controlling the first synchronous service unit to repeatedly forward the message queue to the second synchronous service unit, wherein the second failure signal is used for representing that the forwarding of the first synchronous service unit is unsuccessful.

The embodiment of the invention provides a processor, which is used for running a program, wherein the data synchronization method is executed when the program runs.

Specifically, the data synchronization method includes:

The embodiment of the application provides equipment, which comprises a processor, a memory and a program stored in the memory and capable of running on the processor, wherein the processor realizes at least the following steps when executing the program:

in step S203, the control target database receives the message queue of the message middleware, so as to write the data to be synchronized stored in the message queue into the target database.

The device herein may be a server, PC, PAD, cell phone, etc.

The application also provides a computer program product adapted to perform, when executed on a data processing device, a program initialized with at least the following method steps:

It will be appreciated by those skilled in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than that shown or described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.

From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

1) Firstly, controlling a database to be synchronized to receive service operation data, and screening the service operation data according to a preset interception rule to obtain the data to be synchronized, wherein the service operation data is data for performing service operation on the database, and the service operation at least comprises an adding operation, a deleting operation and a modifying operation; controlling the database to be synchronized to send the data to be synchronized to the message middleware, and storing the data to be synchronized into a message queue of the message middleware; and finally, controlling the target database to receive the message queue of the message middleware so as to write the data to be synchronized stored in the message queue into the target database. The data to be synchronized is obtained by intercepting the service operation data, the service application performance of the database is not affected differently, the intercepted data to be synchronized is synchronized to the target database through the message middleware asynchronously, the cross-database data synchronization is realized, the self resources of the database are not consumed, and the problem of large resource consumption in the process of data synchronization of the database in the prior art is solved.

2) According to the data synchronization device, a first control unit controls a database to be synchronized to receive service operation data, and screens the service operation data according to preset interception rules to obtain the data to be synchronized, wherein the service operation data is data for performing service operation on the database, and the service operation at least comprises an adding operation, a deleting operation and a modifying operation; the second control unit controls the to-be-synchronized database to send to-be-synchronized data to the message middleware and stores the to-be-synchronized data to a message queue of the message middleware; the third control unit controls the target database to receive the message queue of the message middleware so as to write the data to be synchronized stored in the message queue into the target database. The data to be synchronized is obtained by intercepting the service operation data, the service application performance of the database is not affected differently, the intercepted data to be synchronized is synchronized to the target database through the message middleware asynchronously, the cross-database data synchronization is realized, the self resources of the database are not consumed, and the problem of large resource consumption in the process of data synchronization of the database in the prior art is solved.

The above description is only of the preferred embodiments of the present application and is not intended to limit the present application, but various modifications and variations can be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of data synchronization, the method comprising:

controlling a database to be synchronized to receive service operation data, and screening the service operation data according to a preset interception rule to obtain the data to be synchronized, wherein the service operation data is data for performing service operation on the database, and the service operation at least comprises an adding operation, a deleting operation and a modifying operation;

controlling the database to be synchronized to send the data to be synchronized to a message middleware, and storing the data to be synchronized into a message queue of the message middleware;

and controlling a target database to receive the message queue of the message middleware so as to write the data to be synchronized stored in the message queue into the target database.

2. The method of claim 1, wherein the screening the business operation data according to a preset interception rule to obtain data to be synchronized comprises:

and under the condition that the business operation data meets the preset interception rule, determining that the business operation data is the reject data, rejecting the reject data from the business operation data, and the rest of the business operation data is the data to be synchronized.

3. The method of claim 1, wherein controlling the database to be synchronized to send the data to be synchronized to message middleware comprises:

controlling the database to be synchronized to start transactions, and storing the data to be synchronized to a system context;

and controlling the system context to send the data to be synchronized to the message middleware.

4. The method of claim 1, wherein after obtaining the data to be synchronized, before controlling the database to be synchronized to send the data to be synchronized to a message middleware, the method further comprises:

performing corresponding operation on the data of the database to be synchronized corresponding to the data to be synchronized according to the service operation corresponding to the data to be synchronized, receiving return information responding to the operation, and determining whether the operation is successful or not according to the return information;

under the condition that the operation of the database to be synchronized is successful, controlling the database to be synchronized to carry out transaction submission, and outputting a success signal, wherein the success signal is used for representing that the operation of the database to be synchronized is successful according to the data to be synchronized;

and under the condition that the operation of the database to be synchronized is unsuccessful, controlling the database to be synchronized to perform transaction rollback, and outputting a first failure signal, wherein the first failure signal is used for representing that the operation of the database to be synchronized according to the data to be synchronized is unsuccessful.

5. The method of claim 1, wherein the database to be synchronized includes a parameter service unit, a first synchronization service unit, and a first data storage unit, and wherein controlling the database to be synchronized to send the data to be synchronized to a message middleware includes:

controlling the parameter service unit to register as a message producer of the message middleware;

controlling the first synchronization service unit to register as a message consumer of the message middleware.

6. The method of claim 5, wherein the target database comprises a second synchronization service unit and a second data storage unit, wherein the target database is controlled to receive the message queue of the message middleware,

writing the data to be synchronized stored in the message queue to the target database, including:

controlling the parameter service unit to acquire the data to be synchronized stored in the first data storage unit and sending the data to be synchronized to the first synchronization service unit;

controlling the first synchronous service unit to forward the message queue to the second synchronous service unit;

and controlling the second synchronous service unit to write the data to be synchronized stored in the message queue into the second data storage unit.

7. The method of claim 6, wherein after controlling the first synchronization service unit to forward the message queue to the second synchronization service unit, the method further comprises:

determining whether the first synchronous service unit is successfully forwarded;

under the condition that the first synchronous service unit is successful in forwarding, the first synchronous service unit is controlled to stop forwarding;

and under the condition that the forwarding of the first synchronous service unit is unsuccessful, outputting a second failure signal and controlling the first synchronous service unit to repeatedly forward the message queue to the second synchronous service unit, wherein the second failure signal is used for representing that the forwarding of the first synchronous service unit is unsuccessful.

8. A data synchronization device, comprising:

the first control unit is used for controlling the database to be synchronized to receive service operation data, screening the service operation data according to a preset interception rule to obtain the data to be synchronized, wherein the service operation data is data for performing service operation on the database, and the service operation at least comprises an adding operation, a deleting operation and a modifying operation;

The second control unit is used for controlling the database to be synchronized to send the data to be synchronized to the message middleware and storing the data to be synchronized into a message queue of the message middleware;

and the third control unit is used for controlling a target database to receive the message queue of the message middleware so as to write the data to be synchronized stored in the message queue into the target database.

9. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored program, wherein the program, when run, controls a device in which the computer readable storage medium is located to perform the method of any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method according to any of claims 1 to 7 by means of the computer program.