CN112732728A - Data synchronization method and system - Google Patents
Data synchronization method and system Download PDFInfo
- Publication number
- CN112732728A CN112732728A CN201911030607.3A CN201911030607A CN112732728A CN 112732728 A CN112732728 A CN 112732728A CN 201911030607 A CN201911030607 A CN 201911030607A CN 112732728 A CN112732728 A CN 112732728A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- operated
- message
- new value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention discloses a data synchronization method and a data synchronization system, and relates to the technical field of computers. One embodiment of the method comprises: receiving a data operation request, wherein the data operation request indicates data to be operated and a new value of the data to be operated; according to the data operation request, operating the data to be operated in a first database, and generating a log message of the first database after the data to be operated is operated in the first database, wherein the log message indicates the data to be operated in the first database and a new value of the data to be operated; and acquiring the log message of the first database, and indicating a second database to operate the data to be operated according to the log message of the first database. The implementation method has the advantages that the synchronous operation of the database is decoupled, and the consistency of the data stored in the database is ensured.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data synchronization method and system.
Background
With the continuous development of services, the data volume is also increasing, and databases are often required to be sorted and tabulated to reduce the data volume of a single database, thereby improving the performance of a service system. Meanwhile, in order to ensure the reliability of data and the convenience of cross-database data processing, data in a database is usually required to be redundant or backed up, and the consistency of redundant data is ensured.
The existing method for ensuring consistency of redundant data is to synchronously perform data operation in the redundant database under the condition that data is added, deleted, updated and the like in the database according to a data operation request of a user until the data in the database is consistent with the data in the redundant database.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art: data operation in the database and the redundant database needs to be completed synchronously, so that the resource occupation is large and the time is long; when the data in the database is down or restarted, the data in the redundant database cannot be synchronously operated and processed, and the data are inconsistent.
Disclosure of Invention
In view of this, the present invention provides a data synchronization method and system, which can decouple the synchronization operations of different database data, have a small resource occupation ratio and a short time consumption, and avoid inconsistency of redundant data caused by downtime or restart.
To achieve the above object, according to an aspect of the present invention, there is provided a data synchronization method including:
receiving a data operation request, wherein the data operation request indicates data to be operated and a new value of the data to be operated;
according to the data operation request, operating the data to be operated in a first database, and generating a log message of the first database after the data to be operated is operated in the first database, wherein the log message indicates the data to be operated in the first database and a new value of the data to be operated;
and acquiring the log message of the first database, and indicating a second database to operate the data to be operated according to the log message of the first database.
Optionally, the instructing, according to the log message of the first database, the second database to operate the data to be operated further includes:
generating a first data operation message according to the acquired log message of the first database and sending the first data operation message to a message queue, wherein the first data operation message indicates the data to be operated and a new value of the data to be operated;
according to the first data operation message, indicating the second database to operate the data to be operated;
and after the data to be operated is operated in the second database, generating a second data operation message and sending the second data operation message to the message queue, wherein the second data operation message indicates the data to be operated and a new value of the data to be operated.
Optionally, the method further comprises: correspondingly storing the data to be operated of the first database and the new value of the data to be operated, storing the synchronous state of the data to be operated, and recording the synchronous state as unsynchronized;
correspondingly storing the data to be operated and the new value of the data to be operated of the second database, and updating the synchronous state of the data to be operated to be synchronous.
Optionally, the method further comprises: when the synchronous state of the data to be operated is recorded as unsynchronized, starting a first timer to accumulate the duration that the synchronous state is unsynchronized:
starting a second timer under the condition that the synchronization state is still not synchronized when the duration time is accumulated to the threshold time corresponding to the first timer;
under the condition that the synchronous state is recorded as being synchronous within the threshold time corresponding to the first timer, judging whether the stored new value of the data to be operated of the first database is consistent with the stored new value of the data to be operated of the second database, and starting the second timer if the new values are not consistent;
and indicating the second database to try to operate on the data to be operated within the threshold time corresponding to the second timer.
Optionally, the method further comprises: and when the synchronization state is still unsynchronized when the duration is accumulated to the threshold time corresponding to the first timer, or when the stored new value of the data to be operated of the first database is inconsistent with the stored new value of the data to be operated of the second database, sending an exception warning to prompt that the synchronization of the first database and the second database is abnormal.
Optionally, according to the obtained log message of the first database, the first data operation message conforming to the message format of the second database is generated, so as to instruct the second database to operate on the data to be operated according to the first data operation message.
Optionally, a non-relational database or a relational database is used to store the data to be operated, the new value of the data to be operated, and the synchronization state of the data to be operated.
To achieve the above object, according to another aspect of the present invention, there is provided a data synchronization system including: the system comprises a first database, a second database and a data synchronization device; wherein the content of the first and second substances,
the first database is used for receiving a data operation request, wherein the data operation request indicates data to be operated and a new value of the data to be operated; according to the data operation request, operating the data to be operated in a first database, and generating a log message of the first database after the data to be operated is operated in the first database, wherein the log message indicates the data to be operated in the first database and a new value of the data to be operated;
the data synchronization device is used for acquiring the log message of the first database and indicating the second database to operate the data to be operated according to the log message of the first database;
and the second database is used for operating the data to be operated according to the indication of the data synchronization device.
Optionally, the method further comprises: a message queue; the data synchronization device is also used for,
generating a first data operation message according to the acquired log message of the first database and sending the first data operation message to the message queue, wherein the first data operation message indicates the data to be operated and a new value of the data to be operated;
according to the first data operation message, indicating the second database to operate the data to be operated;
and after the data to be operated is operated in the second database, generating a second data operation message and sending the second data operation message to the message queue, wherein the second data operation message indicates the data to be operated and a new value of the data to be operated.
Optionally, the method further comprises: a data storage device; the data storage device is configured to, in response to the request,
correspondingly storing the data to be operated and the new value of the data to be operated, storing the synchronous state of the data to be operated, and recording the synchronous state as unsynchronized;
correspondingly storing the data to be operated and the new value of the data to be operated, and updating the synchronous state to be synchronous.
Optionally, the data synchronization apparatus is further configured to, when the synchronization status of the data to be operated is recorded as unsynchronized, start a first timer, and accumulate the duration that the synchronization status is unsynchronized:
starting a second timer under the condition that the synchronization state is still not synchronized when the duration time is accumulated to the threshold time corresponding to the first timer;
under the condition that the synchronous state is recorded as being synchronous within the threshold time corresponding to the first timer, judging whether the stored new value of the data to be operated of the first database is consistent with the stored new value of the data to be operated of the second database, and starting the second timer if the new values are not consistent;
and indicating the second database to try to operate on the data to be operated within the threshold time corresponding to the second timer.
Optionally, the method further comprises: an abnormality warning device; the abnormality warning device for
And when the synchronization state is still unsynchronized when the duration accumulated value is the threshold time corresponding to the first timer, or when the stored new value of the data to be operated of the first database is inconsistent with the stored new value of the data to be operated of the second database, sending an exception warning to prompt that the synchronization of the first database and the second database is abnormal.
Optionally, the data synchronization apparatus is further configured to generate, according to the acquired log message of the first database, the first data operation message conforming to the second database message format, so as to instruct, according to the first data operation message, the second database to perform an operation on the data to be operated.
Optionally, the data storage device uses a non-relational database or a relational database to store the data to be operated, the new value of the data to be operated, and the synchronization status of the data to be operated.
To achieve the above object, according to still another aspect of the present invention, there is provided a server for implementing data synchronization, including: one or more processors; storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement any of the data synchronization methods described above.
To achieve the above object, according to still another aspect of the present invention, there is provided a computer readable medium having stored thereon a computer program which, when executed by a processor, implements any one of the data synchronization methods described above.
The invention has the following advantages or beneficial effects: by acquiring the log message of the first database and indicating the second database to operate the data to be operated according to the log message, the data consistency of the first database and the second database is realized, the synchronous operation of the first database and the second database is decoupled, and the feedback time of a user waiting for an operation request is shortened. In addition, by continuously updating and judging the synchronous state of the data to be operated and continuously trying to operate the data in the second database under the condition that different states are not synchronous or the data in the first database and the second database are inconsistent, the inconsistency of the data in the first database and the second database caused by restarting, downtime and the like is avoided.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of a main flow of a data synchronization method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a main flow of a data consistency verification method according to an embodiment of the invention;
FIG. 3 is a schematic diagram of the main modules of a data synchronization apparatus according to an embodiment of the present invention;
FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 5 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic main flow diagram of a data synchronization method according to an embodiment of the present invention, and as shown in fig. 1, the method may specifically include the following steps:
step S101, receiving a data operation request, wherein the data operation request indicates data to be operated and a new value of the data to be operated.
And according to different operation types, the new values of the data to be operated stored in the operated database are different. Specifically, the operation types include an update (update) operation, an insert (insert) operation, a delete (delete) operation, and the like. For the updating operation, the new value of the data to be operated is the value of the actually updated data to be operated; for the insertion operation, before the insertion, the value of the data to be operated is null (null), and after the insertion, the new value of the data to be operated is the inserted value; for a delete operation, the new value of the data to be operated on after the deletion is null (null).
Step S102, according to the data operation request, operating the data to be operated in a first database, and after the data to be operated is operated in the first database, generating a log message of the first database, wherein the log message indicates the data to be operated in the first database and the new of the data to be operated.
When the first database, such as the MYSQL database, performs operations such as insertion, deletion, and update on the data to be operated stored therein, a corresponding operation log, that is, a log message of the first database, may be generated, and then new values of the data to be operated and the data to be operated may be obtained by analyzing the log message of the first database, so as to instruct other second databases, which need to synchronize, to perform corresponding operations on the data to be operated according to the obtained new values of the data to be operated and the data to be operated.
Step S103, acquiring the log message of the first database, and indicating a second database to operate the data to be operated according to the log message of the first database.
It is understood that the first database and the second database may be the same type of database or different types of databases, and the number may be one or more. If the first database is Sybase, dBASE, Clipper, FoxPro, CouchDB and the like, and the second database is MySQL, maridb and the like, because the data operation commands or message formats which can be identified by different types of databases are different, in order to indicate the second database to perform corresponding data operation according to the log message of the first database, data synchronization middleware such as canal and the like is needed to perform format conversion on the log message of the first database, and the log message of the first database is converted into a message format which can indicate the second database to perform data operation. Because the second database synchronously operates the data according to the log message of the first database, the consistency of the data stored in the first database and the second database is ensured, and the problems of large resource occupation ratio and long reaction time required by the synchronous operation of the first database and the second database are avoided, especially under the condition of large quantity of data operation requests. In addition, in order to improve the experience of the data operation request sender and shorten the time for feeding back the data operation result to the sender according to the data operation request, the data processing result can be fed back to the data operation request sender when the first database completes the operation on the data to be operated.
In an optional implementation manner, the instructing, according to the log message of the first database, a second database to perform an operation on the data to be operated further includes: generating a first data operation message according to the acquired log message of the first database and sending the first data operation message to a message queue, wherein the first data operation message indicates the data to be operated and a new value of the data to be operated; according to the first data operation message, indicating the second database to operate the data to be operated; and after the data to be operated is operated in the second database, generating a second data operation message and sending the second data operation message to the message queue, wherein the second data operation message indicates the data to be operated and a new value of the data to be operated.
In the process of data synchronization, in order to ensure the consistency of the data stored in the first database and the second database, the data synchronization state in the first database and the second database needs to be monitored so as to process the data in time under the condition of data asynchronization. Specifically, because the first database generates a corresponding log message when the first database completes the operation on the data to be operated, the corresponding first data operation message may be generated to the message queue when the log message of the first database is acquired and the first database is judged to have completed the operation on the data to be operated, so that a consumer in the message queue may judge the synchronization state of the data to be operated of the first database according to the message related to the first database monitored from the message queue. Similarly, the second database also generates a corresponding log message when the operation on the data to be operated is completed, so that a corresponding second data operation message can be generated and pushed to the message queue according to the log message of the second database, so that a consumer of the message queue can judge the synchronization state of the data to be operated of the second database according to the message related to the second database monitored from the message queue, namely, synchronously update the data synchronization state of the first database.
It is understood that message queues that may be used include, but are not limited to: ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocktMQ. In actual implementation, in order to identify a first data operation message from a first database and a second data operation message from a second database, the first data operation message and the second data operation message may be stored in a topic (topic) manner. As illustrated by the example of a Kafka message queue, a first data operation message may be pushed into topic1, while a second data operation message may be pushed into topic 2.
Furthermore, the consumer of the message queue may determine that the first database has completed the operation on the data to be operated according to the first data operation message monitored from the message queue and related to the first database, correspondingly store the data to be operated of the first database and the new value of the data to be operated in other storage areas, store the synchronization state of the data to be operated, and record the synchronization state as unsynchronized; and judging that the second database completes the synchronous processing of the data to be operated according to the second data operation information which is monitored from the information queue and is related to the second database, correspondingly storing the data to be operated of the second database and the new value of the data to be operated in other storage areas, and updating the synchronous state of the data to be operated to be synchronous. On the basis, whether the new value of the data to be operated stored in the first database is consistent with the new value of the data to be operated stored in the second database can be further verified, if not, the second database can be synchronously updated according to the correspondingly stored new values of the data to be operated and the data to be operated, so that the consistency of the data stored in the first database and the second database is ensured.
It is noted that in other storage areas, a non-relational database or a relational database is used to store the data to be operated, the new value of the data to be operated, and the synchronization state of the data to be operated. As described by taking the storage in the form of (key, value) as an example, when the storage is performed according to the first data operation message, key is data to be operated, value is a new value of the data to be operated, correspondingly, another key is in a synchronous state, and value is a value (such as 0, false, etc.) indicating that the data is not synchronized; when the data is stored according to the second data operation message, the key is the redundant data to be operated, and the value is the new value of the data to be operated.
In an optional embodiment, in a case that the synchronization status of the data to be operated is recorded as unsynchronized, a first timer is started to accumulate the duration that the synchronization status is unsynchronized: starting a second timer under the condition that the synchronization state is still not synchronized when the duration time is accumulated to the threshold time corresponding to the first timer; under the condition that the synchronous state is recorded as being synchronous within the threshold time corresponding to the first timer, judging whether the stored new value of the data to be operated of the first database is consistent with the stored new value of the data to be operated of the second database, and starting the second timer if the new values are not consistent; and indicating the second database to try to operate on the data to be operated within the threshold time corresponding to the second timer.
The threshold time corresponding to the first timer and the threshold time of the second timer may be any time set according to actual requirements, such as 3 seconds, 5 seconds, and the like. If the synchronization state is still not synchronized within the time period from the accumulation of the duration time to the threshold value corresponding to the first timer, it indicates that the second database still completes the operation of the data to be operated, and the data operation of the second database is abnormal and cannot be synchronized in time, the second timer needs to be started, that is, within the threshold time of the second timer, the second database is instructed to perform the data operation again by obtaining the new values of the data to be operated and the data to be operated, which are stored correspondingly according to the first data processing message, and the attempt is continuously performed within the threshold time of the second timer until the data operation is successful, otherwise, an abnormal warning can be sent through a mail or the like. And if the synchronization state is changed to be synchronized within the threshold time corresponding to the first timer, indicating that the second database completes the processing of the data to be operated, further verifying the consistency of the data stored in the first database and the second database, and if the data stored in the first database and the data stored in the second database are not consistent, starting the second timer, namely, within the threshold time of the second timer, indicating the second database to perform data operation again by acquiring the data to be operated and new values of the data to be operated, which are correspondingly stored according to the first data processing message, until the data operation is successful. Therefore, when the first database has the problems of restarting, downtime and the like, the consistency of the first database and the second database can still be ensured.
In an optional embodiment, when the synchronization status is still unsynchronized when the duration time is accumulated to the threshold time corresponding to the first timer, or when the stored new value of the data to be operated for the first database is inconsistent with the stored new value of the data to be operated for the second database, an exception warning is sent to prompt that there is an exception in synchronization between the first database and the second database.
Based on the data synchronization method provided by the embodiment, the log message of the first database is acquired, and the second database is instructed to operate the data to be operated according to the log message, so that the data consistency of the first database and the second database is realized, the synchronous operation of the first database and the second database is decoupled, and the feedback time of a user waiting for the operation request is shortened. In addition, by continuously updating and judging the synchronous state of the data to be operated and continuously trying to operate the data in the second database under the condition that different states are not synchronous or the data in the first database and the second database are inconsistent, the inconsistency of the data in the first database and the second database caused by restarting, downtime and the like is avoided.
Referring to fig. 2, on the basis of the above embodiment, the embodiment of the present invention provides a data consistency verification method, which specifically includes the following steps:
step S201, when the synchronization status of the data to be operated is recorded as unsynchronized, starting a first timer to accumulate the duration time of the unsynchronized synchronization status.
Step S202, judging whether the synchronizing state is that the unsynchronized duration time reaches the threshold time corresponding to the first timer; if the duration time is not less than the threshold time, it indicates that the synchronization status has been changed to synchronized, i.e. the second database has completed the operation on the data to be operated, then the following step S203 is executed; if the duration time reaches the threshold time, it indicates that the synchronization status is still unsynchronized, i.e., the second database does not complete the operation on the data to be operated in time, the following step S204 is continuously executed.
Step S203, judging whether the stored new value of the data to be operated of the first database is consistent with the stored new value of the data to be operated of the second database; if they match, the operation is terminated, and if they do not match, the following step S204 is continued.
Step S204, starting the second timer, and indicating the second database to attempt to operate on the data to be operated within a threshold time corresponding to the second timer. That is, within the threshold time of the second timer, the second database is instructed to perform the data operation again by obtaining the data to be operated and the new value of the data to be operated, which are correspondingly stored according to the first data processing message, and the attempt is continuously performed within the threshold time of the second timer until the data operation is successful.
Step S205, sending an exception warning to prompt that there is an exception in the synchronization between the first database and the second database. That is, when the second database fails to synchronize the operation of the first database on the data in time, or when the new value of the data to be operated stored in the first database is inconsistent with the new value of the data to be operated stored in the second database, an abnormality warning is sent by mail, voice, or the like, so as to prompt relevant personnel to investigate the reason of the database synchronization abnormality.
Referring to fig. 3, on the basis of the above embodiment, an embodiment of the present invention provides a database synchronization system 300, including: a first database 301, a second database 302, a data synchronizer 303; the first database 301 is configured to receive a data operation request, where the data operation request indicates data to be operated and a new value of the data to be operated; according to the data operation request, operating the data to be operated in a first database, and generating a log message of the first database after the data to be operated is operated in the first database, wherein the log message indicates the data to be operated in the first database and a new value of the data to be operated; the data synchronization device 303 is configured to obtain a log message of the first database, and instruct the second database to operate on the data to be operated according to the log message of the first database; the second database 302 is configured to operate on the data to be operated according to the instruction of the data synchronization apparatus.
In an optional embodiment, the method further comprises: a message queue 304; the data synchronization device 303 is further configured to generate a first data operation message according to the obtained log message of the first database, and send the first data operation message to the message queue, where the first data operation message indicates the data to be operated and a new value of the data to be operated; according to the first data operation message, indicating the second database to operate the data to be operated; and after the data to be operated is operated in the second database, generating a second data operation message and sending the second data operation message to the message queue, wherein the second data operation message indicates the data to be operated and a new value of the data to be operated.
In an optional embodiment, the method further comprises: data storage device 305; the data storage device 305 is configured to correspondingly store the data to be operated and the new value of the data to be operated, store the synchronization state of the data to be operated, and record the synchronization state as unsynchronized; correspondingly storing the data to be operated and the new value of the data to be operated, and updating the synchronous state to be synchronous.
In an optional implementation manner, the data synchronization device 303 is further configured to, in a case that the synchronization status of the data to be operated is recorded as unsynchronized, start a first timer, and accumulate a duration that the synchronization status is unsynchronized: starting a second timer under the condition that the synchronization state is still not synchronized when the duration time is accumulated to the threshold time corresponding to the first timer; under the condition that the synchronous state is recorded as being synchronous within the threshold time corresponding to the first timer, judging whether the stored new value of the data to be operated of the first database is consistent with the stored new value of the data to be operated of the second database, and starting the second timer if the new values are not consistent; and indicating the second database to try to operate on the data to be operated within the threshold time corresponding to the second timer.
In an optional embodiment, the method further comprises: an abnormality warning device 306; the abnormality warning device 306 is configured to send an abnormality warning to prompt that there is an abnormality in synchronization between the first database and the second database if the synchronization state is still unsynchronized when the duration cumulative value is the threshold time corresponding to the first timer, or if the stored new value of the data to be operated for the first database is inconsistent with the stored new value of the data to be operated for the second database.
In an optional implementation manner, the data synchronization device 303 is further configured to generate, according to the obtained log message of the first database, the first data operation message conforming to the second database message format, so as to instruct, according to the first data operation message, the second database to operate on the data to be operated.
In an alternative embodiment, the data storage device 305 uses a non-relational database or a relational database to store the data to be operated, the new value of the data to be operated, and the synchronization status of the data to be operated.
Fig. 4 illustrates an exemplary system architecture 400 to which the data synchronization method or data synchronization system of embodiments of the present invention may be applied.
As shown in fig. 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405. The network 404 serves as a medium for providing communication links between the terminal devices 401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use terminal devices 401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. The terminal devices 401, 402, 403 may have various communication client applications installed thereon, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, and the like.
The terminal devices 401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 405 may be a server that provides various services, such as a background management server that supports shopping websites browsed by users using the terminal devices 401, 402, and 403. The background management server can analyze and process the received data such as the product information inquiry request and feed back the processing result (the synchronization state of the data to be operated is synchronized) to the terminal equipment.
It should be noted that the data synchronization method provided by the embodiment of the present invention is generally executed by the server 405.
It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 5, shown is a block diagram of a computer system 500 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 501.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: receiving a data operation request, wherein the data operation request indicates data to be operated and a new value of the data to be operated; according to the data operation request, operating the data to be operated in a first database, and generating a log message of the first database after the data to be operated is operated in the first database, wherein the log message indicates the data to be operated in the first database and a new value of the data to be operated; and acquiring the log message of the first database, and indicating a second database to operate the data to be operated according to the log message of the first database.
According to the technical scheme of the embodiment of the invention, the log message of the first database is obtained, and the second database is indicated to operate the data to be operated according to the log message, so that the data consistency of the first database and the second database is realized, the synchronous operation of the first database and the second database is decoupled, and the feedback time of a user waiting for the operation request is shortened. In addition, by continuously updating and judging the synchronous state of the data to be operated and continuously trying to operate the data in the second database under the condition that different states are not synchronous or the data in the first database and the second database are inconsistent, the inconsistency of the data in the first database and the second database caused by restarting, downtime and the like is avoided.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of data synchronization, comprising:
receiving a data operation request, wherein the data operation request indicates data to be operated and a new value of the data to be operated;
according to the data operation request, operating the data to be operated in a first database, and generating a log message of the first database after the data to be operated is operated in the first database, wherein the log message indicates the data to be operated in the first database and a new value of the data to be operated;
and acquiring the log message of the first database, and indicating a second database to operate the data to be operated according to the log message of the first database.
2. The data synchronization method according to claim 1, wherein the instructing, according to the log message of the first database, the second database to operate on the data to be operated further comprises:
generating a first data operation message according to the acquired log message of the first database and sending the first data operation message to a message queue, wherein the first data operation message indicates the data to be operated and a new value of the data to be operated;
according to the first data operation message, indicating the second database to operate the data to be operated;
and after the data to be operated is operated in the second database, generating a second data operation message and sending the second data operation message to the message queue, wherein the second data operation message indicates the data to be operated and a new value of the data to be operated.
3. The data synchronization method according to claim 1 or 2, further comprising:
correspondingly storing the data to be operated of the first database and the new value of the data to be operated, storing the synchronous state of the data to be operated, and recording the synchronous state as unsynchronized;
correspondingly storing the data to be operated and the new value of the data to be operated of the second database, and updating the synchronous state of the data to be operated to be synchronous.
4. The data synchronization method of claim 3, further comprising:
when the synchronous state of the data to be operated is recorded as unsynchronized, starting a first timer to accumulate the duration that the synchronous state is unsynchronized:
starting a second timer under the condition that the synchronization state is still not synchronized when the duration time is accumulated to the threshold time corresponding to the first timer;
under the condition that the synchronous state is recorded as being synchronous within the threshold time corresponding to the first timer, judging whether the stored new value of the data to be operated of the first database is consistent with the stored new value of the data to be operated of the second database, and starting the second timer if the new values are not consistent;
and indicating the second database to try to operate on the data to be operated within the threshold time corresponding to the second timer.
5. The data synchronization method of claim 4, further comprising:
and when the synchronization state is still unsynchronized when the duration is accumulated to the threshold time corresponding to the first timer, or when the stored new value of the data to be operated of the first database is inconsistent with the stored new value of the data to be operated of the second database, sending an exception warning to prompt that the synchronization of the first database and the second database is abnormal.
6. The data synchronization method of claim 2,
and generating the first data operation message which conforms to the message format of the second database according to the acquired log message of the first database, and indicating the second database to operate the data to be operated according to the first data operation message.
7. The data synchronization method of claim 3,
and storing the data to be operated, the new value of the data to be operated and the synchronization state of the data to be operated by using a non-relational database or a relational database.
8. A data synchronization system, comprising: the system comprises a first database, a second database and a data synchronization device; wherein the content of the first and second substances,
the first database is used for receiving a data operation request, wherein the data operation request indicates data to be operated and a new value of the data to be operated; according to the data operation request, operating the data to be operated in a first database, and generating a log message of the first database after the data to be operated is operated in the first database, wherein the log message indicates the data to be operated in the first database and a new value of the data to be operated;
the data synchronization device is used for acquiring the log message of the first database and indicating the second database to operate the data to be operated according to the log message of the first database;
and the second database is used for operating the data to be operated according to the indication of the data synchronization device.
9. A server for implementing data synchronization, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.
10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911030607.3A CN112732728A (en) | 2019-10-28 | 2019-10-28 | Data synchronization method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911030607.3A CN112732728A (en) | 2019-10-28 | 2019-10-28 | Data synchronization method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112732728A true CN112732728A (en) | 2021-04-30 |
Family
ID=75588937
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911030607.3A Pending CN112732728A (en) | 2019-10-28 | 2019-10-28 | Data synchronization method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112732728A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113177093A (en) * | 2021-05-24 | 2021-07-27 | 商汤国际私人有限公司 | Data synchronization method and device, electronic equipment and storage medium |
-
2019
- 2019-10-28 CN CN201911030607.3A patent/CN112732728A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113177093A (en) * | 2021-05-24 | 2021-07-27 | 商汤国际私人有限公司 | Data synchronization method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110262807B (en) | Cluster creation progress log acquisition system, method and device | |
| CN111427701A (en) | Workflow engine system and business processing method | |
| CN110572422A (en) | Data downloading method and device | |
| CN111338834B (en) | Data storage method and device | |
| CN112118352B (en) | Method and device for processing notification trigger message, electronic equipment and computer readable medium | |
| CN112732728A (en) | Data synchronization method and system | |
| CN113535420A (en) | Message processing method and device | |
| CN111767176A (en) | Method and device for remotely controlling terminal equipment | |
| CN115629909A (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN114116675A (en) | Data archiving method and device | |
| CN113742376A (en) | Data synchronization method, first server and data synchronization system | |
| CN114064803A (en) | Data synchronization method and device | |
| CN112749204A (en) | Method and device for reading data | |
| CN109087097B (en) | Method and device for updating same identifier of chain code | |
| CN110048991B (en) | Method and device for realizing signal interaction reliability | |
| CN113722193A (en) | Method and device for detecting page abnormity | |
| CN112783665B (en) | Interface compensation method and device | |
| CN117478535B (en) | Log storage method and device | |
| CN113094211B (en) | Backup data processing method and device | |
| CN110750410B (en) | Method and device for monitoring database logs | |
| CN112181937B (en) | Method and device for transferring data | |
| CN114780593A (en) | Processing method and device for timing task | |
| CN115167769A (en) | Method, device and system for writing data | |
| CN113934548A (en) | Method and device for establishing connection with dependent service | |
| CN113807054A (en) | Method, device, equipment and computer readable medium for processing data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |