CN112612850A

CN112612850A - Data synchronization method and device

Info

Publication number: CN112612850A
Application number: CN202011509378.6A
Authority: CN
Inventors: 韩时通
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Cloud Computing Beijing Co Ltd
Priority date: 2020-12-18
Filing date: 2020-12-18
Publication date: 2021-04-06

Abstract

The embodiment of the application provides a data synchronization method and device. The data synchronization method comprises the following steps: reading a current increment file of a first database cluster through a first thread, wherein the current increment file is used for recording a data writing command executed by the first database cluster; analyzing the current incremental file to acquire a data writing command recorded by the current incremental file; sending a message containing the data writing command to a specified theme of message middleware so that a second thread subscribing to the specified theme acquires the message from the specified theme; and sending the message to a second database cluster through the second thread so that the second database cluster executes and writes a data write command contained in the message into an incremental file of the second database cluster. According to the technical scheme of the embodiment of the application, the data can be rapidly synchronized among the database clusters.

Description

Data synchronization method and device

Technical Field

The present application relates to the field of computer and communication technologies, and in particular, to a data synchronization method and apparatus.

Background

With the rapid development of informatization, the reliability of a single machine room can not be guaranteed, the phenomenon that an optical cable of the machine room is broken by digging and the phenomenon that the machine room is powered off frequently occur, application services and data are required to be deployed into a plurality of machine rooms such as two machine rooms in the same city or three machine rooms in the same city and three machine rooms in different places, and reliable services can still be provided to the outside after a certain machine room is prevented from being broken.

The synchronization of databases across computer rooms, such as a relational database management system mysql and the like, can be well realized through the transmission of binary log binlog, but the synchronization of Key Value pairs of the computer rooms to a Key-Value database redis still depends on manual operation, and the defects of low synchronization efficiency, long time consumption and the like exist.

Disclosure of Invention

Embodiments of the present application provide a data synchronization method and apparatus, so that fast synchronization of data among database clusters can be achieved at least to a certain extent.

Other features and advantages of the present application will be apparent from the following detailed description, or may be learned by practice of the application.

According to an aspect of an embodiment of the present application, there is provided a data synchronization method, including: reading a current increment file of a first database cluster through a first thread, wherein the current increment file is used for recording a data writing command executed by the first database cluster; analyzing the current incremental file to acquire a data writing command recorded by the current incremental file; sending a message containing the data writing command to a specified theme of message middleware so that a second thread subscribing to the specified theme acquires the message from the specified theme; and sending the message to a second database cluster through the second thread, so that the second database cluster executes a data write command contained in the message, and writes the executed data write command into an incremental file of the second database cluster.

According to an aspect of an embodiment of the present application, there is provided a data synchronization apparatus, including: the reading unit is configured to read a current increment file of a first database cluster through a first thread, wherein the current increment file is used for recording a data writing command executed by the first database cluster; the first analysis unit is configured to analyze the current incremental file to acquire a data write-in command recorded by the current incremental file; the first sending unit is configured to send a message containing the data writing command to a specified theme of message middleware so that a second thread subscribing to the specified theme acquires the message from the specified theme; and the second sending unit is configured to send the message to a second database cluster through the second thread, so that the second database cluster executes a data write command contained in the message, and writes the executed data write command into an incremental file of the second database cluster.

In some embodiments of the present application, based on the foregoing solution, the apparatus further includes: a checking unit configured to check whether the first database cluster performs a rewrite operation on the current delta file; a first notification unit configured to notify the first thread to stop reading the current delta file if it is determined that the first database cluster performs a rewrite operation on the current delta file.

In some embodiments of the present application, based on the foregoing solution, the apparatus further includes: the obtaining unit is configured to obtain the size of the incremental file after the rewriting operation is executed last time by the first database cluster; and the determining unit is configured to determine that the first database cluster performs the rewrite operation on the current incremental file if the difference between the size of the incremental file after the previous rewrite operation and the size of the current incremental file exceeds a first specified threshold and the size of the current incremental file reaches a second specified threshold.

In some embodiments of the present application, based on the foregoing solution, the apparatus further includes: the second notification unit is configured to notify the first thread to start reading a new incremental file in the first database cluster after the execution of the rewrite operation is completed, wherein the new incremental file is an incremental file used for replacing the current incremental file after the execution of the rewrite operation is completed; and the second analysis unit is configured to analyze the new incremental file to acquire a data writing command recorded by the new incremental file.

In some embodiments of the present application, based on the foregoing solution, the apparatus further includes: a first deleting unit, configured to delete a message including the data writing command if the message exists in a specified topic of the message middleware after the execution of the rewriting operation is completed; a third sending unit, configured to send a new delta file of the first database cluster to the second database cluster, where the new delta file is a delta file used to replace the current delta file after the execution of the rewrite operation is completed.

In some embodiments of the present application, based on the foregoing solution, before the third sending unit is configured to send the new delta file of the first database cluster to the second database cluster, the apparatus further includes: and the second deleting unit is configured to delete the incremental file of the second database cluster and the snapshot file of the second database cluster, and the snapshot file of the second database cluster is used for recording data corresponding to the second database cluster at a specified time point.

In some embodiments of the present application, based on the foregoing solution, the apparatus further includes: the judging unit is configured to judge whether a message containing the data writing command exists in a specified theme of the message middleware when the first database cluster needs to be switched to the second database cluster; a first switching unit configured to switch from the first database cluster to the second database cluster if there is no message containing the data write command in a specified topic of the message middleware; and the second switching unit is configured to switch from the first database cluster to a second database cluster after the second thread sends all the messages to the second database cluster if the messages containing the data writing command exist in the specified subject of the message middleware.

In some embodiments of the present application, based on the foregoing scheme, the first sending unit is configured to: and sequentially sending the messages containing the data writing commands to the specified subjects of the message middleware according to the acquisition time sequence of the data writing commands.

In some embodiments of the present application, based on the foregoing scheme, the second sending unit is configured to: and sequentially sending the messages to the second database cluster through the second thread in a first-in first-out mode.

According to an aspect of embodiments of the present application, there is provided a computer-readable medium on which a computer program is stored, the computer program, when executed by a processor, implementing the data synchronization method as described in the above embodiments.

According to an aspect of an embodiment of the present application, there is provided an electronic device including: one or more processors; a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the data synchronization method as described in the above embodiments.

According to an aspect of embodiments herein, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the data synchronization method provided in the various alternative embodiments described above.

In the technical solutions provided in some embodiments of the present application, a current delta file of a first database cluster is read through a first thread, and after the current delta file is read, the current delta file is analyzed to obtain a data write command recorded by the current delta file, and then a message including the data write command is sent to a specified subject of a message middleware, and then the message is sent to a second database cluster through a second thread, and the second database cluster executes and writes the data write command included in the message, thereby completing data synchronization processing from the first database cluster to the second database cluster. According to the technical scheme, data synchronization is carried out in a message middleware mode, compared with a manual incremental file copying mode, a fast and safe data synchronization technology is achieved, when a first database cluster breaks down, data transmission work of the message middleware cannot stop, a second database cluster can still acquire a data writing command from the message middleware, and therefore the data loss phenomenon cannot occur, and data safety is guaranteed.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application. It is obvious that the drawings in the following description are only some embodiments of the application, and that for a person skilled in the art, other drawings can be derived from them without inventive effort. In the drawings:

FIG. 1A shows a schematic diagram of an exemplary system architecture to which aspects of embodiments of the present application may be applied;

FIG. 1B is a schematic diagram of a blockchain system to which aspects of one embodiment of the present application may be applied;

FIG. 1C illustrates a block chain diagram to which one embodiment of the present application may be applied;

FIG. 1D is a diagram illustrating the generation of new tiles in a blockchain to which one embodiment of the present application may be applied;

FIG. 2 shows a flow diagram of a data synchronization method according to an embodiment of the present application;

FIG. 3 shows a flow diagram of a data synchronization method according to an embodiment of the present application;

FIG. 4 shows a flow diagram of a data synchronization method according to an embodiment of the present application;

FIG. 5 shows a flow diagram of a data synchronization method according to an embodiment of the present application;

FIG. 6 shows a block diagram of a data synchronization apparatus according to an embodiment of the present application;

FIG. 7 illustrates a schematic structural diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the subject matter of the present application can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the application.

It is to be noted that the terms used in the specification and claims of the present application and the above-described drawings are only for describing the embodiments and are not intended to limit the scope of the present application. It will be understood that the terms "comprises," "comprising," "includes," "including," "has," "having," and the like, when used herein, specify the presence of stated features, integers, steps, operations, elements, components, and/or groups thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It will be further understood that, although the terms first, second, third, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element without departing from the scope of the present invention. Similarly, a second element may be termed a first element. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.

The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

It should be noted that: reference herein to "a plurality" means two or more. "and/or" describe the association relationship of the associated objects, meaning that there may be three relationships, e.g., A and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

Before further detailed description of the embodiments of the present application, terms and expressions referred to in the embodiments of the present application will be described, and the terms and expressions referred to in the embodiments of the present application will be used for the following explanation.

Block chains: the method is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm and the like, is essentially a decentralized database, is a string of data blocks which are associated by using a cryptography method, and each data block contains information for verifying the validity (anti-counterfeiting) of the information and generating a next block.

In one embodiment of the present application, data synchronization may be satisfactory for many application scenarios, including scenarios where data backup is performed to prevent data loss. The method also comprises a scene that different read-write performances are required to be provided for users aiming at the same data, for example, in a configured database for data synchronous storage, the database with high write performance is convenient for users to quickly input data, and the database with high read performance is convenient for users to quickly search data. Or the method also comprises a scene of popularizing data synchronization between a wider database and a newly developed database with higher performance. Specifically, as shown in fig. 1, a data storage system 100A is configured with a first database cluster 101A, a second database cluster 102A, and a message middleware 103A in the embodiment of the present application, and an embodiment of the present application is described below with reference to fig. 1.

The first database cluster 101A may include a plurality of first nodes interconnected with one another to form the first database cluster 101A, the second database cluster 102A may include a plurality of second nodes interconnected with one another to form the second database cluster 102A, and both the first nodes and the second nodes may be key-value type databases, such as Redis databases.

The Redis database is a memory database, stores Data in a memory, and directly reads the Data in the memory to perform corresponding operation processing, however, the memory Data has the characteristic of power failure loss, and once a machine fails or is restarted, the Data in the memory is completely lost, so how to ensure the security of the memory Data is a problem to be solved by the memory database. The mode of the snapshot file is to write the data in the memory into the snapshot file according to a certain strategy, and the mode of the incremental file is to add the write-in command of the memory data into the incremental file in a log mode each time.

Message middleware 103A may be, but is not limited to, a distributed publish-subscribe system kafka. The distributed publish-subscribe message system kafka is a high-throughput distributed publish-subscribe message system, and the topic (topic) is a component of the distributed publish-subscribe message system kafka and is a logical concept of storing messages, i.e. a message collection. Each message sent to the distributed publish-subscribe message system kafka has a topic. The messages corresponding to different themes are stored separately, and each theme can have multiple producers send messages to it or multiple consumers consume the messages. Each topic, in turn, may correspond to multiple partitions in which messages are stored. In the present embodiment, the message containing the data write command can be stored with the same partition specifying the theme by configuring the parameter information of the theme.

The data storage system 100A in the embodiment of the application is designed as a multi-thread running environment, and if data needs to be synchronized between the first database cluster 101A and the second database cluster 102A of the data storage system, the current incremental file of the first database cluster 101A may be read and analyzed by the first thread, and the current incremental file is used to record a data write command executed by the first database cluster 101A. After parsing the data write command, a message containing the data write command may be sent to the same partition of the message middleware that specifies the subject.

Further, the second thread subscribing to the specified topic may acquire a message from the specified topic, and then send the acquired message to the second database cluster 102A, so that the second database cluster 102A executes and writes a data write command included in the message, thereby completing the data synchronization process from the first database cluster 101A to the second database cluster 102A.

In an embodiment of the present application, the first database cluster 101A or the second database cluster 102A may be a blockchain system, see the blockchain system 100B shown in fig. 1B, where the blockchain system 100B refers to a system for performing data sharing between nodes, and each node 101B may receive input information during normal operation and maintain shared data in the blockchain system 100B based on the received input information. In order to ensure the information intercommunication in the blockchain system 100B, there may be an information connection between each node 101B in the blockchain system 100B, and the nodes 101B may transmit information through the information connection. For example, when an input message is received by any node 101B in the blockchain system 100B, other nodes 101B in the blockchain system 100B obtain the input message according to the consensus algorithm, and store the input message as data in the shared data, so that the data stored on all nodes 101B in the blockchain system 100B are consistent.

Each node 101B in the blockchain system 100B has a corresponding node identifier, and each node 101B in the blockchain system 100B may store the node identifiers of other nodes 101B in the blockchain system 100B, so that the generated blocks are broadcast to other nodes 101B in the blockchain system 100B according to the node identifiers of other nodes 101B. Each node 101B may maintain a node identifier list as shown in table 1 below, and store the node name and the node identifier in the node identifier list correspondingly. The node identifier may be an IP (Internet Protocol) address and any other information that can be used to identify the node, and table 1 only illustrates the IP address as an example.

TABLE 1

Node name	Node identification
		Node 1	117.114.151.174
Node 2	117.116.189.145
		…	…
Node N	119.123.789.258

FIG. 1C shows a schematic diagram of a blockchain to which one embodiment of the present application may be applied.

Each node in the blockchain system 100B stores one identical blockchain. The block chain is composed of a plurality of blocks, referring to fig. 1C, the block chain is composed of a plurality of blocks, the starting block includes a block header and a block main body, the block header stores an input information characteristic value, a version number, a timestamp and a difficulty value, and the block main body stores input information; the next block of the starting block takes the starting block as a parent block, the next block also comprises a block head and a block main body, the block head stores the input information characteristic value of the current block, the block head characteristic value of the parent block, the version number, the timestamp and the difficulty value, and the like, so that the block data stored in each block in the block chain is associated with the block data stored in the parent block, and the safety of the input information in the block is ensured.

Fig. 1D shows a schematic diagram of new block generation in a blockchain to which an embodiment of the present application may be applied.

When each block in the block chain is generated, referring to fig. 1D, when a node where the block chain is located receives input information, the input information is verified, after the verification is completed, the input information is stored in the memory pool, and the hash tree for recording the input information is updated; and then, updating the updating time stamp to the time when the input information is received, trying different random numbers, and calculating the characteristic value for multiple times, so that the calculated characteristic value can meet the following formula:

SHA256(SHA256(version+prev_hash+merkle_root+ntime+nbits+x))＜TARGET

wherein, SHA256 is a characteristic value algorithm used for calculating a characteristic value; version is version information of the relevant block protocol in the block chain; prev _ hash is a block head characteristic value of a parent block of the current block; merkle _ root is a characteristic value of the input information; ntime is the update time of the update timestamp; nCits is the current difficulty, is a fixed value within a period of time, and is determined again after the fixed time period is exceeded; x is a random number; TARGET is a feature threshold, which can be determined from nCits.

Therefore, when the random number meeting the formula is obtained through calculation, the information can be correspondingly stored, and the block head and the block main body are generated to obtain the current block. Then, the node where the block chain is located sends the newly generated blocks to other nodes 101B in the block chain system 100B where the newly generated blocks are located respectively according to the node identifiers of the other nodes 101B in the block chain system 100B, the newly generated blocks are verified by the other nodes 101B, and the newly generated blocks are added to the block chain stored in the newly generated blocks after the verification is completed.

Referring to fig. 2 again, a flow chart of a data synchronization method according to an embodiment of the present application is shown, and the data synchronization method according to the embodiment of the present application can be applied to the data storage system 100A shown in fig. 1. Referring to fig. 2, the method includes:

step S210, reading a current increment file of a first database cluster through a first thread, wherein the current increment file is used for recording a data writing command executed by the first database cluster;

step S220, analyzing the current incremental file to obtain a data writing command recorded by the current incremental file;

step S230, sending a message containing the data writing command to a specified theme of a message middleware so that a second thread subscribing to the specified theme acquires the message from the specified theme;

step S240, sending the message to a second database cluster through the second thread, so that the second database cluster executes a data write command included in the message, and writes the executed data write command into an incremental file of the second database cluster.

These steps are described in detail below.

In step S210, a current delta file of a first database cluster is read by a first thread, where the current delta file is used to record a data write command executed by the first database cluster.

The data synchronization method of this embodiment may be applied to a data storage system as shown in fig. 1, where the data storage system is designed as a multi-threaded operating environment, a plurality of first nodes are interconnected with each other to form a first database cluster, and the first node may be a Redis database.

The Redis database is a memory database, is a log-type and Key-Value database which is open-source, written by using ANSI C language, supports network, can be based on memory and can also be persistent. The Redis database stores data in the memory, and directly reads the data in the memory to perform corresponding operation processing, and because the data reading and writing speed of the memory is higher than that of a magnetic disk by several orders of magnitude, the Redis database can greatly improve the processing efficiency of the data by storing the data in the memory.

However, the memory Data has the characteristic of power failure loss, and Data in the memory is completely lost once the machine fails or is restarted, so that the Redis database provides two modes of a snapshot File (Redis Data Base, RDB) and an incremental File (application Only File, AOF) to ensure the security of the memory Data. The snapshot file mode is to write the data in the memory into the snapshot file according to a certain strategy, and the incremental file mode is to add the write-in command of the memory data into the incremental file in a log form each time.

It should be noted that, the incremental file mode is not opened by default in the Redis database, but in order to implement the data synchronization method according to the technical solution of this embodiment, a precondition is that no in a configuration parameter "apendonly no" in a configuration file in the Redis database must be changed to yes, that is, to "apendonly yes", so that the incremental file mode is opened in the Redis database.

After the incremental file mode is started, a current incremental file of the first database cluster can be read through the first thread, and a data write command executed by the first database cluster is recorded in the current incremental file.

It should be further noted that, the incremental file operation modes include the following three types: continuous synchronization (always), synchronization every 1 second (everysec), and non-active synchronization (no). The continuous synchronization strategy is to continuously synchronize the continuous time of the data of the operation of the Redis database to the incremental file of the disk and immediately synchronize the data to the disk, so that the reliability of the data is highest, but the performance of the Redis database is greatly influenced; the synchronization strategy of every 1 second is to synchronize the data of the operation of the Redis database to the incremental file of the disk once every 1 second, the mode has high efficiency and high data reliability, but the data of nearly one second may be lost after the Redis database is down; the inactive synchronization strategy is to synchronize data of the operation on the Redis database to the incremental file of the disk from the inactive mode, and the reliability is poor.

In this embodiment, the operation mode of the incremental file may be set to be synchronous every 1 second (evenysec) by the configuration parameter, that is, the configuration parameter "apendfsync evenysec" is set.

In step S220, the current incremental file is parsed to obtain a data write command recorded by the current incremental file.

Since the data write command executed by the first database cluster is recorded in the current delta file, after the current delta file is read, the data write command can be acquired by parsing the current delta file.

For example, if the format of the read incremental file is as follows, the current incremental file may be analyzed line by line through the first thread, and finally, the data write commands recorded by the current incremental file are select 0, set name jack, and set sex male, respectively.

The 2# current data write command has two parameters

The first parameter of $6# is 6 bytes in length, i.e., select

First parameter value of select # Current data write Command

The second parameter of the $1# current data write command, length is 1, i.e., 0

Second parameter value of 0# current data write command

The 3# current data write command has three parameters

The first parameter of $3# is 3 bytes, i.e., set

set # first parameter value of current data write command

The second parameter size of $4# is 4 bytes, namely name

Second parameter value of name # Current data write Command

The third parameter of $4# is 4 bytes in length, i.e., jack

Third parameter value of jack # Current data write Command

The 3# current data write command has three parameters

The first parameter of $3# is 3 bytes, i.e., set

set # first parameter value of current data write command

The first parameter size of $3# is 3 bytes, i.e. sex

second parameter value of sex # Current data write Command

The third parameter of $4# is 4 bytes in length, i.e., large

Third parameter value of Large # Current data write Command

In step S230, a message including the data write command is sent to a specified topic of the message middleware, so that a second thread subscribing to the specified topic acquires the message from the specified topic.

In this embodiment, the message middleware may be, but is not limited to, a distributed publish-subscribe system kafka. The distributed publish-subscribe messaging system kafka is an open source streaming platform developed by the Apache software foundation, written in Scala and Java. The distributed publish-subscribe message system kafka is a high-throughput distributed publish-subscribe message system that can handle all the action flow data of a consumer in a web site. These data are typically addressed by handling logs and log aggregations due to throughput requirements. The purpose of the distributed publish-subscribe messaging system kafka is to unify online and offline message processing through the parallel loading mechanism of the distributed system infrastructure, and also to provide real-time messages through clustering.

The topic (topic) is a component of the distributed publish-subscribe message system kafka, and is a logical concept of storing messages, that is, a message collection. Each message sent to the distributed publish-subscribe messaging system has a topic. The messages corresponding to different themes are stored separately, and each theme can have multiple producers send messages to it or multiple consumers consume the messages. Each topic may in turn correspond to multiple partitions for storing messages for that topic.

In this embodiment, the message containing the data write command may be stored in the same partition of the specified topic by configuring parameter information of the topic, so that the second thread subscribing to the specified topic may obtain the message from the specified topic.

In the data storage system of the embodiment of the application, a message middleware mechanism is adopted, so that different objects activate the events of the other party by transmitting messages to complete corresponding operations.

In an embodiment of the application, because the data write-in commands recorded in the incremental file have a time sequence, when the first thread parses the incremental file line by line, the obtained data write-in commands also have a time sequence, and therefore the first thread can send messages containing the data write-in commands to the specified subjects of the message middleware in sequence according to the obtaining time sequence of the data write-in commands, so that the sequence of the messages is ensured.

In step S240, the message is sent to a second database cluster through the second thread, so that the second database cluster executes a data write command included in the message, and writes the executed data write command into an incremental file of the second database cluster.

Wherein the second database cluster may include a plurality of second nodes interconnected with each other to form the second database cluster, and the second nodes may be key-value type databases, such as Redis databases.

Specifically, after acquiring the message from the specified topic of the message middleware, the second thread may further send the message to the second database cluster, so that the second database cluster executes and writes the data write command included in the message, thereby completing the data synchronization processing from the first database cluster to the second database cluster.

In one embodiment of the present application, the second thread may send the message to the second database cluster in a first-in-first-out manner. When the messages are sent in a first-in first-out mode, the messages of the specified subjects of the first-in message middleware are acquired by the second thread first and are sent to the second database cluster first, and the messages of the specified subjects of the later-in message middleware are acquired by the second thread and are sent to the second database cluster later.

This embodiment has the advantage that even if the data write command of the first database cluster is finally sent to the second database cluster, it is advantageous for the second database cluster to be able to execute the data write commands in the order of the data write commands, thereby completing the data synchronization, since the data write commands of the current incremental file records of the first database cluster are sent in the order.

Based on the technical scheme of the embodiment, data synchronization is performed in a message middleware mode, and compared with a manual incremental file copying mode, a fast and reliable data synchronization technology is realized.

In an embodiment of the present application, since the data write command executed each time by the first database cluster is maintained in a delta file, the data write command executed each time is maintained in the delta file, and although this may well ensure the security of the data, the data write command is continuously accumulated in the delta file, and the delta file becomes larger and larger as time goes by, and occupies a large amount of storage space. Therefore, it is necessary for the first database cluster to perform the rewrite operation, so as to compress the incremental file volume, and the incremental file with smaller file volume can be loaded by the first database cluster more quickly, in this embodiment, the rewrite operation of the first database cluster can also be checked, as shown in fig. 3, which specifically includes steps S310 to S320, and the detailed description is as follows:

step S310, whether the first database cluster executes rewriting operation on the current incremental file is checked.

Specifically, the system may check, by the first thread, a thread other than the second thread, whether the first database cluster performs a rewrite operation on the current delta file.

For convenience of understanding, the description of the rewrite operation is described below, and the execution of the rewrite operation on the current incremental file by the first database cluster is actually creating a child process by a parent process, where the parent process is a process that executes a data write command and appends the data write command to the current incremental file, and the child process is a process that executes the rewrite operation on the current incremental file. In the process of executing the rewriting operation by the child process, the parent process still continues to execute the data writing command, and appends the data writing command to the incremental file buffer and the incremental file rewriting buffer, and the content of the incremental file buffer is synchronized to the current incremental file. The subprocess starts the rewrite operation and creates a new increment file, the new increment file comprises a data write command which is merged according to the command merging rule before the first database cluster executes the rewrite operation, after the subprocess creates the new increment file, sending a signal to the parent process, the parent process calling a signal processing function after receiving the signal, the function appends the data write command of the delta file rewrite buffer to the new delta file, after the contents in the incremental file rewriting cache area are all written into the new incremental file, the states of the current incremental file, the new incremental file and the first database cluster are completely consistent, and then the parent process renames the new incremental file, covers the current incremental file, and completes the replacement of the current incremental file and the new incremental file, so that the execution of the rewriting operation is completed.

In some embodiments, the condition for the parent process to create a child process for an overwrite operation may be: if the difference between the size of the incremental file after the first database cluster executes the rewriting operation last time and the size of the current incremental file exceeds a first specified threshold, and the size of the current incremental file reaches a second specified threshold, the parent process can create a child process and execute the rewriting operation on the current incremental file.

In this embodiment, the trigger condition of the rewrite operation requires that the difference between the size of the current delta file and the size of the delta file after the rewrite operation was performed last time exceeds a first specified threshold, and that the size of the current delta file reaches a second specified threshold. The first specified threshold and the second specified threshold may be configured by a configuration parameter in a configuration file of the first database cluster, for example, the first specified threshold is configured by a configuration parameter "auto-aof-rewrite-percentage", and the second specified threshold is configured by a configuration parameter "auto-aof-rewrite-min-size".

Step S320, if it is determined that the first database cluster performs a rewrite operation on the current delta file, notifying the first thread to stop reading the current delta file.

If it is determined that the first database cluster performs the rewrite operation on the current incremental file, that is, the states of the current incremental file and the first database cluster are no longer the same, then performing data synchronization on the first database cluster and the second database cluster according to the current incremental file is inaccurate, so that the first thread does not need to read the current incremental file, and the first thread can be notified to stop reading the current incremental file.

In an embodiment of the present application, if the first database cluster performs the rewrite operation on the current incremental file, the data synchronization between the first database cluster and the second database cluster may be performed after the rewrite operation is completed, as shown in fig. 4, the data synchronization method after the rewrite operation is completed may specifically include steps S410 to S420, which are described in detail as follows:

step S410, after the rewrite operation is completed, notifying the first thread to start reading a new incremental file in the first database cluster, where the new incremental file is an incremental file used to replace the current incremental file after the rewrite operation is completed.

It is easy to understand that after the rewriting operation of the current incremental file is completed, a new incremental file inevitably exists in the first database cluster, the new incremental file is also an incremental file used for replacing the current incremental file, the new incremental file is used for recording a data writing command executed by the first database cluster, and meanwhile, a data writing command executed by a subsequent first database cluster is also added to the new incremental file.

Step S420, parsing the new incremental file to obtain a data write command recorded by the new incremental file.

After reading the new incremental file, the first thread may parse the new incremental file to obtain a data write command recorded by the new incremental file.

In another embodiment of the present application, as shown in fig. 5, after the execution of the rewrite operation is completed, the manner of data synchronization between the first database cluster and the second database cluster may include:

step S510, after the execution of the rewriting operation is completed, if the specified subject of the message middleware contains the message containing the data writing command, deleting the message;

step S520, sending a new incremental file of the first database cluster to the second database cluster, where the new incremental file is an incremental file used to replace the current incremental file after the execution of the rewrite operation is completed.

In this embodiment, the data synchronization between the first database cluster and the second database cluster is no longer performed by reading the new incremental file of the first database cluster through the first thread, but the new incremental file is directly sent to the second database cluster, and after receiving the new incremental file, the second database cluster may execute the data write command recorded by the new incremental file to complete data write, and add the data write command to the incremental file of the second database cluster, thereby completing the data synchronization between the first database cluster and the second database cluster.

It should be noted that the new delta file obtained after the rewrite operation is performed is consistent with the state of the first database cluster, so that the new delta file can be directly sent to the second database cluster for data synchronization, and at the same time, it should be considered whether a message containing a data write command exists in the specified subject of the message middleware, because, before the rewrite operation performed by the first database obtains the new delta file, a part of the data write command in the current delta file may have been sent to the specified subject in the message middleware, and with respect to the new delta file, the current delta file belongs to the "old file", the data write command recorded by the current delta file belongs to the "old data write command", and after the new delta file is sent to the second database cluster, that is, the latest state of the first database cluster is synchronized to the second database cluster, there is no need to send the "old data write command" on to the second database cluster, so if there is a message containing a data write command in the specified subject of the message middleware, the message may be deleted to avoid sending the "old data write command" in the message to the second database cluster.

In an embodiment of the present application, before sending the new delta file of the first database cluster to the second database cluster, the delta file of the second database cluster and the snapshot file of the second database cluster may also be deleted, so that the second database cluster only needs to execute and write the data write command of the new delta file record of the first database cluster to complete data synchronization.

The incremental file of the second database cluster is a file for recording a data writing command executed by the second database cluster, and the snapshot file of the second database cluster is a file for recording data corresponding to the second database cluster at the designated time point.

In one embodiment of the present application, if a first database cluster fails and must be switched to a second database cluster, it is necessary to ensure that the state of the first database cluster is consistent with the state of the second database cluster, so as to avoid the occurrence of data loss. The consistency of the two states means that the message of the specified subject sent from the first database cluster to the message middleware is also sent to the second database cluster by the specified subject of the message middleware.

Therefore, in this embodiment, when it is necessary to switch from the first database cluster to the second database cluster, it may be determined whether a message including a data write command exists in the specified topic of the message middleware first, and if the message including the data write command does not exist in the specified topic of the message middleware, the first database cluster may be directly switched to the second database cluster; conversely, if there is a message containing a data write command in the specified subject of the message middleware, a switch may be made from the first database cluster to the second database cluster after the second thread sends the message in its entirety to the second database cluster.

Embodiments of the apparatus of the present application are described below, which may be used to perform the image processing methods in the above-described embodiments of the present application. For details that are not disclosed in the embodiments of the apparatus of the present application, please refer to the embodiments of the image processing method described above in the present application.

Fig. 6 shows a block diagram of an image processing apparatus according to an embodiment of the present application, and referring to fig. 6, a data synchronization apparatus 600 according to an embodiment of the present application includes: a reading unit 602, a first parsing unit 604, a first sending unit 606 and a second sending unit 608.

The reading unit 602 is configured to read, by a first thread, a current delta file of a first database cluster, where the current delta file is used to record a data write command executed by the first database cluster; the first parsing unit 604 is configured to parse the current incremental file to obtain a data write command recorded by the current incremental file; the first sending unit 606 is configured to send a message including the data writing command to a specified topic of a message middleware, so that a second thread subscribing to the specified topic obtains the message from the specified topic; the second sending unit 608 is configured to send the message to a second database cluster through the second thread, so that the second database cluster executes a data write command included in the message, and writes the executed data write command into an incremental file of the second database cluster.

In some embodiments of the present application, the apparatus further comprises: a checking unit configured to check whether the first database cluster performs a rewrite operation on the current delta file; a first notification unit configured to notify the first thread to stop reading the current delta file if it is determined that the first database cluster performs a rewrite operation on the current delta file.

In some embodiments of the present application, the apparatus further comprises: the obtaining unit is configured to obtain the size of the incremental file after the rewriting operation is executed last time by the first database cluster; and the determining unit is configured to determine that the first database cluster performs the rewrite operation on the current incremental file if the difference between the size of the incremental file after the previous rewrite operation and the size of the current incremental file exceeds a first specified threshold and the size of the current incremental file reaches a second specified threshold.

In some embodiments of the present application, the apparatus further comprises: the second notification unit is configured to notify the first thread to start reading a new incremental file in the first database cluster after the execution of the rewrite operation is completed, wherein the new incremental file is an incremental file used for replacing the current incremental file after the execution of the rewrite operation is completed; and the second analysis unit is configured to analyze the new incremental file to acquire a data writing command recorded by the new incremental file.

In some embodiments of the present application, the apparatus further comprises: a first deleting unit, configured to delete a message including the data writing command if the message exists in a specified topic of the message middleware after the execution of the rewriting operation is completed; a third sending unit, configured to send a new delta file of the first database cluster to the second database cluster, where the new delta file is a delta file used to replace the current delta file after the execution of the rewrite operation is completed.

In some embodiments of the present application, before the third sending unit is configured to send the new delta file of the first database cluster to the second database cluster, the apparatus further includes: and the second deleting unit is configured to delete the incremental file of the second database cluster and the snapshot file of the second database cluster, and the snapshot file of the second database cluster is used for recording data corresponding to the second database cluster at a specified time point.

In some embodiments of the present application, the apparatus further comprises: the judging unit is configured to judge whether a message containing the data writing command exists in a specified theme of the message middleware when the first database cluster needs to be switched to the second database cluster; a first switching unit configured to switch from the first database cluster to the second database cluster if there is no message containing the data write command in a specified topic of the message middleware; and the second switching unit is configured to switch from the first database cluster to a second database cluster after the second thread sends all the messages to the second database cluster if the messages containing the data writing command exist in the specified subject of the message middleware.

In some embodiments of the present application, the first sending unit 606 is configured to: and sequentially sending the messages containing the data writing commands to the specified subjects of the message middleware according to the acquisition time sequence of the data writing commands.

In some embodiments of the present application, the second sending unit 608 is configured to: and sequentially sending the messages to the second database cluster through the second thread in a first-in first-out mode.

It should be noted that the computer system 700 of the electronic device shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU)701, which can perform various appropriate actions and processes, such as performing the methods described in the above embodiments, according to a program stored in a Read-Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data necessary for system operation are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An Input/Output (I/O) interface 705 is also connected to the bus 704.

The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a Network interface card such as a LAN (Local Area Network) card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.

In particular, according to embodiments of the application, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising a computer program for performing the method illustrated by the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program executes various functions defined in the system of the present application when executed by a Central Processing Unit (CPU) 701.

It should be noted that the computer readable medium shown in the embodiments of the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM), a flash Memory, an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with a computer program embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.

As another aspect, the present application also provides a computer-readable medium, which may be contained in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by an electronic device, cause the electronic device to implement the method described in the above embodiments.

It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the application. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains.

It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. A method for synchronizing data, the method comprising:

reading a current increment file of a first database cluster through a first thread, wherein the current increment file is used for recording a data writing command executed by the first database cluster;

analyzing the current incremental file to acquire a data writing command recorded by the current incremental file;

sending a message containing the data writing command to a specified theme of message middleware so that a second thread subscribing to the specified theme acquires the message from the specified theme;

and sending the message to a second database cluster through the second thread, so that the second database cluster executes a data write command contained in the message, and writes the executed data write command into an incremental file of the second database cluster.

2. The method of claim 1, further comprising:

checking whether the first database cluster performs a rewrite operation on the current delta file;

and if the first database cluster is determined to execute the rewriting operation on the current incremental file, informing the first thread to stop reading the current incremental file.

3. The method of claim 2, further comprising:

acquiring the size of the incremental file of the first database cluster after the rewriting operation is executed last time;

and if the difference between the size of the incremental file subjected to the previous rewriting operation and the size of the current incremental file exceeds a first specified threshold value, and the size of the current incremental file reaches a second specified threshold value, determining that the first database cluster executes the rewriting operation on the current incremental file.

4. The method of claim 2, further comprising:

after the execution of the rewriting operation is completed, notifying the first thread to start reading a new incremental file in the first database cluster, wherein the new incremental file is used for replacing the current incremental file after the execution of the rewriting operation is completed;

and analyzing the new incremental file to acquire a data writing command recorded by the new incremental file.

5. The method of claim 2, further comprising:

after the execution of the rewriting operation is finished, if a message containing the data writing command exists in the specified theme of the message middleware, deleting the message;

and sending a new incremental file of the first database cluster to the second database cluster, wherein the new incremental file is used for replacing the current incremental file after the rewriting operation is completed.

6. The method of claim 5, wherein prior to sending the new delta file for the first database cluster to the second database cluster, the method further comprises:

and deleting the incremental file of the second database cluster and the snapshot file of the second database cluster, wherein the snapshot file of the second database cluster is used for recording the data corresponding to the second database cluster at the specified time point.

7. The method of claim 1, further comprising:

when the first database cluster needs to be switched to the second database cluster, judging whether a message containing the data writing command exists in a specified subject of the message middleware;

if no message containing the data writing command exists in the specified subject of the message middleware, switching from the first database cluster to the second database cluster;

if the message containing the data write command exists in the specified subject of the message middleware, after the second thread sends all the messages to a second database cluster, switching from the first database cluster to the second database cluster.

8. The method of claim 1, wherein sending the message containing the data write command to a specified topic of message middleware comprises:

and sequentially sending the messages containing the data writing commands to the specified subjects of the message middleware according to the acquisition time sequence of the data writing commands.

9. The method of claim 1, wherein sending the message to a second database cluster via the second thread comprises:

and sequentially sending the messages to the second database cluster through the second thread in a first-in first-out mode.

10. A data synchronization apparatus, the apparatus comprising:

the reading unit is configured to read a current increment file of a first database cluster through a first thread, wherein the current increment file is used for recording a data writing command executed by the first database cluster;

the first analysis unit is configured to analyze the current incremental file to acquire a data write-in command recorded by the current incremental file;

the first sending unit is configured to send a message containing the data writing command to a specified theme of message middleware so that a second thread subscribing to the specified theme acquires the message from the specified theme;

and the second sending unit is configured to send the message to a second database cluster through the second thread, so that the second database cluster executes a data write command contained in the message, and writes the executed data write command into an incremental file of the second database cluster.