CN111953760B - Data synchronization method, device, multi-activity system and storage medium - Google Patents
Data synchronization method, device, multi-activity system and storage medium Download PDFInfo
- Publication number
- CN111953760B CN111953760B CN202010773658.1A CN202010773658A CN111953760B CN 111953760 B CN111953760 B CN 111953760B CN 202010773658 A CN202010773658 A CN 202010773658A CN 111953760 B CN111953760 B CN 111953760B
- Authority
- CN
- China
- Prior art keywords
- message
- distributed system
- processing component
- local
- distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Hardware Redundancy (AREA)
Abstract
The application discloses a data synchronization method, a data synchronization device, a multi-activity system and a storage medium, and relates to the field of communication. The multi-activity system comprises a plurality of distributed systems, a message processing component configured corresponding to each distributed system and a message queue configured corresponding to each distributed system, wherein the method is applied to any message processing component in the multi-activity system, and the method applied to the message processing component comprises the following steps: acquiring messages in message queues of other distributed systems and messages in a local distributed system, wherein the local distributed system is a distributed system corresponding to the message processing component, and the other distributed systems are distributed systems except the local distributed system; if the message is the message in the message queue of other distributed systems, writing the message into the local distributed system; and if the message is generated by the local distributed system, sending the message to a message queue of the local distributed system.
Description
Technical Field
The present application relates to the field of communications technologies, and in particular, to a data synchronization method, apparatus, multi-activity system, and storage medium.
Background
In the distributed system, the services are communicated and cooperated with each other to finish tasks together, so that higher efficiency is achieved. When a plurality of distributed systems work cooperatively, data synchronization among the distributed systems becomes a problem to be solved urgently.
Disclosure of Invention
In view of the above, the present application provides a data synchronization method, apparatus, multi-activity system and storage medium, so as to improve the above-mentioned problems.
In a first aspect, an embodiment of the present application provides a data synchronization method for data synchronization of a multi-active system, where the multi-active system includes a plurality of distributed systems, a message processing component configured corresponding to each distributed system, and a message queue configured corresponding to each distributed system, and the method is applied to any one of the message processing components in the multi-active system, and the method includes: acquiring messages in message queues of other distributed systems and messages in a local distributed system, wherein the local distributed system is a distributed system corresponding to the message processing component, and the other distributed systems are distributed systems except the local distributed system; if the message is the message in the message queue of other distributed systems, writing the message into the local distributed system; and if the message is generated by the local distributed system, sending the message to a message queue of the local distributed system.
In a second aspect, an embodiment of the present application provides a data synchronization method for data synchronization of a multi-active system, where the multi-active system includes a plurality of distributed systems, a message processing component configured corresponding to each distributed system, and a message queue configured corresponding to each distributed system, and the method includes: the distributed system sends the message to a message processing component which is configured correspondingly, the message processing component obtains the message in a message queue of other distributed systems and the message in a local distributed system, the local distributed system is a distributed system corresponding to the message processing component, and the other distributed systems are distributed systems outside the local distributed system; if the message acquired by the message processing component is a message in a message queue of other distributed systems, the message processing component writes the message into the local distributed system; and if the message acquired by the message processing component is a message generated by the local distributed system, sending the message to a message queue of the local distributed system.
In a third aspect, an embodiment of the present application provides a data synchronization apparatus for data synchronization of a multi-active system, where the multi-active system includes a plurality of distributed systems, a message processing component configured corresponding to each distributed system, and a message queue configured corresponding to each distributed system, and the apparatus is applied to any one of the message processing components in the multi-active system, and the apparatus includes: the message acquisition module is used for acquiring messages in message queues of other distributed systems and messages in a local distributed system, wherein the local distributed system is a distributed system corresponding to the message processing component, and the other distributed systems are distributed systems except the local distributed system; the message writing module is used for writing the message into the local distributed system if the message is the message in the message queue of other distributed systems; and the message sending module is used for sending the message to a message queue of the local distributed system if the message is generated by the local distributed system.
In a fourth aspect, an embodiment of the present application provides a multi-active system, including a plurality of distributed systems, a message processing component configured corresponding to each distributed system, and a message queue configured corresponding to each distributed system, where the distributed systems are configured to send a message to the message processing component configured corresponding to each distributed system, the message processing component is configured to obtain a message in a message queue of another distributed system and a message in a local distributed system, where the local distributed system is a distributed system corresponding to the message processing component, and the other distributed systems are distributed systems other than the local distributed system; if the message acquired by the message processing component is a message in a message queue of other distributed systems, the message processing component is used for writing the message into the local distributed system; and if the message acquired by the message processing component is a message generated by a local distributed system, the message processing component is used for sending the message to a message queue of the local distributed system.
In a fifth aspect, embodiments of the present application provide a computer readable storage medium having stored therein program code executable by a processor to invoke the method described above as applied to a message processing component.
The data synchronization method, the data synchronization device, the multi-activity system and the storage medium provided by the embodiment of the application comprise a plurality of distributed systems, and each distributed system is correspondingly configured with a message processing component and a message queue. The message processing component may obtain messages in message queues of other distributed systems and messages in the local distributed system, write messages in message queues of other distributed systems to the local distributed system, and write messages generated by the local distributed system to message queues of the local distributed system. Therefore, the information in the information queue of each distributed system is the information generated by the local distributed system, and the information processing component of each distributed system can acquire the information from the information queues of other distributed systems and write the information into the local distributed system by the other distributed systems, so that the data synchronization among a plurality of distributed systems in the multi-active system is realized.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 shows a schematic diagram of a multiple activity system provided by an embodiment of the present application.
Fig. 2 is a flowchart of a data synchronization method according to an embodiment of the present application.
Fig. 3 is a flowchart of a data synchronization method according to another embodiment of the present application.
Fig. 4 is a flowchart of a data synchronization method according to another embodiment of the present application.
Fig. 5 shows a state change diagram in the data synchronization method according to the embodiment of the present application.
FIG. 6 shows another schematic diagram of a multiple activity system provided by an embodiment of the present application.
Fig. 7 is a functional block diagram of a data synchronization device according to an embodiment of the present application.
Fig. 8 is a memory unit for storing or carrying program codes for implementing a data synchronization method according to an embodiment of the present application.
Detailed Description
In order to enable those skilled in the art to better understand the present application, the following description will make clear and complete descriptions of the technical solutions according to the embodiments of the present application with reference to the accompanying drawings.
The single machine has limited computing capacity and storage capacity, and needs a plurality of machines to cooperatively process to form a distributed system in the face of massive data. A distributed system is a system consisting of a set of computer nodes that communicate over a network to coordinate work in order to accomplish a common task. The distributed system can provide similar single machine service from the macroscopic view, but the distributed system is cooperatively processed by a plurality of machines, so that the processing bottleneck of the single machine can be broken through, the calculation and storage tasks which cannot be completed by a single computer can be completed, more machines can be utilized, and more data can be processed.
High availability in a distributed system is one of the factors that must be considered in constructing a design, and improving availability generally refers to reducing the time by design that the distributed system cannot provide service. For example, if a distributed system is running for 100 time units, 1 time unit may not be available for service, then the availability of the distributed system is 99%.
However, if the high availability is designed only for the deployment of the distributed system in one machine room or one available area, if the machine room is down or the available area is not available, the whole distributed system can not provide service to the outside. Such as: core switches of the machine room fail, ingress fibers are cut off, etc. For this situation, a multi-machine room deployment can be considered, namely, a distributed system is deployed in a plurality of machine rooms, even if one machine room is unavailable, the distributed system can be automatically switched to another machine room to ensure the availability of services; or deployed in a plurality of available areas, even if one machine room is not available, the service availability can be ensured by automatically switching to another available area.
However, when multiple rooms or the same distributed system is deployed across the available areas, there may be cross-room scheduling between services in different rooms in the system, such as cross-room writing between service instances, resulting in higher data latency.
In view of this, the embodiment of the application provides a multi-active system, a plurality of distributed systems are deployed, and the data synchronization of each distributed system in the whole multi-active system is realized by correspondingly configuring a message queue and a message processing component for each distributed system. Each of the distributed systems may independently process requests within the system and synchronize messages with other distributed systems that synchronize data of an unavailable distributed system when the unavailable distributed system fails. As shown in fig. 1, the multi-activity system may include two distributed systems, namely distributed systems a and B, where the distributed systems a and B represent two distributed systems with the same function, and the messages of the distributed systems a and B can be synchronized with each other by corresponding to the message queue a and the message processing component a deployed by the distributed system a and corresponding to the message queue B and the message processing component B deployed by the distributed system B, so as to maintain the data consistency of each distributed system in the multi-activity system, and when the distributed system a fails, the distributed system B can take over the service of the distributed system a and ensure that the service processing is not interrupted; when the distributed system B fails, the distributed system A can take over the service of the distributed system B, and the service processing is ensured not to be interrupted. Three or more distributed systems constitute a multiple active system.
Alternatively, in the embodiment of the present application, the message processing component may be a software program for processing a message, which is generally light and small, and occupies a small amount of data, and occupies a small machine consumption, such as a transaction message processing component (msg handler component), where the message processing component may be deployed in a corresponding machine room configured with a distributed system, or in a certain service instance of the distributed system; message queues are used to place data to be synchronized therein, are typically heavy, large, consume large amounts of data, consume large amounts of occupied machines, and even a cluster, such as a distributed message queue (RocketMQ), may be configured on separate machines.
In the multi-active system, a distributed system may be used to send messages to a correspondingly configured message processing component. The message processing component may be configured to obtain messages in message queues of other distributed systems and messages in a local distributed system, where the local distributed system is a distributed system corresponding to the message processing component, and the other distributed systems are distributed systems other than the local distributed system. If the message acquired by the message processing component is a message in a message queue of other distributed systems, the message processing component can be used for writing the message into the local distributed system; if the message obtained by the message processing component is a message generated by a local distributed system, the message processing component can be used for sending the message to a message queue of the local distributed system.
As shown in fig. 2, the data synchronization method applied to the message processing components of the multi-activity system provided by the embodiment of the application can be applied to any one of the message processing components of the multi-activity system, that is, each message processing component of the multi-activity system realizes that the data of the corresponding distributed system is sent out to enable other distributed systems to acquire through the message processing components, and the message processing components acquire the messages of the other distributed systems, thereby realizing the message synchronization among the distributed systems. As shown in fig. 2, the data synchronization method is described by taking one of the message processing components as an example, and all the message processing components may be executed in the same manner. The data synchronization method may include the following steps.
Step S110: and acquiring messages in message queues of other distributed systems and messages in a local distributed system, wherein the local distributed system is a distributed system corresponding to the message processing component, and the other distributed systems are distributed systems outside the local distributed system.
In the embodiment of the application, the distributed system corresponding to the message processing component for executing the data synchronization method is defined as a local distributed system, and the systems except the local distributed system in the multi-active system are other distributed systems or different distributed systems relative to the local distributed system. For example, the distributed system a shown in fig. 1 is a local distributed system of the message processing component a, and the distributed system B is another distributed system for the message processing component a; distributed system B is the local distributed system of message processing component B, and distributed system a is the other distributed system for message processing component B.
The message processing component may obtain messages in the message queues of each of the other distributed systems and obtain messages in the local distributed system.
When the information of one distributed system is synchronized to other distributed systems, the other distributed systems can take over the service of the distributed system due to the fact that the information of the distributed system is synchronized, and the service of the distributed system can be guaranteed to be processed normally. The specific information included in the message is not limited in the embodiment of the present application, and may be set according to practical situations, for example, the message is a transaction message, and the transaction message includes data creation, deletion and modification.
Step S120: and if the message is the message in the message queue of other distributed systems, writing the message into the local distributed system.
If the information in the information queues of the other distributed systems is acquired, the information is written into the local distributed system, so that the information of the other distributed systems can be synchronized to the local distributed system.
Step S130: and if the message is generated by the local distributed system, sending the message to a message queue of the local distributed system.
If the acquired message is a message generated by the local distributed system, the message is sent to a message queue of the local distributed system, so that message processing components of other distributed systems can acquire the message from the message queue of the local distributed system to synchronize.
It can be understood that, in the embodiment of the present application, the steps S120 and S130 are executed according to conditions, the sequence is not limited, and specifically, the step S120 is executed when the messages in the message queues of other distributed systems are acquired, and the step S130 is executed when the messages generated by the local distributed system are acquired.
In the embodiment of the application, the message processing component writes the message generated by the local distributed system into the message queue of the local distributed system, and the message processing component writes the messages generated by other distributed systems into the local distributed system, so that the synchronization of the messages among the distributed systems is realized.
Alternatively, in the embodiment of the present application, when a message is written, the distributed system may distinguish whether the message is written by the corresponding message processing component or written according to the request of the client. If the information is written according to the request of the client, the information is generated for the local distributed system and is written into a corresponding information processing component; if the message written into the distributed system is the message written by the message processing component, the distributed system is not written into the message processing component any more. Therefore, the message of the local distributed system acquired by the message processing component is the message generated by the local distributed system, and the message written into the message queue of the local distributed system is the message generated by the local distributed system, so that the message is prevented from being written in a circulating way among different distributed systems.
Optionally, in the embodiment of the present application, the messages generated in the distributed system need to be synchronized in the distributed system, and also need to be synchronized between the distributed systems, and the message processing component may acquire the messages needing to be synchronized in the distributed system, and perform synchronization processing on the messages needing to be synchronized between the distributed systems. However, when messages generated by other distributed systems are synchronized to a local distributed system, synchronization within the local distributed system is also required. Thus, the message processing component, upon receiving a message synchronized within the local distributed system, to facilitate synchronizing messages generated by the local distributed system to other distributed systems, may also receive messages of other distributed systems synchronized within the local distributed system.
For example, where a message generated in distributed system B needs to be synchronized to each service instance of distributed system B, the message processing component may receive a message synchronized within distributed system B to facilitate synchronizing messages generated by distributed system B to other distributed systems. However, when the messages from the distributed system a are synchronized to the distributed system B, the messages from the distributed system a need to be synchronized between the service instances in the distributed system B. The message processing component receives a message synchronized within the distributed system B, either a message generated by the distributed system B or a message generated by the distributed system a.
Thus, the message processing component obtains the messages of the local distributed system of all, which may be messages generated by the local distributed system, or messages of other distributed systems obtained by the local distributed system from the message processing component. The message processing component can distinguish whether a message obtained from a local distributed system is a message generated by the local distributed system or a message generated by other distributed systems.
Another embodiment of the present application provides a data synchronization method, which describes data processing of a message processing component. In the method provided by this embodiment, different ways of handling different messages acquired from the local distributed system are included to avoid cyclic writing of messages between the distributed systems. As shown in fig. 3, the method includes:
step S210: and acquiring messages in message queues of other distributed systems and messages in a local distributed system, wherein the local distributed system is a distributed system corresponding to the message processing component, and the other distributed systems are distributed systems outside the local distributed system. When a message from the local distributed system is acquired, step S230 is executed; when a message from another distributed system is acquired, step S220 is performed.
The message processing component acquires the message, and when new messages exist in the message queues of other distributed systems, the message in the message queues of other distributed systems can be acquired; if there is a new write message in the local distributed system, the message in the local distributed system may be acquired. And writing the new messages in the other distributed systems into the message queues for the message processing components of the other distributed systems by the messages generated in the corresponding distributed systems.
Specifically, the message processing component monitors the messages in the message queues of other distributed systems, monitors that new messages exist in the message queues of other distributed systems, and obtains the new messages from the message queues with the new messages.
When the local distributed system has message writing, the message processing component is written at the same time, so that the message processing component acquires the message of the local distributed system.
Step S220: and if the message is the message in the message queue of other distributed systems, writing the message into the local distributed system.
The message processing component acquires the messages in the message queues of other distributed systems, writes the messages into the local distributed system, and realizes the synchronization of data.
Step S230: when a message from the local distributed system is acquired, judging whether the message is generated by the local distributed system or not. If yes, go to step S240; if not, go to step S250.
Among the messages from the local distributed system acquired by the message processing component, the messages may be messages generated by the local distributed system, i.e. the messages are generated by the local distributed system; when messages of other distributed systems are written to the local distributed system, the local distributed system rewrites the messages to the message processing component. For example, the message generated by the local distributed system is a message written by the local distributed system, and in order to ensure that the message generated by the local distributed system is sent to a message queue of the local distributed system, the message written by the local distributed system is written to a message processing component; however, the messages acquired by the message processing component from the message queues of other distributed systems are also written into the local distributed system for synchronization, and the messages written into the local distributed system are written into the message processing component, so that the messages of the local distributed system received by the message processing component also comprise the messages of other distributed systems.
Therefore, when the message processing component acquires the message of the local distributed system, the acquired message may be the message generated by the local distributed system, or may be the local distributed system to which the message generated by other distributed systems is written, and the final generation source of the message needs to be determined and processed correspondingly.
When the message processing component receives a message of the local distributed system, it determines whether the message is a message generated by the local distributed system.
When the message is generated, the identity mark can be written in, and the identity mark represents a distributed system for generating the message, so that the identity mark is carried in the message, and the distributed system for generating the message can be determined through the identity mark.
When judging whether the message is generated by the local distributed system, judging whether the identity carried in the message is the corresponding identity of the local distributed system, if so, determining that the message is generated by the local distributed system, and if not, determining that the message is not generated by the local distributed system.
For example, the message generated by the distributed system a is written into the identity mark a, and the message generated by the distributed system B is written into the identity mark B, so that the message carrying the identity mark a can be determined to be the message generated by the distributed system a, and the message carrying the identity mark B is determined to be the message generated by the distributed system B. When receiving the message of the distributed system A, the message processing component of the distributed system A judges whether the message carries an identity mark A, if so, the message is determined to be generated by a local distributed system; if not, it is determined that the message is not a message generated by the local distributed system, but is a message generated by other distributed systems.
In the embodiment of the present application, the manner of determining whether the acquired message is a message generated by a local distributed system or a message generated by other distributed systems is not limited, and may be differentiated in other manners. For example, the message processing component may add a special tag to a message acquired in the message queues of other distributed systems before or as the message is written to the local distributed system. When the message processing component obtains a message from the local distributed system, the message can be identified, whether the special mark exists in the message or not is determined, and if so, the message is determined to be generated for other distributed systems; if not, the message is determined to be generated for the local distributed system.
Step S240: if yes, the message is sent to a message queue of the local distributed system.
If the message acquired from the local distributed system is a message generated by the local distributed system, the message is sent to a message queue of the local distributed system, so that the message sent to the message queue of the local distributed system is ensured to be the message generated by the local distributed system, and the messages generated by other distributed systems are prevented from returning to other distributed systems through the message queue.
Step S250: if not, discarding the message.
If the message obtained from the local distributed system is not the message generated by the local distributed system, the message can be discarded and no message queue is put in, so that the cyclic writing among the distributed systems is avoided.
In the embodiment of the application, in the messages of the local distributed system acquired by the message processing component, the messages which are not generated by the local distributed system are discarded, and the messages generated by the local distributed system are written into the message queue of the local distributed system, so that the messages generated by the local distributed system are acquired by the message processing components of other distributed systems when the messages are acquired from the message queue of the local distributed system and written into the message queue of the local distributed system.
The application also provides a data synchronization method applied to the multi-activity system. As shown in fig. 4, the method includes the following steps.
Step S310: the distributed system sends messages to the correspondingly configured message processing components.
Each distributed system sends a message requiring data synchronization to a message processing component corresponding to the distributed system configuration. Alternatively, the message sent to the message handling component that requires data synchronization may include only messages that the distributed system needs to synchronize to other distributed systems, i.e., only messages generated by the distributed system. Alternatively, the message to be sent to the message processing component for data synchronization may include a message to be synchronized within the distributed system, including messages generated by the distributed system and messages generated by other distributed systems and synchronized to the distributed system.
Step S320: the message processing component obtains messages in message queues of other distributed systems and messages in a local distributed system, wherein the local distributed system is a distributed system corresponding to the message processing component, and the other distributed systems are distributed systems except the local distributed system.
When there is a message within the local distributed system sent to the message processing component, the message processing component obtains the messages of all local distributed systems. When the message queues of other distributed systems are monitored to have messages, the message processing component acquires the messages in the other distributed systems.
Step S330: if the message acquired by the message processing component is a message in a message queue of other distributed systems, the message processing component writes the message into the local distributed system; and if the message acquired by the message processing component is a message generated by the local distributed system, sending the message to a message queue of the local distributed system.
The message processing component obtains the message generated by the local distributed system and can write the message into the message queue of the local distributed system.
The message processing component may write to the local distributed system when it obtains messages in the message queues of other distributed systems.
According to the data synchronization method provided by the embodiment of the application, each distributed system can send the information to be synchronized to the information processing component, and the information processing component can acquire the information of the distributed system which is correspondingly configured and also can acquire the information in the information queues which are corresponding to other distributed systems. When the message processing component acquires the messages in the message queues corresponding to other distributed systems, the messages can be written into the distributed systems corresponding to the message processing component, and the data of the other distributed systems are synchronized to the distributed systems corresponding to the message processing component. The message processing component obtains the message generated by the corresponding distributed system and can write the message into the message queue corresponding to the distributed system, thereby synchronizing the message generated by the distributed system to other distributed systems.
In the data synchronization method applied to the multi-active system for data synchronization, the distributed system can send the information needed to be synchronized in the distributed system to the corresponding information processing component, the information processing component distinguishes the information generated by the distributed system from the information generated by other distributed systems and synchronizes the information generated by other distributed systems to the distributed system, and the information generated by the distributed system is sent to the corresponding information queue.
In particular, the message generated by the distributed system is a written message for the distributed system; the distributed system receives messages acquired by the message processing component from the message queues of other distributed systems, and for distributed applications, writes messages, which are also a type of written messages. The distributed system sends the written message to the message processing component which is correspondingly configured, and the written message sent to the message processing component comprises the message sent to the distributed system by the message processing component and the message generated by the distributed system according to the writing request. The message of the local distributed system acquired by the specific message processing component is a message generated by the local distributed system or a message synchronized by other distributed systems, and the message is determined according to the actual execution process.
In this embodiment, the message processing component, upon retrieving a message from the local distributed system, may determine whether the message is a message generated by the local distributed system. If yes, the message is sent to a message queue of a local distributed system; if not, discarding the message.
Additionally, optionally, in an embodiment of the present application, each distributed system may write an identity in the message when generating the message, where the identity represents the distributed system that generated the message. The message processing component judges whether the identity mark carried in the message is a mark corresponding to a local distributed system; if yes, determining that the message is generated by the local distributed system, and if not, determining that the message is not generated by the distributed system.
In a particular embodiment, the distributed system generated message may be written in accordance with a write request. For example, the distributed system receives a write request of the client, after determining the write, the distributed system generates a message according to the write request, and adds an identity of the distributed system in the generated message.
In addition, optionally, the identity may be included in the configuration information of the distributed system, so that when the message is generated, writing is performed according to the identity in the configuration information.
Optionally, in the configuration information of the distributed system, the address of the message queue of the local distributed system and the address of the message queue of the other distributed systems may be further included. Analyzing the configuration information and setting the configuration information into a message processing component, wherein the message processing component can determine an identity corresponding to the local distributed system; writing the message into the message queue of the local distributed system according to the address of the message queue of the local distributed system; messages may be read from the message queues of other distributed systems based on the addresses of the message queues of the other distributed systems.
Optionally, in the configuration information of the distributed system, the topic name (topic) of the transmitted message and the topic name of the subscribed message may be further included. Each message belongs to a topic, the topic name of the transmitted message represents the topic name of the message to be transmitted to the message queue of the local distributed system, and the message to be transmitted can be determined according to the topic name; the message to be subscribed to may be determined based on the topic name of the subscribed message.
The message processing component may write to the local distributed system upon retrieving a message in the message queue of the other distributed system. When the local distributed system receives the message written by the message processing component, the message can be synchronized according to the synchronization rule of the local distributed system.
Fig. 5 shows an example of a multi-active system including two distributed systems for data synchronization. The distributed system comprises a distributed system A and a distributed system B, wherein the message processing component A and the message queue A are configured corresponding to the distributed system A, and the message processing component B and the message queue B are configured corresponding to the distributed system B. As shown in fig. 5, when the distributed system a generates the message a, the message a is written to the message processing component a, which writes the message a to the message queue a. The message processing component B monitors the message A in the message queue A, reads the message A from the message queue A and writes the message A into the distributed system B. And the distributed system B receives the message A and synchronizes in the distributed system B to realize the synchronization of the message A in the distributed system B. In addition, during the synchronization process, the message a is written to the message processing component B. The message processing component B judges that the message A is not generated by the distributed system B, discards the message A and does not process the message A, so that the message A is prevented from being written into the distributed system A again.
And when the information B generated by the distributed system B is synchronized to the information processing component A, the information processing component discards the information B and does not process the information B, so that the synchronization of the information B to the distributed system A is realized, the repeated writing is avoided, and the repeated writing is not repeated.
In addition, when three or more distributed systems are included in the multi-active system, the processing procedure of each distributed system is the same as described above. For example, based on the above example, the distributed system C is further included, and the message processing component C and the message queue C configured corresponding to the distributed system C. When a new message A exists in the message queue A, the message processing component C reads the message A from the message queue A and synchronizes the message A; when there is a new message B in the message queue B, the message processing component C reads the message B from the message queue B and synchronizes.
In one embodiment, messages in the message queue are dequeued after they are read, and the message queue is no longer included. In this embodiment, multiple message queues may be configured for each distributed system in the multi-active system, where the number of message queues may be equal to the number of other distributed systems, or the total number of distributed systems in the multi-active system minus one, and the message in each message queue is read by one of the message processing components of the other distributed systems, and the message processing components of different distributed systems read different message queues in the same distributed system. When the message processing component writes the message generated by the local distributed system into the message queues of the local distributed system, one part is written into each message queue. And a plurality of message queues correspondingly configured by one distributed system are respectively read by the message processing components of other distributed systems, so that the message processing components of each distributed system can read new messages. For example, in a multi-active system including three distributed systems A, B and C, two message queues A1 and A2 are configured corresponding to a, when the distributed system generates a new message a, the message processing component a writes the message a into the message queues A1 and A2, respectively, the message processing component of the distributed system B may read the newly generated message a from the message queue A1, and the distributed system C may read the newly generated message a from the message queue A2. Message queue configuration and data processing for distributed systems B and C are the same.
In another embodiment, a message queue is configured for each distributed system, and the messages in the message queue are read by the message processing components of all other available distributed systems and then cleared, so that the synchronization of the data among the distributed systems is ensured.
In the embodiment of the application, the information processing component filters the synchronous information in the distributed system, the information generated by the distributed system is written into the information queue of the distributed system, and the information processing component reads the information in the information queues of other distributed systems, so that the information generated by each distributed system is synchronous to the other distributed systems, and the information generated by each distributed system has backup in each distributed system in the multi-active system.
In addition, when the distributed system in the multi-active system has a fault, after the fault distributed system is recovered, the messages which are generated by other distributed systems and are not synchronized to the distributed system can be read from the message queues of other distributed systems, and finally, the data consistency of the distributed systems is realized.
In addition, in the embodiment of the application, a storage space can be configured corresponding to each distributed system, such as MySQL, ES and Redis, and the message processing component can write the message generated by the local distributed system into the storage space for persistent storage, so that the follow-up export, analysis and the like are facilitated. The message processing component may also write messages obtained from message queues of other distributed systems into the memory space for persistent storage.
In the embodiment of the present application, the distributed system is not limited to the system, and the embodiment of the present application is described with a Zookeeper system. In the Zookeeper system, participants (particidant) and observers (observers) are included. As shown in fig. 6, the participant (particiant) includes a leader (leader) and a follower (follower), the leader is a role finally selected from the participants, responsible for making the initiation and resolution of the vote, and updating the system state; followers are roles of non-leaders in participants, used for receiving client requests and returning results to the clients, and participate in voting in a main selecting process. The observer can accept the client connection and forward the write request to the leader, but the observer does not participate in the voting process, only receives the voting result, synchronizes the state of the leader, and the observer aims to expand the system and improve the reading speed. Each observer and follower connected to the leader are defined as service instances.
As shown in fig. 6, a client (client) is a client of the Zookeeper system, and is a request initiator, and a leader can write a message according to a request initiated by the client.
In the multi-active system, a plurality of ZooKeeper systems can be included, and two ZooKeeper systems are taken as examples for the embodiment of the application. The two ZooKeeper system clusters are used for voting in the respective machine room, and all client communication is performed in the machine room to ensure the maximization of the read-write performance.
And respectively configuring a message processing component and a message queue corresponding to each ZooKeeper system.
In the Zookeeper system, since the observer nodes do not participate in voting, the new addition of all the observer nodes does not affect the throughput of the Zookeeper system cluster, and the pressure of the observer nodes is relatively small, so the message processing component can only send and consume the message in the observer nodes, namely the message processing component can not be independently started, is integrated in one observer node of each Zookeeper system, and newly adds relevant configuration information in the configuration file of the Zookeeper, analyzes the configuration information and sets the configuration information in an object related to the message processing component.
When a client writes data to the Zookeeper system, a write request is sent to a connected service instance. The service instance of the client connection is the observer or follower connected to the client.
The service instance receiving the client write request forwards the request to the leader, and when the leader determines that the write according to the request is needed according to the voting result of the follower, the write is started, and the write is performed to each service instance. The message processing component is configured in the watcher node to receive the written message. The message processing component receives the message from the local Zookeeper system, determines that the message is generated by the local Zookeeper system, and sends the message to a message queue corresponding to the Zookeeper system.
When one message processing component monitors that the message queues of other Zookeeper systems have messages, the message is read from the message queues of other Zookeeper systems, analyzed and sent to a leader for writing. The leader writes to a message processing component configured in one observer node when writing the message to the various service instances for synchronization. The message processing component receives the message from the Zookeeper system, determines that the message is not generated by the local Zookeeper system, discards the message, and places the message in a loop for writing between the distributed systems.
In a multi-activity system comprising a plurality of Zookeeper systems, the final consistency of the transaction information of each cluster is realized by adding a message processing component, so that the multi-activity system is more suitable for reading, writing and finding services, and the like. The double activities of the ZooKeeper clusters can be really realized, and one city, one machine room or the clusters can still provide services for the outside when the machine room or the clusters fail. And the reading and writing are in the ZooKeeper system, and the performance of the ZooKeeper system can be exerted to the maximum.
In addition, the transaction information is buffered and stored through the information queue, so that as long as one cluster fails, the transaction information of a normal cluster can be normally written into the information queue, and after the clusters are recovered, the two clusters are recovered from the last information position to finally be consistent.
In addition, in the embodiment of the application, the deployment complexity is increased by relying on external components such as a RocketMQ message queue, and the RcoketMQ needs to be done by a client in message idempotent, so that other middleware needs to be relied on to do message idempotent, and the message idempotent is realized by designing message storage and consumption. For example, message idempotent is realized through a third party such as Redis; for another example, a time stamp is added when the message is generated, because the writing of the message is sequential and the generation time of different messages is different, the message processing component acquires the message from the message queue, if the message with the same time is acquired and executed, or the message with later time is acquired and executed, the message in the message queue is not acquired any more, or the message in the message queue is discarded.
The embodiment of the application also provides a data synchronization device 500. The data synchronization device may be a message processing component, a software program integrated in the message processing component, or a hardware device integrated in the message processing component. The data synchronization device is used for data synchronization of a multi-activity system, the multi-activity system comprises a plurality of distributed systems, message processing components configured corresponding to each distributed system and message queues configured corresponding to each distributed system, and the method is applied to any message processing component in the multi-activity system. As shown in fig. 7, the apparatus 500 includes: a message obtaining module 510, configured to obtain a message in a message queue of another distributed system and a message in a local distributed system, where the local distributed system is a distributed system corresponding to the message processing component, and the other distributed systems are distributed systems other than the local distributed system; a message writing module 520, configured to write the message into the local distributed system if the message is a message in a message queue of other distributed systems; and the message sending module 530 is configured to send the message to a message queue of the local distributed system if the message is a message generated by the local distributed system.
Optionally, the apparatus may further include a determining module configured to determine, when a message from the local distributed system is acquired, whether the message is a message generated by the local distributed system. If so, the message sending module 530 sends the message to a message queue of the local distributed system; if not, the message sending module 530 discards the message.
Optionally, each message is generated with an identity written to it, said identity representing the distributed system that generated the message. The judging module can judge whether the identity mark carried in the message is the mark corresponding to the local distributed system. If yes, determining that the message is generated by the local distributed system, and if not, determining that the message is not generated by the distributed system.
It will be apparent to those skilled in the art that, for convenience and brevity of description, reference may be made to the above-described embodiments of the method; the specific working process of the above-described device and module may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
In several embodiments provided by the present application, the coupling of the modules to each other may be electrical, mechanical, or other.
In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module. The integrated modules may be implemented in hardware or in software functional modules. The modules may be configured in different electronic devices or may be configured in the same electronic device, and embodiments of the present application are not limited.
Referring to fig. 8, a block diagram of a computer readable storage medium according to an embodiment of the present application is shown. The computer readable storage medium 700 has stored therein program code that can be invoked by a processor to perform the methods described in the method embodiments described above.
The computer readable storage medium 700 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 700 comprises a non-volatile computer readable medium (non-transitory computer-readable storage medium). The computer readable storage medium 700 has memory space for program code 710 that performs any of the method steps described above. The program code can be read from or written to one or more computer program products. Program code 710 may be compressed, for example, in a suitable form.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the application has been described in detail with reference to the foregoing embodiments, it will be appreciated by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not drive the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims (10)
1. A method of data synchronization for a multi-active system, the multi-active system comprising a plurality of distributed systems, a message processing component configured for each distributed system, and a message queue configured for each distributed system, the method being applied to any one of the message processing components in the multi-active system, the method comprising:
acquiring messages in message queues of other distributed systems and messages in a local distributed system, wherein the local distributed system is a distributed system corresponding to the message processing component, and the other distributed systems are distributed systems except the local distributed system;
If the message is the message in the message queue of other distributed systems, writing the message into the local distributed system;
if the message is written according to the request of the client, determining that the message is generated by the local distributed system, sending the message to a message queue of the local distributed system, and writing the message into the message processing component.
2. The method according to claim 1, wherein the method further comprises:
when a message from a local distributed system is acquired, judging whether the message is generated by the local distributed system or not;
if yes, executing the step of sending the message to a message queue of a local distributed system;
if not, discarding the message.
3. The method of claim 2, wherein each message is generated with an identity written thereto, the identity representing a distributed system that generated the message, and wherein determining whether the message is a local distributed system generated message comprises:
judging whether the identity mark carried in the message is a mark corresponding to a local distributed system or not;
if yes, determining the message as a message generated by a local distributed system,
If not, it is determined that the message is not a message generated by the distributed system.
4. A method of data synchronization for a multi-active system, the multi-active system comprising a plurality of distributed systems, a message processing component configured for each distributed system, and a message queue configured for each distributed system, the method comprising:
the distributed system sends messages to the correspondingly configured message processing components,
the message processing component acquires messages in message queues of other distributed systems and messages in a local distributed system, wherein the local distributed system is a distributed system corresponding to the message processing component, and the other distributed systems are distributed systems except the local distributed system;
if the message acquired by the message processing component is a message in a message queue of other distributed systems, the message processing component writes the message into the local distributed system; if the message is written according to the request of the client, determining that the message acquired by the message processing component is a message generated by a local distributed system, sending the message to a message queue of the local distributed system, and writing the message into the message processing component.
5. The method of claim 4, wherein each message is generated with an identity written thereto, the identity representing a distributed system that generated the message, the method further comprising:
the message processing component judges whether the identity mark carried in the message is a mark corresponding to a local distributed system;
if yes, determining the message as a message generated by a local distributed system,
if not, it is determined that the message is not a message generated by the distributed system.
6. The method of claim 5, wherein each message processing component is deployed in a corresponding distributed system, the method further comprising:
the distributed system generates a message according to the writing request, and adds an identity of the distributed system in the generated message;
the distributed system sends a message to a correspondingly configured message processing component, comprising:
the distributed system sends the written message to the message processing component with corresponding configuration, wherein the written message comprises the message sent to the distributed system by the message processing component and the message generated by the distributed system according to the writing request.
7. The method of claim 4, wherein the multiple activity system comprises two distributed systems.
8. A data synchronization apparatus for data synchronization of a multi-active system, the multi-active system including a plurality of distributed systems, a message processing component configured for each distributed system, and a message queue configured for each distributed system, the apparatus being applied to any one of the message processing components in the multi-active system, the apparatus comprising:
the message acquisition module is used for acquiring messages in message queues of other distributed systems and messages in a local distributed system, wherein the local distributed system is a distributed system corresponding to the message processing component, and the other distributed systems are distributed systems except the local distributed system;
the message writing module is used for writing the message into the local distributed system if the message is the message in the message queue of other distributed systems;
and the message sending module is used for determining that the message is generated by the local distributed system if the message is written according to the request of the client, sending the message to a message queue of the local distributed system, and writing the message into the message processing component.
9. A multiple-activity system is characterized by comprising a plurality of distributed systems, a message processing component corresponding to each distributed system configuration and a message queue corresponding to each distributed system configuration, wherein,
the distributed system is operable to send messages to a correspondingly configured message processing component,
the message processing component is used for acquiring messages in message queues of other distributed systems and messages in a local distributed system, the local distributed system is a distributed system corresponding to the message processing component, and the other distributed systems are distributed systems except the local distributed system;
if the message acquired by the message processing component is a message in a message queue of other distributed systems, the message processing component is used for writing the message into the local distributed system; if the message is written according to the request of the client, determining that the message acquired by the message processing component is a message generated by a local distributed system, and simultaneously writing the message into the message processing component, wherein the message processing component is used for sending the message to a message queue of the local distributed system.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a program code, which is callable by a processor for performing the method according to any of the claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010773658.1A CN111953760B (en) | 2020-08-04 | 2020-08-04 | Data synchronization method, device, multi-activity system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010773658.1A CN111953760B (en) | 2020-08-04 | 2020-08-04 | Data synchronization method, device, multi-activity system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111953760A CN111953760A (en) | 2020-11-17 |
| CN111953760B true CN111953760B (en) | 2023-08-11 |
Family
ID=73339397
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010773658.1A Active CN111953760B (en) | 2020-08-04 | 2020-08-04 | Data synchronization method, device, multi-activity system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111953760B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108322358A (en) * | 2017-12-15 | 2018-07-24 | 北京奇艺世纪科技有限公司 | Strange land distributed message transmission, processing, consuming method and device mostly living |
| CN109658225A (en) * | 2018-11-29 | 2019-04-19 | 中国人民银行清算总中心 | Multicenter distributed internet banking system and service message processing method mostly living |
| CN110365750A (en) * | 2019-06-25 | 2019-10-22 | 苏宁云计算有限公司 | Service registration system and method |
| CN110795503A (en) * | 2019-10-18 | 2020-02-14 | 北京达佳互联信息技术有限公司 | Multi-cluster data synchronization method and related device of distributed storage system |
| CN111444274A (en) * | 2020-03-26 | 2020-07-24 | 上海依图网络科技有限公司 | Data synchronization method, data synchronization system, and apparatus, medium, and system thereof |
-
2020
- 2020-08-04 CN CN202010773658.1A patent/CN111953760B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108322358A (en) * | 2017-12-15 | 2018-07-24 | 北京奇艺世纪科技有限公司 | Strange land distributed message transmission, processing, consuming method and device mostly living |
| CN109658225A (en) * | 2018-11-29 | 2019-04-19 | 中国人民银行清算总中心 | Multicenter distributed internet banking system and service message processing method mostly living |
| CN110365750A (en) * | 2019-06-25 | 2019-10-22 | 苏宁云计算有限公司 | Service registration system and method |
| CN110795503A (en) * | 2019-10-18 | 2020-02-14 | 北京达佳互联信息技术有限公司 | Multi-cluster data synchronization method and related device of distributed storage system |
| CN111444274A (en) * | 2020-03-26 | 2020-07-24 | 上海依图网络科技有限公司 | Data synchronization method, data synchronization system, and apparatus, medium, and system thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111953760A (en) | 2020-11-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113014634B (en) | Cluster election processing method, device, equipment and storage medium | |
| CN108647104B (en) | Request processing method, server and computer readable storage medium | |
| CN111091429A (en) | Electronic bill identification distribution method and device and electronic bill generation system | |
| CN110795503A (en) | Multi-cluster data synchronization method and related device of distributed storage system | |
| CN110581782B (en) | Disaster tolerance data processing method, device and system | |
| CN109376197B (en) | Data synchronization method, server and computer storage medium | |
| US20210320977A1 (en) | Method and apparatus for implementing data consistency, server, and terminal | |
| CN109802986B (en) | Equipment management method, system, device and server | |
| CN111198662B (en) | Data storage method, device and computer readable storage medium | |
| CN108712457A (en) | Back-end server dynamic load method of adjustment and device based on Nginx reverse proxys | |
| US8832215B2 (en) | Load-balancing in replication engine of directory server | |
| CN107038192B (en) | Database disaster tolerance method and device | |
| CN111459639A (en) | Distributed task management platform and method supporting global multi-machine-room deployment | |
| CN111414241A (en) | Batch data processing method, device and system, computer equipment and computer readable storage medium | |
| CN108462756B (en) | Data writing method and device | |
| CN113703954A (en) | Message backup method and device, electronic equipment and computer storage medium | |
| CN111541762B (en) | Data processing method, management server, device and storage medium | |
| CN111045602B (en) | Cluster system control method and cluster system | |
| CN111427689B (en) | Cluster keep-alive method and device and storage medium | |
| CN111953760B (en) | Data synchronization method, device, multi-activity system and storage medium | |
| CN109587218B (en) | Cluster election method and device | |
| CN112015561A (en) | Method, apparatus and system for streaming computing services | |
| CN111614702B (en) | Edge calculation method and edge calculation system | |
| CN111541608A (en) | Network communication method, system and related device | |
| CN115396296A (en) | Service processing method and device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |