CN106657314B - Cross-data center data synchronization system and method - Google Patents
Cross-data center data synchronization system and method Download PDFInfo
- Publication number
- CN106657314B CN106657314B CN201611152922.XA CN201611152922A CN106657314B CN 106657314 B CN106657314 B CN 106657314B CN 201611152922 A CN201611152922 A CN 201611152922A CN 106657314 B CN106657314 B CN 106657314B
- Authority
- CN
- China
- Prior art keywords
- data
- message queue
- source
- destination
- data center
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/133—Protocols for remote procedure calls [RPC]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a cross-data center data synchronization system and method, which are used for synchronizing data of a source data center to a target data center, wherein the source data center and the target data center are respectively provided with a plurality of RPC clients and a plurality of RPC servers, a data handling device is arranged in the target data center, and the system also comprises a data handling management device; the data carrying management device is used for managing a source message queue and a destination message queue and controlling the data carrying device to be started or closed to acquire data from the source message queue; and the data handling device is used for acquiring data from the source message queue and storing the acquired data into the destination message queue. The invention can improve the flexibility and reliability of data synchronization.
Description
Technical Field
The invention relates to the technical field of computers and internet, in particular to a cross-data center data synchronization system and a cross-data center data synchronization method.
Background
With the advent of the internet era, interactive websites facing general internet users, such as social networks, microblogs, location services, and the like, are rapidly emerging, and various websites provide interactive services based on the internet and wireless networks to hundreds of millions of users. All the internet users around the world carry out various interactions every day, and various data are manufactured at any time, and the quantity of the data is multiple times of that of the data in the single era.
In order to reliably store the data, the internet company distributes the servers to different data centers in multiple cities, and simultaneously backs up the data in the different data centers, so that the data synchronization between the different data centers is needed. For a large amount of data related to the time series, when the data center synchronization is performed, the synchronization needs to be performed in order according to the generation time of the data, and the accuracy of the original data is damaged by the disordered data. Also, due to the large number of heterogeneous data storage systems, the data synchronization system needs to support multiple languages well.
The existing data synchronization scheme mainly comprises two modes of proprietary data synchronization and application layer data synchronization.
In the dedicated data synchronization mode, databases supporting data synchronization across data centers are provided, such as Mysql, Hbase, and the like, and data synchronization of different data centers is realized by depending on such specific databases. This approach may limit the user's data to be stored only in a particular data storage system, the user may not be able to flexibly select a suitable data storage system, and the data may not be synchronized between heterogeneous data sources.
In an application layer data synchronization mode, application layer double writing and application layer chain multi-writing are required, and data are written to a plurality of data centers through an application layer. Under double-write and chained multi-write, a write success can only be confirmed if data is written successfully in multiple data centers at the same time. However, under the condition of large network jitter, the performance of an application layer is seriously affected by double writing and chained multi-writing, and the ordering of data cannot be guaranteed under abnormal conditions.
Disclosure of Invention
In order to improve flexibility and reliability of data synchronization, embodiments of the present invention provide a cross-data center data synchronization system and method.
According to an aspect of an embodiment of the present invention, a cross-data center data synchronization system is provided, configured to synchronize data of a source data center to a destination data center, where the source data center and the destination data center are both provided with a plurality of RPC clients and a plurality of RPC servers, and the destination data center is provided with a data handling device, and the system further includes a data handling management device; the data carrying management device is used for managing a source message queue and a destination message queue and controlling the data carrying device to be started or closed to acquire data from the source message queue; and the data carrying device is used for acquiring data from the source message queue and storing the acquired data into the destination message queue.
Preferably, the data handling management device is further configured to set a mapping correspondence between the source message queue and the destination message queue; and the data carrying device stores the acquired data into a destination message queue corresponding to the source message queue.
Preferably, the source message queue is set in an RPC server of the source data center, and the destination message queue is set in an RPC server of the destination data center.
Preferably, the source message queue and the destination message queue are distributed queues based on log storage.
Preferably, the data handling device includes a message producing end and a message consuming end, the message consuming end is configured to obtain data from a source message queue, and the message producing end is configured to store the data obtained by the message consuming end into a destination message queue.
Preferably, a communication protocol is defined between the RPC client and the RPC server through IDL, and the RPC client and the RPC server are compiled into language codes through IDL files, so that data communication of the language is supported in the data center.
According to another aspect of the embodiment of the invention, a cross-data center data synchronization method is provided, which is used for synchronizing data of a source data center to a destination data center, wherein the source data center and the destination data center are respectively provided with a plurality of RPC clients and a plurality of RPC servers, and the method comprises the following steps: managing a source message queue of the source data center and a destination message queue of the destination data center, and controlling a data handling device which starts or closes the destination data center to acquire data from the source message queue; and the data handling device acquires data from the source message queue and stores the acquired data into the destination message queue.
Preferably, the method further comprises: setting a mapping corresponding relation between the source message queue and the target message queue; and the data carrying device stores the acquired data into a destination message queue corresponding to the source message queue.
Preferably, the source message queue is arranged in an RPC server of the source data center, and the destination message queue is arranged in an RPC server of the destination data center.
Preferably, the source message queue and the destination message queue are distributed queues based on log storage.
Preferably, the data handling device includes a message producing end and a message consuming end, the message consuming end is configured to obtain data from a source message queue, and the message producing end is configured to store the data obtained by the message consuming end into a destination message queue.
Preferably, the method further comprises: and defining a communication protocol between the RPC client and the RPC server by adopting IDL, and compiling the communication protocol into a language code through an IDL file to realize that the data communication of the language is supported in a data center.
Therefore, the cross-data center data synchronization system provided by the embodiment of the invention realizes multi-language support under heterogeneous data sources by introducing the common RPC protocol, so that the flexibility of data synchronization is enhanced, and the ordered synchronization of time-sequenced (data are collected according to different time points) related data is realized by introducing the distributed message queue system based on log storage.
Drawings
FIG. 1 is a schematic structural diagram of a cross-data center data synchronization system according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the logical processing of a cross-data center data synchronization system provided by one embodiment of the present invention;
fig. 3 is a flowchart of a method for synchronizing data across data centers according to an embodiment of the present invention.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
In view of the defects of the existing data synchronization scheme of the cross-data center, the embodiment of the invention provides a highly reliable and more flexible data synchronization system of the cross-data center, wherein the RPC technology is adopted to realize the support of multiple languages, and the message queue mode is adopted to realize the orderliness and reliability of data synchronization.
Referring to fig. 1, a schematic structural diagram of a cross-data center data synchronization system according to an embodiment of the present invention is provided.
The system comprises a source data center 1, a destination data center 2 and a data handling management device 3, further, the source data center 1 comprises a plurality of RPC clients 101 and a plurality of RPC servers 102, and an active message queue 1021 is arranged in the RPC servers 102. Similarly, the destination data center 2 includes an RPC client 201 and a plurality of RPC servers 202, and a destination message queue 2021 is provided at the RPC server 202, and particularly, the destination data center 2 further includes a data handling device 203, and the data handling device 203 may further include a data consuming end 2031 and a data producing end 2032; the data transport management device 3 is a control end of the data synchronization process, and the data synchronization is started and executed by controlling the source message queue 1021, the destination message queue 2021 and the data transport device 203; the data transfer device 203 is a main body that completes data synchronization, and completes a process of data synchronization.
From a logical functional division, the entire system can be divided into three components (subsystems):
the first part is an RPC subsystem, which includes an RPC client and an RPC server. The RPC subsystem enables support for multiple languages because the communication protocol between the RPC client and the RPC server can be defined by IDL. Therefore, for a new language, the IDL file can be compiled into the code of the language through the compiler of the corresponding language, so that the support of the language is realized. Wherein, idl (interface description language) is an interface description language, which describes the interface in a neutral way, so that the objects running on different platforms and the programs written in different languages can communicate with each other; for example, a first component is written in C + + and a second component is written in Java, and the first component and the second component can communicate with each other via an IDL protocol interface.
The second part is a message queue subsystem, which includes a source message queue 1021 and a destination message queue 2021. In a preferred mode, a message queue based on a distributed message queue system is adopted, so that reliable and ordered storage of data can be realized through a message queue subsystem, because the distributed message queue system performs persistent storage of data through log files, and the log files are characterized by reliability and order.
The third part is a data handling subsystem, which includes a data handling device 203 and a data handling management device 3, where the data handling device 203 implements a synchronization function of data between different data centers, and specifically, the data handling device 203 is composed of a message consumption end 2031 and a message production end 2032 of a message queue. The message consumption end 2031 is responsible for consuming messages from the source data center 1, and the message production end 2032 is responsible for producing messages to the destination data center 2, thereby realizing data synchronization across data centers. The data transport management device 3, working in cooperation with the data transport device 203, can support online expansion and contraction of data transport and ensure that data is not lost during expansion and contraction. The online expansion and contraction of data transportation means that the service instance can be closed off or online without stopping the service, and data is guaranteed not to be lost. Before the service is offline, the read-write permission of the current device is closed through the data carrying device 203, the data stream is shunted to other devices, and the data loss cannot be caused by closing the current device.
The respective main components of the system are described in detail below.
A so-called data center is a complex set of systems that includes not only computer systems and other associated equipment (e.g., communication and storage systems), but also redundant data communication connections, environmental control equipment, monitoring equipment, and various security devices. A data center may be understood as a "multifunctional system, capable of housing a plurality of servers and communication devices". The source data center 1 and the destination data center 2 may be existing conventional data centers or data centers developed in the future.
As mentioned above, the RPC subsystem is composed of an RPC client and an RPC server, and the RPC client is embedded in an external system of the data center to realize communication with the data synchronization system.
Rpc (remote Procedure Call protocol), a protocol that requests services from remote computer programs over a network without knowledge of the underlying network technology. RPC employs a client/server mode. The requesting program is a client and the service providing program is a server. The RPC working process is as follows: first, the client-side calling process sends a calling message with process parameters to the service process, and then waits for a response message. On the server side, the process remains in a sleep state until the call information arrives. When a calling message arrives, the server obtains the process parameters, calculates the result, sends the reply message, then waits for the next calling message, and finally, the client calls the process to receive the reply message, obtain the process result, and then calls execution to continue.
In the embodiment of the invention, data arrives at the source message queue through the RPC server and is finally provided for the destination message queue through the data handling device. As mentioned above, the RPC subsystem has the advantage that multi-language support can be realized by embedding RPC clients in different languages. Preferably, the message queue and the RPC server are deployed on the same device (referring to fig. 1, the source message queue 1021 is deployed on the RPC server 102, and the destination message queue 2021 is deployed on the RPC server 202), so that the network overhead can be reduced. Of course, it will be understood that the source message queue 1021 may also be deployed on devices other than the RPC server 102.
In a preferred mode, both the source message queue 1021 and the destination message queue 2021 adopt a distributed queue system based on log storage, so as to realize storage of actual data in the source data center 1 and the destination data center 2. The log type storage system can meet the orderliness and reliability of data storage, and the distributed message queue cluster can dynamically expand and contract the capacity of the cluster according to the actual data volume. The distributed message queue system carries out data persistence through log files, the log files are characterized by reliability and order, and one piece of data in the log type storage system is added into the files in sequence and is persistently stored in a hard disk, so that the orderliness and the reliability can be met.
The data transfer device 203 mainly performs a function of transferring actual data, and preferably includes a message consumption end 2031 and a message production end 2032. The data-moving device 203 may be understood as a distributed messaging system.
A distributed message system is a system which simultaneously provides temporary storage of messages and distribution of messages on different computers. A typical distributed message system includes a message production end cluster, an agent cluster, and a message consumption end cluster, where a message production end (Producer) constructs messages with different topics, and sends the messages to an agent end (also called Broker) deployed with a message system service in a local area network for temporary storage, and a message consumption end (Customer) obtains a corresponding message from the message system cluster according to a topic subscribed by the Customer for subsequent processing. The distributed message system is characterized in that message services are distributed on a plurality of computers, high availability is achieved through a fault tolerance mechanism, and high throughput is supported through a load balancing strategy.
In the embodiment of the present invention, the message consumption end 2031 is responsible for consuming data from the source data center 1, and at the same time, the message production end 2032 is called to send the acquired message to the destination data center 2. In order to avoid destroying the data order during the data transportation process, it is preferable that all messages in the same queue in the source data center 1 are required to be transported to the same queue in the destination data center. The data transfer management apparatus 3 controls the mapping relationship between the source data queue 1021 and the destination data queue 2021.
Through the data handling device 203, data synchronization between the source data center 1 and the destination data center 2 can be ensured to occur only once, and the overhead of a network dedicated line is reduced. This is because the data of the source data center 1 is transferred to the destination data center 2 by the data transfer device 203, and if the destination data center 2 has a plurality of devices that all need to consume the data of the source data center 1 if the data is not transferred, each device needs to be consumed once for the same data, which is the overhead of occupying the dedicated network line for many times, and after the data is transferred by the data transfer device 203, the device of the destination data center 2 only needs to be consumed from the local data center, and does not need to occupy the dedicated network line resource.
The data transfer management device 3 may be an independent device for managing the data transfer device 203, and it is necessary to control the mapping relationship between the source data queue 1021 and the destination data queue 2021 to ensure the data order. The data transfer management apparatus 3 manages the opening and closing of the data transfer apparatus 203 during dynamic scaling (dynamic scaling refers to changes in the number of deployed servers or other devices), thereby ensuring that data is not lost. When the data transfer device 203 is activated, the data transfer management device 3 loads the correspondence between the source message queue 1021 and the destination message queue 2021, and the subsequent data synchronization process is performed based on this correspondence.
Referring to fig. 2, a schematic logic processing diagram of a cross-data center data synchronization system according to an embodiment of the present invention is provided.
The logic processing process of the cross-data center data synchronization system is as follows:
(1) starting a source message queue and a destination message queue which are deployed in a source data center and a destination data center; starting a data carrying module through the data carrying management device, waiting for data generated in the source data center, and specifying the corresponding relation between the source message queue and the target message queue;
(2) writing data of the source data center into a source message queue of the source data center;
(3) a message consumption end of the data handling module acquires data from a source message queue of a source data center and stores the data into a destination message queue in a destination data center through a message production end; and the client of the target data center acquires the synchronized data from the target message queue of the target data center.
The cross-data center data synchronization system provided by the embodiment of the invention realizes multi-language support under a heterogeneous data source by introducing the universal RPC protocol, thereby enhancing the flexibility of data synchronization, and realizes ordered synchronization of time-series related data by introducing the distributed message queue system based on log storage.
Referring to fig. 3, a flow chart of a cross-data center data synchronization method provided in an embodiment of the present invention is used for synchronizing data of a source data center to a destination data center, where the source data center and the destination data center are both provided with a plurality of RPC clients and a plurality of RPC servers, and the method includes the following steps.
S301: the method comprises the steps of managing a source message queue of a source data center and a destination message queue of a destination data center, and controlling a data handling device of the destination data center to start or stop to acquire data from the source message queue.
S302: the data handling device obtains data from the source message queue and stores the obtained data into the destination message queue.
The source data center and the destination data center involved in the above method can be understood with reference to the cross-data center data synchronization system shown in fig. 1.
The system comprises a source data center 1, a destination data center 2 and a data handling management device 3, further, the source data center 1 comprises a plurality of RPC clients 101 and a plurality of RPC servers 102, and an active message queue 1021 is arranged on the RPC server 102, similarly, the destination data center 2 comprises an RPC client 201 and a plurality of RPC servers 202, and an destination message queue 2021 is arranged on the RPC server 202, and particularly, the destination data center 2 further comprises a data handling device 203, and the data handling device 203 can further comprise a data consumption end 2031 and a data production end 2032; the data transport management device 3 is a control end of the data synchronization process, and the data synchronization is started and executed by controlling the source message queue 1021, the destination message queue 2021 and the data transport device 203; the data transfer device 203 is a main body that completes data synchronization, and completes a process of data synchronization.
From a logical functional division, the entire system can be divided into three components (subsystems):
the first part is an RPC subsystem, which includes an RPC client and an RPC server. The RPC subsystem realizes the support of multiple languages, because the communication protocol between the RPC client and the RPC server can be defined by IDL, so that for a new language, the IDL file can be compiled into the code of the language through the compiler of the corresponding language, thereby realizing the support of the language. Wherein, idl (interface description language) is an interface description language, which describes the interface in a neutral way, so that the objects running on different platforms and the programs written in different languages can communicate with each other; for example, one component is written in C + + and another component is written in Java.
The second part is a message queue subsystem, which includes a source message queue 1021 and a destination message queue 2021. In a preferred mode, a message queue based on a distributed message queue system is adopted, so that reliable and ordered storage of data can be realized through a message queue subsystem, because the distributed message queue system performs data persistence through log files, and the log files are characterized by reliability and order.
The third part is a data handling subsystem, which includes a data handling device 203 and a data handling management device 3, where the data handling device 203 implements a synchronization function of data between different data centers, and specifically, the data handling device 203 is composed of a message consumption end 2031 and a message production end 2032 of a message queue. The message consumption end 2031 is responsible for consuming messages from the source data center 1, and the message production end 2032 is responsible for producing messages to the destination data center 2, thereby realizing data synchronization across data centers. The data transport management device 3, working in cooperation with the data transport device 203, can support online expansion and contraction of data transport and ensure that data is not lost during expansion and contraction. The online expansion and contraction of data transportation means that the service instance can be closed off or online without stopping the service, and data is guaranteed not to be lost. Before the service is offline, the read-write permission of the current device is closed through the data carrying device 203, the data stream is shunted to other devices, and the data loss cannot be caused by closing the current device.
Preferably, the method further comprises: setting a mapping corresponding relation between the source message queue and the target message queue; and the data handling device stores the acquired data into a destination message queue corresponding to the source message queue.
Preferably, the source message queue is arranged in an RPC server of the source data center, and the destination message queue is arranged in an RPC server of the destination data center.
Preferably, the source message queue and the destination message queue are distributed queues based on log storage.
Preferably, the data handling device includes a message producing end and a message consuming end, the message consuming end is configured to obtain data from a source message queue, and the message producing end is configured to store the data obtained by the message consuming end into a destination message queue.
Preferably, the method further comprises: and defining a communication protocol between the RPC client and the RPC server by adopting IDL, and compiling the communication protocol into a language code through an IDL file to realize that the data communication of the language is supported in a data center.
It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the illustrated order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments of the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no particular act is required to implement the invention.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the embodiments of the invention.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.
The method and the system for data synchronization across data centers provided by the invention are described in detail, specific examples are applied in the text to explain the principle and the implementation mode of the invention, and the description of the above embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (12)
1. A cross-data center data synchronization system is used for synchronizing data of a source data center to a target data center, and is characterized in that the source data center and the target data center are both provided with a plurality of RPC clients and a plurality of RPC servers, a data handling device is arranged in the target data center, and the system further comprises a data handling management device;
the data carrying management device is used for managing a source message queue and a destination message queue and controlling the data carrying device to be started or closed to acquire data from the source message queue;
the data handling device is used for acquiring data from the source message queue and storing the acquired data into the destination message queue;
the data handling management device is further configured to control a mapping correspondence between the source message queue and the destination message queue, and manage startup and shutdown of the data handling device during dynamic expansion and contraction.
2. The system according to claim 1, wherein the data handling management device is further configured to set a mapping correspondence between the source message queue and the destination message queue;
and the data carrying device stores the acquired data into a destination message queue corresponding to the source message queue.
3. The system of claim 1, wherein the source message queue is set in an RPC server of the source data center and the destination message queue is set in an RPC server of the destination data center.
4. The system of claim 1, wherein the source message queue and the destination message queue are both log storage based distributed queues.
5. The system of claim 1, wherein the data handling device comprises a message producer and a message consumer, the message consumer is configured to obtain data from a source message queue, and the message producer is configured to store the data obtained by the message consumer into a destination message queue.
6. The system of any one of claims 1-5, wherein the RPC client and the RPC server define a communication protocol via IDL, and are compiled into language code via IDL files, so as to support data communication of the language in the data center.
7. A cross-data center data synchronization method is used for synchronizing data of a source data center to a target data center, and is characterized in that the source data center and the target data center are provided with a plurality of RPC clients and a plurality of RPC servers, and the method comprises the following steps:
managing a source message queue of the source data center and a destination message queue of the destination data center, and controlling a data handling device which starts or closes the destination data center to acquire data from the source message queue;
the data handling device acquires data from the source message queue and stores the acquired data into the destination message queue;
the step of managing the source message queue of the source data center and the destination message queue of the destination data center, and controlling the data handling device of the destination data center to start or stop acquiring data from the source message queue comprises:
and controlling the mapping corresponding relation between the source message queue and the destination message queue, and managing the starting and the closing of the data handling device during dynamic expansion and contraction.
8. The method of claim 7, further comprising: setting a mapping corresponding relation between the source message queue and the target message queue;
and the data carrying device stores the acquired data into a destination message queue corresponding to the source message queue.
9. The method of claim 7, wherein the source message queue is located in an RPC server of the source data center and the destination message queue is located in an RPC server of the destination data center.
10. The method of claim 7, wherein the source message queue and the destination message queue are both log storage based distributed queues.
11. The method of claim 7, wherein the data handling device comprises a message producer and a message consumer, the message consumer is configured to obtain data from a source message queue, and the message producer is configured to store the data obtained by the message consumer into a destination message queue.
12. The method according to any one of claims 7-11, further comprising:
and defining a communication protocol between the RPC client and the RPC server by adopting IDL, and compiling the communication protocol into a language code through an IDL file to realize that the data communication of the language is supported in a data center.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611152922.XA CN106657314B (en) | 2016-12-14 | 2016-12-14 | Cross-data center data synchronization system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611152922.XA CN106657314B (en) | 2016-12-14 | 2016-12-14 | Cross-data center data synchronization system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106657314A CN106657314A (en) | 2017-05-10 |
| CN106657314B true CN106657314B (en) | 2020-10-27 |
Family
ID=58822114
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611152922.XA Active CN106657314B (en) | 2016-12-14 | 2016-12-14 | Cross-data center data synchronization system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106657314B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108573006B (en) * | 2017-06-06 | 2021-03-26 | 北京金山云网络技术有限公司 | Cross-machine-room data synchronization system, method and device and electronic equipment |
| CN107180116A (en) * | 2017-06-28 | 2017-09-19 | 努比亚技术有限公司 | A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium |
| CN107330089B (en) * | 2017-07-04 | 2020-06-23 | 中国联合网络通信集团有限公司 | Cross-network structured data collection system |
| CN107404540B (en) * | 2017-09-01 | 2021-06-18 | 北京奇虎科技有限公司 | Data synchronization method and device |
| CN107920136B (en) * | 2017-12-29 | 2019-09-17 | Oppo广东移动通信有限公司 | Data synchronization control method, device and server |
| CN108256002B (en) * | 2017-12-31 | 2021-05-11 | Oppo广东移动通信有限公司 | Cross-machine-room data synchronization method, device, system and server |
| CN108762947A (en) * | 2018-05-16 | 2018-11-06 | 武汉微创光电股份有限公司 | A kind of distributed system remote invocation method, apparatus and system |
| CN110865985B (en) * | 2018-08-28 | 2024-04-16 | 阿里巴巴新加坡控股有限公司 | Data synchronization method, device, electronic equipment and storage medium |
| CN109165261A (en) * | 2018-10-10 | 2019-01-08 | 深圳市欧瑞博科技有限公司 | Method of data synchronization, device, server and computer storage medium |
| CN109815035A (en) * | 2019-02-27 | 2019-05-28 | 深信服科技股份有限公司 | A kind of message delivery method and relevant apparatus of message queue |
| CN113296971B (en) * | 2020-07-14 | 2024-04-19 | 阿里巴巴集团控股有限公司 | Method, device and equipment for expanding and shrinking volume and processing message queue |
| CN111930843B (en) * | 2020-08-11 | 2023-07-28 | 中国工商银行股份有限公司 | Database-based data system and processing method |
| CN112818054B (en) * | 2020-10-15 | 2022-05-06 | 广州南天电脑系统有限公司 | Data synchronization method and device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101014002A (en) * | 2006-12-12 | 2007-08-08 | 华为技术有限公司 | Cluster message transmitting method and distributed cluster system |
| CN103546557A (en) * | 2013-10-22 | 2014-01-29 | 乐视致新电子科技（天津）有限公司 | Data interaction method of platform-level services, service platform and systems |
| EP2899947A1 (en) * | 2013-08-22 | 2015-07-29 | Inspur Electronic Information Industry Co., Ltd. | Component oriented hybrid cloud operating system architecture and communication method thereof |
| CN105262831A (en) * | 2015-10-30 | 2016-01-20 | 北京奇艺世纪科技有限公司 | Method, device, and synchronizing system for synchronizing data among memory systems |
-
2016
- 2016-12-14 CN CN201611152922.XA patent/CN106657314B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101014002A (en) * | 2006-12-12 | 2007-08-08 | 华为技术有限公司 | Cluster message transmitting method and distributed cluster system |
| EP2899947A1 (en) * | 2013-08-22 | 2015-07-29 | Inspur Electronic Information Industry Co., Ltd. | Component oriented hybrid cloud operating system architecture and communication method thereof |
| CN103546557A (en) * | 2013-10-22 | 2014-01-29 | 乐视致新电子科技（天津）有限公司 | Data interaction method of platform-level services, service platform and systems |
| CN105262831A (en) * | 2015-10-30 | 2016-01-20 | 北京奇艺世纪科技有限公司 | Method, device, and synchronizing system for synchronizing data among memory systems |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106657314A (en) | 2017-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106657314B (en) | Cross-data center data synchronization system and method | |
| US8954786B2 (en) | Failover data replication to a preferred list of instances | |
| US10528527B2 (en) | File management in thin provisioning storage environments | |
| US9497288B2 (en) | Subscriber based priority of messages in a publisher-subscriber domain | |
| CN108023953B (en) | High-availability implementation method and device for FTP service | |
| US20140115044A1 (en) | Stream processing using a client-server architecture | |
| US20150026237A1 (en) | Push notification via file sharing service synchronization | |
| EP2215773A1 (en) | Method and system for handling failover in a distributed environment that uses session affinity | |
| EP3066578B1 (en) | Strict queue ordering in a distributed system | |
| US11093279B2 (en) | Resources provisioning based on a set of discrete configurations | |
| CN113553178A (en) | Task processing method and device and electronic equipment | |
| CN106605213B (en) | System for supporting with intermittent connectivity, corresponding local device and corresponding cloud computing platform | |
| US8832215B2 (en) | Load-balancing in replication engine of directory server | |
| CN111338834B (en) | Data storage method and device | |
| US9894143B1 (en) | Pre-processing and processing pipeline for queue client | |
| CN110233791B (en) | Data deduplication method and device | |
| CN111444148B (en) | Data transmission method and device based on MapReduce | |
| CN115686813A (en) | Resource scheduling method and device, electronic equipment and storage medium | |
| CN106933654B (en) | Virtual machine starting method based on cache | |
| CN113472638B (en) | Edge gateway control method, system, device, electronic equipment and storage medium | |
| US10616317B2 (en) | Method and system for affinity load balancing | |
| US9348672B1 (en) | Singleton coordination in an actor-based system | |
| US20110208930A1 (en) | Providing Shared Access to Data Storage Resources Across Cluster Computing Environment Boundaries | |
| CN112749042B (en) | Application running method and device | |
| CN112583614B (en) | Host cluster management method, device and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |