CN108256002B - Cross-machine-room data synchronization method, device, system and server - Google Patents
Cross-machine-room data synchronization method, device, system and server Download PDFInfo
- Publication number
- CN108256002B CN108256002B CN201711496252.8A CN201711496252A CN108256002B CN 108256002 B CN108256002 B CN 108256002B CN 201711496252 A CN201711496252 A CN 201711496252A CN 108256002 B CN108256002 B CN 108256002B
- Authority
- CN
- China
- Prior art keywords
- data
- message queue
- synchronized
- queue device
- service server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Hardware Redundancy (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses a cross-machine-room data synchronization method, device, system and server. The method comprises the following steps: the data migration equipment detects whether a source end stores data to be synchronized; when detecting that the source end stores data to be synchronized, acquiring the data to be synchronized; and writing the data to be synchronized into a message queue device of a destination end so that a service server of the destination end reads the data to be synchronized from the message queue device of the destination end. According to the method, in the data synchronization process, the data migration equipment does not need to directly transmit the acquired data to be synchronized to the service server in the destination end, the data to be synchronized is firstly written into the message queue device, and the service server reads the data to be synchronized from the message queue device, so that the service server and the cross-machine-room data migration equipment are decoupled from each other, the mutual influence of the service server and the cross-machine-room data migration equipment is avoided, and the reliability of the whole system is improved.
Description
Technical Field
The present application relates to the field of data synchronization, and more particularly, to a method, an apparatus, a system, and a server for cross-machine room data synchronization.
Background
Some internet sites deploy multiple sites at different locations for the purpose of data security backup or improving access speed of users at various locations. Each site is a server or a cluster consisting of a plurality of servers. Data synchronization is maintained between the sites so that part of the data content or all of the data content stored therein is consistent. However, in this data synchronization, the reliability of the data synchronization still needs to be improved.
Disclosure of Invention
In view of the above problems, the present application provides a method, an apparatus, a system and a server for cross-machine room data synchronization, so as to improve reliability in a data synchronization process.
In a first aspect, the present application provides a cross-machine-room data synchronization method, which is applied to a cross-machine-room data synchronization system, where the cross-machine-room data synchronization system includes a data migration device, a message queue device, and a service server, all of which are disposed at a destination end; the data migration device is configured to migrate data to be synchronized, which is stored at a source end, to the message queue device, and the service server is configured to read data from the message queue device and process the data; the method comprises the following steps: the data migration equipment detects whether the source end stores data to be synchronized; when detecting that the source end has data to be synchronized, acquiring the data to be synchronized; and writing the data to be synchronized into the message queue device so that the service server reads the data to be synchronized from the message queue device.
In a second aspect, the present application provides a cross-machine-room data synchronization method, which is applied to a cross-machine-room data synchronization system, where the cross-machine-room data synchronization system includes a data migration device, a message queue device, and a service server, all of which are disposed at a destination; the data migration device is configured to migrate data to be synchronized, which is stored at a source end, to the message queue device, and the service server is configured to read data from the message queue device and process the data; the method comprises the following steps: the data migration equipment detects whether the source end stores data to be synchronized; when detecting that the source end stores data to be synchronized, acquiring the data to be synchronized; writing the data to be synchronized to the message queue device; and the service server reads the data to be synchronized from the message queue device.
In a third aspect, the present application provides a cross-machine-room data synchronization apparatus, which is a data migration device operating in a cross-machine-room data synchronization system, where the cross-machine-room data synchronization system further includes a message queue apparatus and a service server; the message queue device, the data migration device and the service server are all arranged at a destination end, the data migration device is used for migrating data to be synchronized stored at a source end to the message queue device, and the service server is used for reading data from the message queue device for processing; the device comprises: the data detection unit is used for detecting whether the source end stores data to be synchronized; the data acquisition unit is used for acquiring the data to be synchronized when the source end is detected to store the data to be synchronized; and the data writing unit is used for writing the data to be synchronized into the message queue device so that the service server can read the data to be synchronized from the message queue device.
In a fourth aspect, the present application provides a cross-room data synchronization system, including a data migration device, a message queue device, and a service server; the message queue device, the data migration device and the service server are all arranged at a destination end, the data migration device is used for migrating data to be synchronized stored at a source end to the message queue device, and the service server is used for reading data from the message queue device for processing; the data migration device is configured to detect whether the source end stores data to be synchronized; when detecting that the source end stores data to be synchronized, acquiring the data to be synchronized; writing the data to be synchronized to the message queue device; and the service server is used for reading the data to be synchronized from the message queue device.
In a fifth aspect, the present application provides a server comprising one or more processors and memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to perform the methods described above.
In a sixth aspect, the present application provides a computer readable storage medium having program code executable by a processor, the program code causing the processor to perform the above-described method.
According to the cross-machine-room data synchronization method, device, system and server, when cross-machine-room data migration equipment detects that the source end stores data to be synchronized, the data to be synchronized is acquired. And writing the data to be synchronized into the message queue device so that the service server reads the data to be synchronized from the message queue device. In the data synchronization process, the data migration equipment does not need to directly transmit the acquired data to be synchronized to the service server, but writes the data into the message queue device first, and the service server reads the data to be synchronized from the message queue device, so that the service server and the cross-machine-room data migration equipment are decoupled from each other, the mutual influence of the service server and the cross-machine-room data migration equipment is avoided, and the reliability of the whole system is improved.
These and other aspects of the present application will be more readily apparent from the following description of the embodiments.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic diagram illustrating an environment of a cross-room data synchronization system proposed in the present application;
fig. 2 shows a block diagram of a cross-room data synchronization system proposed in the present application;
fig. 3 shows a flowchart of a cross-room data synchronization method proposed in the present application;
FIG. 4 is a flow chart illustrating another cross-room data synchronization method proposed in the present application;
FIG. 5 shows a timing diagram of another cross-room data synchronization method proposed in the present application;
fig. 6 shows a block diagram of a cross-room data synchronization apparatus proposed in the present application;
fig. 7 shows a block diagram of another cross-room data synchronization apparatus proposed in the present application;
fig. 8 shows a block diagram of a server according to the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Usually, an internet company deploys a plurality of sites distributed in different areas, and sometimes data synchronization is performed between different sites in order to ensure that data between the deployed different sites is the same or partial data in the sites is the same. For example, in order to facilitate the user to log in when accessing each site, login information such as user names and passwords of all users may be synchronized between different sites.
Usually, a site is built in a computer room, so when data synchronization is performed between different sites, it can also be regarded as data synchronization between different computer rooms. The inventor finds that the control on the data synchronization efficiency cannot be realized in the data synchronization process, and further potential risks exist in the system during operation.
Therefore, the inventor provides a cross-computer-room data synchronization method, a cross-computer-room data synchronization device and a cross-computer-room data synchronization server, which are used for solving the problem that the improvement cannot realize control over data synchronization efficiency, so that a potential risk exists in the system during operation.
The following description is first made on an environment of a cross-machine-room data synchronization system in which the cross-machine-room data synchronization method and apparatus provided by the present application operate.
As shown in fig. 1, the cross-room data synchronization system 100 shown in fig. 1 includes a first room 110 and a second room 120. The first computer room 110 and the second computer room 120 perform data interaction through the network 130. A server 111 and a message queue means 112 in the first room 110 are deployed in the first room 110. A data migration device 121, a message queue apparatus 122, and a server 123 are disposed in the second machine room 120. It should be noted that, as shown in fig. 2, the message queue apparatus 112 and the message queue apparatus 122 may include a plurality of message queues during the data synchronization process.
The message queue is a channel for transmitting messages between applications by the applications. Applications communicate by writing and retrieving queued data (messages) for the application without requiring a dedicated connection to link them. Common message queues are rabbitmq, rocktmq, kafka, and the like.
Referring to fig. 2 again, in the first computer room 110, an SDK (software Development kit) component runs in the server 111, and the SDK component may subscribe to service data generated by the data producer and write the service data into the message queue device 112 in the first computer room 110. In the second computer room 120, the data migration device 121 may migrate the data of the message queue apparatus 112 in the first computer room 110 to the message queue apparatus 122, and the server 123 may read the data from the message queue apparatus 122 for processing. As one mode, a management node and a plurality of processing nodes run in the server 123, for example, a sender node in fig. 2 is the management node, and a receive node is the processing node. The ETCD is responsible for electing the management node as a processing node and coordinating a plurality of processing nodes to process data in the message queue.
Next, a data transmission flow in the following synchronization process will be described by taking the synchronization of data from the first machine room 110 to the second machine room 120 as an example. In this case, the first machine room 110 is a source end of a service data sender, and the second machine room 120 is a destination end of a service data receiver. The SDK component, which is run by the server 111 in the first computer room 110, writes the received synchronous data to the message queue in the message queue device 112 in an asynchronous thread manner. The data migration device 121 at the destination subscribes to the data in the message queue apparatus 112 in advance, and when the data migration device 121 detects that the message queue in the message queue apparatus 112 has synchronous data written therein, the data in the message queue is migrated to the message queue in the message queue apparatus 122 of the second computer room 120. The server 123 of the second machine room 120 reads the synchronization data to be processed from the message queue in the message queue device 122 for processing, so as to implement writing to the local, thereby completing the synchronization of the data from the first machine room 110 to the second machine room 120.
As an embodiment in the present application, in such a case that the first computer room 110 sends the synchronous data to the second computer room 120, the server 123 in the second computer room 120 is a service server of a service receiver. In this case, the data migration device 121 is configured to detect whether the first machine room 110 as the source end stores data to be synchronized. When detecting that the source end stores data to be synchronized, the source end obtains the data to be synchronized, and writes the data to be synchronized into the message queue device 122. The service server is configured to read the data to be synchronized from the message queue device 122.
It should be noted that the message queue apparatus 112 may be carried by one physical device, for example, one server, or may be carried by a plurality of physical devices, for example, a plurality of servers. The same is true for the message queue means 122. Furthermore, when the data migration apparatus 121 is implemented by a server and the message queue device 122 is also carried by a physical apparatus, the message queue device 122 may be deployed in the data migration apparatus 121, that is, the functions of the data migration apparatus 121 and the message queue device 122 are implemented by a server.
Embodiments of the present application will be described in detail below with reference to the accompanying drawings.
Referring to fig. 3, the cross-machine-room data synchronization method provided in the present application is applied to a cross-machine-room data synchronization system, where the cross-machine-room data synchronization system includes a data migration device, a message queue device, and a service server; the message queue device and the service server are arranged at a destination end, the data migration equipment is used for migrating data to be synchronized stored at a source end to the message queue device, and the service server is used for reading data from the message queue device for processing; the method comprises the following steps:
step S110: the data migration equipment detects whether the source end stores data to be synchronized.
Step S120: and when detecting that the source end stores the data to be synchronized, acquiring the data to be synchronized.
Step S130: and writing the data to be synchronized into the message queue device so that the service server reads the data to be synchronized from the message queue device.
According to the cross-machine-room data synchronization method, in the data synchronization process, the data migration equipment does not need to directly transmit the acquired data to be synchronized to the service server, but writes the data into the message queue device first, and the service server reads the data to be synchronized from the message queue device, so that the service server and the cross-machine-room data migration equipment are decoupled from each other, the data are not influenced with each other, and the reliability of the whole system is improved.
Referring to fig. 4, the cross-machine-room data synchronization method provided in the present application is applied to a cross-machine-room data synchronization system, where the cross-machine-room data synchronization system includes a data migration device, a message queue device, and a service server; the message queue device and the service server are arranged at a destination end, the data migration equipment is used for migrating data to be synchronized stored at a source end to the message queue device, and the service server is used for reading data from the message queue device for processing; the method comprises the following steps:
step S210: the data migration device detects whether a data synchronization task exists.
As one way, after the data migration device is started, it may first detect whether the data migration device is assigned with a data synchronization task. The data migration device can detect whether the data migration device is allocated with a data synchronization task by inquiring whether a data synchronization instruction is stored locally. The locally stored data synchronization instruction may be sent by the external device after detecting a certain state, for example, after detecting that the service sender starts to produce service data, the locally stored data synchronization instruction may also be sent by the external device after receiving a synchronization instruction triggered by a manager. In addition, after the data migration device detects a certain state, whether a data synchronization task exists or not can be triggered and detected. For example, when the data migration device detects that the network with a certain computer room is connected or the network state with a certain computer room achieves the expected effect, it starts to detect whether there is a data synchronization task. The network state can be characterized by network delay and network packet loss rate. Moreover, the data migration device can also directly send a query request to the external device after being started to query whether a data synchronization task exists.
Step S220: and when the data migration equipment detects that the data synchronization task exists, detecting whether the source end stores data to be synchronized.
A server and a message queue device are typically included in the source peer. For example, when the first computer room 110 shown in fig. 1 is used as a source, after receiving data produced by a service party, an SDK component running in the server 111 of the first computer room 110 writes the received data as data to be synchronized into the message queue apparatus 112. The data migration device may perform an initialization operation on the message queue device after detecting the data synchronization task related to the source end, and then subscribe the data to be synchronized to the message queue device in the source end, so that the data migration device may detect the data to be synchronized after the data to be synchronized is written in the message queue device.
As a mode, when the data migration device detects that there is the data synchronization task, a message queue matching with the task requirement of the data synchronization task may be configured in the message queue device, so that the data transmission performance that can be achieved by the data synchronization system across the computer room may match with the task requirement. It should be noted that, when the data migration apparatus detects that there is an existing message queue in the message queue device, the existing message queue is reconfigured as a message queue matching the task requirement of the data synchronization task. And when detecting that the message queue device does not create the queue, creating a message queue matched with the task requirement of the data synchronization task.
The task requirements include transmission efficiency, estimated total amount of transmitted data, and the like. For example, when it is detected that the data transmission efficiency required for the data synchronization task is a data transmission efficiency of a data per second, the data migration apparatus may configure the number of message queues corresponding to a data transmission efficiency of a data per second in the message queue device of the source end and the message queue device of the destination end. For another example, when the data migration device detects that there is a data synchronization task pending, and the task requirement of the data synchronization task is data with a total data transmission amount B, the data migration device may configure the number of message queues in the message queue device in the source end or the destination end based on the total data amount B, so as to complete the data transmission with the total data transmission amount B within a limited time.
Step S230: and when detecting that the source end stores the data to be synchronized, acquiring the data to be synchronized.
And when the data migration equipment detects that the data to be synchronized in the message queue device is written, the detected data to be synchronized is migrated to the data migration equipment. It should be noted that the message queue device includes a plurality of message queues for data transmission, and the data migration device acquires data from a message queue to which data needs to be synchronized after detecting that data needs to be synchronized and written in a certain message queue in the message queue device.
Step S240: and writing the data to be synchronized into the message queue device so that the service server reads the data to be synchronized from the message queue device.
As a mode, the data migration device directly transmits the acquired to-be-processed synchronous data to the message queue device of the destination end in a transparent transmission mode.
It should be noted that, after the data to be synchronized is transmitted and written into the message queue device of the destination, the data migration device does not process the data to be synchronized written into the message queue device of the destination any more, and then the subsequent processing of the data to be synchronized in the message queue device is performed by the service server of the destination, so that the message queue device can isolate the data migration device from the service server of the destination, and further the flow of writing the data to be synchronized into the message queue device by the data migration device and the flow of reading the data to be synchronized from the message queue device by the service server are not affected by each other, so that even when the data is read from the message queue device by the service server for processing, the data to be synchronized can still be normally written into the message queue device by the data migration device.
As one way, when the data migration apparatus detects that the data transmission efficiency of the message queue device does not match the expected data transmission efficiency, the data migration apparatus may regulate the number of message queues in the message queue device so that the data transmission efficiency of the message queue device matches the expected data transmission efficiency.
As one way, the data migration apparatus may precedent an exception data container in the message queue device before writing data to the message queue device of the destination. For example, a message queue may be separately established to store data that is abnormal when the subsequent service server performs processing, so that even when the subsequent service server obtains data from the message queue device and performs processing, the data that is abnormal in processing may be migrated to the separately established message queue, and further, the data migration device is prevented from being influenced to write new data to be synchronized into the message queue device, thereby further improving data transmission efficiency in the data synchronization process.
According to the cross-machine-room data synchronization method, when cross-machine-room data migration equipment detects that the source end stores data to be synchronized, the data to be synchronized are obtained. And writing the data to be synchronized into the message queue device so that the service server reads the data to be synchronized from the message queue device. In the data synchronization process, the data migration equipment does not need to directly transmit the data to be synchronized in the source end to the service server in the destination end, but writes the data into the message queue device first, and the service server reads the data to be synchronized from the message queue device, so that the service server and the cross-machine room data migration equipment are decoupled from each other, the data to be synchronized are not influenced with each other, and the reliability of the whole system is improved.
Referring to fig. 5, the cross-machine-room data synchronization method provided in the present application is applied to a cross-machine-room data synchronization system, where the cross-machine-room data synchronization system includes a data migration device, a message queue device, and a service server; the message queue device, the data migration device and the service server are used for being arranged at a destination end, the data migration device is used for migrating data to be synchronized stored at a source end to the message queue device, and the service server is used for reading data from the message queue device for processing; the method comprises the following steps:
step S310: the data migration equipment detects whether the source end stores data to be synchronized;
step S320: when detecting that the source end stores data to be synchronized, acquiring the data to be synchronized;
step S330: and writing the data to be synchronized into the message queue device.
Step S340: and the service server reads the data to be synchronized from the message queue device.
Referring to fig. 6, a cross-machine-room data synchronization apparatus 400 provided in the present application is a data migration device operating in a cross-machine-room data synchronization system, where the cross-machine-room data synchronization system further includes a message queue apparatus and a service server; the message queue device and the service server are arranged at a destination end, the data migration equipment is used for migrating data to be synchronized stored at a source end to the message queue device, and the service server is used for reading data from the message queue device for processing; the apparatus 400 comprises: a data detection unit 410, a data acquisition unit 420, and a data writing unit 430.
The data detection unit 410 is configured to detect whether the source stores data to be synchronized.
A data obtaining unit 420, configured to obtain the data to be synchronized when it is detected that the source end stores the data to be synchronized.
A data writing unit 430, configured to write the data to be synchronized into the message queue device, so that the service server reads the data to be synchronized from the message queue device.
Referring to fig. 7, a cross-machine-room data synchronization apparatus 500 provided in the present application is a data migration device operating in a cross-machine-room data synchronization system, where the cross-machine-room data synchronization system further includes a message queue apparatus and a service server; the message queue device and the service server are arranged at a destination end, the data migration equipment is used for migrating data to be synchronized stored at a source end to the message queue device, and the service server is used for reading data from the message queue device for processing; the apparatus 500 comprises: a data detection unit 510, a data acquisition unit 520, and a data writing unit 530.
The data detection unit 510 is configured to detect whether the source stores data to be synchronized.
By way of one approach, the data detection unit 510 includes: a task detection subunit 511, configured to detect whether there is a data synchronization task; a data detecting subunit 512, configured to subscribe to data to be synchronized to the source end when the task detecting subunit detects that the data synchronization task exists, and detect whether the source end stores the data to be synchronized. As one way, when the task detection subunit 511 detects that there is the data synchronization task, the data detection subunit 512 may configure a message queue matching the task requirement of the data synchronization task in the message queue device, so that the data transmission performance that can be achieved by the data synchronization system across the computer room may match the task requirement. The task requirements include transmission efficiency, estimated total amount of transmitted data, and the like. For example, when the data detection subunit 512 detects that the data transmission efficiency required by the data synchronization task is a data transmission efficiency of a data per second, the data detection subunit 512 may configure the number of message queues appropriate for a data transmission per second in the message queue device of the source end and the message queue device of the destination end. For another example, when the data detection subunit 512 detects that there is a data synchronization task pending, and the task requirement of the data synchronization task is data with a total data transmission amount B, the data detection subunit 512 may configure the number of message queues in the message queue device in the source end or the destination end based on the total data amount B.
A data obtaining unit 520, configured to obtain the data to be synchronized when it is detected that the source end stores the data to be synchronized.
A data writing unit 530, configured to write the data to be synchronized into the message queue device, so that the service server reads the data to be synchronized from the message queue device.
As one way, the apparatus 500 further includes a transmission rate regulating unit 540, configured to regulate the number of message queues in the message queue apparatus when detecting that the data transmission efficiency of the message queue apparatus does not match the expected data transmission efficiency, so that the data transmission efficiency of the message queue apparatus matches the expected data transmission efficiency.
Referring to fig. 8, based on the above data synchronization control method and apparatus, the present application further provides a server 140, where the server 140 may be a server 111 or a server 123 of the system shown in fig. 1, may also be a physical device bearing a message queue apparatus 122 or a message queue apparatus 112, and is further used as a data migration device in the present application.
It should be noted that, when the message queue apparatus 112 or the message queue apparatus 122 is both installed on a device, the message queue apparatus 112 operates in a server 140, and the message queue apparatus 122 also operates in a server 140. When the message queue device 112 or the message queue device 122 is distributed in a plurality of physical devices, the message queue device 112 is distributed in a plurality of servers 140, and the message queue device 122 is also distributed in a plurality of servers 140.
The server 140 includes one or more processors 222 (only one of which is shown), memory 224, a network module 228, and a peripheral interface 230.
Those skilled in the art will appreciate that all other components are peripheral devices with respect to the processor 222, and the processor 222 is coupled to the peripheral devices through a plurality of peripheral interfaces 230. The peripheral interface 230 may be implemented based on the following standards: universal Asynchronous Receiver/Transmitter (UART), General Purpose Input/Output (GPIO), Serial Peripheral Interface (SPI), and Inter-Integrated Circuit (I2C), but the present invention is not limited to these standards. In some examples, the peripheral interface 230 may include only a bus. These controllers may also be separate from the peripheral interface 230 and integrated within the processor 222 or a corresponding peripheral.
The memory 224 may be used to store software programs and modules, and the processor 222 executes various functional applications and data processing by executing the software programs and modules stored in the memory 224. The memory 224 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory. In some examples, the memory 224 may further include memory that is remotely located from the processor 222. In addition, the memory 224 may also store some data to be processed. For example, when message queue device 112 or message queue device 122 is carried by server 140, memory 224 may store data in a message queue.
The network module 228 is configured to receive and send network data to enable data interaction with other devices through a network.
The peripheral interface 230 is used for the server 140 to communicate with external devices in a wired manner. The peripheral interface can be a USB interface or an RS232 interface and the like.
Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (server) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments. For example, when the program is executed, the executed contents are the contents of step S110 to step S130 in the foregoing contents. In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not necessarily depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (5)
1. A cross-machine-room data synchronization method is characterized by being applied to a cross-machine-room data synchronization system, wherein the cross-machine-room data synchronization system comprises data migration equipment, a message queue device and a service server which are all arranged at a destination end; the data migration device is configured to migrate data to be synchronized, which is stored at a source end, to the message queue device, and the service server is configured to read data from the message queue device and process the data; the method comprises the following steps:
the data migration equipment detects whether a data synchronization task exists;
when the data synchronization task is detected, subscribing data to be synchronized to the source end, and detecting whether the source end stores the data to be synchronized, wherein when the data synchronization task is detected, a message queue matched with a task requirement of the data synchronization task is configured in the message queue device, and the task requirement comprises transmission efficiency and estimated total transmission data;
when detecting that the source end has data to be synchronized, acquiring the data to be synchronized;
writing the data to be synchronized into the message queue device so that the service server reads the data to be synchronized from the message queue device;
the method further comprises the following steps:
when the data migration equipment detects that the data transmission efficiency of the message queue device is not matched with the expected data transmission efficiency; and regulating the number of message queues in the message queue device so as to enable the data transmission efficiency of the message queue device to be matched with the expected data transmission efficiency.
2. A cross-machine-room data synchronization device is characterized in that the cross-machine-room data synchronization device is operated in a data migration device in a cross-machine-room data synchronization system, and the cross-machine-room data synchronization system further comprises a message queue device and a service server; the message queue device, the data migration device and the service server are all arranged at a destination end, the data migration device is used for migrating data to be synchronized stored at a source end to the message queue device, and the service server is used for reading data from the message queue device for processing; the device comprises:
the data detection unit is used for detecting whether a data synchronization task exists;
when the data synchronization task is detected, subscribing data to be synchronized to the source end, and detecting whether the source end stores the data to be synchronized, wherein when the data synchronization task is detected, a message queue matched with a task requirement of the data synchronization task is configured in the message queue device, and the task requirement comprises transmission efficiency and estimated total transmission data;
the data acquisition unit is used for acquiring the data to be synchronized when the source end is detected to store the data to be synchronized;
a data writing unit, configured to write the data to be synchronized into the message queue device, so that the service server reads the data to be synchronized from the message queue device;
a transmission rate regulating unit, configured to detect that data transmission efficiency of the message queue apparatus does not match expected data transmission efficiency; and regulating the number of message queues in the message queue device so as to enable the data transmission efficiency of the message queue device to be matched with the expected data transmission efficiency.
3. A cross-machine-room data synchronization system is characterized by comprising data migration equipment, a message queue device and a service server; the message queue device, the data migration device and the service server are all arranged at a destination end, the data migration device is used for migrating data to be synchronized stored at a source end to the message queue device, and the service server is used for reading data from the message queue device for processing;
the data migration equipment is used for detecting whether a data synchronization task exists; when the data synchronization task is detected, subscribing data to be synchronized to the source end, and detecting whether the source end stores the data to be synchronized, wherein when the data synchronization task is detected, a message queue matched with a task requirement of the data synchronization task is configured in the message queue device, and the task requirement comprises transmission efficiency and estimated total transmission data; when detecting that the source end stores data to be synchronized, acquiring the data to be synchronized; writing the data to be synchronized to the message queue device;
the service server is used for reading the data to be synchronized from the message queue device;
the data migration device is further configured to detect that the data transmission efficiency of the message queue apparatus does not match an expected data transmission efficiency; and regulating the number of message queues in the message queue device so as to enable the data transmission efficiency of the message queue device to be matched with the expected data transmission efficiency.
4. A server, comprising one or more processors and memory;
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to perform the method of claim 1.
5. A computer-readable storage medium storing program code executable by a processor, the program code causing the processor to perform the method of claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711496252.8A CN108256002B (en) | 2017-12-31 | 2017-12-31 | Cross-machine-room data synchronization method, device, system and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711496252.8A CN108256002B (en) | 2017-12-31 | 2017-12-31 | Cross-machine-room data synchronization method, device, system and server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108256002A CN108256002A (en) | 2018-07-06 |
| CN108256002B true CN108256002B (en) | 2021-05-11 |
Family
ID=62724730
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711496252.8A Active CN108256002B (en) | 2017-12-31 | 2017-12-31 | Cross-machine-room data synchronization method, device, system and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108256002B (en) |
Families Citing this family (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110209526B (en) * | 2018-08-14 | 2023-06-06 | 腾讯科技（深圳）有限公司 | Storage layer synchronization system and storage medium |
| CN109726046B (en) * | 2018-11-23 | 2021-01-08 | 网联清算有限公司 | Machine room switching method and device |
| CN109710688B (en) * | 2018-11-26 | 2021-10-08 | 福建天泉教育科技有限公司 | Data real-time comparison and verification method and message middleware |
| CN111381977A (en) * | 2018-12-29 | 2020-07-07 | 北大方正集团有限公司 | Message processing method and device |
| CN109905457B (en) * | 2019-01-09 | 2022-01-11 | 广州视源电子科技股份有限公司 | Master-slave server data synchronization method and device, computer equipment and storage medium |
| CN110069670A (en) * | 2019-04-30 | 2019-07-30 | 深圳前海微众银行股份有限公司 | Purpose data classifying method, device, equipment and computer readable storage medium |
| CN110149382A (en) * | 2019-05-08 | 2019-08-20 | 拉扎斯网络科技（上海）有限公司 | Data synchronization method, system, main server, synchronization client and medium |
| CN110807018A (en) * | 2019-10-17 | 2020-02-18 | 上海易点时空网络有限公司 | Method, device, equipment and storage medium for migrating master-slave architecture of business data to cluster architecture |
| CN110795503A (en) * | 2019-10-18 | 2020-02-14 | 北京达佳互联信息技术有限公司 | Multi-cluster data synchronization method and related device of distributed storage system |
| CN110765206A (en) * | 2019-10-21 | 2020-02-07 | 广东小天才科技有限公司 | Data synchronization system, method, device, equipment and storage medium |
| CN110933150A (en) * | 2019-11-18 | 2020-03-27 | 拉扎斯网络科技（上海）有限公司 | Data synchronization method, system, device, electronic equipment and computer storage medium |
| CN110989933A (en) * | 2019-12-05 | 2020-04-10 | 北京首汽智行科技有限公司 | Message queue RockMq smooth migration method |
| CN111586117A (en) * | 2020-04-26 | 2020-08-25 | 特瓦特能源科技有限公司 | Equipment communication method and device |
| CN115023929B (en) * | 2020-05-06 | 2024-07-05 | 深圳市欢太科技有限公司 | Data synchronization method, device, system, electronic equipment and storage medium |
| CN111581244B (en) * | 2020-05-14 | 2023-04-25 | 厦门熵基科技有限公司 | Method, system and equipment for orderly synchronizing service transaction data of heterogeneous system |
| CN111813868B (en) * | 2020-08-13 | 2023-11-10 | 中国工商银行股份有限公司 | Data synchronization method and device |
| CN112182095A (en) * | 2020-08-31 | 2021-01-05 | 福州智象信息技术有限公司 | Method, device, equipment and medium for synchronizing service data among multiple machine rooms |
| CN112052104A (en) * | 2020-09-01 | 2020-12-08 | 掌阅科技股份有限公司 | Message queue management method based on multi-computer-room realization and electronic equipment |
| CN112068972A (en) * | 2020-09-01 | 2020-12-11 | 掌阅科技股份有限公司 | Message queue consumption method based on multi-computer room realization and electronic equipment |
| CN112187889A (en) * | 2020-09-15 | 2021-01-05 | 中信银行股份有限公司 | Data synchronization method, device and storage medium |
| CN112187916B (en) * | 2020-09-27 | 2023-12-05 | 中国银联股份有限公司 | Cross-system data synchronization method and device |
| CN112632032B (en) * | 2020-12-18 | 2022-12-27 | 华人运通(上海)云计算科技有限公司 | Data migration method and device, storage medium and terminal equipment |
| CN112817988B (en) * | 2021-01-06 | 2022-11-15 | 贵阳迅游网络科技有限公司 | Synchronous acceleration method for enterprise business |
| CN113407629A (en) * | 2021-06-18 | 2021-09-17 | 湖南快乐阳光互动娱乐传媒有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN117370310B (en) * | 2023-10-19 | 2024-05-28 | 中电云计算技术有限公司 | Distributed file system cross-cluster data increment migration method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105162879A (en) * | 2015-09-24 | 2015-12-16 | 北京奇虎科技有限公司 | Method, device and system for realizing data consistency among plurality of machine rooms |
| CN105989123A (en) * | 2015-02-13 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Data synchronization method, apparatus and system |
| CN106657314A (en) * | 2016-12-14 | 2017-05-10 | 北京奇艺世纪科技有限公司 | Cross-data center data synchronization system and method |
| CN106713487A (en) * | 2017-01-16 | 2017-05-24 | 腾讯科技（深圳）有限公司 | Data synchronization method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8949182B2 (en) * | 2011-06-17 | 2015-02-03 | International Business Machines Corporation | Continuous and asynchronous replication of a consistent dataset |
-
2017
- 2017-12-31 CN CN201711496252.8A patent/CN108256002B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105989123A (en) * | 2015-02-13 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Data synchronization method, apparatus and system |
| CN105162879A (en) * | 2015-09-24 | 2015-12-16 | 北京奇虎科技有限公司 | Method, device and system for realizing data consistency among plurality of machine rooms |
| CN106657314A (en) * | 2016-12-14 | 2017-05-10 | 北京奇艺世纪科技有限公司 | Cross-data center data synchronization system and method |
| CN106713487A (en) * | 2017-01-16 | 2017-05-24 | 腾讯科技（深圳）有限公司 | Data synchronization method and device |
Non-Patent Citations (1)
| Title |
|---|
| 支持动态负载平衡的分层消息队列模型;鲍春健;《计算机工程与应用》;20170131(第1期);正文第155-158页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108256002A (en) | 2018-07-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108256002B (en) | Cross-machine-room data synchronization method, device, system and server | |
| CN108347476B (en) | Cross-machine-room data synchronization method and device and server | |
| US11940965B2 (en) | Data migration method and apparatus | |
| US9706002B2 (en) | Push notification via file sharing service synchronization | |
| CN109067877B (en) | Control method for cloud computing platform deployment, server and storage medium | |
| EP3575975B1 (en) | Method and apparatus for operating smart network interface card | |
| CN108696581B (en) | Distributed information caching method and device, computer equipment and storage medium | |
| CN108347477B (en) | Data transmission method, device and server | |
| CN107959582B (en) | Slice instance management method and device | |
| US20130179558A1 (en) | Server management using a baseboard management controller to establish a wireless network | |
| US20130290947A1 (en) | Termimal and method for updating firmware of baseboard management controller | |
| CN110493028A (en) | A kind of clustered deploy(ment) method, system, device and computer readable storage medium | |
| CN109361525B (en) | Method, device, control terminal and medium for restarting distributed deployment of multiple services | |
| CN107360633B (en) | Pairing connection method and device of virtual reality system and virtual reality system | |
| US10404568B2 (en) | Agent manager for distributed transaction monitoring system | |
| US10282120B2 (en) | Method, apparatus and system for inserting disk | |
| US10102088B2 (en) | Cluster system, server device, cluster system management method, and computer-readable recording medium | |
| CN107920136B (en) | Data synchronization control method, device and server | |
| CN111935029B (en) | Gateway load balancing method and device, storage medium and electronic equipment | |
| CN112968812A (en) | Network performance testing method, device, equipment and storage medium | |
| CN111919419A (en) | Node, network system and data synchronization method | |
| CN108234216B (en) | Method and device for downloading data message | |
| JP6380774B1 (en) | Computer system, server device, program, and failure detection method | |
| CN111629054A (en) | Message processing method, device and system, electronic equipment and readable storage medium | |
| CN104052776A (en) | VM boot method, VMM, shared storage system, VDI system and physical host |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Changan town in Guangdong province Dongguan 523860 usha Beach Road No. 18 Applicant after: GUANGDONG OPPO MOBILE TELECOMMUNICATIONS Corp.,Ltd. Address before: Changan town in Guangdong province Dongguan 523860 usha Beach Road No. 18 Applicant before: GUANGDONG OPPO MOBILE TELECOMMUNICATIONS Corp.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |