CN108881379B - Method and device for data synchronization between server clusters - Google Patents
Method and device for data synchronization between server clusters Download PDFInfo
- Publication number
- CN108881379B CN108881379B CN201810414297.4A CN201810414297A CN108881379B CN 108881379 B CN108881379 B CN 108881379B CN 201810414297 A CN201810414297 A CN 201810414297A CN 108881379 B CN108881379 B CN 108881379B
- Authority
- CN
- China
- Prior art keywords
- data
- server cluster
- server
- type
- clusters
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Hardware Redundancy (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a method and a device for data synchronization among server clusters, which relate to the technical field of cluster storage, and the method comprises the following steps: monitoring the data transmission state of each server cluster when the server cluster acquires the first type of data from the first data source equipment and the health degree of each server cluster, then determining a first backup server cluster according to the data transmission state and the health degree of each server cluster, and finally controlling the first backup server cluster to synchronize the acquired first type of data to other server clusters except the first backup server cluster. Because the backup server clusters are determined according to the data transmission states and the health degrees of the server clusters, the data acquired by the backup server clusters from the data source equipment is complete data, and the integrity of the data of the server clusters after synchronization is ensured. And secondly, synchronizing the data acquired by the backup server cluster to other server clusters, thereby realizing the consistency of the data among the server clusters.
Description
Technical Field
The embodiment of the invention relates to the technical field of cluster storage, in particular to a method and a device for data synchronization among server clusters.
Background
As China enters the big data era, the demands for data management and storage are increasing. In order to satisfy the requirement of storing and managing large-scale data, a scheme for building a server cluster to manage and store the data has appeared in recent years. In the prior art, in order to prevent the problem that a server cluster is unavailable due to a machine room problem, a backbone network problem and the like, a plurality of server clusters are generally arranged to acquire data of the same data source for storage and management. However, in the process of requesting data from a server cluster by a user, if the server cluster is switched to another server cluster, the requested data may be inconsistent or inconsistent, thereby affecting the user experience.
Disclosure of Invention
The embodiment of the invention provides a method and a device for data synchronization among server clusters, which are used for solving the problem of inconsistent data among the server clusters in the prior art.
In one aspect, an embodiment of the present invention provides a method for data synchronization between server clusters, including:
monitoring a data transmission state of each server cluster when the server cluster acquires first-class data from first data source equipment and the health degree of each server cluster, then determining the server cluster with a normal data transmission state and a health degree meeting preset conditions from each server cluster, then determining a first backup server cluster from the server clusters with a normal data transmission state and a health degree meeting the preset conditions, and finally controlling the first backup server cluster to synchronize the acquired first-class data to other server clusters except the first backup server cluster in each server cluster. The first backup server cluster is determined by monitoring the data transmission state of each server cluster when the first type of data is acquired and the health degree of each server cluster, so that the data acquired from the data source equipment by the server cluster for backup is complete data, and the integrity of the data of each server cluster after synchronization is ensured. And secondly, synchronizing the data acquired by the first backup server cluster to other server clusters, thereby realizing the consistency of the data among the server clusters.
Optionally, the determining a first backup server cluster from the server clusters with normal data transmission states and health degrees meeting preset conditions includes: and determining the server cluster with the highest priority in the server clusters with normal data transmission states and health degrees meeting preset conditions as a first backup server cluster. Because the data transmission link corresponding to the server cluster with higher priority is more stable, the health degree is better, so that the server cluster with the highest priority is selected as the first backup server cluster from the server clusters with normal data transmission state and health degree meeting the preset condition, and the integrity of the data in the first backup server cluster is improved.
Optionally, a second type of data stored in each server cluster is obtained, where the second type of data is obtained from a second data source device. And when the second type of data stored in each server cluster is determined to be inconsistent, determining a second backup server cluster according to the data transmission state of each server cluster when the second type of data is acquired and the health degree of each server cluster, and then controlling the second backup server cluster to synchronize the stored second type of data to other server clusters except the second backup server cluster in each server cluster. By comparing the data stored in the server clusters, the data of the second backup server cluster can be synchronized to other server clusters in time when the data in the server clusters are inconsistent, so that the consistency of the data among the server clusters is ensured. And secondly, determining a second backup server cluster according to the data transmission state of the server cluster when the second type of data is acquired and the health degree of each server cluster, thereby ensuring the integrity of the data acquired from the data source equipment by the second backup server cluster.
Optionally, for any one of the server clusters, the server cluster receives a first data request sent by a user terminal, where the first data request is used to request to acquire first-class data, and then determines whether the first-class data synchronized from the first backup server cluster is stored; if so, sending the first type of data synchronized from the first backup server cluster to the user terminal; otherwise, the first type data acquired from the first data source equipment is sent to the user terminal. After receiving a data request of a user terminal, the server cluster judges whether the stored synchronous data is stored or not, if yes, the server cluster sends the synchronous data to the user terminal, consistency and coherence of the data acquired by the user terminal from each server cluster are guaranteed, otherwise, the server cluster sends the data acquired from the data source equipment, and the server cluster can also provide the data for the user terminal before data synchronization.
Optionally, the method further comprises: and the server cluster receives a second data request sent by the user terminal, wherein the second data request is used for requesting to acquire second-class data, and sends the stored second-class data to the user terminal, and the second-class data stored by the server cluster is acquired from the second data source equipment or the second backup server cluster. For the second type of data, the synchronous data directly covers the data acquired by the server cluster from the data source equipment, so that when the user terminal requests the data, the server cluster directly sends the data stored by the cluster to the user terminal without judgment, thereby improving the speed of responding to the data request of the user terminal on one hand and saving the storage space of the server cluster on the other hand.
On one hand, the embodiment of the invention provides a device for data synchronization among server clusters, which comprises a monitoring module, a first processing module and a synchronization module, wherein the monitoring module is used for monitoring the data transmission state of each server cluster when acquiring first-class data from a first data source device and the health degree of each server cluster; the first processing module is used for determining a server cluster with a normal data transmission state and a health degree meeting a preset condition from the server clusters; determining a first backup server cluster from the server clusters with normal data transmission states and health degrees meeting preset conditions; and the synchronization module is used for controlling the first backup server cluster to synchronize the acquired first type of data to other server clusters except the first backup server cluster in each server cluster.
Optionally, the first processing module is specifically configured to: and determining the server cluster with the highest priority in the server clusters with normal data transmission states and health degrees meeting preset conditions as a first backup server cluster.
Optionally, the server cluster data synchronization apparatus further includes a query module, where the query module is configured to obtain second type data stored in each server cluster, where the second type data is obtained from a second data source device; the first processing module is further configured to determine a second backup server cluster according to a data transmission state of the server clusters when the second type of data stored in the server clusters is determined to be inconsistent and the health degree of the server clusters; the synchronization module is further configured to control the second backup server cluster to synchronize the stored second type of data to other server clusters in the server clusters except for the second backup server cluster.
Optionally, the method further comprises: the device comprises a receiving module and a second processing module, wherein the receiving module is used for receiving a first data request sent by a user terminal, and the first data request is used for requesting to acquire first-class data; the second processing module is used for judging whether the first type of data synchronized from the first backup server cluster is stored or not; if so, sending the first type of data synchronized from the first backup server cluster to the user terminal; otherwise, the first type data acquired from the first data source equipment is sent to the user terminal.
Optionally, the receiving module is further configured to receive a second data request sent by the user terminal, where the second data request is used to request to acquire second-class data; the second processing module is further configured to send the stored second type of data to the user terminal, where the second type of data stored by the server cluster is obtained from the second data source device or obtained from the second backup server cluster.
In another aspect, an embodiment of the present invention provides a terminal device, which includes at least one processing unit, a display unit, and at least one storage unit, where the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit is caused to execute the steps of any one of the above methods.
In yet another aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program executable by a terminal device, and when the program runs on the terminal device, the program causes the terminal device to execute the steps of any one of the above-mentioned methods.
According to the technical scheme provided by the embodiment of the invention, the server cluster with the highest priority in the server clusters with normal data transmission states and health degrees meeting the preset conditions is determined as the backup server cluster by monitoring the data transmission states of the server clusters when the server clusters acquire data and the health degrees of the server clusters, so that compared with other server clusters, the data acquired by the backup server cluster from the data source equipment is more complete data, the data of the backup server cluster is synchronized to other server clusters, and the integrity of the data of each server cluster after synchronization is ensured. Secondly, the data acquired by the backup server cluster is synchronized to other server clusters, so that the consistency of the data among the server clusters is realized, and the consistency of the requested data can be ensured even if a user meets the cluster switching condition when requesting the data from the server clusters, thereby improving the user experience.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
FIG. 1 is a diagram of a system architecture according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of a method for data synchronization between server clusters according to an embodiment of the present invention;
fig. 3 is a schematic flowchart of a method for data synchronization between server clusters according to an embodiment of the present invention;
fig. 4 is a schematic flowchart of a method for data synchronization between server clusters according to an embodiment of the present invention;
fig. 5 is a schematic flowchart of a method for data synchronization between server clusters according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a server cluster according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a server cluster according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of a device for data synchronization between server clusters according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of a terminal device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more clearly apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In this document, it is to be understood that any number of elements in the figures are provided by way of illustration and not limitation, and any nomenclature is used for differentiation only and not in any limiting sense. For convenience of understanding, terms referred to in the embodiments of the present invention are explained below.
Server clustering: multiple servers collectively perform the same service, appearing to the client as if there was only one server. The cluster can use a plurality of computers to perform parallel computation so as to obtain high computation speed, and can also use a plurality of computers to perform backup so as to ensure that any one server breaks down the whole system or can normally run.
Fig. 1 schematically illustrates a system architecture applicable to the embodiment of the present invention, and as shown in fig. 1, the system architecture applicable to the embodiment of the present invention includes a user terminal 110, a data source device 120, at least two server clusters 130, and a control center 140. The control center 140 may be a stand-alone server, and may be integrated on the server cluster 130 or other devices integrated in an existing CDN network.
The user terminal 110 is an electronic device with network communication capability, and the electronic device is a smart phone, a tablet computer, a portable personal computer, or the like. The user terminal 110 is connected to the server cluster 130 and the control center 140 through a wireless network.
The data source device 120 provides raw data to the server cluster 130, and the data format may be text, pictures, video, web pages, http information, and the like. The data source device 120 is connected to the server cluster 130 and the control center 140 through a wireless network.
The server cluster 130 obtains and stores data from the data source device 120, and provides data to the user terminal 110 in response to a data request after receiving the data request from the user terminal 110.
After monitoring the data transmission state of each server cluster 130 when acquiring data from the data source device 120 and the health degree of each server cluster 130 by the monitoring device, the control center 140 determines a backup server cluster from the server clusters 130 whose data transmission state is normal and whose health degree satisfies a preset condition, and then controls the backup server cluster to synchronize the acquired first type of data to other server clusters 130 in each server cluster 130 except the backup server cluster, so that the consistency of data among the server clusters is ensured. When the user terminal 110 requests data from the server clusters 130, the data request is sent to the control center 140, then the control center 140 allocates one server cluster 130 to the user terminal 110 according to the load condition of each server cluster 130, and the user terminal 110 requests data from the allocated server cluster 130. When the server cluster 130 providing data for the user terminal 110 fails, the control center 140 controls the user terminal 110 to be switched to another server cluster 130, and the other server cluster 130 continues to provide data for the user terminal 110. The control center 140 is a server or a server cluster composed of several servers or a cloud computing center.
Optionally, the wireless or wired networks described above use standard communication techniques and/or protocols. The Network is typically the Internet, but may be any Network including, but not limited to, a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a mobile, wireline or wireless Network, a private Network, or any combination of virtual private networks.
Based on the system architecture diagram illustrated in fig. 1, an embodiment of the present invention provides a method for data synchronization between server clusters, where a flow of the method may be executed by a device for data synchronization between server clusters, as illustrated in fig. 2, and the method includes the following steps:
step S201, monitoring a data transmission state of each server cluster when the server cluster acquires the first type of data from the first data source device and a health level of each server cluster itself.
In a possible implementation manner, the first type of data is important data, the important data is determined according to specific service requirements, and the first data source device is a device that provides the important data, it should be noted that the first type of data source device may only generate the important data, or may generate the important data and the non-important data at the same time.
In another possible implementation, the first type of data real-time data is data that a user requests from a server cluster in real time, for example, in a live broadcast process, a guest needs to obtain a video of a main broadcast in real time, and then the data that the main broadcast sends to the server cluster as a data source is the first type of data. The first data source device is a device for providing real-time data, and the first data source device generates first-type data and then sends the first-type data to each server cluster. And each server cluster receives the first type of data sent by the first data source equipment and then stores the first type of data. The monitoring device monitors the data transmission state of each server cluster when the server cluster acquires the first type of data and the health degree of each server cluster, the data transmission state includes but is not limited to network transmission state, transmission component state, machine downtime and other sudden failures, and factors influencing the health degree of the server cluster include but are not limited to: hard disk, memory, Central Processing Unit (CPU), motherboard, I/O board, and power supply.
Step S202, determining the server clusters with normal data transmission states and health degrees meeting preset conditions from all the server clusters.
Step S203, determining a first backup server cluster from the server clusters with normal data transmission states and health degrees meeting the preset conditions.
In a possible embodiment, when the data transmission state of only one server cluster in each service cluster is normal and the health degree meets a preset condition, the server cluster is determined as a first backup server cluster.
In another possible embodiment, when the data transmission state of at least two server clusters in each service cluster is normal and the health degree meets a preset condition, the first backup service cluster is determined according to the priority of the server clusters, for example, the server cluster with the highest priority in the server clusters with the normal data transmission state and the health degree meeting the preset condition is determined as the first backup server cluster. Because the data transmission link corresponding to the server cluster with higher priority is more stable, the health degree is better, so that the server cluster with the highest priority is selected as the first backup server cluster from the server clusters with normal data transmission state and health degree meeting the preset condition, and the integrity of the data in the first backup server cluster is improved.
Step S204, controlling the first backup server cluster to synchronize the acquired first type of data to other server clusters except the first backup server cluster in each server cluster.
The first backup server cluster distributes the acquired first type of data to other server clusters, and the other server clusters store the first type of data distributed by the first backup server cluster after receiving the first type of data, so that the data among the server clusters are consistent, and when a user switches from one server cluster to another server cluster, the consistency and the continuity of the data requested by the user can be ensured, thereby improving the user experience. In addition, the other server clusters store the first type of data distributed by the first backup server cluster and also store the first type of data acquired from the first data source device, so that even if the first backup server cluster does not finish distributing the first type of data, the other server clusters can firstly send the first type of data acquired from the first data source device to the user terminal when receiving a data request of the user terminal, and a user can conveniently acquire the first type of data in real time. Secondly, if the first backup server cluster fails to distribute the first type of data, the backup server cluster can be determined from other server clusters again, and the first type of data acquired from the first data source device is sent to other server clusters, so that the stability of data backup is improved.
Based on the system architecture diagram illustrated in fig. 1, an embodiment of the present invention further provides a method for data synchronization between server clusters, where a flow of the method may be executed by a device for data synchronization between server clusters, as illustrated in fig. 3, and the method includes the following steps:
step S301, obtaining second type data stored in each server cluster, where the second type data is obtained from a second data source device.
In a possible implementation manner, the second type of data is non-important data, the non-important data is determined according to specific service requirements, and the second data source device is a device for providing the non-important data, it should be noted that the second type of data source device may generate only the non-important data, or may generate both the important data and the non-important data. In addition, the first data source device and the second data source device may be the same data source device or different data source devices.
In another possible implementation, the second type of data is non-real-time data, that is, data that is periodically updated by the server cluster, and the user knows that the type of data is periodically updated in advance, for example, the second type of data is data that is updated every day, and after the second type of data source device sends the second type of data of the current day to the server cluster, if the user needs to query the second type of data of the current day, the query may take one day before. The second data source device is a device for providing non-real-time data, and the second data source device generates the non-real-time data and then sends the non-real-time data to each server cluster. It should be noted that the second type of data source device may generate only the non-real-time data, or may generate the real-time data and the non-real-time data simultaneously, which is not limited in the embodiment of the present invention. The monitoring equipment monitors the data transmission state of each server cluster when the server cluster acquires the second type of data and the health degree of each server cluster in real time and stores the data transmission state and the health degree, and each server cluster directly stores the second type of data sent by the second data source equipment after receiving the second type of data, without judging the data transmission state in real time and carrying out data synchronization. And then querying second-type data acquired and stored by each server cluster in the same time period, wherein the querying mode can be manual querying by a User or regular automatic querying, and then displaying and comparing the second data acquired by querying each server cluster on a User Interface (UI) to judge whether the data acquired and stored by each server cluster in the same time period are consistent.
Step S302, when the second type data stored in each server cluster is determined to be inconsistent, a second backup server cluster is determined according to the data transmission state when the second type data is acquired by each server cluster and the health degree of each server cluster.
When the server clusters acquire the second type data, the monitoring device monitors the data transmission state of the server clusters acquiring the second type data and the health degree of the server clusters in real time, so that when the second type data acquired and stored by the server clusters in the same time period are determined to be inconsistent after comparison, the data transmission state of the server clusters acquiring the second type data and the health degree of the server clusters monitored by the monitoring device in the time period can be acquired, and then the second backup server cluster is determined from the server clusters with normal data transmission state and health degree meeting preset conditions. The method for determining the second backup server cluster from the server clusters with normal data transmission states and health degrees meeting the preset conditions is the same as the method for determining the first backup server cluster from the server clusters with normal data transmission states and health degrees meeting the preset conditions, and the description is omitted here.
Step S303, controlling the second backup server cluster to synchronize the stored second type data to other server clusters except the second backup server cluster in each server cluster.
The second backup server cluster distributes the stored second-class data to other server clusters, and the other server clusters store the received second-class data and delete the second-class data acquired from the second-class data source equipment. By comparing the data stored in the server clusters, the data of the second backup server cluster can be synchronized to other server clusters in time when the data in the server clusters are inconsistent, so that the consistency of the data among the server clusters is ensured. It should be noted that, the first-type data in each server cluster is also applicable to the scheme provided in the embodiment of the present invention, that is, whether the data are consistent is periodically queried and compared, and if the data are not consistent, data synchronization is performed, but the first-type data acquired from the first data source device is not overwritten during synchronization.
Based on the system architecture diagram illustrated in fig. 1, an embodiment of the present invention further provides a method for data synchronization between server clusters, where a flow of the method may be executed by any one of the server clusters, as illustrated in fig. 4, and the method includes the following steps:
step S401, the server cluster receives a first data request sent by the user terminal, where the first data request is used to request to acquire first type data.
The server cluster simultaneously stores first-class data acquired from the first-class data source equipment and first-class data distributed by the first backup server cluster, and the first-class data are stored in different distributed databases. It should be noted that, if the server cluster is the first backup server cluster, only the first type of data acquired from the first type of data source device needs to be saved.
In step S402, the server cluster determines whether the first type of data synchronized from the first backup server cluster is stored.
Step S403, if yes, sending the first type of data synchronized from the first backup server cluster to the user terminal.
And step S404, otherwise, sending the first type of data acquired from the first data source equipment to the user terminal.
The server cluster receives a first data request sent by a user terminal in real time, and at the moment, the first backup server cluster may not completely backup first-class data to other service server clusters, so that when the server cluster receives the first data request, whether the first-class data synchronized from the first backup server cluster is stored or not is judged firstly, namely whether the first-class data is distributed by the first backup server is finished or not is judged, if yes, the first-class data synchronized from the first backup server cluster is sent to the user terminal, otherwise, the first-class data acquired from the first data source device is sent to the user terminal, and therefore consistency and continuity of data acquired by the user terminal from each server cluster are guaranteed, and data requests of real-time response user terminals are guaranteed.
Based on the system architecture diagram illustrated in fig. 1, an embodiment of the present invention further provides a method for synchronizing data among server clusters, where a flow of the method may be executed by any one of the server clusters, as illustrated in fig. 5, and the method includes the following steps:
step S501, the server cluster receives a second data request sent by the user terminal, where the second data request is used to request to acquire second-class data.
The server cluster stores the second type data acquired from the second type data source device or the second type data distributed by the second backup server cluster.
Step S502, the server cluster sends the stored second type data to the user terminal.
The second type of data stored by the server cluster is obtained from the second data source device or obtained from the second backup server cluster. For the second type of data, because the user terminal requests the data in non-real time, the synchronous data directly covers the data acquired by the server cluster from the data source equipment, and when the user terminal requests the data, the server cluster directly sends the stored data to the user terminal without judgment, so that the speed of responding to the data request of the user terminal is improved on one hand, and the storage space of the server cluster is saved on the other hand.
For better explaining the embodiment of the present invention, a method for synchronizing data between server clusters according to the embodiment of the present invention is described below through a specific implementation scenario, as shown in fig. 6, for a first type of data, a server cluster 1, a server cluster 2, and a server cluster 3 are set to obtain the first type of data from a first data source device, the server cluster 1 includes a receiving module, a processing module, a caching module, a distributing module, a storage module, and a query module, and structures and working processes of the server cluster 2 and the server cluster 3 are the same as those of the server cluster 1. First, a receiving module receives first-class data sent by a first data source device, and the data format can be diverse and can be files, http information and the like. The first type data may then be accessed and sent to different destinations, for example, the first type data may be sent to kafka for processing, and the receiving module may simultaneously copy the first type data to the distributed file system for data repair and verification. And then the receiving module sends the first type of data to the processing module, the processing module performs extraction (extract), conversion (transform), loading (load) or aggregation and other operations on the first type of data before data storage, the used processing engines comprise JSTTORM (Java Server instruction), SPARKSTREAMING (spare instruction), and the like, and the processing logic is customized according to business requirements or self-definition. The cache module caches the results of the processing module for storage, so as to prevent the phenomena of storage accumulation and the like, and a storage system such as CODIS or Aeroskip can be adopted. And if the distribution module of the server cluster 1 receives a distribution instruction sent by the control center, the first type of data is fished from the cache module and is distributed to the server cluster 2 and the server cluster 3. And if the distribution module of the server cluster 1 does not receive the distribution instruction sent by the control center, receiving and storing the first type of data distributed by the server cluster 2 or the server cluster 3. Fig. 6 illustrates a process in which the distribution module of the server cluster 1 distributes the first type of data. For consistency and reliability, the control center only sends a distribution instruction to the first backup server cluster in a time period, and further controls the first backup server cluster to synchronize the first type of data acquired from the first type of data source equipment to other server clusters. Optionally, the distribution module of the server cluster 1 may also obtain, from the monitoring device of the control center, a data transmission state when each server cluster obtains the first type of data and a health degree of each server cluster itself, and then judge whether each server cluster is a first backup server according to the data transmission state, the health degree, and the priority of each server cluster, if so, the first type of data is retrieved from the cache module and distributed to the server cluster 2 and the server cluster 3, otherwise, the first type of data distributed by the server cluster 2 or the server cluster 3 is received and stored. The storage module is generally a distributed database, which may be a phoenix, hbase, mongodb, or the like, and the specific selection is determined according to a service scenario. And the query module is used for external data query.
As shown in fig. 7, for the second type of data, the server cluster 1, the server cluster 2, and the server cluster 3 are set to obtain the second type of data from the second data source device and store the data in the storage module. The control center regularly acquires second data from the storage module of each server cluster through the query module of the server cluster to perform UI display and comparison, judges whether the second data stored in the server cluster 1, the server cluster 2 and the server cluster 3 are consistent or not, if yes, the control center directly ends the UI display, otherwise, determines a second backup server cluster according to the data transmission state when the server cluster acquires the second data and the self health degree of each server cluster, and sends a synchronization instruction to the second backup server cluster so that the second backup server cluster synchronizes the stored second data to other server clusters, wherein the second backup server cluster is one of the server cluster 1, the server cluster 2 and the server cluster 3.
According to the technical scheme provided by the embodiment of the invention, the server cluster with the highest priority in the server clusters with normal data transmission states and health degrees meeting the preset conditions is determined as the backup server cluster by monitoring the data transmission states of the server clusters when the server clusters acquire data and the health degrees of the server clusters, so that compared with other server clusters, the data acquired by the backup server cluster from the data source equipment is more complete data, the data of the backup server cluster is synchronized to other server clusters, and the integrity of the data of each server cluster after synchronization is ensured. Secondly, the data acquired by the backup server cluster is synchronized to other server clusters, so that the consistency of the data among the server clusters is realized, and the consistency of the requested data can be ensured even if a user meets the cluster switching condition when requesting the data from the server clusters, thereby improving the user experience.
Based on the same technical concept, an embodiment of the present invention further provides a device for data synchronization between server clusters, as shown in fig. 8, where the device 800 includes a monitoring module 801, a first processing module 802, and a synchronization module 803, where the monitoring module 801 is configured to monitor a data transmission state when each server cluster acquires first type data from a first data source device, and a health degree of each server cluster; a first processing module 802, configured to determine, from the server clusters, a server cluster in which a data transmission state is normal and a health degree meets a preset condition; determining a first backup server cluster from the server clusters with normal data transmission states and health degrees meeting preset conditions; a synchronizing module 803, configured to control the first backup server cluster to synchronize the acquired first type of data to other server clusters in the server clusters except the first backup server cluster.
Optionally, the first processing module 802 is specifically configured to: and determining the server cluster with the highest priority in the server clusters with normal data transmission states and health degrees meeting preset conditions as a first backup server cluster.
Optionally, the server cluster data synchronization apparatus further includes a query module 804, where the query module 804 is configured to obtain second type data stored in each server cluster, where the second type data is obtained from a second data source device; the first processing module 802 is further configured to determine, when it is determined that the second type of data stored in each server cluster is inconsistent, a second backup server cluster according to a data transmission state of the server cluster when acquiring the second type of data and a health degree of the server cluster; the synchronization module 803 is further configured to control the second backup server cluster to synchronize the stored second type of data to other server clusters in the server clusters except the second backup server cluster.
Optionally, the apparatus further includes a receiving module 805, configured to receive a first data request sent by a user terminal, where the first data request is used to request to acquire first type data; a second processing module 806, configured to determine whether the first type of data synchronized from the first backup server cluster is stored; if so, sending the first type of data synchronized from the first backup server cluster to the user terminal; otherwise, the first type data acquired from the first data source equipment is sent to the user terminal.
Optionally, the receiving module 805 is further configured to receive a second data request sent by the user terminal, where the second data request is used to request to acquire second-class data; the second processing module 806 is further configured to send the stored second type data to the user terminal, where the second type data stored by the server cluster is obtained from the second data source device or obtained from the second backup server cluster.
The embodiment of the invention provides terminal equipment, which comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the steps of the method for synchronizing the data among the server clusters. As shown in fig. 9, which is a schematic diagram of a hardware structure of a terminal device according to an embodiment of the present invention, the terminal device may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, and the like. Specifically, the terminal device may include a memory 901, a processor 902, and a computer program stored on the memory, where the processor 902 implements the steps of the method for synchronizing data between any server clusters in the foregoing embodiments when executing the program. Memory 901 may include, among other things, Read Only Memory (ROM) and Random Access Memory (RAM), and provides processor 902 with program instructions and data stored in memory 901.
Further, the terminal device described in this embodiment of the application may further include an input device 903, an output device 904, and the like. The input device 903 may include a keyboard, mouse, touch screen, etc.; the output device 904 may include a Display device such as a Liquid Crystal Display (LCD), a Cathode Ray Tube (CRT), a touch screen, or the like. The memory 901, the processor 902, the input device 903 and the output device 904 may be connected by a bus or other means, and fig. 9 illustrates the connection by a bus as an example. The processor 902 calls the program instructions stored in the memory 901 and executes the method for data synchronization between server clusters provided by the above-mentioned embodiments according to the obtained program instructions.
An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a terminal device, and when the program runs on the terminal device, the terminal device is caused to execute the steps of the method for synchronizing data between server clusters.
It should be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. A method for data synchronization between server clusters, comprising:
the method comprises the steps that a control center monitors the data transmission state of each server cluster when first-class data are acquired from first data source equipment and the health degree of each server cluster;
determining a server cluster with a normal data transmission state and a health degree meeting a preset condition from the server clusters;
determining a first backup server cluster from the server clusters with normal data transmission states and health degrees meeting preset conditions;
controlling the first backup server cluster to synchronize the acquired first type of data to other server clusters except the first backup server cluster in each server cluster;
when a data request sent by a user terminal is received, determining a target server cluster for providing data for the user terminal from all the server clusters according to the load condition of all the server clusters, so that the user terminal requests the data from the target server cluster.
2. The method of claim 1, wherein determining a first backup server cluster from the server clusters with normal data transmission status and health satisfying a predetermined condition comprises:
and determining the server cluster with the highest priority in the server clusters with normal data transmission states and health degrees meeting preset conditions as a first backup server cluster.
3. The method of claim 1, further comprising:
acquiring second-class data stored in each server cluster, wherein the second-class data is acquired from second data source equipment;
when the second type of data stored in each server cluster is determined to be inconsistent, determining a second backup server cluster according to the data transmission state of each server cluster when the second type of data is acquired and the health degree of each server cluster;
and controlling the second backup server cluster to synchronize the stored second type data to other server clusters except the second backup server cluster in each server cluster.
4. The method of claim 1, further comprising:
for any one server cluster in the server clusters, the server cluster receives a first data request sent by a user terminal, and the first data request is used for requesting to acquire first-class data;
the server cluster judges whether the first type of data synchronized from the first backup server cluster is stored or not;
if so, sending the first type of data synchronized from the first backup server cluster to the user terminal;
otherwise, the first type data acquired from the first data source equipment is sent to the user terminal.
5. The method of claim 4, further comprising:
the server cluster receives a second data request sent by the user terminal, wherein the second data request is used for requesting to acquire second-class data;
and the server cluster sends the stored second-class data to the user terminal, wherein the second-class data stored by the server cluster is acquired from the second data source equipment or the second backup server cluster.
6. An apparatus for data synchronization between server clusters, comprising:
the monitoring module is used for monitoring the data transmission state of each server cluster when the first type of data is acquired from the first data source equipment and the health degree of each server cluster;
the first processing module is used for determining a server cluster with a normal data transmission state and a health degree meeting a preset condition from the server clusters; determining a first backup server cluster from the server clusters with normal data transmission states and health degrees meeting preset conditions;
the synchronization module is used for controlling the first backup server cluster to synchronize the acquired first type of data to other server clusters except the first backup server cluster in each server cluster; when a data request sent by a user terminal is received, determining a target server cluster for providing data for the user terminal from all the server clusters according to the load condition of all the server clusters, so that the user terminal requests the data from the target server cluster.
7. The apparatus of claim 6, wherein the first processing module is specifically configured to:
and determining the server cluster with the highest priority in the server clusters with normal data transmission states and health degrees meeting preset conditions as a first backup server cluster.
8. The apparatus of claim 6, further comprising a query module;
the query module is used for acquiring second-class data stored in each server cluster, wherein the second-class data is acquired from second data source equipment;
the first processing module is further configured to determine a second backup server cluster according to a data transmission state of the server clusters when the second type of data stored in the server clusters is determined to be inconsistent and the health degree of the server clusters;
the synchronization module is further configured to control the second backup server cluster to synchronize the stored second type of data to other server clusters in the server clusters except for the second backup server cluster.
9. The apparatus of claim 6, further comprising:
the device comprises a receiving module, a sending module and a receiving module, wherein the receiving module is used for receiving a first data request sent by a user terminal, and the first data request is used for requesting to acquire first-class data;
the second processing module is used for judging whether the first type of data synchronized from the first backup server cluster is stored or not;
if so, sending the first type of data synchronized from the first backup server cluster to the user terminal;
otherwise, the first type data acquired from the first data source equipment is sent to the user terminal.
10. The apparatus of claim 9, wherein the receiving module is further configured to receive a second data request sent by the ue, where the second data request is used to request to obtain a second type of data;
the second processing module is further configured to send the stored second type of data to the user terminal, where the second type of data stored by the server cluster is obtained from the second data source device or obtained from the second backup server cluster.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810414297.4A CN108881379B (en) | 2018-05-03 | 2018-05-03 | Method and device for data synchronization between server clusters |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810414297.4A CN108881379B (en) | 2018-05-03 | 2018-05-03 | Method and device for data synchronization between server clusters |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108881379A CN108881379A (en) | 2018-11-23 |
| CN108881379B true CN108881379B (en) | 2021-04-16 |
Family
ID=64327362
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810414297.4A Expired - Fee Related CN108881379B (en) | 2018-05-03 | 2018-05-03 | Method and device for data synchronization between server clusters |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108881379B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109614271B (en) * | 2018-12-14 | 2023-06-13 | 深圳前海微众银行股份有限公司 | Method, device, equipment and storage medium for controlling consistency of multiple cluster data |
| CN109660560A (en) * | 2019-01-23 | 2019-04-19 | 视联动力信息技术股份有限公司 | A kind of method of data synchronization and system |
| CN111371900B (en) * | 2020-03-13 | 2022-07-12 | 北京奇艺世纪科技有限公司 | Method and system for monitoring health state of synchronous link |
| CN111901389B (en) * | 2020-07-03 | 2023-07-04 | 北京达佳互联信息技术有限公司 | Data updating method, device, server and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102682117A (en) * | 2012-05-15 | 2012-09-19 | 浪潮电子信息产业股份有限公司 | Method for quickly copying cluster data in database |
| CN103391299A (en) * | 2012-05-08 | 2013-11-13 | 深圳市腾讯计算机系统有限公司 | Load balance method and load balance system |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102025630A (en) * | 2010-12-14 | 2011-04-20 | 成都市华为赛门铁克科技有限公司 | Load balancing method and load balancing system |
| US8949828B2 (en) * | 2011-01-11 | 2015-02-03 | International Business Machines Corporation | Single point, scalable data synchronization for management of a virtual input/output server cluster |
| CN104536971A (en) * | 2014-12-02 | 2015-04-22 | 北京锐安科技有限公司 | High-availability database |
| CN104462483A (en) * | 2014-12-18 | 2015-03-25 | 北京奇虎科技有限公司 | Data synchronizing method, device and system |
| CN106888245B (en) * | 2016-06-07 | 2020-04-24 | 阿里巴巴集团控股有限公司 | Data processing method, device and system |
| CN105915646A (en) * | 2016-06-20 | 2016-08-31 | 飞天诚信科技股份有限公司 | Method and device for dispersing load of servers |
-
2018
- 2018-05-03 CN CN201810414297.4A patent/CN108881379B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103391299A (en) * | 2012-05-08 | 2013-11-13 | 深圳市腾讯计算机系统有限公司 | Load balance method and load balance system |
| CN102682117A (en) * | 2012-05-15 | 2012-09-19 | 浪潮电子信息产业股份有限公司 | Method for quickly copying cluster data in database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108881379A (en) | 2018-11-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108881379B (en) | Method and device for data synchronization between server clusters | |
| EP2904763B1 (en) | Load-balancing access to replicated databases | |
| CN108418900B (en) | Caching method, write-in point client and read client in server cluster system | |
| CN111078147B (en) | Processing method, device and equipment for cache data and storage medium | |
| US9413807B1 (en) | Browser rendering and computation | |
| CN102546256B (en) | System and method used for monitoring cloud computation service | |
| US20120278344A1 (en) | Proximity grids for an in-memory data grid | |
| CN103559072A (en) | Method and system for implementing bidirectional auto scaling service of virtual machines | |
| CN109783151B (en) | Method and device for rule change | |
| US9792185B2 (en) | Directed backup for massively parallel processing databases | |
| CN110825704B (en) | Data reading method, data writing method and server | |
| CN110784498B (en) | Personalized data disaster tolerance method and device | |
| CN105227489A (en) | A kind of bandwidth management method and electronic equipment | |
| US20160234129A1 (en) | Communication system, queue management server, and communication method | |
| CN113094430B (en) | Data processing method, device, equipment and storage medium | |
| CN104750757A (en) | Data storage method and equipment based on HBase | |
| CN115309497A (en) | Request scheduling method, device, equipment and storage medium | |
| CN111726388A (en) | Cross-cluster high-availability implementation method, device, system and equipment | |
| CN111600794A (en) | Server switching method, terminal, server and storage medium | |
| KR20150007698A (en) | Load distribution system for virtual desktop service | |
| CN106570006B (en) | Multimedia file recommendation system and delivery information storage method | |
| CN111225003B (en) | NFS node configuration method and device | |
| CN116069493A (en) | Data processing method, device, equipment and readable storage medium | |
| CN106257447A (en) | The video storage of cloud storage server and search method, video cloud storage system | |
| CN112631756A (en) | Distributed regulation and control method and device applied to space flight measurement and control software |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210416 |