WO2023019953A1 - 数据同步方法、系统、服务器及存储介质 - Google Patents
数据同步方法、系统、服务器及存储介质 Download PDFInfo
- Publication number
- WO2023019953A1 WO2023019953A1 PCT/CN2022/083501 CN2022083501W WO2023019953A1 WO 2023019953 A1 WO2023019953 A1 WO 2023019953A1 CN 2022083501 W CN2022083501 W CN 2022083501W WO 2023019953 A1 WO2023019953 A1 WO 2023019953A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- cluster
- data
- mirror
- synchronization method
- main
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 230000001360 synchronised effect Effects 0.000 claims description 17
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 6
- 238000004891 communication Methods 0.000 abstract description 4
- 238000011084 recovery Methods 0.000 description 25
- 238000010586 diagram Methods 0.000 description 11
- 230000000694 effects Effects 0.000 description 5
- 230000005856 abnormality Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000006855 networking Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
Definitions
- the embodiments of the present application relate to the technical field of communications, and in particular to a data synchronization method, system, server, and storage medium.
- the cloud platform In order to achieve disaster recovery, the cloud platform usually realizes by forming replicas in different data centers. When a data center fails, the replicas of other data centers are activated to realize disaster recovery.
- the disaster recovery method of the cloud platform is based on extended clusters, that is, multiple data centers share a storage cluster.
- extended clusters that is, multiple data centers share a storage cluster.
- the embodiment of the present application provides a data synchronization method, including: detecting whether there is data written in the main cluster; when detecting that the main cluster has data written, synchronizing the data to the mirror cluster, the main The cluster and the mirrored cluster are pre-created in the same storage environment.
- the embodiment of the present application also provides a data synchronization system, including a master cluster and a mirror cluster, the master cluster and the mirror cluster are pre-created in the same storage environment; the mirror cluster is used to detect whether the master cluster has data written input, and when it is detected that data is written in the primary cluster, the data is synchronized to the mirror cluster.
- a data synchronization system including a master cluster and a mirror cluster, the master cluster and the mirror cluster are pre-created in the same storage environment; the mirror cluster is used to detect whether the master cluster has data written input, and when it is detected that data is written in the primary cluster, the data is synchronized to the mirror cluster.
- the embodiment of the present application also provides a server, including: at least one processor; and a memory connected in communication with the at least one processor; wherein, the memory stores instructions that can be executed by the at least one processor , the instructions are executed by the at least one processor, so that the at least one processor can execute the above data synchronization method.
- the embodiment of the present application also provides a computer-readable storage medium storing a computer program, and implementing the above data synchronization method when the computer program is executed by a processor.
- FIG. 1 is a schematic flow diagram of a data synchronization method provided in an embodiment of the present application
- FIG. 2 is a schematic flow diagram of the application of the data synchronization method provided by the embodiment of the present application on the cloud platform;
- FIG. 3 is an example diagram of the data synchronization method provided by the embodiment of the present application when the mirroring service is enabled;
- FIG. 4 is an example diagram of the data synchronization method provided by the embodiment of the present application in disaster recovery configuration
- FIG. 5 is an example diagram of the principle architecture of the data synchronization method provided by the embodiment of the present application.
- FIG. 6 is a schematic diagram of the principle of updating storage pool data using mirroring services in the data synchronization method provided by the embodiment of the present application;
- FIG. 7 is a schematic structural diagram of a data synchronization system provided by an embodiment of the present application.
- FIG. 8 is a schematic structural diagram of a server provided by an embodiment of the present application.
- the main purpose of the embodiments of the present application is to provide a data synchronization method, system, server, and storage medium, which can improve the stability of a cloud platform cluster during disaster recovery.
- it relates to a data synchronization method.
- the data is synchronized to the mirror cluster, wherein the main cluster and the mirror cluster are in the same storage
- the environment is pre-created, so that when an exception occurs in the main cluster, the read and write objects can be switched to the mirror cluster for disaster recovery. Since the disaster recovery is realized by using the mirrored cluster in the same storage environment, the stability of the cluster in use will not be affected during the disaster recovery, thereby improving the stability of the cluster during the disaster recovery of the cloud platform.
- the execution subject of the data synchronization method provided in the embodiment of the present application may be a mirroring cluster.
- the master cluster and the mirror cluster are relative concepts, which refer to a master-slave relationship.
- the master cluster acts as the master role, and the mirror cluster acts as the slave role.
- the mirror cluster can be switched to the master role, and the master cluster can be switched to the slave role.
- the mirror cluster can be the new master cluster, and the original master cluster can be the new mirror cluster.
- the execution subject of the data synchronization method provided by the embodiment of the present application may be a new mirror cluster, and the new mirror cluster detects the new master cluster, and when it detects that the new master cluster has data written, the data is synchronized to the new mirrored cluster.
- the main cluster and the mirror cluster correspond to different server groups.
- the main cluster and the mirror cluster belong to different data centers (DCs), so as to achieve the effect of remote disaster recovery.
- the main cluster can write the data to be backed up to a specified location, and the mirror cluster can detect the specified location of the main cluster.
- the data when performing data backup, the data is actively backed up to different data centers, and the original data and the backup data are updated synchronously.
- the main cluster does not actively back up data to the mirror cluster, but the mirror cluster detects the main cluster, and only when it detects that the main cluster has data written, the data is backed up.
- the original data and backup data are updated asynchronously. It is understandable that the IO delay between different data centers is relatively poor.
- the existing technology adopts a synchronous update method, and data backup will have a certain impact on the efficiency of data update.
- the data synchronization method provided by the embodiment of the present application adopts an asynchronous update method, which will not affect the data update efficiency in the main cluster due to data backup, so there is no need to increase hardware costs because of the need to reduce the delay of different data centers .
- the same storage environment can refer to the storage environment under the same management portal, so that the mirror cluster synchronizes data from the main cluster.
- synchronizing data to the mirror cluster before synchronizing data to the mirror cluster, it also includes: enabling the ceph-based mirror service of the mirror cluster; when synchronizing data to the mirror cluster, the mirror service of the mirror cluster is used to synchronize the data to the mirror cluster .
- the main cluster and the mirror cluster are pre-created based on the ceph distributed storage environment, and the mirror cluster synchronizes the data of the main cluster to the mirror cluster based on the mirror service provided by ceph technology.
- the mirror service of the master cluster can be enabled, so that the data of the mirror cluster can be synchronized to the master cluster by using the mirror service.
- the mirroring service can be enabled at the same time when creating the master cluster and the mirror cluster.
- ceph is an open source software (software-defined storage) storage platform for object, block-level and file-level storage.
- the main cluster and the mirror cluster are pre-created in different virtual data centers (Virtual Data Center, VDC for short) in the cloud platform, and/or, the main cluster and the mirror cluster are pre-created based on a hyper-converged scenario.
- VDC Virtual Data Center
- the data synchronization method provided by the embodiment of the present application can be realized on the cloud platform environment, and then the disaster recovery on the cloud platform can be realized; at the same time, the main cluster and the mirror image Clusters corresponding to different virtual data centers are also conducive to improving disaster recovery capabilities.
- disaster recovery is usually implemented through disk array replication.
- the data synchronization method provided by the embodiment of the present application pre-creates the main cluster and the mirror cluster based on the hyper-converged scenario. Since the server is both a computing node and a storage node in the hyper-converged scenario, no additional storage device is required, and the networking is relatively It is relatively simple, more convenient to deploy, and has better scalability.
- Fig. 2 is a schematic flow diagram of the application of the data synchronization method provided by the embodiment of the application on the cloud platform, which roughly includes the following flow: 1. Build a hyper-converged environment; 2. Set up a ceph distributed storage cluster (including the main cluster and Mirror cluster); 3. Create a storage pool according to the data that needs to be backed up; 4. Enable the mirror service of the main cluster and the mirror cluster (ceph-rbd-mirror@admin.service), and its specific implementation can refer to Figure 3; 5. Binding The main cluster and the mirror cluster (and the storage pool) are configured for disaster recovery, and its specific implementation can refer to Figure 4; 5. The cloud platform adds the storage libraries of the main cluster and the mirror cluster respectively in different virtual centers.
- the main cluster and the mirror cluster include several first storage pools with the same name.
- the data is synchronized to the mirror cluster, including:
- the data is synchronized to the first storage pool of the mirror cluster through the mirror service of the mirror cluster.
- the first storage pool of the main cluster needs to perform data backup.
- the mirror cluster uses the mirror service to synchronize the data to the first storage pool of the mirror cluster and the main cluster. in the first storage pool with the same name. For example, if there is data written to the first storage pool A in the main cluster, the mirror cluster uses the mirroring service to synchronize the data to the first storage pool A of the mirror cluster; if there is data written to the first storage pool B in the main cluster , the mirroring cluster uses the mirroring service to synchronize data to the first storage pool B of the mirroring cluster.
- a storage pool with the same name as the storage pool is created in the mirror cluster, and these two storage pools are the first storage pools with the same name.
- first in the first storage pool is only used to refer to the storage pool that needs to be backed up, and does not serve as other restrictions such as sequence or symbol.
- the newly created rbd (rados block device, Rados block device) under the first storage pool in the main cluster needs to enable the journal feature and exclusive-lock feature.
- the mirroring state When the mirroring state is enabled, it indicates that the rbd is in normal data Synchronization state, when mirroring primary is true, it indicates that the rbd is the primary role.
- an rbd with the same name will also be created in the first storage pool in the mirror cluster, and mirroring primary is false, indicating that it is the slave role of the master rbd.
- the first storage pools of the primary cluster or the mirror cluster can be distributed in different data centers, for example, the first storage pool A is distributed in data center A, the first storage pool B is distributed in data center B, etc., so that more Good remote disaster recovery effect.
- FIG. 5 is an example diagram of the principle architecture of the data synchronization method provided by the embodiment of the present application.
- the main cluster and the mirror cluster are distributed in different DCs (the main cluster is distributed in DC1, and the mirror cluster is distributed in DC2), and the mirror cluster realizes data synchronization with the main cluster through the mirror service (rbd-mirror in the figure).
- the main cluster and the mirror cluster include N storage pools (pool1, pool2, pool3...) with the same name.
- the mirror cluster synchronizes data to the storage pool with the same name as the main cluster through the mirror service, which can realize the data synchronization between the mirror cluster and the storage pool in the main cluster, and also facilitate the main cluster to specify the data to be backed up by specifying the storage pool.
- detecting whether there is data writing in the main cluster may include: detecting whether there is data update in the journal of the first storage pool of the main cluster through the mirror service of the mirror cluster; When there is data written in the storage pool, the data is synchronized to the first storage pool of the mirror cluster through the mirroring service, which may include: if it is detected that there is data update in the journal of the first storage pool of the main cluster, then using the mirroring service from The updated data read in the journal is written to the first storage pool of the mirror cluster.
- FIG. 6 is a schematic diagram of the principle of using the mirroring service to update storage pool data in the data synchronization method provided by the embodiment of the present application.
- the data will first be written into the image journal of the main cluster, and when the journal is successfully written, the data in the journal will be written into the rbd of the main cluster and a response will be replied.
- the rbd-mirror process (mirror service) of the mirror cluster detects that the journal of the main cluster has data updates, it reads data from the journal of the main cluster and writes it into the rbd image of the mirror cluster, thereby completing data synchronization.
- the metadata in the main cluster will be updated, indicating that the IO journal has been synchronized this time.
- the mirror service is used to detect whether there is data update in the journal of the first storage pool of the main cluster. When data update is detected, the data is synchronized to the first storage pool of the mirror cluster through the mirror service.
- the active detection of cluster data updates realizes asynchronous data updates and reduces the IO delay requirements between the mirror cluster and the main cluster.
- after synchronizing the data to the mirror cluster further includes: switching the read and write objects to the mirror cluster or the first storage pool of the mirror cluster if an abnormality is detected in the master cluster.
- the mirror cluster or the first storage pool in the mirror cluster is promoted to the primary role to read and write data.
- the main cluster is abnormal, only some of the first storage pools may be abnormal, so the corresponding first storage pool in the mirror cluster can be promoted to the master role to read and write data.
- it may be to detect whether the first storage pool of the primary cluster is abnormal.
- the read-write object When switching the read-write object to the mirror cluster or the first storage pool of the mirror cluster, it can be automatically implemented by configuring the mirror cluster. Since the exception of the main cluster may not be a real exception, in order to reduce the switching of reading and writing objects, switching can be done manually.
- the data synchronization method provided by the embodiment of this application detects whether there is data writing in the main cluster, and when it detects that there is data writing in the main cluster, the data is synchronized to the mirror cluster. Created so that when an exception occurs in the primary cluster, the read and write objects can be switched to the mirror cluster for disaster recovery. Since the disaster recovery is realized by using the mirrored cluster in the same storage environment, the stability of the cluster in use will not be affected during the disaster recovery, thereby improving the stability of the cluster during the disaster recovery of the cloud platform.
- a data synchronization system 200 as shown in FIG. 7 , including a main cluster 201 and a mirror cluster 202.
- the main cluster 201 and the mirror cluster 202 are pre-created in the same storage environment. The details are as follows:
- the mirror cluster 202 is used to detect whether there is data written in the master cluster 201, and when it is detected that the master cluster 201 has data written, the data is synchronized to the mirror cluster 202.
- the mirroring cluster 202 is also used to start a ceph-based mirroring service of the mirroring cluster, and use the mirroring service to synchronize data to the mirroring cluster 202 .
- the primary cluster 201 and the mirroring cluster 202 include several first storage pools with the same name, and the mirroring cluster 202 is also used to synchronize data through the mirroring service when it detects that data is written into the first storage pool of the primary cluster 201 to the first storage pool of the mirror cluster 202.
- the mirror cluster 202 is also used to detect whether there is data update in the journal of the first storage pool of the main cluster 201 through the mirror service; if it is detected that there is data update in the journal, then utilize the mirror service to read the updated Data is written into the first storage pool of the mirror cluster 202 .
- the mirror cluster 202 is further configured to switch the read and write objects to the mirror cluster 202 or the first storage pool of the mirror cluster 202 if an abnormality of the master cluster is detected.
- the main cluster 201 and the mirror cluster 202 are pre-created in different virtual data centers in the cloud platform.
- the main cluster 201 and the mirror cluster 202 are pre-created based on a hyper-converged scenario.
- this embodiment is a system embodiment corresponding to the foregoing method embodiments, and this embodiment can be implemented in cooperation with the foregoing method embodiments.
- the relevant technical details mentioned in the foregoing method embodiments are still valid in this embodiment, and will not be repeated here in order to reduce repetition.
- the relevant technical details mentioned in this embodiment may also be applied to the foregoing method embodiments.
- a server as shown in FIG. 8 , including: at least one processor 301; and a memory 302 connected in communication with at least one processor 301; wherein, the memory 302 stores information that can be processed by at least one processor.
- the instructions executed by the processor 301 are executed by at least one processor 301, so that the at least one processor 301 can execute the above data synchronization method.
- the memory and the processor are connected by a bus
- the bus may include any number of interconnected buses and bridges, and the bus connects one or more processors and various circuits of the memory together.
- the bus may also connect together various other circuits such as peripherals, voltage regulators, and power management circuits, all of which are well known in the art and therefore will not be further described herein.
- the bus interface provides an interface between the bus and the transceivers.
- a transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing means for communicating with various other devices over a transmission medium.
- the data processed by the processor is transmitted on the wireless medium through the antenna, further, the antenna also receives the data and transmits the data to the processor.
- the processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Instead, memory can be used to store data that the processor uses when performing operations.
- it relates to a computer readable storage medium storing a computer program.
- the above method embodiments are implemented when the computer program is executed by the processor.
- a storage medium includes several instructions to make a device ( It may be a single-chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the various embodiments of the present application.
- the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc., which can store program codes. .
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本申请实施例涉及通信技术领域，公开了一种数据同步方法、系统、服务器及存储介质，所述数据同步方法包括：检测主集群是否有数据写入；在检测到所述主集群有数据写入时，将所述数据同步至所述镜像集群，所述主集群与所述镜像集群在同一存储环境下预先创建。
Description
相关申请的交叉引用
本申请基于申请号为“202110939058.2”、申请日为2021年08月16日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此以引入方式并入本申请。
本申请实施例涉及通信技术领域，特别涉及一种数据同步方法、系统、服务器及存储介质。
云平台为了实现容灾，通常是通过在不同的数据中心形成副本来实现的，当一个数据中心发生故障时，启用其它数据中心的副本来实现容灾功能。
然而，云平台该种实现容灾的方式是基于延展集群的，即多个数据中心共用一个存储集群，当一个数据中心发生故障时，虽然可以使用其它数据中心的副本来进行容灾，但是其集群的稳定性已经受到了影响。
发明内容
本申请实施例提供了一种数据同步方法，包括：检测主集群是否有数据写入；在检测到所述主集群有数据写入时，将所述数据同步至所述镜像集群，所述主集群与所述镜像集群在同一存储环境下预先创建。
本申请实施例还提供了一种数据同步系统，包括主集群和镜像集群，所述主集群和镜像集群在同一存储环境下预先创建；所述镜像集群用于检测所述主集群是否有数据写入，在检测到所述主集群有数据写入时，将所述数据同步至所述镜像集群。
本申请实施例还提供了一种服务器，包括：至少一个处理器；以及，与所述至少一个处理器通信连接的存储器；其中，所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行上述的数据同步方法。
本申请实施例还提供了一种计算机可读存储介质，存储有计算机程序，所述计算机程序被处理器执行时实现上述的数据同步方法。
一个或多个实施例通过与之对应的附图中的图片进行示例性说明，这些示例性说明并不构成对实施例的限定。
图1是本申请实施例提供的数据同步方法的流程示意图；
图2是本申请实施例提供的数据同步方法在云平台上应用的流程示意图；
图3是本申请实施例提供的数据同步方法在开启镜像服务的示例图；
图4是本申请实施例提供的数据同步方法在进行容灾配置的示例图；
图5是本申请实施例提供的数据同步方法的原理架构示例图；
图6是本申请实施例提供的数据同步方法中利用镜像服务更新存储池数据的原理示意图；
图7是本申请实施例提供的数据同步系统的结构示意图；
图8是本申请实施例提供的服务器的结构示意图。
本申请实施例的主要目的在于提出一种数据同步方法、系统、服务器及存储介质，可以提高云平台的集群在容灾时的稳定性。
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合附图对本申请的各实施例进行详细的阐述。然而，本领域的普通技术人员可以理解，在本申请各实施例中，为了使读者更好地理解本申请而提出了许多技术细节。但是，即使没有这些技术细节和基于以下各实施例的种种变化和修改，也可以实现本申请所要求保护的技术方案。以下各个实施例的划分是为了描述方便，不应对本申请的具体实现方式构成任何限定，各个实施例在不矛盾的前提下可以相互结合相互引用。
在一个实施例中，涉及一种数据同步方法，通过检测主集群是否有数据写入，在检测到主集群有数据写入时，将数据同步至镜像集群，其中主集群与镜像集群在同一存储环境下预先创建，从而在主集群发生异常时，可以通过将读写对象切换至镜像集群进行容灾。由于是利用同一存储环境下的镜像集群来实现容灾的，因此在容灾时，正在使用的集群的稳定性不会受到影响，从而提高了云平台容灾时集群的稳定性。
应当说明的是，本申请实施例提供的数据同步方法的执行主体可以为镜像集群。可以理解的是，主集群与镜像集群是相对概念，是指一种主从关系，主集群作为主角色，镜像集群作为从角色。当主集群异常时，可以将镜像集群切换为主角色，将主集群切换为从角色，此时可以将镜像集群为新的主集群，而原来的主集群则为新的镜像集群。此时本申请实施例提供的数据同步方法的执行主体可以为新的镜像集群，由新的镜像集群对新的主集群进行检测，在检测到新的主集群有数据写入时，将数据同步至新的镜像集群。具体实现中，主集群与镜像集群对应不同的服务器群，可选地，主集群与镜像集群属于不同的数据中心(DC)，从而实现异地容灾的效果。
本申请实施例提供的数据同步方法的具体流程如图1所示，包括以下步骤：
S101：检测主集群是否有数据写入。
由于主集群写入的数据并不一定都需要进行备份以达到容灾的效果，因此，在检测主集群是否有数据写入时，可以是检测是否有需要备份的数据写入。具体实现时，主集群可以将需要备份的数据写入一个指定的位置，镜像集群对主集群该指定的位置进行检测即可。
现有技术中在进行数据备份时，均是主动将数据备份至不同的数据中心，原数据与备份数据是同步更新的。而本申请实施例提供的数据同步方法，并不是由主集群主动将数据备份至镜像集群，而是由镜像集群对主集群进行检测，在检测到主集群有数据写入时，才将数据备份至镜像集群，即原数据与备份数据是异步更新的。可以理解的是，不同数据中心之间的IO时延是比较差的，现有技术采用同步更新的方式，数据的备份会对数据更新的效率造成一定的影响，若想降低不同数据中心的时延，则需要增加硬件成本，例如数据中心之间使用万兆网络。而本申请实施例提供的数据同步方法，采用异步更新的方式，不会因为数据的备份 对主集群中的数据更新效率造成影响，因此不需要因为需要降低不同数据中心的时延而增加硬件成本。
S102：在检测到主集群有数据写入时，将数据同步至镜像集群，主集群与镜像集群在同一存储环境下预先创建。
同一存储环境可以是指同一管理门户下的存储环境，从而使镜像集群从主集群中同步数据。
在一个具体的例子中，在将数据同步至镜像集群之前，还包括：开启镜像集群基于ceph的镜像服务；在将数据同步至镜像集群时，是利用镜像集群的镜像服务将数据同步至镜像集群。
即主集群与镜像集群是基于ceph分布式存储环境下预先创建的，镜像集群基于ceph技术提供的镜像服务将主集群的数据同步至镜像集群。可选地，当主集群与镜像集群发生主从角色的切换时，可以开启主集群的镜像服务，从而利用镜像服务将镜像集群的数据同步至主集群。为了保证在主从角色切换时主集群的镜像服务是开启的，可以在创建主集群和镜像集群时同时开启镜像服务。其中，ceph是一种开源软件(软件定义的存储)存储平台，用于对象、块级和文件级存储。
在一个具体的例子中，主集群与镜像集群在云平台中不同的虚拟数据中心(Virtual Data Center，简称VDC)中预先创建，和/或，主集群和镜像集群基于超融合场景预先创建。通过在云平台中不同的虚拟数据中心中创建主集群和镜像集群，可以在云平台环境上实现本申请实施例提供的数据同步方法，进而实现云平台上的容灾；同时，主集群与镜像集群对应不同的虚拟数据中心也有利于提高容灾能力。另外，现有技术中，在实现容灾时，通常是通过磁阵复制的方式来实现的，这样的方案虽然通常适合较大的局点，但因为需要单独的存储设备，因而组网比较复杂。而本申请实施例提供的数据同步方法，在基于超融合场景预先创建主集群和镜像集群，由于在超融合场景下服务器既是计算节点，也是存储节点，因此不需要额外的存储设备，组网相对比较简单，部署上更加便捷，扩展性也较好。
请参考图2，其为本申请实施例提供的数据同步方法在云平台上应用的流程示意图，大概包括以下流程：1、搭建超融合环境；2、搭建ceph分布式存储集群(包括主集群和镜像集群)；3、根据需要备份的数据创建存储池；4、开启主集群和镜像集群的镜像服务(ceph-rbd-mirror@admin.service)，其具体实现可以参考图3；5、绑定主集群和镜像集群(及存储池)进行容灾配置，其具体实现可以参考图4；5、云平台在不同的虚拟中心分别添加主集群和镜像集群的存储库。
在一个具体的例子中，主集群和镜像集群包括若干名称相同的第一存储池，在检测到主集群有数据写入时，将数据同步至镜像集群，包括：在检测到主集群的第一存储池有数据写入时，通过镜像集群的镜像服务将数据同步至镜像集群的第一存储池中。
也即，主集群的第一存储池是需要进行数据备份的，当主集群的第一存储池有数据写入时，镜像集群利用镜像服务将数据同步至镜像集群中与主集群的第一存储池名称相同的第一存储池中。例如，若主集群中第一存储池A有数据写入时，镜像集群利用镜像服务将数据同步至镜像集群的第一存储池A中；若主集群中第一存储池B有数据写入时，镜像集群利用镜像服务将数据同步至镜像集群的第一存储池B中。具体实现中，若主集群中某一存储池的数据需要备份，则在镜像集群中创建与该存储池名称相同的存储池，而这两个存储池即为名称 相同的第一存储池。应当说明的是，第一存储池中的“第一”仅用于指代需要进行数据备份的存储池，并不作为顺序或符号等其它限制。
具体实现中，主集群中的第一存储池下的新创建的rbd(rados block device，Rados块设备)需要开启journal特性和exclusive-lock特性，当mirroring state为enabled时，表明该rbd处于正常的数据同步状态，mirroring primary为true时表明该rbd为主角色。同时，镜像集群中的第一存储池中也会创建出名称相同的rbd，mirroring primary为false，表明其为主rbd的从角色。
可以理解的是，主集群与镜像集群对应不同的数据中心可以达到异地容灾的效果。
示例性地，主集群或镜像集群的第一存储池可以分布于不同的数据中心，例如第一存储池A分布于数据中心A，第一存储池B分布于数据中心B等，这样可以达到更好的异地容灾效果。
请参考图5，其为本申请实施例提供的数据同步方法的原理架构示例图。具体地，主集群与镜像集群分布于不同的DC中(主集群分布于DC1，而镜像集群分布于DC2),镜像集群通过镜像服务(图中的rbd-mirror)实现与主集群的数据同步，主集群与镜像集群中包括N个名称相同的存储池(pool1，pool2，pool3……)。
镜像集群通过镜像服务将数据同步至主集群名称相同的存储池，可以实现镜像集群与主集群中存储池的数据同步，同时也方便主集群通过指定存储池的方式来指定需要备份的数据。
在一个具体的例子中，检测主集群是否有数据写入，可以包括：通过镜像集群的镜像服务检测主集群的第一存储池的journal中是否有数据更新；而在检测到主集群的第一存储池有数据写入时，通过镜像服务将数据同步至镜像集群的第一存储池中，则可以包括：若检测到主集群的第一存储池的journal中有数据更新，则利用镜像服务从journal中读取更新的数据写入镜像集群的第一存储池中。
请参考图6，其为本申请实施例提供的数据同步方法中利用镜像服务更新存储池数据的原理示意图。具体地，当产生I/O时，数据首先会写入主集群的image journal中，当journal写入成功后，journal中的数据会写入主集群的rbd中并回复响应。同时，镜像集群的rbd-mirror进程(镜像服务)检测到主集群的journal有数据更新后，便从主集群的journal读取数据，写入镜像集群的rbd image中，从而完成数据的同步。当镜像集群中的image写入成功后，会更新主集群中的元数据，表示这次IO的journal已经同步完成。另外，可以通过查看存储池中的image状态是否处于正常的容灾模式(replaying)和存储池的health是否为OK来实时监控镜像存储池的容灾状态。
通过镜像服务来检测主集群的第一存储池的journal中是否有数据更新，在检测到有数据更新时，通过镜像服务将数据同步至镜像集群的第一存储池中，可以实现镜像集群对主集群数据更新的主动检测，实现数据的异步更新，降低镜像集群与主集群之间的IO时延需求。
在一个具体的例子中，在将数据同步至镜像集群之后，还包括：若检测到主集群异常，则将读写对象切换至镜像集群或镜像集群的第一存储池。
即在检测到主集群异常时，将镜像集群或镜像集群中的第一存储池提升为主角色进行数据的读写。由于主集群异常时，可能仅为部分第一存储池发生异常，则可以将镜像集群中相应的第一存储池提升为主角色进行数据读写即可。相应地，检测主集群是否异常时，可以是检测主集群的第一存储池是否异常。
在将读写对象切换至镜像集群或镜像集群的第一存储池时，可以通过配置镜像集群来自动实现。由于主集群异常可能并不是真正的异常，因此为了减少切换读写对象，可以通过手动进行切换。
通过在检测到主集群异常时，将读写对象切换至镜像集群或镜像集群的第一存储池，可以有效地实现容灾的效果。
本申请实施例提供的数据同步方法，通过检测主集群是否有数据写入，在检测到主集群有数据写入时，将数据同步至镜像集群，其中主集群与镜像集群在同一存储环境下预先创建，从而在主集群发生异常时，可以将读写对象切换至镜像集群进行容灾。由于是利用同一存储环境下的镜像集群来实现容灾的，因此在容灾时，正在使用的集群的稳定性不会受到影响，从而提高了云平台容灾时集群的稳定性。
此外，本领域技术人员可以理解，上面各种方法的步骤划分，只是为了描述清楚，实现时可以合并为一个步骤或者对某些步骤进行拆分，分解为多个步骤，只要包括相同的逻辑关系，都在本专利的保护范围内；对算法中或者流程中添加无关紧要的修改或者引入无关紧要的设计，但不改变其算法和流程的核心设计都在该专利的保护范围内。
在一个实施例中，涉及一种数据同步系统200，如图7所示，包括主集群201和镜像集群202，主集群201和镜像集群202在同一存储环境下预先创建，详细说明如下：
示例性地，镜像集群202用于检测主集群201是否有数据写入，在检测到主集群201有数据写入时，将数据同步至镜像集群202。
示例性地，镜像集群202还用于开启镜像集群基于ceph的镜像服务，利用镜像服务将数据同步至镜像集群202。
示例性地，主集群201和镜像集群202包括若干名称相同的第一存储池，镜像集群202还用于在检测到主集群201的第一存储池有数据写入时，通过镜像服务将数据同步至镜像集群202的第一存储池中。
示例性地，镜像集群202还用于通过镜像服务检测主集群201的第一存储池的journal中是否有数据更新；若检测到journal中有数据更新，则利用镜像服务从journal中读取更新的数据写入镜像集群202的第一存储池中。
示例性地，镜像集群202还用于若检测到主集群异常，则将读写对象切换至镜像集群202或镜像集群202的第一存储池。
示例性地，主集群201与镜像集群202在云平台中不同的虚拟数据中心中预先创建。
示例性地，主集群201和镜像集群202基于超融合场景预先创建。
不难发现，本实施例为与前述方法的实施例相对应的系统实施例，本实施例可与前述方法的实施例互相配合实施。前述方法的实施例中提到的相关技术细节在本实施例中依然有效，为了减少重复，这里不再赘述。相应地，本实施例中提到的相关技术细节也可应用在前述方法的实施例中。
在一个实施例中，涉及一种服务器，如图8所示，包括：至少一个处理器301；以及，与至少一个处理器301通信连接的存储器302；其中，存储器302存储有可被至少一个处理器301执行的指令，指令被至少一个处理器301执行，以使至少一个处理器301能够执行上 述的数据同步方法。
其中，存储器和处理器采用总线方式连接，总线可以包括任意数量的互联的总线和桥，总线将一个或多个处理器和存储器的各种电路连接在一起。总线还可以将诸如外围设备、稳压器和功率管理电路等之类的各种其他电路连接在一起，这些都是本领域所公知的，因此，本文不再对其进行进一步描述。总线接口在总线和收发机之间提供接口。收发机可以是一个元件，也可以是多个元件，比如多个接收器和发送器，提供用于在传输介质上与各种其他装置通信的单元。经处理器处理的数据通过天线在无线介质上进行传输，进一步，天线还接收数据并将数据传送给处理器。
处理器负责管理总线和通常的处理，还可以提供各种功能，包括定时，外围接口，电压调节、电源管理以及其他控制功能。而存储器可以被用于存储处理器在执行操作时所使用的数据。
在一个实施例中，涉及一种计算机可读存储介质，存储有计算机程序。计算机程序被处理器执行时实现上述方法实施例。
即，本领域技术人员可以理解，实现上述实施例方法中的全部或部分步骤是可以通过程序来指令相关的硬件来完成，该程序存储在一个存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片等)或处理器(processor)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
本领域的普通技术人员可以理解，上述各实施例是实现本申请的具体实施例，而在实际应用中，可以在形式上和细节上对其作各种改变，而不偏离本申请的精神和范围。
Claims (10)
- 一种数据同步方法，包括：检测主集群是否有数据写入；在检测到所述主集群有数据写入时，将所述数据同步至镜像集群，所述主集群与所述镜像集群在同一存储环境下预先创建。
- 根据权利要求1所述的数据同步方法，其中，在所述数据同步至镜像集群之前，还包括：开启所述镜像集群基于ceph的镜像服务；所述将所述数据同步至镜像集群，包括：利用所述镜像服务将所述数据同步至所述镜像集群。
- 根据权利要求2所述的数据同步方法，其中，所述主集群和所述镜像集群包括若干名称相同的第一存储池；所述在检测到主集群有数据写入时，将所述数据同步至所述镜像集群，包括：在检测到所述主集群的第一存储池有数据写入时，通过所述镜像服务将所述数据同步至所述镜像集群的第一存储池中。
- 根据权利要求3所述的数据同步方法，其中，所述检测主集群是否有数据写入，包括：通过所述镜像服务检测所述主集群的第一存储池的journal中是否有数据更新；所述在检测到所述主集群的第一存储池有数据写入时，通过所述镜像服务将所述数据同步至所述镜像集群的第一存储池中，包括：若检测到所述journal中有数据更新，则利用所述镜像服务从所述journal中读取更新的数据写入所述镜像集群的第一存储池中。
- 根据权利要求3所述的数据同步方法，其中，在所述将所述数据同步至所述镜像集群之后，还包括：若检测到所述主集群异常，则将读写对象切换至所述镜像集群或所述镜像集群的第一存储池。
- 根据权利要求1至5中任一项所述的数据同步方法，其中，所述主集群与所述镜像集群在云平台中不同的虚拟数据中心中预先创建。
- 根据权利要求1至5中任一项所述的数据同步方法，其中，所述主集群和所述镜像集群基于超融合场景预先创建。
- 一种数据同步系统，包括主集群和镜像集群，所述主集群和镜像集群在同一存储环境下预先创建；所述镜像集群用于检测所述主集群是否有数据写入，在检测到所述主集群有数据写入时，将所述数据同步至所述镜像集群。
- 一种服务器，包括：至少一个处理器；以及，与所述至少一个处理器通信连接的存储器；其中，所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行如权利要求1至7任一项所述的数据同步方法。
- 一种计算机可读存储介质，存储有计算机程序，所述计算机程序被处理器执行时实现如权利要求1至7任一项所述的数据同步方法。
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110939058.2A CN115705269A (zh) | 2021-08-16 | 2021-08-16 | 数据同步方法、系统、服务器及存储介质 |
| CN202110939058.2 | 2021-08-16 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2023019953A1 true WO2023019953A1 (zh) | 2023-02-23 |
Family
ID=85181235
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2022/083501 WO2023019953A1 (zh) | 2021-08-16 | 2022-03-28 | 数据同步方法、系统、服务器及存储介质 |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN115705269A (zh) |
| WO (1) | WO2023019953A1 (zh) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117573295A (zh) * | 2024-01-16 | 2024-02-20 | 之江实验室 | 一种基于k8s的多集群组件管理方法、装置及计算机设备 |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108512693A (zh) * | 2018-02-24 | 2018-09-07 | 国家计算机网络与信息安全管理中心 | 一种跨区域容灾方法和装置 |
| CN109408280A (zh) * | 2017-08-17 | 2019-03-01 | 北京金山云网络技术有限公司 | 数据备份方法、装置及系统 |
| CN110442646A (zh) * | 2019-07-29 | 2019-11-12 | 北京易捷思达科技发展有限公司 | 一种ceph数据同步模块主端写性能优化系统及方法 |
| CN110456984A (zh) * | 2019-06-21 | 2019-11-15 | 南京壹进制信息科技有限公司 | 一种对Ceph存储的块设备进行连续数据保护的方法 |
| CN111611109A (zh) * | 2020-05-22 | 2020-09-01 | 苏州浪潮智能科技有限公司 | 一种分布式集群的备份方法、系统、设备以及介质 |
| US20200333977A1 (en) * | 2019-04-17 | 2020-10-22 | Vmware, Inc. | Throttling resynchronization operations in a data store cluster based on i/o bandwidth limits |
| CN111857969A (zh) * | 2020-07-30 | 2020-10-30 | 江苏方天电力技术有限公司 | 一种数据灾备重建方法、系统、装置及存储介质 |
| CN111917846A (zh) * | 2020-07-19 | 2020-11-10 | 中信银行股份有限公司 | 一种Kafka集群切换方法、装置、系统、电子设备及可读存储介质 |
-
2021
- 2021-08-16 CN CN202110939058.2A patent/CN115705269A/zh active Pending
-
2022
- 2022-03-28 WO PCT/CN2022/083501 patent/WO2023019953A1/zh unknown
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109408280A (zh) * | 2017-08-17 | 2019-03-01 | 北京金山云网络技术有限公司 | 数据备份方法、装置及系统 |
| CN108512693A (zh) * | 2018-02-24 | 2018-09-07 | 国家计算机网络与信息安全管理中心 | 一种跨区域容灾方法和装置 |
| US20200333977A1 (en) * | 2019-04-17 | 2020-10-22 | Vmware, Inc. | Throttling resynchronization operations in a data store cluster based on i/o bandwidth limits |
| CN110456984A (zh) * | 2019-06-21 | 2019-11-15 | 南京壹进制信息科技有限公司 | 一种对Ceph存储的块设备进行连续数据保护的方法 |
| CN110442646A (zh) * | 2019-07-29 | 2019-11-12 | 北京易捷思达科技发展有限公司 | 一种ceph数据同步模块主端写性能优化系统及方法 |
| CN111611109A (zh) * | 2020-05-22 | 2020-09-01 | 苏州浪潮智能科技有限公司 | 一种分布式集群的备份方法、系统、设备以及介质 |
| CN111917846A (zh) * | 2020-07-19 | 2020-11-10 | 中信银行股份有限公司 | 一种Kafka集群切换方法、装置、系统、电子设备及可读存储介质 |
| CN111857969A (zh) * | 2020-07-30 | 2020-10-30 | 江苏方天电力技术有限公司 | 一种数据灾备重建方法、系统、装置及存储介质 |
Non-Patent Citations (1)
| Title |
|---|
| ANONYMOUS: " Distributed Storage Ceph RBD-Mirror Disaster Recovery Solution", 20 April 2021 (2021-04-20), XP093037434, Retrieved from the Internet <URL:https://blog.51cto.com/u_15174006/2718987> [retrieved on 20230404] * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117573295A (zh) * | 2024-01-16 | 2024-02-20 | 之江实验室 | 一种基于k8s的多集群组件管理方法、装置及计算机设备 |
| CN117573295B (zh) * | 2024-01-16 | 2024-05-28 | 之江实验室 | 一种基于k8s的多集群组件管理方法、装置及计算机设备 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115705269A (zh) | 2023-02-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3062226B1 (en) | Data replication method and storage system | |
| US6950915B2 (en) | Data storage subsystem | |
| US10678663B1 (en) | Synchronizing storage devices outside of disabled write windows | |
| CN104156361B (zh) | 一种实现数据同步的方法及系统 | |
| JP4170056B2 (ja) | 複製ボリューム間でのバックアップ・リストア管理方法およびこの方法に用いる記憶制御装置 | |
| US20010049749A1 (en) | Method and system for storing duplicate data | |
| KR20150035507A (ko) | 데이터 송신 방법, 데이터 수신 방법, 및 저장 장치 | |
| US9483367B1 (en) | Data recovery in distributed storage environments | |
| KR20110044858A (ko) | 데이터 센터들에 걸쳐 데이터 서버들내 데이터 무결정의 유지 | |
| US10459813B2 (en) | System and device for synchronizing data in a plurality of devices | |
| US9841923B2 (en) | Storage apparatus and storage system | |
| CN101808127A (zh) | 数据备份方法、系统和服务器 | |
| CN109254873B (zh) | 数据备份方法、相关装置及系统 | |
| CN113010549A (zh) | 基于异地多活系统的数据处理方法、相关设备及存储介质 | |
| CN115292408A (zh) | MySQL数据库的主从同步方法、装置、设备及介质 | |
| WO2023019953A1 (zh) | 数据同步方法、系统、服务器及存储介质 | |
| CN103986771A (zh) | 一种不依赖于共享存储的高可用集群管理方法 | |
| CN113064766A (zh) | 数据备份方法、装置、设备及存储介质 | |
| CN110121712A (zh) | 一种日志管理方法、服务器和数据库系统 | |
| WO2023151443A1 (zh) | 同步主备数据库 | |
| CN116389233A (zh) | 容器云管理平台主备切换系统、方法、装置和计算机设备 | |
| CN115563221A (zh) | 数据同步方法、存储系统、设备及存储介质 | |
| US11416160B1 (en) | Volume operation propagation for synchronous remote copy | |
| WO2024093263A1 (zh) | 数据处理系统、方法、装置及相关设备 | |
| US11379328B2 (en) | Transitioning from a donor four site data replication system to a target four site data replication system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22857272 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |