CN112527562A - Data backup method and device of distributed storage system and computer equipment - Google Patents
Data backup method and device of distributed storage system and computer equipment Download PDFInfo
- Publication number
- CN112527562A CN112527562A CN202011476984.2A CN202011476984A CN112527562A CN 112527562 A CN112527562 A CN 112527562A CN 202011476984 A CN202011476984 A CN 202011476984A CN 112527562 A CN112527562 A CN 112527562A
- Authority
- CN
- China
- Prior art keywords
- data
- backup
- snapshot
- block
- center
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 239000012634 fragment Substances 0.000 claims abstract description 27
- 238000013467 fragmentation Methods 0.000 claims abstract description 9
- 238000006062 fragmentation reaction Methods 0.000 claims abstract description 9
- 238000004590 computer program Methods 0.000 claims description 26
- 239000003795 chemical substances by application Substances 0.000 description 14
- 230000005540 biological transmission Effects 0.000 description 9
- 230000006870 function Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000011084 recovery Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000013468 resource allocation Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 206010033799 Paralysis Diseases 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/128—Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Abstract
The application relates to a data backup method and device of a distributed storage system, computer equipment and a storage medium. The method comprises the following steps: the method comprises the steps that fragmentation is carried out on data stored on a first piece of equipment to obtain fragmentation data; generating a snapshot for the fragment data to obtain first snapshot data; the first snapshot data and the metadata of the first block of equipment are uploaded to a disaster recovery center, and the data are fragmented and then regenerated into snapshots, so that the snapshot time is shortened, the backup time is shortened, and the backup efficiency is improved; and the influence on the I/O function of the block device is reduced, so that the stability during backup is good.
Description
Technical Field
The present application relates to the field of cloud storage technologies, and in particular, to a data backup method and apparatus for a distributed storage system, a computer device, and a storage medium.
Background
With the development of cloud storage technology and the continuous increase of various business scales, a single data center backup cannot meet the disaster tolerance requirements of businesses, for example, double-eleven activities of domestic fire-blast tianmao and black friday abroad, the explosive data request amount in a certain period may cause the whole data center to be paralyzed only by the operation of a single data center, so that many cloud computing companies propose disaster tolerance solutions. When a disaster happens, the data center can be automatically switched, and the continuity of the service is guaranteed.
In the existing disaster recovery backup mode based on the ceph storage system, the function is unstable and the backup efficiency is low.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data backup method, apparatus, computer device and storage medium for a distributed storage system, which can improve backup efficiency.
A data backup method of a distributed storage system is applied to a local production center and comprises the following steps:
fragmenting data stored on a first piece of equipment to obtain fragmented data;
generating a snapshot from the fragment data to obtain first snapshot data;
and uploading the first snapshot data and the metadata of the first block device to the disaster recovery center.
In one embodiment, after the snapshot generating the sharded data to obtain the first snapshot data, the method further includes:
generating a first check value according to the first snapshot data;
uploading the first snapshot data and the metadata of the first block device to a disaster recovery center, comprising:
and uploading the first snapshot data, the first check value and the metadata of the first block device to the disaster recovery center.
In one embodiment, uploading the first snapshot data, the first check value, and the metadata of the first block device to the disaster recovery center includes:
compressing the first snapshot data, the first check value and the metadata of the first block device to obtain compressed data;
and uploading the compressed data to a disaster recovery center.
In one embodiment, the slicing data obtained by slicing the data stored in the first block device includes: and receiving a backup instruction sent by the disaster backup center, and segmenting the data stored on the first block of equipment at preset time according to the backup instruction to obtain segmented data.
A data backup method of a distributed storage system is applied to a disaster recovery center, and comprises the following steps:
acquiring first snapshot data and metadata of a first block device, which are sent by a local production center;
inquiring to obtain information of a second block of equipment according to the metadata of the first block of equipment and a backup relation, wherein the backup relation refers to the backup relation between the first block of equipment of the local production center and the second block of equipment of the disaster backup center;
and writing the first snapshot data into a second block device corresponding to the second device information.
In one embodiment, the obtaining the first snapshot data and the metadata of the first block device sent by the local production center includes:
acquiring first snapshot data, a first check value and metadata of a first block of equipment, which are sent by a local production center;
writing the first snapshot data into a second block device corresponding to the second device information, including:
generating a second check value according to the first snapshot data;
checking whether the first check value is consistent with the second check value;
and if the first check value is consistent with the second check value, writing the first snapshot data into a second block device corresponding to the second device information.
In one embodiment, the method further comprises:
and sending a backup instruction to a local production center at preset time, so that the local production center generates the first snapshot data and the metadata of the first block of equipment according to the backup instruction and sends the first snapshot data and the metadata to the disaster recovery center.
A data backup device of a distributed storage system is applied to a local production center, and comprises:
the fragmentation module is used for fragmenting data stored on the first block of equipment to obtain fragmented data;
the snapshot module is used for generating a snapshot from the fragment data to obtain first snapshot data;
and the uploading module is used for uploading the first snapshot data and the metadata of the first block device to the disaster recovery center.
A data backup device of a distributed storage system is applied to a disaster recovery center, and comprises:
the acquisition module is used for acquiring first snapshot data and metadata of a first block of equipment, which are sent by a local production center;
the searching module is used for inquiring a backup relationship according to the metadata of the first block device to obtain information of a second block device, wherein the backup relationship refers to the backup relationship between the first block device of the local production center and the second block device of the disaster backup center;
and the writing module is used for writing the first snapshot data into a second block device corresponding to the second device information.
A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the data backup method of the distributed storage system provided by the invention when executing the computer program.
A computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements a data backup method of a distributed storage system provided above according to the present invention.
According to the data backup method and device, the computer equipment and the storage medium of the distributed storage system, the local production center fragments the data stored on the first piece of equipment to obtain fragmented data; generating a snapshot from the fragment data to obtain first snapshot data; uploading the first snapshot data and the metadata of the first block device to a disaster recovery center; after receiving the first snapshot data and the metadata of the first block device, the disaster backup center queries in a backup relationship according to the metadata of the first block device to obtain information of a second block device; compared with the existing backup method, the method has the advantages that the first snapshot data are written into the second block device corresponding to the second device information, and the data stored on the first block device are fragmented and then regenerated into the snapshot, so that the snapshot time is shortened, the backup time is shortened, and the backup efficiency is improved; and the influence on the I/O function of the block device is reduced, so that the stability of the backup process is good.
Drawings
FIG. 1 is a diagram of an application environment of a data backup method of a distributed storage system in one embodiment;
FIG. 2 is a flowchart illustrating a data backup method for a distributed storage system in one embodiment;
FIG. 3 is a flowchart illustrating a data backup method for a distributed storage system according to another embodiment;
FIG. 4 is a block diagram of a data backup apparatus of a distributed storage system in one embodiment;
FIG. 5 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The data backup method of the distributed storage system provided by the application can be applied to the application environment shown in fig. 1. Wherein the local production center 102 communicates with the disaster recovery center 104 over a gigabit network. The local production center is connected with the client to receive the data and write the data into the block equipment of the distributed storage system of the local production center; the block devices of the distributed storage system of disaster recovery center 104 are used to backup the block devices of the local production center. The local production center 102 and the disaster recovery center 104 can be implemented by independent servers or a server cluster composed of a plurality of servers.
In one embodiment, as shown in fig. 2, a data backup method of a distributed storage system is provided, which is described by taking the method as an example of being applied to the local production center in fig. 1, and includes the following steps:
The local production center is used for receiving the I/O data generated by the client and storing the I/O data in a first piece of equipment of a distributed storage system of the local production center. A block is a sequence of bytes (e.g., a 512-byte block of data). A Block Device (RBD), which is one type of I/O Device, stores information in blocks of a fixed size, each Block having its own address, and can read data of a certain length at any position of the Device, and the Block Device can be fully compatible with existing applications, and thus has a wide range of use.
Specifically, the data stored in the first block device of the local production center is fragmented to obtain fragmented data, the data stored in the first block device may be fragmented according to a preset fragmentation size, and the data stored in the first block device is divided into fragmented data of the same size, for example, the fragmentation size may be 64MB or 128MB, and may be set according to a user requirement, which is not limited herein.
The snapshot is a data copy technology based on a time point, and refers to a read-only copy of data at a certain time point. The purpose of the snapshot is to record and save data information at a certain moment, and if some faults occur later and data recovery is needed, the data can be recovered to the state of the previous time point through the snapshot.
In particular, a first block device of the local production center may create a snapshot for the data to preserve its history. If the first block device still performs the I/O operation during the snapshot, the snapshot may not obtain the accurate or latest data on the first block device, so the I/O operation may be stopped before the snapshot is performed on the first block device. After data stored in a first block device of a local production center is fragmented, the fragmented data is snapshot to obtain first snapshot data, and the snapshot data of each fragmented data has a starting address and an ending address in the first block device, so that the continuity of the data is ensured, and the subsequent backup is convenient according to the snapshot data. By making a snapshot on the fragment data, the efficiency of making the snapshot is improved, so that the backup efficiency is improved; meanwhile, the influence on the I/O performance of the whole first block device is reduced, so that the stability of the backup process is improved.
The metadata of the first block device stores first block device information, which includes a name and an ID number of the first block device, and the ID number is a unique identifier of the first block device.
Specifically, after the fragment data on the first block device is snapshot, the first snapshot data and the metadata of the first block device are uploaded to the disaster recovery center together, and the block device information of the fragment data, namely the name and the ID number of the first block device, can be obtained through the metadata of the first block device, so that the corresponding backup relationship can be found according to the first block device information, and the snapshot data is backed up to the disaster recovery center, so that the disaster recovery center backs up the snapshot data.
In the data backup method of the distributed storage system, fragmented data is obtained by fragmenting data stored on the first block of equipment; generating a snapshot from the fragment data to obtain first snapshot data; the first snapshot data and the metadata of the first block of equipment are uploaded to a disaster recovery center, and the data are fragmented and then regenerated into snapshots, so that the snapshot time is shortened, the backup time is shortened, and the backup efficiency is improved; in addition, the influence on the I/O function of the block device is reduced, so that the stability in the backup process is better.
In an embodiment, after the snapshot generating the sharded data to obtain the first snapshot data, the method further includes:
generating a first check value according to the first snapshot data;
uploading the first snapshot data and the metadata of the first block device to a disaster recovery center, comprising:
and uploading the first snapshot data, the first check value and the metadata of the first block device to the disaster recovery center.
In particular, the first check value may be used as a unique "digital fingerprint" of the first snapshot data to verify the integrity and consistency of the first snapshot data during transmission. For example, the first check value may be an MD5 value, a fragment MD5 value may be generated for snapshot data of each fragment data, and finally all the fragment MD5 values on the first block device are processed together as an MD5 value of the entire first block device, and it is known whether the first snapshot data has changed by whether the MD5 values calculated before and after transmission have changed.
The accuracy and the integrity of data transmission can be ensured by checking the check value.
In one embodiment, uploading the first snapshot data, the first check value, and the metadata of the first block device to the disaster recovery center includes:
compressing the first snapshot data, the first check value and the metadata of the first block device to obtain compressed data;
and uploading the compressed data to a disaster recovery center.
Specifically, the first snapshot data, the generated first check value with the check function and the metadata carrying the first piece of equipment information are compressed to obtain compressed data, the compressed data are uploaded to the disaster recovery center through a gigabit network, network bandwidth is saved in the transmission process of the compressed data, the data volume is small after the data are compressed, and the transmission speed is high; therefore, the backup method provided by the implementation has smaller requirements on network bandwidth and time delay.
In one embodiment, fragmenting data stored on a first block device to obtain fragmented data includes:
and receiving a backup instruction sent by the disaster backup center, and segmenting data stored on the first block of equipment at preset time according to the backup instruction to obtain segmented data.
Specifically, the local production center performs data storage through a first distributed cluster, where the first distributed cluster may be a ceph cluster, and the first block device may be a ceph block device. And configuring a scheduler component and an agent component on the ceph cluster. The scheduler component is a scheduling component and is used for sending a backup instruction; the scheduler component can operate in a local production center or a disaster recovery center, and compared with the disaster recovery center, the local production center often breaks down, so the scheduler component mainly operates in the disaster recovery center, when the disaster recovery center breaks down, the scheduler component operates in the local production center, the scheduler component is used for sending a backup instruction to an agent component (located in the local production center), the agent component executes a task of data backup on a first piece of equipment at preset time, the preset time can be determined according to a time period with less I/O of the local production center obtained through data statistics, for example, nine night every three weeks or two early morning every day, and backup can be more intelligent and flexible by setting backup cycle and time point in a customized manner. Firstly, fragmenting data stored on a first piece of equipment through an agent component to obtain fragmented data; taking a snapshot of the fragment data to generate first snapshot data, and generating a first check value according to the first snapshot data; and finally, compressing the first snapshot data, the first check value and the metadata of the first block of equipment, and uploading the compressed data to the disaster recovery center. In this embodiment, after the first snapshot data is obtained, the first snapshot data of this time is compared with the first snapshot data backed up at the previous preset time, a difference part is found, an MD5 value is generated for the data of the difference part, and the data of the difference part, the MD5 value of the data of the difference part and the metadata of the first block device are compressed together and then uploaded to the disaster recovery center.
The backup instruction is sent to the agent component at the preset time through the scheduler component located in the disaster recovery center, and then the agent component executes tasks such as data fragmentation and snapshot on fragmented data, so that the time can be flexibly adjusted, peak time processing is avoided, the flexibility of resource allocation is improved, and the intelligence and the flexibility of backup are improved.
In one embodiment, as shown in fig. 3, a data backup method of a distributed storage system is provided, which is described by taking the method as an example of being applied to the disaster recovery center in fig. 1, and includes the following steps:
Specifically, the disaster recovery center provides a receiving window to acquire data transmitted from the local production center. And receiving the compressed data sent by the local production center through the receiving window, decompressing the compressed data, and obtaining the first snapshot data and the metadata of the first block of equipment sent by the local production center.
The first snapshot data is snapshot data obtained by snapshot of sharded data of data shards stored on the first block device.
And step 304, obtaining information of the second block of equipment according to the metadata of the first block of equipment and a backup relationship, wherein the backup relationship refers to the backup relationship between the first block of equipment of the local production center and the second block of equipment of the disaster backup center.
The backup relationship refers to a backup relationship between a first block of equipment of the local production center and a second block of equipment of the disaster recovery center, is stored in a database, and can be obtained through a scheduler component. For example, the name of the first block device of the production center is my _ test, the ID number is 123456, the first block device is used as disaster recovery, the name of the second block device of the corresponding disaster recovery center is my _ test _ bak, the ID number is 654321, and the information of the corresponding backup block device or the corresponding disaster recovery block device can be found according to the backup relationship and the information of any block device in the backup relationship.
Specifically, the name and the ID number of the first block device are obtained according to the metadata of the first block device, a backup relationship in the database is obtained through the scheduler component, and second block device information of the disaster recovery center corresponding to the first block device, namely the name and the ID number of the second block device, is found through the backup relationship.
Specifically, the disaster recovery center writes the decompressed first snapshot data into the second block device determined according to the backup relationship.
In one embodiment, obtaining the first snapshot data and the metadata of the first block device sent by the local production center comprises:
acquiring first snapshot data, a first check value and metadata of a first block of equipment, which are sent by a local production center;
writing the first snapshot data into a second block device corresponding to the second device information, including:
generating a second check value according to the first snapshot data;
checking whether the first check value is consistent with the second check value;
and if the first check value is consistent with the second check value, writing the first snapshot data into a second block device corresponding to the second device information.
Specifically, the data uploaded by the local production center further includes a first check value, the compressed data sent by the local production center is received through the receiving window, the compressed data is decompressed to obtain first snapshot data, the first check value and metadata of the first block device, the MD5 value, namely a second check value, is generated according to the decompressed first snapshot data, whether the first check value and the second check value are consistent or not is compared, if so, it is indicated that the first snapshot data is not changed in the transmission process, the data is complete and is consistent with the first snapshot data of the local production center, and the decompressed first snapshot data is written into the second block device.
The first snapshot data received by the disaster recovery center is verified, so that the accuracy and the integrity of data transmission can be ensured, the first snapshot data which is verified to be error-free is written into the disaster recovery center, and the consistency of the data of the local production center and the disaster recovery center can be ensured.
In one embodiment, the method further comprises:
and sending a backup instruction to the local production center at a preset time.
Specifically, the disaster recovery center performs data storage through a second distributed cluster, where the second distributed cluster may be a ceph cluster, and the second device may be a ceph device. And configuring a scheduler component and a backup component on the ceph cluster, wherein the scheduler component is a scheduling component and can send a backup instruction to the local production center agent component through a network at preset time. After the local production center finishes the data snapshot, sending the snapshot data to the disaster recovery center, and after the backup component of the disaster recovery center receives the snapshot data sent by the local production center, executing a task of data backup on a second piece of equipment, wherein the preset time can be determined according to a time period with less local production center I/O obtained by data statistics, for example, nine nights every three weeks or two nights every morning, and the backup period and time point can be set by self-defining, so that the backup is more intelligent and flexible. Firstly, a backup component receives compressed data sent by a local production center, then decompresses the compressed data to obtain first snapshot data, a first check value and metadata of a first block of equipment, generates an MD5 value, namely a second check value, according to the decompressed first snapshot data, compares whether the first check value is consistent with the second check value, and if so, indicates that the first snapshot data is not changed in the transmission process; the scheduler component inquires the backup relation according to the metadata of the first piece of equipment to obtain second piece of equipment information; and the backup component writes the decompressed first snapshot data into a second block device corresponding to the second device information.
The deployment scheduler assembly is used for managing and controlling other assemblies, a backup instruction is sent to the agent assembly of the local production center at preset time, data backup and the like are carried out through the backup assembly, the time can be flexibly adjusted, peak time processing is avoided, the flexibility of resource allocation is improved, and meanwhile the intelligence and the flexibility of backup are improved.
In order to easily understand the technical solution provided by the embodiment of the present application, a data backup method of a distributed storage system provided by the embodiment of the present application is briefly described in a data backup process of a complete distributed storage system:
(1) a scheduler component of the disaster recovery center stores the backup relation in a database;
(2) a scheduler component of the disaster recovery center sends a backup instruction to a local production center agent component at preset time;
(3) after receiving the backup instruction, the agent component of the local production center fragments the data stored on the first piece of equipment to obtain fragmented data;
(4) the agent component generates snapshot for the fragment data to obtain first snapshot data, and generates a first check value according to the first snapshot data;
(5) the agent component compresses the first snapshot data, the first check value and the metadata of the first block of equipment to obtain compressed data, and uploads the compressed data to the disaster recovery center;
(6) a backup component located in the disaster recovery center receives compressed data uploaded by the local production center, and decompresses the compressed data to obtain first snapshot data, a first check value and metadata of the first block of equipment;
(7) a backup component located in the disaster recovery center obtains information of a second piece of equipment according to the metadata of the first piece of equipment and the backup relation stored in a database by the scheduler component;
(8) a backup component of the disaster recovery center generates a second check value according to the first snapshot data and checks whether the first check value is consistent with the second check value; and if the first check value is consistent with the second check value, writing the first snapshot data into a second block device corresponding to the second device information.
In the embodiment, the backup instruction is sent to the agent component of the local production center at the preset time through the scheduler component of the disaster recovery center, and the agent component fragments the data on the first block of equipment and then generates the snapshot according to the backup instruction, so that the snapshot time is shortened, the influence on the I/O function of the block of equipment is reduced, in addition, the time can be flexibly adjusted through the preset time backup, the peak time period processing is avoided, and the flexibility of resource allocation is improved; moreover, the first snapshot data, the first check value and the metadata of the first block device are compressed and then transmitted, so that the network bandwidth is saved, the data volume is small after the data is compressed, and the transmission speed is high, so that the backup method has lower requirements on the network bandwidth and the time delay; after receiving the first snapshot data, the backup component of the disaster recovery center verifies the first snapshot data, so that the integrity and consistency of the data of the local production center and the disaster recovery center can be ensured; after the data is checked to be correct, a backup relation is found according to the metadata of the first block device, corresponding second block device information is found according to the backup relation, the first snapshot data is written into the corresponding second block device, backup time is shortened, backup is more intelligent and flexible, and backup efficiency is improved.
It should be understood that although the various steps in the flow charts of fig. 2-3 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-3 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least some of the other steps.
In one embodiment, as shown in fig. 4, a data backup apparatus of a distributed storage system is provided, which is applied to a local production center, and includes: a sharding module 402, a snapshot module 404, and an upload module 406, wherein:
the fragmentation module 402 is configured to fragment data stored in the first block device to obtain fragmented data.
The snapshot module 404 is configured to generate a snapshot from the sliced data to obtain first snapshot data.
And an uploading module 406, configured to upload the first snapshot data and the metadata of the first block device to the disaster recovery center.
In one embodiment, the snapshot module 404 is further configured to generate a first check value according to the first snapshot data;
the uploading module 406 is further configured to upload the first snapshot data, the first check value, and the metadata of the first block device to the disaster recovery center.
In an embodiment, the uploading module 406 is further configured to compress the first snapshot data, the first check value, and the metadata of the first block device to obtain compressed data; the compressed data is then uploaded to the disaster recovery center.
In an embodiment, the fragmentation module 402 is further configured to receive a backup instruction sent by the disaster recovery center, and fragment data stored in the first block of device at preset time according to the backup instruction to obtain fragmented data.
In one embodiment, a data backup device of a distributed storage system is provided, which is applied to a disaster recovery center, and includes: the device comprises an acquisition module, an inquiry module and a write-in module, wherein:
the acquisition module is used for acquiring the first snapshot data and the metadata of the first block device sent by the local production center.
And the query module is used for querying to obtain the information of the second block of equipment according to the metadata of the first block of equipment and the backup relationship, wherein the backup relationship refers to the backup relationship between the first block of equipment of the local production center and the second block of equipment of the disaster backup center.
And the writing module is used for writing the first snapshot data into a second block device corresponding to the second device information.
In one embodiment, the obtaining module is further configured to obtain first snapshot data, a first check value, and metadata of the first block device, where the first snapshot data, the first check value, and the metadata are sent by the local production center;
the writing module is also used for generating a second check value according to the first snapshot data; then checking whether the first check value is consistent with the second check value; and if the first check value is consistent with the second check value, the disaster backup center backup module is used for writing the first snapshot data into a second block of equipment corresponding to the second equipment information.
In one embodiment, the writing module is further configured to send a backup instruction to the local production center at a preset time.
For specific limitations of the data backup apparatus of the distributed storage system, reference may be made to the above limitations of the data backup method of the distributed storage system, and details are not described here. The modules in the data backup device of the distributed storage system can be implemented in whole or in part by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing backup relationship data and block device data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data backup method for a distributed storage system.
Those skilled in the art will appreciate that the architecture shown in fig. 5 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided for use in a local production center, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:
fragmenting data stored on a first piece of equipment to obtain fragmented data;
generating a snapshot from the fragment data to obtain first snapshot data;
and uploading the first snapshot data and the metadata of the first block device to the disaster recovery center.
In one embodiment, the processor, when executing the computer program, further performs the steps of: after the first snapshot data is obtained by generating the snapshot of the fragment data, the method further comprises the following steps: generating a first check value according to the first snapshot data; uploading the first snapshot data and the metadata of the first block device to a disaster recovery center, comprising: and uploading the first snapshot data, the first check value and the metadata of the first block device to the disaster recovery center.
In one embodiment, the processor, when executing the computer program, further performs the steps of: uploading the first snapshot data, the first check value and the metadata of the first block device to a disaster recovery center, comprising: compressing the first snapshot data, the first check value and the metadata of the first block device to obtain compressed data; and uploading the compressed data to a disaster recovery center.
In one embodiment, the processor, when executing the computer program, further performs the steps of: the method for fragmenting data stored in a first block of equipment to obtain fragmented data includes: and receiving a backup instruction sent by the disaster backup center, and segmenting the data stored on the first block of equipment at preset time according to the backup instruction to obtain segmented data.
In one embodiment, a computer device is provided, which is applied to a disaster recovery center, and comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to realize the following steps: acquiring first snapshot data and metadata of a first block device, which are sent by a local production center; inquiring to obtain information of a second block of equipment according to the metadata of the first block of equipment and a backup relation, wherein the backup relation refers to the backup relation between the first block of equipment of the local production center and the second block of equipment of the disaster backup center; and writing the first snapshot data into a second block device corresponding to the second device information.
In one embodiment, the processor, when executing the computer program, further performs the steps of: generating a snapshot for the fragment data, and after obtaining the first snapshot data, further comprising: the method for acquiring the first snapshot data and the metadata of the first block device sent by the local production center comprises the following steps: acquiring first snapshot data, a first check value and metadata of a first block of equipment, which are sent by a local production center; writing the first snapshot data into a second block device corresponding to the second device information, including: generating a second check value according to the first snapshot data; checking whether the first check value is consistent with the second check value; and if the first check value is consistent with the second check value, writing the first snapshot data into a second block device corresponding to the second device information.
In one embodiment, the processor, when executing the computer program, further performs the steps of: and sending a backup instruction to a local production center at preset time, so that the local production center generates the first snapshot data and the metadata of the first block of equipment according to the backup instruction and sends the first snapshot data and the metadata to the disaster recovery center.
In one embodiment, a computer-readable storage medium is provided for use in a local production center, having a computer program stored thereon, which when executed by a processor, performs the steps of:
fragmenting data stored on a first piece of equipment to obtain fragmented data;
generating a snapshot from the fragment data to obtain first snapshot data;
and uploading the first snapshot data and the metadata of the first block device to the disaster recovery center.
In one embodiment, the computer program when executed by the processor further performs the steps of: after the first snapshot data is obtained by generating the snapshot of the fragment data, the method further comprises the following steps: generating a first check value according to the first snapshot data; uploading the first snapshot data and the metadata of the first block device to a disaster recovery center, comprising: and uploading the first snapshot data, the first check value and the metadata of the first block device to the disaster recovery center.
In one embodiment, the computer program when executed by the processor further performs the steps of: uploading the first snapshot data, the first check value and the metadata of the first block device to a disaster recovery center, comprising: compressing the first snapshot data, the first check value and the metadata of the first block device to obtain compressed data; and uploading the compressed data to a disaster recovery center.
In one embodiment, the computer program when executed by the processor further performs the steps of: the method for fragmenting data stored in a first block of equipment to obtain fragmented data includes: and receiving a backup instruction sent by the disaster backup center, and segmenting the data stored on the first block of equipment at preset time according to the backup instruction to obtain segmented data.
In one embodiment, a computer-readable storage medium is provided for use in a disaster recovery center, having a computer program stored thereon, which when executed by a processor, performs the steps of: acquiring first snapshot data and metadata of a first block device, which are sent by a local production center; inquiring to obtain information of a second block of equipment according to the metadata of the first block of equipment and a backup relation, wherein the backup relation refers to the backup relation between the first block of equipment of the local production center and the second block of equipment of the disaster backup center; and writing the first snapshot data into a second block device corresponding to the second device information.
In one embodiment, the computer program when executed by the processor further performs the steps of: generating a snapshot for the fragment data, and after obtaining the first snapshot data, further comprising: the method for acquiring the first snapshot data and the metadata of the first block device sent by the local production center comprises the following steps: acquiring first snapshot data, a first check value and metadata of a first block of equipment, which are sent by a local production center; writing the first snapshot data into a second block device corresponding to the second device information, including: generating a second check value according to the first snapshot data; checking whether the first check value is consistent with the second check value; and if the first check value is consistent with the second check value, writing the first snapshot data into a second block device corresponding to the second device information.
In one embodiment, the computer program when executed by the processor further performs the steps of: and sending a backup instruction to a local production center at preset time, so that the local production center generates the first snapshot data and the metadata of the first block of equipment according to the backup instruction and sends the first snapshot data and the metadata to the disaster recovery center.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (11)
1. A data backup method of a distributed storage system is applied to a local production center, and comprises the following steps:
fragmenting data stored on a first piece of equipment to obtain fragmented data;
generating a snapshot from the fragment data to obtain first snapshot data;
and uploading the first snapshot data and the metadata of the first block device to a disaster recovery center.
2. The method according to claim 1, wherein after snapshotting the sliced data into the first snapshot data, further comprising:
generating a first check value according to the first snapshot data;
the uploading the first snapshot data and the metadata of the first block device to a disaster recovery center includes:
and uploading the first snapshot data, the first check value and the metadata of the first block of equipment to a disaster recovery center.
3. The method of claim 2, wherein uploading the first snapshot data, the first check value, and the metadata of the first block device to a disaster recovery center comprises:
compressing the first snapshot data, the first check value and the metadata of the first block device to obtain compressed data;
and uploading the compressed data to a disaster recovery center.
4. The method of claim 1, wherein the fragmenting the data stored on the first block device to obtain fragmented data comprises:
and receiving a backup instruction sent by the disaster backup center, and segmenting data stored on the first block of equipment at preset time according to the backup instruction to obtain segmented data.
5. A data backup method of a distributed storage system is applied to a disaster recovery center, and comprises the following steps:
acquiring first snapshot data and metadata of a first block device, which are sent by a local production center;
inquiring to obtain information of a second block of equipment according to the metadata of the first block of equipment and a backup relation, wherein the backup relation refers to the backup relation between the first block of equipment of the local production center and the second block of equipment of the disaster backup center;
and writing the first snapshot data into a second block device corresponding to the second device information.
6. The method of claim 5, wherein the obtaining the first snapshot data and the metadata of the first block device sent by the local production center comprises:
acquiring the first snapshot data, the first check value and the metadata of the first block device, which are sent by a local production center;
the writing the first snapshot data into a second block device corresponding to the second device information includes:
generating a second check value according to the first snapshot data;
checking whether the first check value and the second check value are consistent;
and if the first check value is consistent with the second check value, writing the first snapshot data into a second block device corresponding to the second device information.
7. The method of claim 5, further comprising:
and sending a backup instruction to a local production center at preset time, so that the local production center generates the first snapshot data and the metadata of the first block of equipment according to the backup instruction and sends the first snapshot data and the metadata to the disaster recovery center.
8. A data backup device of a distributed storage system, applied to a local production center, the device comprising:
the fragmentation module is used for fragmenting data stored on the first block of equipment to obtain fragmented data;
the snapshot module is used for generating a snapshot from the fragment data to obtain first snapshot data;
and the uploading module is used for uploading the first snapshot data and the metadata of the first block of equipment to the disaster recovery center.
9. The utility model provides a distributed storage system's data backup device which characterized in that is applied to disaster recovery center, the device includes:
the acquisition module is used for acquiring first snapshot data and metadata of a first block of equipment, which are sent by a local production center;
the query module is used for querying according to the metadata of the first block device and a backup relationship to obtain information of a second block device, wherein the backup relationship refers to the backup relationship between the first block device of the local production center and the second block device of the disaster backup center;
and the writing module is used for writing the first snapshot data into a second block device corresponding to the second device information.
10. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 7.
11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011476984.2A CN112527562A (en) | 2020-12-15 | 2020-12-15 | Data backup method and device of distributed storage system and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011476984.2A CN112527562A (en) | 2020-12-15 | 2020-12-15 | Data backup method and device of distributed storage system and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112527562A true CN112527562A (en) | 2021-03-19 |
Family
ID=75000056
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011476984.2A Pending CN112527562A (en) | 2020-12-15 | 2020-12-15 | Data backup method and device of distributed storage system and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112527562A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113568788A (en) * | 2021-09-26 | 2021-10-29 | 成都云祺科技有限公司 | Snapshot method, system and storage medium for Linux non-logical volume block device |
| CN114661527A (en) * | 2022-05-23 | 2022-06-24 | 武汉四通信息服务有限公司 | Data backup method and device, electronic equipment and storage medium |
| CN114968671A (en) * | 2022-07-28 | 2022-08-30 | 云和恩墨(北京)信息技术有限公司 | Method and device for realizing clone volume |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150081993A1 (en) * | 2013-09-13 | 2015-03-19 | Vmware, Inc. | Incremental backups using retired snapshots |
| CN104932956A (en) * | 2015-06-19 | 2015-09-23 | 华南理工大学 | Big-data-oriented cloud disaster tolerant backup method |
| CN109284205A (en) * | 2018-09-21 | 2019-01-29 | 平安科技（深圳）有限公司 | Snap backup method, device, computer equipment and storage medium |
| CN111625396A (en) * | 2019-02-27 | 2020-09-04 | 阿里巴巴集团控股有限公司 | Backup data verification method, server and storage medium |
-
2020
- 2020-12-15 CN CN202011476984.2A patent/CN112527562A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150081993A1 (en) * | 2013-09-13 | 2015-03-19 | Vmware, Inc. | Incremental backups using retired snapshots |
| CN104932956A (en) * | 2015-06-19 | 2015-09-23 | 华南理工大学 | Big-data-oriented cloud disaster tolerant backup method |
| CN109284205A (en) * | 2018-09-21 | 2019-01-29 | 平安科技（深圳）有限公司 | Snap backup method, device, computer equipment and storage medium |
| CN111625396A (en) * | 2019-02-27 | 2020-09-04 | 阿里巴巴集团控股有限公司 | Backup data verification method, server and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 杨皓森等: "面向OpenStack/Ceph的虚拟机备份系统研究", 《计算机系统应用》, vol. 27, no. 11, pages 96 - 102 * |
| 黎婷婷;李;: "一种改进的CDP快照方法", 信息安全研究, no. 03, pages 276 - 280 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113568788A (en) * | 2021-09-26 | 2021-10-29 | 成都云祺科技有限公司 | Snapshot method, system and storage medium for Linux non-logical volume block device |
| CN114661527A (en) * | 2022-05-23 | 2022-06-24 | 武汉四通信息服务有限公司 | Data backup method and device, electronic equipment and storage medium |
| CN114968671A (en) * | 2022-07-28 | 2022-08-30 | 云和恩墨(北京)信息技术有限公司 | Method and device for realizing clone volume |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11327799B2 (en) | Dynamic allocation of worker nodes for distributed replication | |
| US20200348852A1 (en) | Distributed object replication architecture | |
| CN112527562A (en) | Data backup method and device of distributed storage system and computer equipment | |
| US10585746B2 (en) | Tracking status and restarting distributed replication | |
| US11349915B2 (en) | Distributed replication and deduplication of an object from a source site to a destination site | |
| US11429566B2 (en) | Approach for a controllable trade-off between cost and availability of indexed data in a cloud log aggregation solution such as splunk or sumo | |
| US7657533B2 (en) | Data management systems, data management system storage devices, articles of manufacture, and data management methods | |
| CN108762984B (en) | Method and device for continuous data backup | |
| US20190227710A1 (en) | Incremental data restoration method and apparatus | |
| CN113064760B (en) | Database synthesis backup method and device, computer equipment and storage medium | |
| CN108810055B (en) | Large file transmission method and device | |
| CN113051102B (en) | File backup method, device, system, storage medium and computer equipment | |
| US10884650B1 (en) | Opportunistic compression of replicated data in a content addressable storage system | |
| CN114138549A (en) | Data backup and recovery method based on kubernets system | |
| CN112579550B (en) | Metadata information synchronization method and system of distributed file system | |
| CN114020522A (en) | Data backup method and device, electronic equipment and system | |
| US20180307437A1 (en) | Backup control method and backup control device | |
| CN113348450A (en) | Vehicle-mounted data storage method and system | |
| CN112698987A (en) | On-line backup method, device, equipment and storage medium for snapshot-free operating system | |
| CN114020691B (en) | Read-write separated data updating method and device and KV storage system | |
| CN113704027B (en) | File aggregation compatible method and device, computer equipment and storage medium | |
| CN112698926B (en) | Data processing method, device, equipment, storage medium and system | |
| US10684922B2 (en) | Enhanced data storage using compressed data | |
| CN112966046B (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN114840054B (en) | Time synchronization method, device, system, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |