WO2022198429A1 - Data redundancy backup method and apparatus, storage medium, and electronic device - Google Patents

Data redundancy backup method and apparatus, storage medium, and electronic device Download PDF

Info

Publication number
WO2022198429A1
WO2022198429A1 PCT/CN2021/082263 CN2021082263W WO2022198429A1 WO 2022198429 A1 WO2022198429 A1 WO 2022198429A1 CN 2021082263 W CN2021082263 W CN 2021082263W WO 2022198429 A1 WO2022198429 A1 WO 2022198429A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
data block
redundant
redundant data
blocks
Prior art date
Application number
PCT/CN2021/082263
Other languages
French (fr)
Chinese (zh)
Inventor
唐之享
Original Assignee
深圳市欢太科技有限公司
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市欢太科技有限公司, Oppo广东移动通信有限公司 filed Critical 深圳市欢太科技有限公司
Priority to CN202180090137.9A priority Critical patent/CN116710899A/en
Priority to PCT/CN2021/082263 priority patent/WO2022198429A1/en
Publication of WO2022198429A1 publication Critical patent/WO2022198429A1/en

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14Error detection or correction of the data by redundancy in operation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00Arrangements for detecting or preventing errors in the information received

Definitions

  • the present application relates to the field of computer technology, and in particular, to a data redundancy backup method, device, storage medium and electronic device.
  • the current cloud storage system has three availability zones.
  • the data to be saved is redundantly backed up by the erasure code strategy, and the data blocks and check blocks are averaged. Stored in three availability zones.
  • the damaged data blocks can be recovered by extracting the original number of data blocks and check blocks, but this method should also be used when only a single data block is damaged. Restoration takes a lot of computing memory of the cloud storage system and requires a lot of repair costs.
  • Embodiments of the present application provide a data redundancy backup method, device, storage medium, and electronic device. By first splitting the original data, and then performing an erasure code operation on the split data, the first data When there is less damaged data in the segment and the second data segment, the data in the availability zone can be used for repair, thereby saving the cost of data repair.
  • the technical solution is as follows:
  • an embodiment of the present application provides a data redundancy backup method, the method comprising:
  • the first redundant data, the second redundant data and the third redundant data are respectively stored in different availability zones in the set of availability zones.
  • an embodiment of the present application provides a data redundancy backup device, and the device includes:
  • a data splitting module for splitting the original data into a first data segment and a second data segment
  • an erasure code operation module configured to perform an erasure code operation on the first data segment and the second data segment respectively to obtain first redundant data and second redundant data;
  • a logic operation module configured to generate third redundant data based on the logical relationship between the first redundant data and the second redundant data
  • the first redundant data, the second redundant data and the third redundant data are respectively stored in different availability zones in the set of availability zones.
  • an embodiment of the present application provides a computer storage medium, where the computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the foregoing method steps.
  • an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein, the memory stores a computer program, and the computer program is adapted to be loaded by the processor and execute the above method steps .
  • the original data is split into a first data segment and a second data segment, and an erasure code operation is performed on the first data segment and the second data segment respectively to obtain the first data segment and the second data segment.
  • a redundant data and a second redundant data, and the third redundant data is generated based on the logical relationship between the first redundant data and the second redundant data.
  • FIG. 1 is a schematic diagram of an example of a data redundancy backup device used for cloud storage provided by an embodiment of the present application
  • FIG. 2 is a schematic flowchart of a data redundancy backup method provided by an embodiment of the present application
  • FIG. 3 is a schematic flowchart of a data redundancy backup method provided by an embodiment of the present application.
  • 3a is a schematic diagram illustrating an example of an erasure code operation rule provided by an embodiment of the present application
  • FIG. 3b is an exemplary schematic diagram of adding a supplementary data block provided by an embodiment of the present application.
  • 3c is an exemplary schematic diagram of a third redundant data generation provided by an embodiment of the present application.
  • FIG. 4 is a schematic flowchart of a data restoration process provided by an embodiment of the present application.
  • Fig. 4a is a schematic diagram of an example of erasure correction code reconstruction provided by an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of a data redundancy backup device provided by an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of a data redundancy backup device provided by an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of a data splitting module provided by an embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of an erasure code computing module provided by an embodiment of the present application.
  • FIG. 9 is a schematic structural diagram of a logic operation module provided by an embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of a first data restoration module provided by an embodiment of the present application.
  • FIG. 11 is a schematic structural diagram of a second data restoration module provided by an embodiment of the present application.
  • FIG. 12 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • a plurality means two or more.
  • “And/or”, which describes the association relationship of the associated objects means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone.
  • the character “/” generally indicates that the associated objects are an "or" relationship.
  • the data redundancy backup method provided by the embodiments of the present application can be implemented by relying on a computer program, and can be run on a data redundancy backup device based on the von Neumann system.
  • the computer program can be integrated into an application or run as a stand-alone utility application.
  • the data redundancy backup device in the embodiment of the present application may be a cloud storage device, such as a server cluster, which may be used to provide data storage and access services, ensure data security, and save storage space.
  • the data redundancy backup device may It includes at least three availability zones. After the data redundancy backup device receives the uploaded original data, it can perform redundant backup processing on the original data, and split the processed data into at least three availability zones.
  • the original data when receiving the access service for the original data, the original data can be obtained from these at least three availability zones; when the original data in the availability zone is damaged, the data in the same availability zone or other availability zones can also be used restore the original data.
  • the method for processing the redundant backup of the original data may be to perform an erasure code operation on the original data.
  • the erasure code operation is a data protection method, which can divide the data into The data blocks are expanded, encoded, and stored in different locations, such as disks, storage nodes, or other geographic locations.
  • redundant data blocks may be stored in different availability zones.
  • FIG. 1 provides an exemplary schematic diagram of a data redundancy backup device for cloud storage according to an embodiment of the application.
  • the data redundancy backup device includes three availability zones, and each availability zone may be a storage server.
  • the redundant backup device can transmit the original data with multiple terminal devices through the network, and the terminal device can upload the original data to the data redundant backup device.
  • the data redundancy backup device In the availability zone, when the terminal device accesses the data redundancy backup device, the data redundancy backup device will obtain the original data from the availability zone and return the original data to the terminal device.
  • FIG. 2 a schematic flowchart of a data redundancy backup method is provided in an embodiment of the present application. As shown in FIG. 2 , the method in this embodiment of the present application may include the following steps S101-S103.
  • users can connect their terminal equipment to the data redundancy backup device through the network, and upload data to the data redundancy backup device to complete cloud storage, reducing the storage space occupied by the terminal equipment.
  • the data redundancy backup device receives When it comes to the original data uploaded by the terminal device connected to it through the network, the original data can be split into two parts, namely the first data segment and the second data segment, and the data in the first data segment and the second data segment are different.
  • the first data segment and the second data segment may form complete original data, and the lengths of the first data segment and the second data segment may be the same or different.
  • S102 Perform erasure code operation on the first data segment and the second data segment respectively to obtain first redundant data and second redundant data.
  • the data redundancy backup device performs an erasure code operation on the first data segment, aligns it, expands and encodes it to obtain the first redundant data, and then also performs an erasure code operation on the second redundant data to obtain the second redundant data.
  • data, the first redundant data and the second redundant data can be stored in different availability zones respectively.
  • the first redundant data can be directly used to reconstruct the erasure code
  • the data restoration process is performed on the first data segment, and when the same second data segment is damaged, the second redundant data can be directly used to restore it.
  • S103 Generate third redundant data based on the logical relationship between the first redundant data and the second redundant data, and combine the first redundant data, the second redundant data, and the first redundant data with the second redundant data.
  • the three redundant data are stored in different Availability Zones in the Availability Zone set.
  • the data redundancy backup device since the first redundant data and the second redundant data are stored in different availability zones, the data redundancy backup device needs to have at least three availability zones in order to realize cloud storage to ensure a large amount of data corruption. The data in other availability zones can be called for restoration, so as to ensure data security. Therefore, the data redundancy backup device calculates the third redundancy according to the logical relationship between the first redundant data and the second redundant data. data, and save the third redundant data in a different availability zone from the first redundant data and the second redundant data.
  • the logical relationship and the third redundant data can be used to restore the data.
  • the second redundant data can be restored by using the logical relationship between the first redundant data and the third redundant data, and the second redundant The logical relationship between the data and the third redundant data restores the first redundant data.
  • the logical relationship includes an XOR relationship. For example, when the lengths of the first redundant data and the second redundant data are equal, the first redundant data and the second redundant data are bit-by-bit XORed to obtain the third redundant data. .
  • the original data is split into a first data segment and a second data segment, and erasure code operations are performed on the first data segment and the second data segment respectively to obtain the first redundant data and second redundant data, generating third redundant data based on the logical relationship between the first redundant data and the second redundant data.
  • FIG. 3 a schematic flowchart of a data redundancy backup method is provided in an embodiment of the present application. As shown in FIG. 3 , the method in this embodiment of the present application may include the following steps S201-S205.
  • the original data is divided into at least two original data blocks, and the original data is divided into a first data segment consisting of a original data blocks and a second data segment consisting of b original data blocks, so Both the a and the b are positive integers.
  • the data redundancy backup device is a cloud storage device, which can be connected to multiple terminal devices through a network.
  • the user can upload original data to the data redundancy backup device on the terminal device, and save the original data in the data redundancy backup device.
  • the backup device thereby saving the storage space of the terminal device. Therefore, after the data redundancy backup device receives the original data uploaded by the terminal device, the original data can be divided into at least two original data blocks, and the size of each original data block is the same.
  • the setting is performed on the backup device, or the user can perform the setting on the terminal device and send the setting to the data redundancy backup device.
  • the data redundancy backup device splits the original data into two segments, one segment is the first data segment consisting of a original data blocks, and the other is the second data segment consisting of b original data blocks, where a and b are Both are positive integers, the data in the first data segment and the second data segment are different, and the first data segment and the second data segment can form a complete original data.
  • the original data can be equally divided into 2N original data blocks, where N is a positive integer, the data composed of the first N original data blocks in the original data is confirmed as the first data segment, and the last N original data
  • S202 perform erasure coding operation on the first data segment to obtain m first check data blocks, and generate a+m first redundant data based on a original data blocks and m first check data blocks block, perform erasure coding operation on the second data segment to obtain n second check data blocks, and generate b+n second redundant data based on a original data blocks and m first check data blocks piece.
  • the data redundancy backup device performs erasure code operation on a original data blocks in the first data segment, obtains m check data blocks, and confirms a original data blocks and m check data blocks as a +m first redundant data blocks; the data redundancy backup device performs erasure code operation on b original data blocks in the second data segment to obtain n check data blocks, and converts b original data blocks and n The check data block is confirmed as the second redundant data block, wherein m is a positive integer not greater than a, and n is a positive integer not greater than b.
  • the same erasure coding operation rule can be used to perform the first data segment and the second data segment. Perform erasure code operation to generate the same number of check data blocks, and wait until the same number of first redundant data blocks and second redundant data blocks. It can be understood that when using the first redundant data to When performing data restoration processing on the second data segment and using the second redundant data for data restoration processing on the second data segment, the erasure code reconstruction can be performed based on the same set of erasure code operation rules, which reduces the computational burden of the data redundancy backup device. .
  • a1, a2, a3, a4, a5, a6 in 3a are the 6 original data blocks in the first data segment, which form a vector [a1, a2, a3, a4, a5, a6], which is a unit of a*a.
  • the matrix operation can be obtained to obtain the vector [a1, a2, a3, a4, a5, a6, c1, c2, c3], wherein c1, c2, c3 are the obtained m check data blocks.
  • the B matrix is the encoding matrix in the erasure code operation, and the Vandermonde matrix can be used, because any sub-matrix in the Vandermonde matrix is invertible, and it is convenient to use the encoding matrix for erasure code reconstruction.
  • the data redundancy backup device will add the first redundant data block or add the second redundant data block, so that the first redundant data block and the second redundant data If the number of blocks is the same, the first redundant data block is composed of the first redundant data, and the second redundant data block is composed of the second redundant data.
  • the method for adding the first redundant data block or adding the second redundant data block is: when a+m is less than b+n, the data redundancy backup device adds at least one supplementary data block of preset content to the In the first redundant data block, the number of the first redundant data blocks is increased to b+n; when a+m is greater than b+n, the data redundancy backup device will supplement the data of at least one preset content.
  • the block is added to the second redundant data block, so that the number of the second redundant data block is increased to a+m, and the preset content can be set in the data redundant backup device, or can be set by the user in the Setting in the terminal device and sending the setting to the data redundancy backup device, the preset content may be a data block with all 0s or all 1s, or a first redundant data block or a second redundant data Part of the data block in the block, and the size of the supplementary data block is the same as the original data block and the check data block.
  • Add supplementary data blocks are supplementary data blocks of y preset contents, where y is a positive integer.
  • y a+ m-(b+n)
  • adding y supplementary data blocks with preset contents to b+n second redundant data blocks to obtain a+m second redundant data blocks.
  • X1 to Xy may be all 0 or all 1 data blocks, or may be any y identical or different data blocks among the b original data blocks and the m check data blocks in the second data segment.
  • S205 perform an XOR operation on the first redundant data block and the second redundant data block one by one to generate a third redundant data block, and obtain a third redundant data block composed of the third redundant data block data, the first redundant data, the second redundant data and the third redundant data are respectively stored in different availability zones in the availability zone set.
  • the data redundancy backup device performs an XOR operation on the first redundant data block and the second redundant data block one by one to generate a third redundant data block, and the obtained third redundant data block forms a third redundant data block. data.
  • the data redundancy backup device numbers the first redundant data block and the second redundant data block respectively according to a preset sequence
  • the preset sequence can be set in the data redundancy backup device, or can be set by the data redundancy backup device.
  • the user sets the terminal equipment and sends the setting to the data redundancy backup device.
  • the preset sequence can be that the original data block is in the front, the verification data block is in the middle, and the supplementary data block is at the end, which are sequentially numbered;
  • the data redundancy backup device performs the XOR operation on the first redundant data block and the second redundant data block with the same number one by one to generate the third redundant data block with the corresponding number.
  • the number of the remaining data blocks is the same, so all corresponding XOR operations can be performed to generate the same number of third redundant data blocks, and the third redundant data blocks are formed into third redundant data according to the preset order.
  • the third redundant data is stored in a different availability zone from the first redundant data and the second redundant data.
  • data restoration can be performed by using the XOR relationship and the third redundant data.
  • the second redundant data can be restored by using the XOR relationship between the first redundant data and the third redundant data, and the second redundant data and The exclusive OR relationship of the third redundant data restores the first redundant data.
  • FIG. 3c provides a schematic diagram of an example of third redundant data generation for an embodiment of the present application.
  • the data redundancy backup device makes the first redundant data block and the second redundant data After the number of redundant data blocks is a+m, A1, A2, A3, A4...Aa+m are the first redundant data blocks numbered 1 to a+m, B1, B2, B3, B4.. .Ba+m is the second redundant data block numbered 1 to a+m, and the first redundant data block and the second redundant data block with the same number are XORed bit by bit to generate the third redundant data block with the corresponding number.
  • the remaining data blocks for example, A1 and B1 perform an exclusive OR operation to generate a third redundant data block C1 numbered 1, thereby obtaining C1, C2, C3, C4...Ca+m, that is, the numbers 1 ⁇ a+m
  • the a+m third redundant data blocks form third redundant data according to the sequence of numbers.
  • the original data is split into a first data segment and a second data segment, and erasure code operations are performed on the first data segment and the second data segment respectively to obtain the first redundant data and the second redundant data, and add supplementary data blocks so that the number of the first redundant data blocks is the same as the number of the second redundant data blocks.
  • the OR operation generates the same number of third redundant data blocks, which makes the XOR operation more convenient and faster.
  • the data redundancy backup method of S201-S205 can be used to save and backup the original data uploaded by the user.
  • the data redundancy backup device can perform data restoration processing on the original data.
  • FIG. 4 a schematic flowchart of data restoration processing is provided in this embodiment of the present application. As shown in FIG. 4 , the data restoration processing in this embodiment of the present application may include the following steps S301 - S306 .
  • S301 Acquire the verification quantity of the verification data blocks and the original quantity of the original data blocks in the target availability zone.
  • the data redundancy backup device when the data redundancy backup device receives the access service for the original data from the terminal device, the data redundancy backup device will obtain the original data block in the availability zone where the first redundant data and the second redundant data are stored to The original data is formed and returned to the terminal equipment.
  • the data redundant backup device When the original data block in the first redundant data or the second redundant data is damaged and the data redundant backup device cannot directly form the original data, the data redundant backup device will damage the data. Perform data repair processing on the block, and confirm the availability zone with damaged data blocks as the target availability zone. When the number of damaged data blocks is less than or equal to the damage threshold, the data redundancy backup device directly uses the redundant data blocks in the target availability zone.
  • the redundant data block is the first redundant data block, the second redundant data block or the third redundant data block.
  • the data redundancy backup device will obtain the check number of the check data blocks in the target availability zone and the original number of the original data blocks, because in the erasure code reconstruction, it is necessary to use the original number of redundant data blocks in the target availability zone.
  • Data recovery processing so if you want to directly use redundant data blocks in the target availability zone to perform data recovery processing on damaged data, the data redundancy backup device can tolerate at most the number of damaged data blocks in the target availability zone, so the damage threshold is The maximum value can be equal to the number of checks.
  • the data redundancy backup device can directly use the redundant data blocks in the target availability zone to perform data repair processing on the damaged data, and the data redundancy The backup device will obtain the original number of remaining data blocks in the target availability zone, and the remaining data blocks are other redundant data blocks in the target availability zone except for the damaged data blocks.
  • S303 Perform data repair processing on the damaged data block based on the remaining data block to obtain a repaired data block, and replace the damaged data block with the repaired data block.
  • the data redundancy backup device uses the remaining data blocks and the erasure code operation rules used in the redundant backup of the original data in S201-S205 to perform erasure code reconstruction, and performs data repair processing on the damaged data blocks to obtain Correspondingly repaired data blocks, the data redundancy backup device replaces the damaged data blocks with the repaired data blocks to obtain all the original data blocks, and combines the original data blocks to obtain the original data, and the data redundancy backup device can return the original data. to the terminal device.
  • FIG. 4a provides a schematic diagram of an example of erasure code reconstruction for this embodiment of the application.
  • the damaged data block is When it is a1, a2, and a3, delete the spliced matrix used for the erasure code operation in steps S201-S205, and delete the row data corresponding to the damaged data block to obtain a new matrix B', and calculate the inverse of the matrix B' matrix, and then the data redundancy backup device can perform matrix operation on the vector composed of the inverse matrix of matrix B' and the remaining data blocks, and can obtain the vector [a1, a2, a3, a4, a5, a6], where the calculated a1, a2 and a3 are repaired data blocks, and the data redundancy backup device replaces the repaired data blocks with the damaged data blocks to obtain complete original data.
  • the data redundancy backup device cannot use the redundant data in the target availability zone to directly perform data repair processing on the damaged data blocks, and needs to obtain the first
  • the redundant data blocks in the remaining availability zone and the second remaining availability zone are used to perform data repair on the damaged data blocks, and the first remaining availability zone and the second remaining availability zone are the available zone sets except the target availability zone.
  • Other Availability Zones The data redundancy backup unit obtains the numbers of the damaged data blocks and confirms these numbers as target numbers.
  • the data redundancy backup device will obtain the first target data block corresponding to the target number in the first remaining available zone, and obtain the second target data block corresponding to the target number in the second remaining available zone.
  • S306 Perform data repair processing on the damaged data block based on the redundant data block corresponding to the target number to obtain a repaired data block, and replace the damaged data block with the repaired data block.
  • the third redundant data block is obtained by performing the XOR operation on the first redundant data block and the second redundant data block with the same number.
  • the XOR operation between the second redundant data block and the third redundant data block can obtain another redundant data block.
  • the third redundant data block and the second redundant data block with the same number are XORed. operation, the first redundant data block with the same number can be obtained. Therefore, the data redundancy backup device performs the XOR operation on the first target data block and the second target data block with the same number one by one, so as to generate the repair data block with the corresponding number, and replace the repair data block with the damaged data block. Get complete raw data.
  • the data redundancy backup device can also perform self-checking according to a preset time, and the damaged data blocks are limited to the original data.
  • the method of S301-S306 can also be used for data restoration processing.
  • the number of original data blocks in the first data segment and the second data segment is the same, and the same erasure coding operation rules are used to perform erasure coding operations on the first data segment and the second data segment, the The third redundant data block generated by the residual data block and the second redundant data block can also be restored by using the same set of erasure coding operation rules.
  • the original data is split into a first data segment and a second data segment, and erasure code operations are performed on the first data segment and the second data segment respectively to obtain the first redundant data and the second redundant data, and add supplementary data blocks so that the number of the first redundant data blocks is the same as the number of the second redundant data blocks.
  • the OR operation generates the same number of third redundant data blocks, which makes the XOR operation more convenient and faster.
  • the data redundancy backup device provided by the embodiment of the present application will be described in detail below with reference to FIG. 5 to FIG. 11 . It should be noted that the data redundancy backup devices in FIG. 5 to FIG. 11 are used to execute the methods of the embodiments shown in FIG. 2 to FIG. 4 of the present application. For convenience of description, only the same embodiment as the embodiment of the present application is shown. For relevant parts, if the specific technical details are not disclosed, please refer to the embodiments shown in FIG. 2 to FIG. 4 of the present application.
  • FIG. 5 shows a schematic structural diagram of a data redundancy backup device provided by an exemplary embodiment of the present application.
  • the data redundancy backup device can be implemented as a whole or a part of the device through software, hardware or a combination of the two.
  • the device 1 includes a data splitting module 11 , an erasure code operation module 12 , a logic operation module 13 and a data storage module 14 .
  • a data splitting module 11 for splitting the original data into a first data segment and a second data segment
  • an erasure code operation module 12 configured to perform an erasure code operation on the first data segment and the second data segment respectively to obtain first redundant data and second redundant data;
  • a logical operation module 13 configured to generate third redundant data based on the logical relationship between the first redundant data and the second redundant data
  • the data saving module 14 is configured to save the first redundant data, the second redundant data and the third redundant data in different availability zones in the availability zone set, respectively.
  • the original data is split into a first data segment and a second data segment, and erasure code operations are performed on the first data segment and the second data segment respectively to obtain the first redundant data and
  • third redundant data is generated based on the logical relationship between the first redundant data and the second redundant data.
  • FIG. 6 shows a schematic structural diagram of a data redundancy backup apparatus provided by an exemplary embodiment of the present application.
  • the data redundancy backup device can be implemented as a whole or a part of the device through software, hardware or a combination of the two.
  • the device 1 includes a data splitting module 11 , an erasure code operation module 12 , a logic operation module 13 , a data storage module 14 , a first data repair module 15 and a second data repair module 16 .
  • a data splitting module 11 for splitting the original data into a first data segment and a second data segment
  • FIG. 7 provides a schematic structural diagram of a data splitting module in an embodiment of the present application.
  • the data splitting module 11 may include:
  • a first splitting unit 111 configured to equally divide the original data into at least two original data blocks
  • the second splitting unit 112 is configured to split the original data into a first data segment consisting of a original data blocks and a second data segment consisting of b original data blocks, the a and the b All are positive integers.
  • an erasure code operation module 12 configured to perform an erasure code operation on the first data segment and the second data segment respectively to obtain first redundant data and second redundant data;
  • the erasure code operation module 12 may include:
  • the first operation unit 121 is configured to perform an erasure code operation on the first data segment, obtain m first check data blocks, and generate a+m based on a original data blocks and m first check data blocks a first redundant data block, the m is a positive integer not greater than a;
  • the second operation unit 122 is configured to perform an erasure code operation on the second data segment, obtain n second check data blocks, and generate b+n based on a original data blocks and m first check data blocks a second redundant data block, the n is a positive integer not greater than b;
  • the first data generating unit 123 is configured to generate first redundant data based on the first redundant data block and generate second redundant data based on the second redundant data block when a+m is equal to b+n. remaining data;
  • the second data generating unit 124 is configured to add a supplementary data block to the first redundant data block or the second redundant data block when a+m and b+n are not equal, so that after the addition, the The number of the first redundant data block and the second redundant data block is the same, the first redundant data is generated based on the first redundant data block, and the second redundant data is generated based on the second redundant data block. remaining data.
  • the second data generating unit 124 is specifically configured to add at least one supplementary data block of preset content to the first redundant data block when a+m is less than b+n, so that the The number of first redundant data blocks is increased to b+n;
  • At least one supplementary data block of preset content is added to the second redundant data block, so that the number of the second redundant data blocks is increased to a+m.
  • a logical operation module 13 configured to generate third redundant data based on the logical relationship between the first redundant data and the second redundant data
  • the logic operation module 13 is specifically configured to perform an XOR operation on the first redundant data block and the second redundant data block one by one to generate a third redundant data block, and obtain the first redundant data block.
  • the third redundant data is composed of three redundant data blocks.
  • FIG. 9 provides a schematic structural diagram of a logic operation module according to an embodiment of the present application.
  • the logic operation module 13 may include:
  • Numbering unit 131 used to respectively number the first redundant data block and the second redundant data block according to a preset sequence
  • the XOR operation unit 132 is used to perform XOR operation on the first redundant data block and the second redundant data block with the same number one by one to generate the third redundant data block of the corresponding number;
  • the third data generating unit 133 is configured to form the third redundant data blocks according to the preset sequence into third redundant data.
  • a data saving module 14 configured to save the first redundant data, the second redundant data and the third redundant data in different availability zones in the availability zone set;
  • the first data repair module 15 is configured to perform data repair processing on the damaged data blocks based on redundant data blocks in the target availability zone when the damaged quantity of the damaged data blocks in the target availability zone is less than or equal to the damage threshold to obtain a repaired data block, and replace the repaired data block with the damaged data block;
  • the first data repair module 15 may include:
  • the first quantity obtaining unit 151 is used to obtain the check quantity of the check data block and the original quantity of the original data block in the target availability zone;
  • the first data block obtaining unit 152 is configured to obtain, in the target availability zone, the remainder indicated by the original quantity in the target availability zone when the damaged quantity of the damaged data blocks in the target availability zone is less than or equal to the check quantity data block;
  • the first replacement unit 153 is configured to perform data repair processing on the damaged data block based on the remaining data block to obtain a repaired data block, and use the repaired data block to replace the damaged data block.
  • the second data repair module 16 is configured to, when the damaged quantity of the damaged data blocks in the target availability zone is greater than the damage threshold value, perform an analysis on the damaged data blocks based on the redundant data blocks in the first remaining availability zone and the second remaining availability zone performing data repair processing to obtain repair data blocks, and using the repair data blocks to replace the damaged data blocks;
  • the target availability zone is any availability zone in the availability zone set, and the first remaining availability zone and the second remaining availability zone are the other two available zones in the availability zone set except the target availability zone area, the remaining data blocks are other redundant data blocks in the target availability zone except the damaged data block, and the redundant data blocks are the first redundant data block, the second redundant data block data block or the third redundant data block.
  • the second data repair module 16 may include:
  • the second quantity obtaining unit 161 is used to obtain the check quantity of check data blocks in the target availability zone
  • a number obtaining unit 162 configured to obtain the target number of the damaged data block when the damaged quantity of the damaged data block in the target availability zone is greater than the check quantity
  • the second data block obtaining unit 163 is configured to obtain redundant data blocks corresponding to the target numbers in the first remaining available zone and the second remaining available zone respectively;
  • the second data block obtaining unit 163 is specifically configured to obtain the first target data block corresponding to the target number in the first remaining available zone, and the second data block corresponding to the target number in the second remaining available zone. target data block.
  • the second replacement unit 164 is configured to perform data repair processing on the damaged data block based on the redundant data block corresponding to the target number to obtain a repaired data block, and use the repaired data block to replace the damaged data block .
  • the second replacement unit 164 is specifically configured to perform an XOR operation on the first target data block and the second target data block with the same number one by one, and generate a repair data block with a corresponding number, using the The repair data block replaces the damaged data block.
  • the original data is split into a first data segment and a second data segment, and erasure code operations are performed on the first data segment and the second data segment respectively to obtain the first redundant data and
  • the second redundant data is added, and the supplementary data blocks are added so that the number of the first redundant data blocks is the same as the number of the second redundant data blocks, and then the first redundant data block and the second redundant data block are XORed one by one operation to generate the same number of third redundant data blocks, which makes the XOR operation more convenient and fast.
  • the data redundancy backup device when the data redundancy backup device provided in the above embodiments executes the data redundancy backup method, only the division of the above functional modules is used as an example for illustration.
  • the functional modules of the device are completed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
  • the data redundancy backup device and the data redundancy backup method provided by the above embodiments belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.
  • An embodiment of the present application further provides a computer storage medium, where the computer storage medium can store a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the above-described embodiments shown in FIG. 1 to FIG. 4a.
  • the computer storage medium can store a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the above-described embodiments shown in FIG. 1 to FIG. 4a.
  • the data redundancy backup method for the specific execution process, reference may be made to the specific description of the embodiment shown in FIG. 1 to FIG. 4 a , which will not be repeated here.
  • the present application also provides a computer program product, the computer program product stores at least one instruction, and the at least one instruction is loaded by the processor and executes the data redundancy in the embodiments shown in the foregoing FIG. 1 to FIG. 4a.
  • the specific execution process can refer to the specific description of the embodiment shown in FIG. 1-FIG. 4a, which will not be repeated here.
  • the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , user interface 1003 , memory 1005 , and at least one communication bus 1002 .
  • the communication bus 1002 is used to realize the connection and communication between these components.
  • the user interface 1003 may include a display screen (Display) and a camera (Camera), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.
  • Display display screen
  • Camera Camera
  • the optional user interface 1003 may also include a standard wired interface and a wireless interface.
  • the network interface 1004 may optionally include a standard wired interface and a wireless interface (eg, a WI-FI interface).
  • the processor 1001 may include one or more processing cores.
  • the processor 1001 uses various excuses and lines to connect various parts of the entire server 1000, and executes the server by running or executing the instructions, programs, code sets or instruction sets stored in the memory 1005, and calling the data stored in the memory 1005. 1000s of various functions and processing data.
  • the processor 1001 may adopt at least one of digital signal processing (Digital Signal Processing, DSP), field-programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA).
  • DSP Digital Signal Processing
  • FPGA Field-Programmable Gate Array
  • PLA programmable logic array
  • a hardware form is implemented.
  • the processor 1001 may integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), a modem, and the like.
  • the CPU mainly deals with the operating system, user interface and application programs, etc.; the GPU is used for rendering and drawing the content that needs to be displayed on the display screen; the modem is used for processing wireless communication. It can be understood that, the above-mentioned modem may not be integrated into the processor 1001, but is implemented by a single chip.
  • the memory 1005 may include random access memory (Random Access Memory, RAM), or may include read-only memory (Read-Only Memory).
  • the memory 1005 includes a non-transitory computer-readable storage medium.
  • Memory 1005 may be used to store instructions, programs, codes, sets of codes, or sets of instructions.
  • the memory 1005 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), Instructions and the like used to implement the above method embodiments; the storage data area may store the data and the like involved in the above method embodiments.
  • the memory 1005 may also be at least one storage device located away from the aforementioned processor 1001 .
  • the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a data redundancy backup application program.
  • the user interface 1003 is mainly used to provide an input interface for the user to obtain the data input by the user; and the processor 1001 can be used to call the data redundancy backup application program stored in the memory 1005, And do the following specifically:
  • the first redundant data, the second redundant data and the third redundant data are respectively stored in different availability zones in the set of availability zones.
  • the processor 1001 when the processor 1001 splits the original data into the first data segment and the second data segment, the processor 1001 specifically performs the following operations:
  • the original data is divided into a first data segment consisting of a original data blocks and a second data segment consisting of b original data blocks, where a and b are both positive integers.
  • the processor 1001 when the processor 1001 performs an erasure code operation on the first data segment and the second data segment respectively to obtain the first redundant data and the second redundant data, the processor 1001 specifically executes Do the following:
  • Erasure code operation is performed on the first data segment to obtain m first check data blocks, and a+m first redundant data blocks are generated based on a original data blocks and m first check data blocks,
  • the m is a positive integer not greater than a;
  • n is a positive integer not greater than b;
  • a supplementary data block is added to the first redundant data block or the second redundant data block, so that after the addition, the first redundant data block and all redundant data blocks are added.
  • the number of the second redundant data blocks is the same, the first redundant data is generated based on the first redundant data blocks, and the second redundant data is generated based on the second redundant data blocks.
  • the processor 1001 adds a supplementary data block to the first redundant data block or the second redundant data block when a+m and b+n are not equal, so that After adding, the first redundant data blocks and the second redundant data blocks have the same number, first redundant data is generated based on the first redundant data blocks, and first redundant data is generated based on the second redundant data blocks
  • first redundant data is generated based on the second redundant data blocks
  • At least one supplementary data block of preset content is added to the second redundant data block, so that the number of the second redundant data blocks is increased to a+m.
  • the processor 1001 when the processor 1001 performs generating the third redundant data based on the logical relationship between the first redundant data and the second redundant data, the processor 1001 specifically performs the following operations:
  • the processor 1001 performs XOR budgeting on the first redundant data block and the second redundant data block one by one, generates a third redundant data block, and obtains the first redundant data block from the second redundant data block.
  • the third redundant data is composed of three redundant data blocks, the following operations are specifically performed:
  • the first redundant data block and the second redundant data block with the same number are XORed one by one to generate the third redundant data block of the corresponding number;
  • the third redundant data blocks are formed into third redundant data according to the preset sequence.
  • the processor 1001 when the processor 1001 executes the data redundancy backup method, the processor 1001 further executes the following operations:
  • the target availability zone is any availability zone in the availability zone set, and the first remaining availability zone and the second remaining availability zone are the other two available zones in the availability zone set except the target availability zone area, the remaining data blocks are other redundant data blocks in the target availability zone except the damaged data block, and the redundant data blocks are the first redundant data block, the second redundant data block data block or the third redundant data block.
  • the processor 1001 executes, when the number of corrupted data blocks in the target availability zone is less than or equal to a corruption threshold, the corrupted data blocks are determined based on redundant data blocks in the target availability zone. Perform data repair processing to obtain a repaired data block, and when using the repaired data block to replace the damaged data block, specifically perform the following operations:
  • the processor 1001 performs, when the damaged quantity of the damaged data blocks in the target availability zone is greater than the damage threshold, the pair of all data blocks based on the redundant data blocks in the first remaining availability zone and the second remaining availability zone. Perform data repair processing on the damaged data block to obtain a repaired data block, and when using the repaired data block to replace the damaged data block, specifically perform the following operations:
  • the processor 1001 when the processor 1001 acquires the redundant data blocks corresponding to the target numbers in the first remaining availability zone and the second remaining availability zone respectively, the processor 1001 specifically performs the following operations:
  • the XOR operation is performed on the first target data block and the second target data block with the same number one by one to generate a repair data block with a corresponding number, and the damaged data block is replaced by the repair data block.
  • the original data is split into a first data segment and a second data segment, and erasure code operations are performed on the first data segment and the second data segment respectively to obtain the first redundant data and
  • the second redundant data is added, and the supplementary data blocks are added so that the number of the first redundant data blocks is the same as the number of the second redundant data blocks, and then the first redundant data block and the second redundant data block are XORed one by one operation to generate the same number of third redundant data blocks, which makes the XOR operation more convenient and fast.
  • the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. During execution, the processes of the embodiments of the above-mentioned methods may be included.
  • the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, or a random storage memory, and the like.

Abstract

Disclosed in the present application are a data redundancy backup method and apparatus, a storage medium, and an electronic device. The method comprises: splitting original data into a first data segment and a second data segment; separately performing erasure coding operation on the first data segment and the second data segment to obtain first redundant data and second redundant data; and generating third redundant data on the basis of a logical relationship between the first redundant data and the second redundant data. By means of the present application, original data is split first, and then erasure coding operation is separately performed on the split data, such that when there is less damaged data in the first data segment and the second data segment, the damaged data can be repaired by using data in available areas where the first data segment and the second data segment are located, thereby reducing data repair cost.

Description

一种数据冗余备份方法、装置、存储介质及电子设备A data redundancy backup method, device, storage medium and electronic device 技术领域technical field
本申请涉及计算机技术领域,尤其涉及一种数据冗余备份方法、装置、存储介质及电子设备。The present application relates to the field of computer technology, and in particular, to a data redundancy backup method, device, storage medium and electronic device.
背景技术Background technique
随着互联网的飞速发展,人们每天都会在互联网上产生大量的数据,现如今会将大部分数据都存储于云存储系统中,人们访问互联网时能直接从云存储系统中获取数据,而不需要保存在个人的本地存储中。为了防止存储机器故障、停电、地震等原因导致的损坏,现在的云存储系统都有着三个可用区,将需要保存的数据利用纠删码策略进行冗余备份,将数据块与校验块平均保存在三个可用区内,当出现数据块损坏时,提取原有数据块数目的数据块和校验块便能恢复损坏的数据块,但是在只有单个数据块损坏时也要用这样的方式进行复原,十分耗费云存储系统的运算内存,需要付出很大的修复代价。With the rapid development of the Internet, people generate a large amount of data on the Internet every day. Nowadays, most of the data is stored in the cloud storage system. When people access the Internet, they can directly obtain data from the cloud storage system without needing to Saved in personal local storage. In order to prevent damage caused by storage machine failures, power outages, earthquakes, etc., the current cloud storage system has three availability zones. The data to be saved is redundantly backed up by the erasure code strategy, and the data blocks and check blocks are averaged. Stored in three availability zones. When data blocks are damaged, the damaged data blocks can be recovered by extracting the original number of data blocks and check blocks, but this method should also be used when only a single data block is damaged. Restoration takes a lot of computing memory of the cloud storage system and requires a lot of repair costs.
发明内容SUMMARY OF THE INVENTION
本申请实施例提供了一种数据冗余备份方法、装置、存储介质及电子设备,可以通过先将原始数据进行拆分,再分别对拆分后的数据进行纠删码运算,使得第一数据段和第二数据段中损坏数据较少时可以利用其所在可用区内的数据进行修复,节省数据修复成本。所述技术方案如下:Embodiments of the present application provide a data redundancy backup method, device, storage medium, and electronic device. By first splitting the original data, and then performing an erasure code operation on the split data, the first data When there is less damaged data in the segment and the second data segment, the data in the availability zone can be used for repair, thereby saving the cost of data repair. The technical solution is as follows:
第一方面,本申请实施例提供了一种数据冗余备份方法,所述方法包括:In a first aspect, an embodiment of the present application provides a data redundancy backup method, the method comprising:
将原始数据拆分为第一数据段和第二数据段;split the original data into a first data segment and a second data segment;
分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据;Perform erasure coding operations on the first data segment and the second data segment respectively to obtain first redundant data and second redundant data;
基于所述第一冗余数据和所述第二冗余数据之间的逻辑关系生成第三冗余数据;generating third redundant data based on the logical relationship between the first redundant data and the second redundant data;
将所述第一冗余数据、所述第二冗余数据和所述第三冗余数据分别保存在可用区集合中不同的可用区中。The first redundant data, the second redundant data and the third redundant data are respectively stored in different availability zones in the set of availability zones.
第二方面,本申请实施例提供了一种数据冗余备份装置,所述装置包括:In a second aspect, an embodiment of the present application provides a data redundancy backup device, and the device includes:
数据拆分模块,用于将原始数据拆分为第一数据段和第二数据段;a data splitting module for splitting the original data into a first data segment and a second data segment;
纠删码运算模块,用于分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据;an erasure code operation module, configured to perform an erasure code operation on the first data segment and the second data segment respectively to obtain first redundant data and second redundant data;
逻辑运算模块,用于基于所述第一冗余数据和所述第二冗余数据之间的逻辑关系生成第三冗余数据;a logic operation module, configured to generate third redundant data based on the logical relationship between the first redundant data and the second redundant data;
将所述第一冗余数据、所述第二冗余数据和所述第三冗余数据分别保存在可用区集合中不同的可用区中。The first redundant data, the second redundant data and the third redundant data are respectively stored in different availability zones in the set of availability zones.
第三方面,本申请实施例提供一种计算机存储介质,所述计算机存储介质存储有多条指令, 所述指令适于由处理器加载并执行上述的方法步骤。In a third aspect, an embodiment of the present application provides a computer storage medium, where the computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the foregoing method steps.
第四方面,本申请实施例提供一种电子设备,可包括:处理器和存储器;其中,所述存储器存储有计算机程序,所述计算机程序适于由所述处理器加载并执行上述的方法步骤。In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein, the memory stores a computer program, and the computer program is adapted to be loaded by the processor and execute the above method steps .
本申请一些实施例提供的技术方案带来的有益效果至少包括:The beneficial effects brought by the technical solutions provided by some embodiments of the present application include at least:
在本申请一个或多个实施例中,将原始数据拆分为第一数据段和第二数据段,分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据,基于所述第一冗余数据和所述第二冗余数据之间的逻辑关系生成第三冗余数据。通过先将原始数据进行拆分,再分别对拆分后的数据进行纠删码运算,使得第一数据段和第二数据段中损坏数据较少时可以利用其所在可用区内的数据进行修复,节省了数据修复成本。In one or more embodiments of the present application, the original data is split into a first data segment and a second data segment, and an erasure code operation is performed on the first data segment and the second data segment respectively to obtain the first data segment and the second data segment. A redundant data and a second redundant data, and the third redundant data is generated based on the logical relationship between the first redundant data and the second redundant data. By first splitting the original data, and then performing erasure code operation on the split data, the first data segment and the second data segment can be repaired by using the data in the availability zone when there is less damaged data. , saving data repair costs.
附图说明Description of drawings
为了更清楚地说明本申请实施例或现有技术中的技术方案,下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.
图1是本申请实施例提供的一种数据冗余备份装置用于云存储的举例示意图;1 is a schematic diagram of an example of a data redundancy backup device used for cloud storage provided by an embodiment of the present application;
图2是本申请实施例提供的一种数据冗余备份方法的流程示意图;2 is a schematic flowchart of a data redundancy backup method provided by an embodiment of the present application;
图3是本申请实施例提供的一种数据冗余备份方法的流程示意图;3 is a schematic flowchart of a data redundancy backup method provided by an embodiment of the present application;
图3a是本申请实施例提供的一种纠删码运算规则的举例示意图;3a is a schematic diagram illustrating an example of an erasure code operation rule provided by an embodiment of the present application;
图3b是本申请实施例提供的一种添加补充数据块的举例示意图;FIG. 3b is an exemplary schematic diagram of adding a supplementary data block provided by an embodiment of the present application;
图3c是本申请实施例提供的一种第三冗余数据生成的举例示意图;3c is an exemplary schematic diagram of a third redundant data generation provided by an embodiment of the present application;
图4是本申请实施例提供的一种数据修复处理的流程示意图;4 is a schematic flowchart of a data restoration process provided by an embodiment of the present application;
图4a是本申请实施例提供的一种纠删码重建的举例示意图;Fig. 4a is a schematic diagram of an example of erasure correction code reconstruction provided by an embodiment of the present application;
图5是本申请实施例提供的一种数据冗余备份装置的结构示意图;5 is a schematic structural diagram of a data redundancy backup device provided by an embodiment of the present application;
图6是本申请实施例提供的一种数据冗余备份装置的结构示意图;6 is a schematic structural diagram of a data redundancy backup device provided by an embodiment of the present application;
图7是本申请实施例提供的一种数据拆分模块的结构示意图;7 is a schematic structural diagram of a data splitting module provided by an embodiment of the present application;
图8是本申请实施例提供的一种纠删码运算模块的结构示意图;8 is a schematic structural diagram of an erasure code computing module provided by an embodiment of the present application;
图9是本申请实施例提供的一种逻辑运算模块的结构示意图;9 is a schematic structural diagram of a logic operation module provided by an embodiment of the present application;
图10是本申请实施例提供的一种第一数据修复模块的结构示意图;10 is a schematic structural diagram of a first data restoration module provided by an embodiment of the present application;
图11是本申请实施例提供的一种第二数据修复模块的结构示意图;11 is a schematic structural diagram of a second data restoration module provided by an embodiment of the present application;
图12是本申请实施例提供的一种电子设备的结构示意图。FIG. 12 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
具体实施方式Detailed ways
下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅仅是本申请一部分实施例,而不是全部的实施例。基于本申请中的实施 例,本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例,都属于本申请保护的范围。The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.
在本申请的描述中,需要理解的是,术语“第一”、“第二”等仅用于描述目的,而不能理解为指示或暗示相对重要性。在本申请的描述中,需要说明的是,除非另有明确的规定和限定,“包括”和“具有”以及它们任何变形,意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、系统、产品或设备没有限定于已列出的步骤或单元,而是可选地还包括没有列出的步骤或单元,或可选地还包括对于这些过程、方法、产品或设备固有的其他步骤或单元。对于本领域的普通技术人员而言,可以具体情况理解上述术语在本申请中的具体含义。此外,在本申请的描述中,除非另有说明,“多个”是指两个或两个以上。“和/或”,描述关联对象的关联关系,表示可以存在三种关系,例如,A和/或B,可以表示:单独存在A,同时存在A和B,单独存在B这三种情况。字符“/”一般表示前后关联对象是一种“或”的关系。In the description of the present application, it should be understood that the terms "first", "second" and the like are used for descriptive purposes only, and should not be construed as indicating or implying relative importance. In the description of the present application, it should be noted that, unless otherwise expressly specified and defined, "including" and "having" and any modifications thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally also includes For other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood in specific situations. Also, in the description of the present application, unless otherwise specified, "a plurality" means two or more. "And/or", which describes the association relationship of the associated objects, means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the associated objects are an "or" relationship.
本申请实施例提供的数据冗余备份方法可依赖于计算机程序实现,可运行于基于冯诺依曼体系的数据冗余备份装置上。该计算机程序可集成在应用中,也可作为独立的工具类应用运行。本申请实施例中的数据冗余备份装置可以是一种云存储装置,例如服务器集群,可以用于提供数据存储和访问业务,保证数据的安全性,并节约存储空间,数据冗余备份装置可以包括有至少三个可用区,当数据冗余备份装置接收到上传的原始数据后,可以对原始数据进行冗余备份处理,将冗余备份处理后的数据拆分保存在至少三个可用区中,当接收到针对原始数据的访问业务时,可以用这至少三个可用区中获取原始数据;当可用区中的原始数据发生了损坏时,还可以利用同一可用区中的数据或其他可用区中的数据对原始数据进行复原。本申请实施例中对原始数据就那些冗余备份处理的方法可以是对原始数据进行纠删码运算,所述纠删码运算是一种数据保护方法,可以将数据分割成片段,把冗余数据块扩展、编码,并将其存储在不同的位置,比如磁盘、存储节点或者其它地理位置,例如本申请实施例中可以将冗余数据块保存在不同的可用区中。The data redundancy backup method provided by the embodiments of the present application can be implemented by relying on a computer program, and can be run on a data redundancy backup device based on the von Neumann system. The computer program can be integrated into an application or run as a stand-alone utility application. The data redundancy backup device in the embodiment of the present application may be a cloud storage device, such as a server cluster, which may be used to provide data storage and access services, ensure data security, and save storage space. The data redundancy backup device may It includes at least three availability zones. After the data redundancy backup device receives the uploaded original data, it can perform redundant backup processing on the original data, and split the processed data into at least three availability zones. , when receiving the access service for the original data, the original data can be obtained from these at least three availability zones; when the original data in the availability zone is damaged, the data in the same availability zone or other availability zones can also be used restore the original data. In the embodiment of the present application, the method for processing the redundant backup of the original data may be to perform an erasure code operation on the original data. The erasure code operation is a data protection method, which can divide the data into The data blocks are expanded, encoded, and stored in different locations, such as disks, storage nodes, or other geographic locations. For example, in this embodiment of the present application, redundant data blocks may be stored in different availability zones.
请参见图1,为本申请实施例提供了一种数据冗余备份装置用于云存储的举例示意图,数据冗余备份装置中包含有三个可用区,每个可用区可以是一个存储服务器,数据冗余备份装置可以与多个终端设备通过网络进行原始数据传输,终端设备可以将原始数据上传至数据冗余备份装置,数据冗余备份装置将原始数据进行冗余备份并拆分保存与三个可用区中,当终端设备向数据冗余备份装置进行访问业务时,数据冗余备份装置会从可用区中获取原始数据并将原始数据返回给终端设备。Please refer to FIG. 1, which provides an exemplary schematic diagram of a data redundancy backup device for cloud storage according to an embodiment of the application. The data redundancy backup device includes three availability zones, and each availability zone may be a storage server. The redundant backup device can transmit the original data with multiple terminal devices through the network, and the terminal device can upload the original data to the data redundant backup device. In the availability zone, when the terminal device accesses the data redundancy backup device, the data redundancy backup device will obtain the original data from the availability zone and return the original data to the terminal device.
下面结合具体的实施例对本申请提供的数据冗余备份方法进行详细说明。The data redundancy backup method provided by the present application will be described in detail below with reference to specific embodiments.
请参见图2,为本申请实施例提供了一种数据冗余备份方法的流程示意图。如图2所示,本申请实施例的所述方法可以包括以下步骤S101-S103。Referring to FIG. 2 , a schematic flowchart of a data redundancy backup method is provided in an embodiment of the present application. As shown in FIG. 2 , the method in this embodiment of the present application may include the following steps S101-S103.
S101,将原始数据拆分为第一数据段和第二数据段。S101, split the original data into a first data segment and a second data segment.
具体的,用户可以将自己的终端设备通过网络与数据冗余备份装置相连接,并向数据冗余备份装置上传数据来完成云存储,减少占用终端设备的存储空间,当数据冗余备份装置接收到与其通过网络连接的终端设备上传的原始数据时,可以将原始数据拆分为两部分,分别是第一数据段 和第二数据段,第一数据段和第二数据段中的数据不相同且第一数据段和第二数据段可以组成完整的原始数据,第一数据段和第二数据段的长度可以相同也可以不相同。Specifically, users can connect their terminal equipment to the data redundancy backup device through the network, and upload data to the data redundancy backup device to complete cloud storage, reducing the storage space occupied by the terminal equipment. When the data redundancy backup device receives When it comes to the original data uploaded by the terminal device connected to it through the network, the original data can be split into two parts, namely the first data segment and the second data segment, and the data in the first data segment and the second data segment are different. In addition, the first data segment and the second data segment may form complete original data, and the lengths of the first data segment and the second data segment may be the same or different.
S102,分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据。S102: Perform erasure code operation on the first data segment and the second data segment respectively to obtain first redundant data and second redundant data.
具体的,数据冗余备份设备对第一数据段进行纠删码运算,对齐进行拓展、编码从而得到第一冗余数据,然后对第二冗余数据同样进行纠删码运算得到第二冗余数据,可以将第一冗余数据和第二冗余数据分别保存在不同的可用区中,当第一数据段中数据出现损坏时,可以直接用第一冗余数据采用纠删码重建的方法对第一数据段进行数据修复处理,同样的第二数据段出现损坏时可以直接用第二冗余数据对其进行复原。Specifically, the data redundancy backup device performs an erasure code operation on the first data segment, aligns it, expands and encodes it to obtain the first redundant data, and then also performs an erasure code operation on the second redundant data to obtain the second redundant data. data, the first redundant data and the second redundant data can be stored in different availability zones respectively. When the data in the first data segment is damaged, the first redundant data can be directly used to reconstruct the erasure code The data restoration process is performed on the first data segment, and when the same second data segment is damaged, the second redundant data can be directly used to restore it.
S103,基于所述第一冗余数据和所述第二冗余数据之间的逻辑关系生成第三冗余数据,将所述第一冗余数据、所述第二冗余数据和所述第三冗余数据分别保存在可用区集合中不同的可用区中。S103. Generate third redundant data based on the logical relationship between the first redundant data and the second redundant data, and combine the first redundant data, the second redundant data, and the first redundant data with the second redundant data. The three redundant data are stored in different Availability Zones in the Availability Zone set.
具体的,由于第一冗余数据和第二冗余数据分别保存在了不同的可用区中,而数据冗余备份装置想要实现云存储需要有至少三个可用区,才能保证数据出现大量损坏时可以调用其他可用区中的数据进行复原,从而保证数据的安全性,所以数据冗余备份装置根据第一冗余数据和第二冗余数据之间的逻辑关系,计算得出第三冗余数据,并将第三冗余数据保存在与第一冗余数据和第二冗余数据不同的可用区中,当第一冗余数据或第二冗余数据出现了大量损坏,导致不能直接用自身的数据进行复原时,可以利用逻辑关系和第三冗余数据进行数据复原,例如可以用第一冗余数据和第三冗余数据的逻辑关系复原第二冗余数据,用第二冗余数据和第三冗余数据的逻辑关系复原第一冗余数据。Specifically, since the first redundant data and the second redundant data are stored in different availability zones, the data redundancy backup device needs to have at least three availability zones in order to realize cloud storage to ensure a large amount of data corruption. The data in other availability zones can be called for restoration, so as to ensure data security. Therefore, the data redundancy backup device calculates the third redundancy according to the logical relationship between the first redundant data and the second redundant data. data, and save the third redundant data in a different availability zone from the first redundant data and the second redundant data. When the first redundant data or the second redundant data is damaged in large quantities, the When restoring its own data, the logical relationship and the third redundant data can be used to restore the data. For example, the second redundant data can be restored by using the logical relationship between the first redundant data and the third redundant data, and the second redundant The logical relationship between the data and the third redundant data restores the first redundant data.
所述逻辑关系包括异或关系,例如当第一冗余数据和第二冗余数据长度相等时,将第一冗余数据和第二冗余数据逐位进行异或运算得到第三冗余数据。The logical relationship includes an XOR relationship. For example, when the lengths of the first redundant data and the second redundant data are equal, the first redundant data and the second redundant data are bit-by-bit XORed to obtain the third redundant data. .
在本申请实施例中,将原始数据拆分为第一数据段和第二数据段,分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据,基于所述第一冗余数据和所述第二冗余数据之间的逻辑关系生成第三冗余数据。通过先将原始数据进行拆分,再分别对拆分后的数据进行纠删码运算,使得第一数据段和第二数据段中损坏数据较少时可以利用其所在可用区内的数据进行修复,节省了数据修复成本。In the embodiment of the present application, the original data is split into a first data segment and a second data segment, and erasure code operations are performed on the first data segment and the second data segment respectively to obtain the first redundant data and second redundant data, generating third redundant data based on the logical relationship between the first redundant data and the second redundant data. By first splitting the original data, and then performing erasure code operation on the split data, the first data segment and the second data segment can be repaired by using the data in the availability zone when there is less damaged data. , saving data repair costs.
请参见图3,为本申请实施例提供了一种数据冗余备份方法的流程示意图。如图3所示,本申请实施例的所述方法可以包括以下步骤S201-S205。Referring to FIG. 3 , a schematic flowchart of a data redundancy backup method is provided in an embodiment of the present application. As shown in FIG. 3 , the method in this embodiment of the present application may include the following steps S201-S205.
S201,将原始数据均分为至少两个原始数据块,将所述原始数据拆分为由a个原始数据块组成的第一数据段和由b个原始数据块组成的第二数据段,所述a和所述b均为正整数。S201, the original data is divided into at least two original data blocks, and the original data is divided into a first data segment consisting of a original data blocks and a second data segment consisting of b original data blocks, so Both the a and the b are positive integers.
具体的,数据冗余备份装置是一种云存储装置,可以与多个终端设备通过网络相连接,用户在终端设备上可以向数据冗余备份装置上传原始数据,将原始数据保存在数据冗余备份装置中,从而节省终端设备的存储空间。故在数据冗余备份装置接收到终端设备上传的原始数据后,可以将原始数据均分为至少两个原始数据块,每个原始数据块的大小相同,原始数据块的大小可以在 数据冗余备份装置上进行设置,也可以由用户在终端设备上进行设置并将该设置发送给数据冗余备份装置。然后数据冗余备份装置将原始数据拆分成两段,一段是由a个原始数据块组成的第一数据段,一份是由b个原始数据块组成的第二数据段,其中a和b均为正整数,第一数据段和第二数据段中的数据不相同,且第一数据段和第二数据段能够组成一个完整的原始数据。Specifically, the data redundancy backup device is a cloud storage device, which can be connected to multiple terminal devices through a network. The user can upload original data to the data redundancy backup device on the terminal device, and save the original data in the data redundancy backup device. In the backup device, thereby saving the storage space of the terminal device. Therefore, after the data redundancy backup device receives the original data uploaded by the terminal device, the original data can be divided into at least two original data blocks, and the size of each original data block is the same. The setting is performed on the backup device, or the user can perform the setting on the terminal device and send the setting to the data redundancy backup device. Then, the data redundancy backup device splits the original data into two segments, one segment is the first data segment consisting of a original data blocks, and the other is the second data segment consisting of b original data blocks, where a and b are Both are positive integers, the data in the first data segment and the second data segment are different, and the first data segment and the second data segment can form a complete original data.
可选的,可以将原始数据均分为2N个原始数据块,其中N为正整数,将原始数据中的前N个原始数据块组成的数据确认为第一数据段,将后N个原始数据块组成的数据确认为第二数据段,即a=b=N。Optionally, the original data can be equally divided into 2N original data blocks, where N is a positive integer, the data composed of the first N original data blocks in the original data is confirmed as the first data segment, and the last N original data The block-composed data is identified as the second data segment, ie a=b=N.
S202,对所述第一数据段进行纠删码运算,获得m个第一校验数据块,基于a个原始数据块和m个第一校验数据块生成a+m个第一冗余数据块,对所述第二数据段进行纠删码运算,获得n个第二校验数据块,基于a个原始数据块和m个第一校验数据块生成b+n个第二冗余数据块。S202, perform erasure coding operation on the first data segment to obtain m first check data blocks, and generate a+m first redundant data based on a original data blocks and m first check data blocks block, perform erasure coding operation on the second data segment to obtain n second check data blocks, and generate b+n second redundant data based on a original data blocks and m first check data blocks piece.
具体的,数据冗余备份装置对第一数据段中的a个原始数据块进行纠删码运算,获得m个校验数据块,将a个原始数据块和m个校验数据块确认为a+m个第一冗余数据块;数据冗余备份装置对第二数据段中的b个原始数据块进行纠删码运算,获得n个校验数据块,将b个原始数据块和n个校验数据块确认为第二冗余数据块,其中m为不大于a的正整数,n为不大于b的正整数。Specifically, the data redundancy backup device performs erasure code operation on a original data blocks in the first data segment, obtains m check data blocks, and confirms a original data blocks and m check data blocks as a +m first redundant data blocks; the data redundancy backup device performs erasure code operation on b original data blocks in the second data segment to obtain n check data blocks, and converts b original data blocks and n The check data block is confirmed as the second redundant data block, wherein m is a positive integer not greater than a, and n is a positive integer not greater than b.
可选的,当a与b相等,即第一数据段和第二数据段中的原始数据块的数量相同时,可以用相同的纠删码运算规则来对第一数据段和第二数据段进行纠删码运算,从而生成相同数量的校验数据块,等到相同数量的第一冗余数据块和第二冗余数据块,可以理解的是,在利用第一冗余数据对第一数据段进行数据修复处理,和利用第二冗余数据对第二数据段进行数据修复处理时,可以基于同一套纠删码运算规则来进行纠删码重建,减少了数据冗余备份装置的运算负担。Optionally, when a and b are equal, that is, when the number of original data blocks in the first data segment and the second data segment is the same, the same erasure coding operation rule can be used to perform the first data segment and the second data segment. Perform erasure code operation to generate the same number of check data blocks, and wait until the same number of first redundant data blocks and second redundant data blocks. It can be understood that when using the first redundant data to When performing data restoration processing on the second data segment and using the second redundant data for data restoration processing on the second data segment, the erasure code reconstruction can be performed based on the same set of erasure code operation rules, which reduces the computational burden of the data redundancy backup device. .
请参见图3a,为本申请实施例提供了一种纠删码运算规则的举例示意图,以第一数据段为例,第一数据段中包含有a个原始数据块,其中a=6,图3a中的a1、a2、a3、a4、a5、a6为第一数据段中的6个原始数据块,组成向量【a1,a2,a3,a4,a5,a6】,将一个a*a的单位矩阵和一个a*m的B矩阵如图3a进行拼接,其中m=3,将拼接后的矩阵与原始数据块组成的向量按照如图3a的方式进行矩阵运算,可以得到向量【a1,a2,a3,a4,a5,a6,c1,c2,c3】,其中c1,c2,c3为得到的m个校验数据块。其中B矩阵为纠删码运算中的编码矩阵,可以采用范德蒙矩阵,因为范德蒙矩阵中任意的子矩阵都是可逆的,方便利用编码矩阵进行纠删码重建。Please refer to FIG. 3a, which provides a schematic diagram of an example of an erasure code operation rule for an embodiment of the present application. Taking the first data segment as an example, the first data segment includes a original data blocks, where a=6. a1, a2, a3, a4, a5, a6 in 3a are the 6 original data blocks in the first data segment, which form a vector [a1, a2, a3, a4, a5, a6], which is a unit of a*a. The matrix and an a*m B matrix are spliced as shown in Figure 3a, where m=3, and the vector composed of the spliced matrix and the original data block is performed in the manner of Figure 3a. The matrix operation can be obtained to obtain the vector [a1, a2, a3, a4, a5, a6, c1, c2, c3], wherein c1, c2, c3 are the obtained m check data blocks. The B matrix is the encoding matrix in the erasure code operation, and the Vandermonde matrix can be used, because any sub-matrix in the Vandermonde matrix is invertible, and it is convenient to use the encoding matrix for erasure code reconstruction.
可以理解的是,当a与b相等,即第一数据段和第二数据段中的原始数据块的数量相同时,第二数据段也可以利用相同的B矩阵来进行纠删码运算,得到同样数量的校验数据块,即n=m,在对第二数据段进行数据修复处理时,也可以基于相同的B矩阵来进行纠删码重建。It can be understood that when a and b are equal, that is, when the number of original data blocks in the first data segment and the second data segment is the same, the second data segment can also use the same B matrix to perform erasure coding operations, and obtain The same number of check data blocks, that is, n=m, can also be used for erasure code reconstruction based on the same B matrix when performing data repair processing on the second data segment.
S203,当a+m与b+n相等时,基于所述第一冗余数据块生成第一冗余数据,并基于所述第二冗余数据块生成第二冗余数据。S203, when a+m is equal to b+n, generate first redundant data based on the first redundant data block, and generate second redundant data based on the second redundant data block.
具体的,在数据冗余备份装置分别对第一数据段和第二数据段进行纠删码运算之后,生成了a+m个第一冗余数据块和b+n个第二冗余数据块,当a+m=b+n时,将a+m个第一冗余数据块组成第一冗余数据,将b+n个第二冗余数据块组成第二冗余数据。Specifically, after the data redundancy backup device performs erasure coding operations on the first data segment and the second data segment, respectively, a+m first redundant data blocks and b+n second redundant data blocks are generated , when a+m=b+n, a+m first redundant data blocks are formed into first redundant data, and b+n second redundant data blocks are formed into second redundant data.
S204,当a+m与b+n不相等时,在所述第一冗余数据块或所述第二冗余数据块中添加补充数 据块,使得在添加后所述第一冗余数据块和所述第二冗余数据块的数量相同,基于所述第一冗余数据块生成第一冗余数据,并基于所述第二冗余数据块生成第二冗余数据。S204, when a+m and b+n are not equal, add a supplementary data block to the first redundant data block or the second redundant data block, so that after adding the first redundant data block The same as the number of the second redundant data blocks, the first redundant data is generated based on the first redundant data blocks, and the second redundant data is generated based on the second redundant data blocks.
具体的,当a+m与b+n不相等时,数据冗余备份装置会添加第一冗余数据块或添加第二冗余数据块,使得第一冗余数据块与第二冗余数据块的数量相同,再将第一冗余数据块组成为第一冗余数据,将第二冗余数据块组成第二冗余数据。Specifically, when a+m and b+n are not equal, the data redundancy backup device will add the first redundant data block or add the second redundant data block, so that the first redundant data block and the second redundant data If the number of blocks is the same, the first redundant data block is composed of the first redundant data, and the second redundant data block is composed of the second redundant data.
可选的,添加第一冗余数据块或添加第二冗余数据块的方法为:当a+m小于b+n时,数据冗余备份装置将至少一个预设内容的补充数据块添加至第一冗余数据块中,使得所述第一冗余数据块的数量增加至b+n个;当a+m大于b+n时,数据冗余备份装置将至少一个预设内容的补充数据块添加至第二冗余数据块中,使得所述第二冗余数据块的数量增加至a+m个,所述预设内容可以在数据冗余备份装置中进行设置,也可以由用户在终端设备中进行设置并将该设置发送给数据冗余备份装置,所述预设内容可以是全为0或全为1的数据块,也可以是第一冗余数据块或第二冗余数据块中的部分数据块,且补充数据块的大小与原始数据块和校验数据块相同。Optionally, the method for adding the first redundant data block or adding the second redundant data block is: when a+m is less than b+n, the data redundancy backup device adds at least one supplementary data block of preset content to the In the first redundant data block, the number of the first redundant data blocks is increased to b+n; when a+m is greater than b+n, the data redundancy backup device will supplement the data of at least one preset content. The block is added to the second redundant data block, so that the number of the second redundant data block is increased to a+m, and the preset content can be set in the data redundant backup device, or can be set by the user in the Setting in the terminal device and sending the setting to the data redundancy backup device, the preset content may be a data block with all 0s or all 1s, or a first redundant data block or a second redundant data Part of the data block in the block, and the size of the supplementary data block is the same as the original data block and the check data block.
请参见图3b,为本申请实施例提供了一种添加补充数据块的举例示意图,以a+m大于b+n为例,此时需要数据冗余备份装置需要向第二冗余数据块中添加补充数据块,X1~Xy为y个预设内容的补充数据块,其中y为正整数,为了使得第一冗余数据块与第二冗余数据块的数量相等,则需要y=a+m-(b+n),在b+n个第二冗余数据块中加入y个预设内容的补充数据块,得到a+m个第二冗余数据块。其中X1~Xy可以为全为0或全为1的数据块,也可以为第二数据段中b个原始数据块和m个校验数据块中的任意y个相同或不相同的数据块。Referring to FIG. 3b, an example schematic diagram of adding a supplementary data block is provided for an embodiment of the present application. Taking a+m greater than b+n as an example, in this case, a data redundancy backup device needs to be added to the second redundant data block. Add supplementary data blocks, X1~Xy are supplementary data blocks of y preset contents, where y is a positive integer. In order to make the number of the first redundant data block and the second redundant data block equal, y=a+ m-(b+n), adding y supplementary data blocks with preset contents to b+n second redundant data blocks to obtain a+m second redundant data blocks. Wherein X1 to Xy may be all 0 or all 1 data blocks, or may be any y identical or different data blocks among the b original data blocks and the m check data blocks in the second data segment.
S205,将所述第一冗余数据块与所述第二冗余数据块逐一进行异或运算,生成第三冗余数据块,获得由所述第三冗余数据块组成的第三冗余数据,将所述第一冗余数据、所述第二冗余数据和所述第三冗余数据分别保存在可用区集合中不同的可用区中。S205, perform an XOR operation on the first redundant data block and the second redundant data block one by one to generate a third redundant data block, and obtain a third redundant data block composed of the third redundant data block data, the first redundant data, the second redundant data and the third redundant data are respectively stored in different availability zones in the availability zone set.
具体的,数据冗余备份装置将第一冗余数据块和第二冗余数据块逐一进行异或运算,生成第三冗余数据块,将得到的第三冗余数据块组成第三冗余数据。Specifically, the data redundancy backup device performs an XOR operation on the first redundant data block and the second redundant data block one by one to generate a third redundant data block, and the obtained third redundant data block forms a third redundant data block. data.
可选的,数据冗余备份装置按照预设顺序分别给第一冗余数据块和第二冗余数据块进行编号,所述预设顺序可以在数据冗余备份装置中进行设置,也可以由用户在终端设备中进行设置并将该设置发送给数据冗余备份装置,例如所述预设顺序可以是原始数据块在前,校验数据块在中间,补充数据块在最后,依次进行编号;数据冗余备份装置对编号相同的第一冗余数据块和第二冗余数据块逐一进行异或运算,生成对应编号的第三冗余数据块,由于第一冗余数据块和第二冗余数据块的数量相同,所以能够全部对应进行异或运算,生成相同数量的第三冗余数据块,将所述第三冗余数据块按照所述预设顺序组成第三冗余数据,将第三冗余数据保存在与第一冗余数据和第二冗余数据不同的可用区中,当第一冗余数据或第二冗余数据出现了大量损坏,导致不能直接用自身的数据进行复原时,可以利用异或关系和第三冗余数据进行数据复原,例如可以用第一冗余数据和第三冗余数据的异或关系复原第二冗余数据,用第二冗余数据和第三冗余数据的异或关系复原第一冗余数据。Optionally, the data redundancy backup device numbers the first redundant data block and the second redundant data block respectively according to a preset sequence, and the preset sequence can be set in the data redundancy backup device, or can be set by the data redundancy backup device. The user sets the terminal equipment and sends the setting to the data redundancy backup device. For example, the preset sequence can be that the original data block is in the front, the verification data block is in the middle, and the supplementary data block is at the end, which are sequentially numbered; The data redundancy backup device performs the XOR operation on the first redundant data block and the second redundant data block with the same number one by one to generate the third redundant data block with the corresponding number. The number of the remaining data blocks is the same, so all corresponding XOR operations can be performed to generate the same number of third redundant data blocks, and the third redundant data blocks are formed into third redundant data according to the preset order. The third redundant data is stored in a different availability zone from the first redundant data and the second redundant data. When the first redundant data or the second redundant data is damaged a lot, it is impossible to directly use its own data for processing. During restoration, data restoration can be performed by using the XOR relationship and the third redundant data. For example, the second redundant data can be restored by using the XOR relationship between the first redundant data and the third redundant data, and the second redundant data and The exclusive OR relationship of the third redundant data restores the first redundant data.
请参见图3c,为本申请实施例提供了一种第三冗余数据生成的举例示意图,以a+m大于b+n为例,数据冗余备份装置使得第一冗余数据块和第二冗余数据块的数量均为a+m后,A1、A2、 A3、A4...Aa+m为编号1~a+m的第一冗余数据块,B1、B2、B3、B4...Ba+m为编号1~a+m的第二冗余数据块,将编号相同的第一冗余数据块和第二冗余数据块逐位进行异或运算,生成对应编号的第三冗余数据块,例如A1和B1进行异或运算生成编号为1的第三冗余数据块C1,由此获得C1、C2、C3、C4...Ca+m,即编号1~a+m的第三冗余数据块,将这a+m个第三冗余数据块按照编号顺序组成第三冗余数据。Please refer to FIG. 3c, which provides a schematic diagram of an example of third redundant data generation for an embodiment of the present application. Taking a+m greater than b+n as an example, the data redundancy backup device makes the first redundant data block and the second redundant data After the number of redundant data blocks is a+m, A1, A2, A3, A4...Aa+m are the first redundant data blocks numbered 1 to a+m, B1, B2, B3, B4.. .Ba+m is the second redundant data block numbered 1 to a+m, and the first redundant data block and the second redundant data block with the same number are XORed bit by bit to generate the third redundant data block with the corresponding number. The remaining data blocks, for example, A1 and B1 perform an exclusive OR operation to generate a third redundant data block C1 numbered 1, thereby obtaining C1, C2, C3, C4...Ca+m, that is, the numbers 1~a+m For the third redundant data block, the a+m third redundant data blocks form third redundant data according to the sequence of numbers.
在本申请实施例中,将原始数据拆分为第一数据段和第二数据段,分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据,并且加入补充数据块使得第一冗余数据块的数量与第二冗余数据块的数量相同,再逐一对第一冗余数据块和第二冗余数据块进行异或运算,生成相同数量的第三冗余数据块,使得异或运算更加方便快捷。通过先将原始数据进行拆分,再分别对拆分后的数据进行纠删码运算,使得第一数据段和第二数据段中损坏数据较少时可以利用其所在可用区内的数据进行修复,节省了数据修复成本,单个可用区可自行进行数据修复处理,提高了数据冗余备份装置的可靠性,并且通过异或运算的方式生成第三冗余数据,是的冗余备份数据的生成更加方便简单、快捷。In the embodiment of the present application, the original data is split into a first data segment and a second data segment, and erasure code operations are performed on the first data segment and the second data segment respectively to obtain the first redundant data and the second redundant data, and add supplementary data blocks so that the number of the first redundant data blocks is the same as the number of the second redundant data blocks. The OR operation generates the same number of third redundant data blocks, which makes the XOR operation more convenient and faster. By first splitting the original data, and then performing erasure code operation on the split data, the first data segment and the second data segment can be repaired by using the data in the availability zone when there is less damaged data. , saves the cost of data restoration, a single availability zone can perform data restoration processing by itself, improves the reliability of the data redundant backup device, and generates third redundant data through XOR operation, which is the generation of redundant backup data. More convenient, simple and fast.
可以理解的是,采用S201-S205的数据冗余备份方法可以将用户上传的原始数据进行保存、备份,当用户需要通过数据冗余备份装置来获取原始数据,而由于可用区断网、发生自然灾害导致服务器损坏等原因,导致原始数据发生了损坏时,数据冗余备份装置可以对原始数据进行数据修复处理。请参见图4,为本申请实施例提供了一种数据修复处理的流程示意图,如图4所示,本申请实施例的所述数据修复处理可以包括以下步骤S301-S306。It can be understood that the data redundancy backup method of S201-S205 can be used to save and backup the original data uploaded by the user. When the user needs to obtain the original data through the data redundancy backup device, due to the availability zone When the original data is damaged due to the server damage caused by the disaster, the data redundancy backup device can perform data restoration processing on the original data. Referring to FIG. 4 , a schematic flowchart of data restoration processing is provided in this embodiment of the present application. As shown in FIG. 4 , the data restoration processing in this embodiment of the present application may include the following steps S301 - S306 .
S301,获取目标可用区中校验数据块的校验数量和原始数据块的原始数量。S301: Acquire the verification quantity of the verification data blocks and the original quantity of the original data blocks in the target availability zone.
具体的,当数据冗余备份装置接收到终端设备针对原始数据的访问业务时,数据冗余备份装置会在保存有第一冗余数据和第二冗余数据的可用区中获取原始数据块来组成原始数据并返回给终端设备,当第一冗余数据或第二冗余数据中的原始数据块发生损坏导致数据冗余备份装置无法直接组成原始数据时,数据冗余备份装置会对损坏数据块进行数据修复处理,将出现损坏数据块的可用区确认为目标可用区,当损坏数据块的损坏数量小于或等于损坏阈值时,数据冗余备份装置直接利用目标可用区内的冗余数据块对损坏数据块进行数据修复处理;当损坏数据块的损坏数量大于损坏阈值时,数据冗余备份装置会调用除了目标可用区的其他可用区中的冗余数据块对损坏数据块进行数据修复处理。所述冗余数据块为所述第一冗余数据块、所述第二冗余数据块或所述第三冗余数据块。Specifically, when the data redundancy backup device receives the access service for the original data from the terminal device, the data redundancy backup device will obtain the original data block in the availability zone where the first redundant data and the second redundant data are stored to The original data is formed and returned to the terminal equipment. When the original data block in the first redundant data or the second redundant data is damaged and the data redundant backup device cannot directly form the original data, the data redundant backup device will damage the data. Perform data repair processing on the block, and confirm the availability zone with damaged data blocks as the target availability zone. When the number of damaged data blocks is less than or equal to the damage threshold, the data redundancy backup device directly uses the redundant data blocks in the target availability zone. Perform data restoration processing on damaged data blocks; when the number of damaged data blocks is greater than the damage threshold, the data redundancy backup device will call redundant data blocks in other availability zones except the target availability zone to perform data restoration processing on damaged data blocks . The redundant data block is the first redundant data block, the second redundant data block or the third redundant data block.
数据冗余备份装置会获取目标可用区中校验数据块的校验数量和原始数据块的原始数量,因为在纠删码重建中,需要利用目标可用区中原始数量的冗余数据块才能进行数据修复处理,所以若要直接利用目标可用区中的冗余数据块对损坏数据进行数据修复处理,数据冗余备份装置最多可以容忍目标可用区中出现校验数量的损坏数据块,所以损坏阈值的最大值可以与校验数量相等。The data redundancy backup device will obtain the check number of the check data blocks in the target availability zone and the original number of the original data blocks, because in the erasure code reconstruction, it is necessary to use the original number of redundant data blocks in the target availability zone. Data recovery processing, so if you want to directly use redundant data blocks in the target availability zone to perform data recovery processing on damaged data, the data redundancy backup device can tolerate at most the number of damaged data blocks in the target availability zone, so the damage threshold is The maximum value can be equal to the number of checks.
S302,当所述目标可用区中的所述损坏数据块的损坏数量小于或等于所述校验数量时,在所述目标可用区中获取所述原始数量指示的剩余数据块。S302, when the damaged quantity of the damaged data blocks in the target availability zone is less than or equal to the check quantity, obtain the remaining data blocks indicated by the original quantity in the target availability zone.
具体的,当目标可用区中损坏数据块的损坏数据小于或等于校验数量时,数据冗余备份装置 可以直接用目标可用区中的冗余数据块对损坏数据进行数据修复处理,数据冗余备份装置会在目标可用区中获取原始数量只是的剩余数据块,所述剩余数据块为目标可用区中除损坏数据块以外的其他冗余数据块。Specifically, when the damaged data of the damaged data blocks in the target availability zone is less than or equal to the number of checksums, the data redundancy backup device can directly use the redundant data blocks in the target availability zone to perform data repair processing on the damaged data, and the data redundancy The backup device will obtain the original number of remaining data blocks in the target availability zone, and the remaining data blocks are other redundant data blocks in the target availability zone except for the damaged data blocks.
S303,基于所述剩余数据块对所述损坏数据块进行数据修复处理,以获取修复数据块,采用所述修复数据块对所述损坏数据块进行替换。S303: Perform data repair processing on the damaged data block based on the remaining data block to obtain a repaired data block, and replace the damaged data block with the repaired data block.
具体的,数据冗余备份装置利用剩余数据块和S201-S205中对原始数据进行冗余备份时所采用的纠删码运算规则来进行纠删码重建,对损坏数据块进行数据修复处理,得到相应的修复数据块,数据冗余备份装置将修复数据块对损坏数据块进行替换以得到全部的原始数据块,将原始数据块进行组合获取原始数据,数据冗余备份装置就可以将原始数据返回给终端设备。Specifically, the data redundancy backup device uses the remaining data blocks and the erasure code operation rules used in the redundant backup of the original data in S201-S205 to perform erasure code reconstruction, and performs data repair processing on the damaged data blocks to obtain Correspondingly repaired data blocks, the data redundancy backup device replaces the damaged data blocks with the repaired data blocks to obtain all the original data blocks, and combines the original data blocks to obtain the original data, and the data redundancy backup device can return the original data. to the terminal device.
请一并参见图4a,为本申请实施例提供了一种纠删码重建的举例示意图,以保存有第一冗余数据块的目标可用区中出现了损坏数据块为例,当损坏数据块为a1、a2、a3时,将S201-S205步骤中用于纠删码运算的拼接后的矩阵,对应损坏数据块的行数据删除,得到新的矩阵B’,并计算得到矩阵B’的逆矩阵,然后数据冗余备份装置可以将矩阵B’的逆矩阵与剩余数据块组成的向量进行矩阵运算,可以得到向量【a1,a2,a3,a4,a5,a6】,其中计算得到的a1,a2,a3即为修复数据块,数据冗余备份装置将修复数据块与损坏数据块进行替换,即可得到完整的原始数据。Please refer to FIG. 4a together, which provides a schematic diagram of an example of erasure code reconstruction for this embodiment of the application. Taking a damaged data block in the target availability zone where the first redundant data block is stored as an example, when the damaged data block is When it is a1, a2, and a3, delete the spliced matrix used for the erasure code operation in steps S201-S205, and delete the row data corresponding to the damaged data block to obtain a new matrix B', and calculate the inverse of the matrix B' matrix, and then the data redundancy backup device can perform matrix operation on the vector composed of the inverse matrix of matrix B' and the remaining data blocks, and can obtain the vector [a1, a2, a3, a4, a5, a6], where the calculated a1, a2 and a3 are repaired data blocks, and the data redundancy backup device replaces the repaired data blocks with the damaged data blocks to obtain complete original data.
S304,当所述目标可用区中的所述损坏数据块的损坏数量大于所述校验数量时,获取所述损坏数据块的目标编号。S304, when the damaged quantity of the damaged data block in the target availability zone is greater than the check quantity, obtain the target number of the damaged data block.
具体的,当目标可用区中损坏数据块的损坏数量大于校验数据时,数据冗余备份装置无法使用目标可用区中的冗余数据亏直接对损坏数据块进行数据修复处理,需要获取第一剩余可用区和第二剩余可用区中的冗余数据块来对损坏数据块进行数据修复,所述第一剩余可用区和所述第二剩余可用区为可用区集合中除目标可用区外的其他可用区。数据冗余备份装置会获取损坏数据块的编号,将这些编号确认为目标编号。Specifically, when the number of damaged data blocks in the target availability zone is greater than the check data, the data redundancy backup device cannot use the redundant data in the target availability zone to directly perform data repair processing on the damaged data blocks, and needs to obtain the first The redundant data blocks in the remaining availability zone and the second remaining availability zone are used to perform data repair on the damaged data blocks, and the first remaining availability zone and the second remaining availability zone are the available zone sets except the target availability zone. Other Availability Zones. The data redundancy backup unit obtains the numbers of the damaged data blocks and confirms these numbers as target numbers.
S305,分别在第一剩余可用区和第二剩余可用区中获取所述目标编号对应的冗余数据块。S305: Obtain redundant data blocks corresponding to the target numbers in the first remaining available area and the second remaining available area, respectively.
具体的,数据冗余备份装置会在第一剩余可用区中获取目标编号对应的第一目标数据块,在第二剩余可用区中获取目标编号对应的第二目标数据块。Specifically, the data redundancy backup device will obtain the first target data block corresponding to the target number in the first remaining available zone, and obtain the second target data block corresponding to the target number in the second remaining available zone.
S306,基于所述目标编号对应的冗余数据块对所述损坏数据块进行数据修复处理,以获取修复数据块,采用所述修复数据块对所述损坏数据块进行替换。S306: Perform data repair processing on the damaged data block based on the redundant data block corresponding to the target number to obtain a repaired data block, and replace the damaged data block with the repaired data block.
具体的,第三冗余数据块是由编号相同的第一冗余数据块和第二冗余数据块进行异或运算得到的,可以理解的是,编号相同的第一冗余数据块、第二冗余数据块和第三冗余数据块两两之间进行异或运算都能得到另外一个冗余数据块,例如编号相同的第三冗余数据块和第二冗余数据块进行异或运算,可以得到编号相同的第一冗余数据块。所以,数据冗余备份装置对编号相同的第一目标数据块和第二目标数据块逐一进行异或运算,可以生成对应编号的修复数据块,将修复数据块与损坏数据块进行替换,即可得到完整的原始数据。Specifically, the third redundant data block is obtained by performing the XOR operation on the first redundant data block and the second redundant data block with the same number. The XOR operation between the second redundant data block and the third redundant data block can obtain another redundant data block. For example, the third redundant data block and the second redundant data block with the same number are XORed. operation, the first redundant data block with the same number can be obtained. Therefore, the data redundancy backup device performs the XOR operation on the first target data block and the second target data block with the same number one by one, so as to generate the repair data block with the corresponding number, and replace the repair data block with the damaged data block. Get complete raw data.
可以理解的是,数据冗余备份装置除了由于终端设备的访问业务发现原始数据中出现了损坏数据块,数据冗余备份装置还可以按照预设时间进行自检,损坏数据块也仅限于原始数据块,校验数据块出现损坏时也可以采用S301-S306的方法进行数据修复处理。当第一数据段和第二数据 段中的原始数据块的数量相同,并且用相同的纠删码运算规则来对第一数据段和第二数据段进行纠删码运算时,基于第一冗余数据块和第二冗余数据块生成的第三冗余数据块,也可以用同一套纠删码运算规则来进行复原。It is understandable that, in addition to finding damaged data blocks in the original data due to the access service of the terminal equipment, the data redundancy backup device can also perform self-checking according to a preset time, and the damaged data blocks are limited to the original data. When the check data block is damaged, the method of S301-S306 can also be used for data restoration processing. When the number of original data blocks in the first data segment and the second data segment is the same, and the same erasure coding operation rules are used to perform erasure coding operations on the first data segment and the second data segment, the The third redundant data block generated by the residual data block and the second redundant data block can also be restored by using the same set of erasure coding operation rules.
在本申请实施例中,将原始数据拆分为第一数据段和第二数据段,分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据,并且加入补充数据块使得第一冗余数据块的数量与第二冗余数据块的数量相同,再逐一对第一冗余数据块和第二冗余数据块进行异或运算,生成相同数量的第三冗余数据块,使得异或运算更加方便快捷。通过先将原始数据进行拆分,再分别对拆分后的数据进行纠删码运算,使得第一数据段和第二数据段中损坏数据较少时可以利用其所在可用区内的数据进行修复,节省了数据修复成本,单个可用区可自行进行数据修复处理,提高了数据冗余备份装置的可靠性,并且通过异或运算的方式生成第三冗余数据,是的冗余备份数据的生成更加方便简单、快捷。In the embodiment of the present application, the original data is split into a first data segment and a second data segment, and erasure code operations are performed on the first data segment and the second data segment respectively to obtain the first redundant data and the second redundant data, and add supplementary data blocks so that the number of the first redundant data blocks is the same as the number of the second redundant data blocks. The OR operation generates the same number of third redundant data blocks, which makes the XOR operation more convenient and faster. By first splitting the original data, and then performing erasure code operation on the split data, the first data segment and the second data segment can be repaired by using the data in the availability zone when there is less damaged data. , saves the cost of data restoration, a single availability zone can perform data restoration processing by itself, improves the reliability of the data redundant backup device, and generates third redundant data through XOR operation, which is the generation of redundant backup data. More convenient, simple and fast.
下面将结合附图5-附图11,对本申请实施例提供的数据冗余备份装置进行详细介绍。需要说明的是,附图5-附图11中的数据冗余备份装置,用于执行本申请图2-图4所示实施例的方法,为了便于说明,仅示出了与本申请实施例相关的部分,具体技术细节未揭示的,请参照本申请图2-图4所示的实施例。The data redundancy backup device provided by the embodiment of the present application will be described in detail below with reference to FIG. 5 to FIG. 11 . It should be noted that the data redundancy backup devices in FIG. 5 to FIG. 11 are used to execute the methods of the embodiments shown in FIG. 2 to FIG. 4 of the present application. For convenience of description, only the same embodiment as the embodiment of the present application is shown. For relevant parts, if the specific technical details are not disclosed, please refer to the embodiments shown in FIG. 2 to FIG. 4 of the present application.
请参见图5,其示出了本申请一个示例性实施例提供的数据冗余备份装置的结构示意图。该数据冗余备份装置可以通过软件、硬件或者两者的结合实现成为装置的全部或一部分。该装置1包括数据拆分模块11、纠删码运算模块12、逻辑运算模块13和数据保存模块14。Please refer to FIG. 5 , which shows a schematic structural diagram of a data redundancy backup device provided by an exemplary embodiment of the present application. The data redundancy backup device can be implemented as a whole or a part of the device through software, hardware or a combination of the two. The device 1 includes a data splitting module 11 , an erasure code operation module 12 , a logic operation module 13 and a data storage module 14 .
数据拆分模块11,用于将原始数据拆分为第一数据段和第二数据段;A data splitting module 11, for splitting the original data into a first data segment and a second data segment;
纠删码运算模块12,用于分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据;an erasure code operation module 12, configured to perform an erasure code operation on the first data segment and the second data segment respectively to obtain first redundant data and second redundant data;
逻辑运算模块13,用于基于所述第一冗余数据和所述第二冗余数据之间的逻辑关系生成第三冗余数据;a logical operation module 13, configured to generate third redundant data based on the logical relationship between the first redundant data and the second redundant data;
数据保存模块14,用于将所述第一冗余数据、所述第二冗余数据和所述第三冗余数据分别保存在可用区集合中不同的可用区中。The data saving module 14 is configured to save the first redundant data, the second redundant data and the third redundant data in different availability zones in the availability zone set, respectively.
在本实施例中,将原始数据拆分为第一数据段和第二数据段,分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据,基于所述第一冗余数据和所述第二冗余数据之间的逻辑关系生成第三冗余数据。通过先将原始数据进行拆分,再分别对拆分后的数据进行纠删码运算,使得第一数据段和第二数据段中损坏数据较少时可以利用其所在可用区内的数据进行修复,节省了数据修复成本。In this embodiment, the original data is split into a first data segment and a second data segment, and erasure code operations are performed on the first data segment and the second data segment respectively to obtain the first redundant data and For second redundant data, third redundant data is generated based on the logical relationship between the first redundant data and the second redundant data. By first splitting the original data, and then performing erasure code operation on the split data, the first data segment and the second data segment can be repaired by using the data in the availability zone when there is less damaged data. , saving data repair costs.
请参见图6,其示出了本申请一个示例性实施例提供的数据冗余备份装置的结构示意图。该数据冗余备份装置可以通过软件、硬件或者两者的结合实现成为装置的全部或一部分。该装置1包括数据拆分模块11、纠删码运算模块12、逻辑运算模块13、、数据保存模块14、第一数据修复模块15和第二数据修复模块16。Please refer to FIG. 6 , which shows a schematic structural diagram of a data redundancy backup apparatus provided by an exemplary embodiment of the present application. The data redundancy backup device can be implemented as a whole or a part of the device through software, hardware or a combination of the two. The device 1 includes a data splitting module 11 , an erasure code operation module 12 , a logic operation module 13 , a data storage module 14 , a first data repair module 15 and a second data repair module 16 .
数据拆分模块11,用于将原始数据拆分为第一数据段和第二数据段;A data splitting module 11, for splitting the original data into a first data segment and a second data segment;
具体的,请一并参见图7,为本申请实施例提供了一种数据拆分模块的结构示意图。如图7所示,所述数据拆分模块11可以包括:Specifically, please refer to FIG. 7 together, which provides a schematic structural diagram of a data splitting module in an embodiment of the present application. As shown in Figure 7, the data splitting module 11 may include:
第一拆分单元111,用于将原始数据均分为至少两个原始数据块;a first splitting unit 111, configured to equally divide the original data into at least two original data blocks;
第二拆分单元112,用于将所述原始数据拆分为由a个原始数据块组成的第一数据段和由b个原始数据块组成的第二数据段,所述a和所述b均为正整数。The second splitting unit 112 is configured to split the original data into a first data segment consisting of a original data blocks and a second data segment consisting of b original data blocks, the a and the b All are positive integers.
纠删码运算模块12,用于分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据;an erasure code operation module 12, configured to perform an erasure code operation on the first data segment and the second data segment respectively to obtain first redundant data and second redundant data;
具体的,请一并参见图8,为本申请实施例提供了一种纠删码运算模块的结构示意图。如图8所示,所述纠删码运算模块12可以包括:Specifically, please refer to FIG. 8 together, which provides a schematic structural diagram of an erasure correction code operation module according to an embodiment of the present application. As shown in FIG. 8, the erasure code operation module 12 may include:
第一运算单元121,用于对所述第一数据段进行纠删码运算,获得m个第一校验数据块,基于a个原始数据块和m个第一校验数据块生成a+m个第一冗余数据块,所述m为不大于a的正整数;The first operation unit 121 is configured to perform an erasure code operation on the first data segment, obtain m first check data blocks, and generate a+m based on a original data blocks and m first check data blocks a first redundant data block, the m is a positive integer not greater than a;
第二运算单元122,用于对所述第二数据段进行纠删码运算,获得n个第二校验数据块,基于a个原始数据块和m个第一校验数据块生成b+n个第二冗余数据块,所述n为不大于b的正整数;The second operation unit 122 is configured to perform an erasure code operation on the second data segment, obtain n second check data blocks, and generate b+n based on a original data blocks and m first check data blocks a second redundant data block, the n is a positive integer not greater than b;
第一数据生成单元123,用于当a+m与b+n相等时,基于所述第一冗余数据块生成第一冗余数据,并基于所述第二冗余数据块生成第二冗余数据;The first data generating unit 123 is configured to generate first redundant data based on the first redundant data block and generate second redundant data based on the second redundant data block when a+m is equal to b+n. remaining data;
第二数据生成单元124,用于当a+m与b+n不相等时,在所述第一冗余数据块或所述第二冗余数据块中添加补充数据块,使得在添加后所述第一冗余数据块和所述第二冗余数据块的数量相同,基于所述第一冗余数据块生成第一冗余数据,并基于所述第二冗余数据块生成第二冗余数据。The second data generating unit 124 is configured to add a supplementary data block to the first redundant data block or the second redundant data block when a+m and b+n are not equal, so that after the addition, the The number of the first redundant data block and the second redundant data block is the same, the first redundant data is generated based on the first redundant data block, and the second redundant data is generated based on the second redundant data block. remaining data.
可选的,所述第二数据生成单元124具体用于当a+m小于b+n时,将至少一个预设内容的补充数据块添加至所述第一冗余数据块中,使得所述第一冗余数据块的数量增加至b+n个;Optionally, the second data generating unit 124 is specifically configured to add at least one supplementary data block of preset content to the first redundant data block when a+m is less than b+n, so that the The number of first redundant data blocks is increased to b+n;
当a+m大于b+n时,将至少一个预设内容的补充数据块添加至所述第二冗余数据块中,使得所述第二冗余数据块的数量增加至a+m个。When a+m is greater than b+n, at least one supplementary data block of preset content is added to the second redundant data block, so that the number of the second redundant data blocks is increased to a+m.
逻辑运算模块13,用于基于所述第一冗余数据和所述第二冗余数据之间的逻辑关系生成第三冗余数据;a logical operation module 13, configured to generate third redundant data based on the logical relationship between the first redundant data and the second redundant data;
可选的,所述逻辑运算模块13具体用于将所述第一冗余数据块与所述第二冗余数据块逐一进行异或运算,生成第三冗余数据块,获得由所述第三冗余数据块组成的第三冗余数据。Optionally, the logic operation module 13 is specifically configured to perform an XOR operation on the first redundant data block and the second redundant data block one by one to generate a third redundant data block, and obtain the first redundant data block. The third redundant data is composed of three redundant data blocks.
具体的,请一并参见图9,为本申请实施例提供了一种逻辑运算模块的结构示意图。如图9所示,所述逻辑运算模块13可以包括:Specifically, please refer to FIG. 9 together, which provides a schematic structural diagram of a logic operation module according to an embodiment of the present application. As shown in FIG. 9, the logic operation module 13 may include:
编号单元131,用于按照预设顺序分别给所述第一冗余数据块和所述第二冗余数据块进行编号; Numbering unit 131, used to respectively number the first redundant data block and the second redundant data block according to a preset sequence;
异或运算单元132,用于对编号相同的所述第一冗余数据块和所述第二冗余数据块逐一进行异或运算,生成对应编号的第三冗余数据块;The XOR operation unit 132 is used to perform XOR operation on the first redundant data block and the second redundant data block with the same number one by one to generate the third redundant data block of the corresponding number;
第三数据生成单元133,用于将所述第三冗余数据块按照所述预设顺序组成第三冗余数据。The third data generating unit 133 is configured to form the third redundant data blocks according to the preset sequence into third redundant data.
数据保存模块14,用于将所述第一冗余数据、所述第二冗余数据和所述第三冗余数据分别保存在可用区集合中不同的可用区中;a data saving module 14, configured to save the first redundant data, the second redundant data and the third redundant data in different availability zones in the availability zone set;
第一数据修复模块15,用于当目标可用区中的损坏数据块的损坏数量小于或等于损坏阈值时,基于所述目标可用区中的冗余数据块对所述损坏数据块进行数据修复处理以获取修复数据块,采用所述修复数据块与所述损坏数据块进行替换;The first data repair module 15 is configured to perform data repair processing on the damaged data blocks based on redundant data blocks in the target availability zone when the damaged quantity of the damaged data blocks in the target availability zone is less than or equal to the damage threshold to obtain a repaired data block, and replace the repaired data block with the damaged data block;
具体的,请一并参见图10,为本申请实施例提供了一种第一数据修复模块的结构示意图。如图10所示,所述第一数据修复模块15可以包括:Specifically, please refer to FIG. 10 together, which provides a schematic structural diagram of a first data restoration module according to an embodiment of the present application. As shown in FIG. 10 , the first data repair module 15 may include:
第一数量获取单元151,用于获取目标可用区中校验数据块的校验数量和原始数据块的原始数量;The first quantity obtaining unit 151 is used to obtain the check quantity of the check data block and the original quantity of the original data block in the target availability zone;
第一数据块获取单元152,用于当所述目标可用区中的所述损坏数据块的损坏数量小于或等于所述校验数量时,在所述目标可用区获取所述原始数量指示的剩余数据块;The first data block obtaining unit 152 is configured to obtain, in the target availability zone, the remainder indicated by the original quantity in the target availability zone when the damaged quantity of the damaged data blocks in the target availability zone is less than or equal to the check quantity data block;
第一替换单元153,用于基于所述剩余数据块对所述损坏数据块进行数据修复处理,以获取修复数据块,采用所述修复数据块对所述损坏数据块进行替换。The first replacement unit 153 is configured to perform data repair processing on the damaged data block based on the remaining data block to obtain a repaired data block, and use the repaired data block to replace the damaged data block.
第二数据修复模块16,用于当目标可用区中的损坏数据块的损坏数量大于损坏阈值时,基于第一剩余可用区和第二剩余可用区中的冗余数据块对所述损坏数据块进行数据修复处理以获取修复数据块,采用所述修复数据块与所述损坏数据块进行替换;The second data repair module 16 is configured to, when the damaged quantity of the damaged data blocks in the target availability zone is greater than the damage threshold value, perform an analysis on the damaged data blocks based on the redundant data blocks in the first remaining availability zone and the second remaining availability zone performing data repair processing to obtain repair data blocks, and using the repair data blocks to replace the damaged data blocks;
所述目标可用区为所述可用区集合中的任一可用区,所述第一剩余可用区和第二剩余可用区为所述可用区集合中除所述目标可用区外的其它两个可用区,所述剩余数据块为所述目标可用区中除所述损坏数据块外的其他冗余数据块,所述冗余数据块为所述第一冗余数据块、所述第二冗余数据块或所述第三冗余数据块。The target availability zone is any availability zone in the availability zone set, and the first remaining availability zone and the second remaining availability zone are the other two available zones in the availability zone set except the target availability zone area, the remaining data blocks are other redundant data blocks in the target availability zone except the damaged data block, and the redundant data blocks are the first redundant data block, the second redundant data block data block or the third redundant data block.
具体的,请一并参见图11,为本申请实施例提供了一种第二数据修复模块的结构示意图。如图11所示,所述第二数据修复模块16可以包括:Specifically, please refer to FIG. 11 together, which provides a schematic structural diagram of a second data restoration module according to an embodiment of the present application. As shown in FIG. 11 , the second data repair module 16 may include:
第二数量获取单元161,用于获取目标可用区中的校验数据块的校验数量;The second quantity obtaining unit 161 is used to obtain the check quantity of check data blocks in the target availability zone;
编号获取单元162,用于当所述目标可用区中的所述损坏数据块的损坏数量大于所述校验数量时,获取所述损坏数据块的目标编号;a number obtaining unit 162, configured to obtain the target number of the damaged data block when the damaged quantity of the damaged data block in the target availability zone is greater than the check quantity;
第二数据块获取单元163,用于分别在第一剩余可用区和第二剩余可用区中获取所述目标编号对应的冗余数据块;The second data block obtaining unit 163 is configured to obtain redundant data blocks corresponding to the target numbers in the first remaining available zone and the second remaining available zone respectively;
可选的,所述第二数据块获取单元163具体用于获取第一剩余可用区中所述目标编号对应的第一目标数据块,和第二剩余可用区中所述目标编号对应的第二目标数据块。Optionally, the second data block obtaining unit 163 is specifically configured to obtain the first target data block corresponding to the target number in the first remaining available zone, and the second data block corresponding to the target number in the second remaining available zone. target data block.
第二替换单元164,用于基于所述目标编号对应的冗余数据块对所述损坏数据块进行数据修复处理,以获取修复数据块,采用所述修复数据块对所述损坏数据块进行替换。The second replacement unit 164 is configured to perform data repair processing on the damaged data block based on the redundant data block corresponding to the target number to obtain a repaired data block, and use the repaired data block to replace the damaged data block .
可选的,所述第二替换单元164具体用于对编号相同的所述第一目标数据块和所述第二目标数据块逐一进行异或运算,生成对应编号的修复数据块,采用所述修复数据块对所述损坏数据块进行替换。Optionally, the second replacement unit 164 is specifically configured to perform an XOR operation on the first target data block and the second target data block with the same number one by one, and generate a repair data block with a corresponding number, using the The repair data block replaces the damaged data block.
在本实施例中,将原始数据拆分为第一数据段和第二数据段,分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据,并且加入补充数据块使得第一 冗余数据块的数量与第二冗余数据块的数量相同,再逐一对第一冗余数据块和第二冗余数据块进行异或运算,生成相同数量的第三冗余数据块,使得异或运算更加方便快捷。通过先将原始数据进行拆分,再分别对拆分后的数据进行纠删码运算,使得第一数据段和第二数据段中损坏数据较少时可以利用其所在可用区内的数据进行修复,节省了数据修复成本,单个可用区可自行进行数据修复处理,提高了数据冗余备份装置的可靠性,并且通过异或运算的方式生成第三冗余数据,是的冗余备份数据的生成更加方便简单、快捷。In this embodiment, the original data is split into a first data segment and a second data segment, and erasure code operations are performed on the first data segment and the second data segment respectively to obtain the first redundant data and The second redundant data is added, and the supplementary data blocks are added so that the number of the first redundant data blocks is the same as the number of the second redundant data blocks, and then the first redundant data block and the second redundant data block are XORed one by one operation to generate the same number of third redundant data blocks, which makes the XOR operation more convenient and fast. By first splitting the original data, and then performing erasure code operation on the split data, the first data segment and the second data segment can be repaired by using the data in the availability zone when there is less damaged data. , saves the cost of data restoration, a single availability zone can perform data restoration processing by itself, improves the reliability of the data redundant backup device, and generates third redundant data through XOR operation, which is the generation of redundant backup data. More convenient, simple and fast.
需要说明的是,上述实施例提供的数据冗余备份装置在执行数据冗余备份方法时,仅以上述各功能模块的划分进行举例说明,实际应用中,可以根据需要而将上述功能分配由不同的功能模块完成,即将设备的内部结构划分成不同的功能模块,以完成以上描述的全部或者部分功能。另外,上述实施例提供的数据冗余备份装置与数据冗余备份方法实施例属于同一构思,其体现实现过程详见方法实施例,这里不再赘述。It should be noted that, when the data redundancy backup device provided in the above embodiments executes the data redundancy backup method, only the division of the above functional modules is used as an example for illustration. The functional modules of the device are completed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the data redundancy backup device and the data redundancy backup method provided by the above embodiments belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.
上述本申请实施例序号仅仅为了描述,不代表实施例的优劣。The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.
本申请实施例还提供了一种计算机存储介质,所述计算机存储介质可以存储有多条指令,所述指令适于由处理器加载并执行如上述图1-图4a所示实施例的所述数据冗余备份方法,具体执行过程可以参见图1-图4a所示实施例的具体说明,在此不进行赘述。An embodiment of the present application further provides a computer storage medium, where the computer storage medium can store a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the above-described embodiments shown in FIG. 1 to FIG. 4a. For the data redundancy backup method, for the specific execution process, reference may be made to the specific description of the embodiment shown in FIG. 1 to FIG. 4 a , which will not be repeated here.
本申请还提供了一种计算机程序产品,该计算机程序产品存储有至少一条指令,所述至少一条指令由所述处理器加载并执行如上述图1-图4a所示实施例的所述数据冗余备份方法,具体执行过程可以参见图1-图4a所示实施例的具体说明,在此不进行赘述。The present application also provides a computer program product, the computer program product stores at least one instruction, and the at least one instruction is loaded by the processor and executes the data redundancy in the embodiments shown in the foregoing FIG. 1 to FIG. 4a. For the redundant backup method, the specific execution process can refer to the specific description of the embodiment shown in FIG. 1-FIG. 4a, which will not be repeated here.
请参见图12,为本申请实施例提供了一种电子设备的结构示意图。如图12所示,所述电子设备1000可以包括:至少一个处理器1001,至少一个网络接口1004,用户接口1003,存储器1005,至少一个通信总线1002。Referring to FIG. 12 , a schematic structural diagram of an electronic device is provided in an embodiment of the present application. As shown in FIG. 12 , the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , user interface 1003 , memory 1005 , and at least one communication bus 1002 .
其中,通信总线1002用于实现这些组件之间的连接通信。Among them, the communication bus 1002 is used to realize the connection and communication between these components.
其中,用户接口1003可以包括显示屏(Display)、摄像头(Camera),可选用户接口1003还可以包括标准的有线接口、无线接口。The user interface 1003 may include a display screen (Display) and a camera (Camera), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.
其中,网络接口1004可选的可以包括标准的有线接口、无线接口(如WI-FI接口)。Wherein, the network interface 1004 may optionally include a standard wired interface and a wireless interface (eg, a WI-FI interface).
其中,处理器1001可以包括一个或者多个处理核心。处理器1001利用各种借口和线路连接整个服务器1000内的各个部分,通过运行或执行存储在存储器1005内的指令、程序、代码集或指令集,以及调用存储在存储器1005内的数据,执行服务器1000的各种功能和处理数据。可选的,处理器1001可以采用数字信号处理(Digital Signal Processing,DSP)、现场可编程门阵列(Field-Programmable Gate Array,FPGA)、可编程逻辑阵列(Programmable Logic Array,PLA)中的至少一种硬件形式来实现。处理器1001可集成中央处理器(Central Processing Unit,CPU)、图像处理器(Graphics Processing Unit,GPU)和调制解调器等中的一种或几种的组合。其中,CPU 主要处理操作系统、用户界面和应用程序等;GPU用于负责显示屏所需要显示的内容的渲染和绘制;调制解调器用于处理无线通信。可以理解的是,上述调制解调器也可以不集成到处理器1001中,单独通过一块芯片进行实现。The processor 1001 may include one or more processing cores. The processor 1001 uses various excuses and lines to connect various parts of the entire server 1000, and executes the server by running or executing the instructions, programs, code sets or instruction sets stored in the memory 1005, and calling the data stored in the memory 1005. 1000s of various functions and processing data. Optionally, the processor 1001 may adopt at least one of digital signal processing (Digital Signal Processing, DSP), field-programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). A hardware form is implemented. The processor 1001 may integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), a modem, and the like. Among them, the CPU mainly deals with the operating system, user interface and application programs, etc.; the GPU is used for rendering and drawing the content that needs to be displayed on the display screen; the modem is used for processing wireless communication. It can be understood that, the above-mentioned modem may not be integrated into the processor 1001, but is implemented by a single chip.
其中,存储器1005可以包括随机存储器(Random Access Memory,RAM),也可以包括只读存储器(Read-Only Memory)。可选的,该存储器1005包括非瞬时性计算机可读介质(non-transitory computer-readable storage medium)。存储器1005可用于存储指令、程序、代码、代码集或指令集。存储器1005可包括存储程序区和存储数据区,其中,存储程序区可存储用于实现操作系统的指令、用于至少一个功能的指令(比如触控功能、声音播放功能、图像播放功能等)、用于实现上述各个方法实施例的指令等;存储数据区可存储上面各个方法实施例中涉及到的数据等。存储器1005可选的还可以是至少一个位于远离前述处理器1001的存储装置。如图12所示,作为一种计算机存储介质的存储器1005中可以包括操作系统、网络通信模块、用户接口模块以及数据冗余备份应用程序。The memory 1005 may include random access memory (Random Access Memory, RAM), or may include read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. Memory 1005 may be used to store instructions, programs, codes, sets of codes, or sets of instructions. The memory 1005 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), Instructions and the like used to implement the above method embodiments; the storage data area may store the data and the like involved in the above method embodiments. Optionally, the memory 1005 may also be at least one storage device located away from the aforementioned processor 1001 . As shown in FIG. 12 , the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a data redundancy backup application program.
在图12所示的电子设备1000中,用户接口1003主要用于为用户提供输入的接口,获取用户输入的数据;而处理器1001可以用于调用存储器1005中存储的数据冗余备份应用程序,并具体执行以下操作:In the electronic device 1000 shown in FIG. 12 , the user interface 1003 is mainly used to provide an input interface for the user to obtain the data input by the user; and the processor 1001 can be used to call the data redundancy backup application program stored in the memory 1005, And do the following specifically:
将原始数据拆分为第一数据段和第二数据段;split the original data into a first data segment and a second data segment;
分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据;Perform erasure coding operations on the first data segment and the second data segment respectively to obtain first redundant data and second redundant data;
基于所述第一冗余数据和所述第二冗余数据之间的逻辑关系生成第三冗余数据;generating third redundant data based on the logical relationship between the first redundant data and the second redundant data;
将所述第一冗余数据、所述第二冗余数据和所述第三冗余数据分别保存在可用区集合中不同的可用区中。The first redundant data, the second redundant data and the third redundant data are respectively stored in different availability zones in the set of availability zones.
在一个实施例中,所述处理器1001在执行将原始数据拆分为第一数据段和第二数据段时,具体执行以下操作:In one embodiment, when the processor 1001 splits the original data into the first data segment and the second data segment, the processor 1001 specifically performs the following operations:
将原始数据均分为至少两个原始数据块;Divide the original data into at least two original data blocks equally;
将所述原始数据拆分为由a个原始数据块组成的第一数据段和由b个原始数据块组成的第二数据段,所述a和所述b均为正整数。The original data is divided into a first data segment consisting of a original data blocks and a second data segment consisting of b original data blocks, where a and b are both positive integers.
在一个实施例中,所述处理器1001在执行分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据时,具体执行以下操作:In one embodiment, when the processor 1001 performs an erasure code operation on the first data segment and the second data segment respectively to obtain the first redundant data and the second redundant data, the processor 1001 specifically executes Do the following:
对所述第一数据段进行纠删码运算,获得m个第一校验数据块,基于a个原始数据块和m个第一校验数据块生成a+m个第一冗余数据块,所述m为不大于a的正整数;Erasure code operation is performed on the first data segment to obtain m first check data blocks, and a+m first redundant data blocks are generated based on a original data blocks and m first check data blocks, The m is a positive integer not greater than a;
对所述第二数据段进行纠删码运算,获得n个第二校验数据块,基于a个原始数据块和m个第一校验数据块生成b+n个第二冗余数据块,所述n为不大于b的正整数;Perform erasure coding operation on the second data segment to obtain n second check data blocks, and generate b+n second redundant data blocks based on a original data blocks and m first check data blocks, The n is a positive integer not greater than b;
当a+m与b+n相等时,基于所述第一冗余数据块生成第一冗余数据,并基于所述第二冗余数据块生成第二冗余数据;When a+m is equal to b+n, generating first redundant data based on the first redundant data block, and generating second redundant data based on the second redundant data block;
当a+m与b+n不相等时,在所述第一冗余数据块或所述第二冗余数据块中添加补充数据块,使得在添加后所述第一冗余数据块和所述第二冗余数据块的数量相同,基于所述第一冗余数据块 生成第一冗余数据,并基于所述第二冗余数据块生成第二冗余数据。When a+m is not equal to b+n, a supplementary data block is added to the first redundant data block or the second redundant data block, so that after the addition, the first redundant data block and all redundant data blocks are added. The number of the second redundant data blocks is the same, the first redundant data is generated based on the first redundant data blocks, and the second redundant data is generated based on the second redundant data blocks.
在一个实施例中,所述处理器1001在执行当a+m与b+n不相等时,在所述第一冗余数据块或所述第二冗余数据块中添加补充数据块,使得在添加后所述第一冗余数据块和所述第二冗余数据块的数量相同,基于所述第一冗余数据块生成第一冗余数据,并基于所述第二冗余数据块生成第二冗余数据时,具体执行以下操作:In one embodiment, the processor 1001 adds a supplementary data block to the first redundant data block or the second redundant data block when a+m and b+n are not equal, so that After adding, the first redundant data blocks and the second redundant data blocks have the same number, first redundant data is generated based on the first redundant data blocks, and first redundant data is generated based on the second redundant data blocks When generating the second redundant data, specifically perform the following operations:
当a+m小于b+n时,将至少一个预设内容的补充数据块添加至所述第一冗余数据块中,使得所述第一冗余数据块的数量增加至b+n个;When a+m is less than b+n, adding at least one supplementary data block of preset content to the first redundant data block, so that the number of the first redundant data blocks is increased to b+n;
当a+m大于b+n时,将至少一个预设内容的补充数据块添加至所述第二冗余数据块中,使得所述第二冗余数据块的数量增加至a+m个。When a+m is greater than b+n, at least one supplementary data block of preset content is added to the second redundant data block, so that the number of the second redundant data blocks is increased to a+m.
在一个实施例中,所述处理器1001在执行基于所述第一冗余数据和所述第二冗余数据之间的逻辑关系生成第三冗余数据时,具体执行以下操作:In one embodiment, when the processor 1001 performs generating the third redundant data based on the logical relationship between the first redundant data and the second redundant data, the processor 1001 specifically performs the following operations:
将所述第一冗余数据块与所述第二冗余数据块逐一进行异或运算,生成第三冗余数据块,获得由所述第三冗余数据块组成的第三冗余数据。Perform an exclusive OR operation on the first redundant data block and the second redundant data block one by one to generate a third redundant data block, and obtain third redundant data composed of the third redundant data block.
在一个实施例中,所述处理器1001在执行将所述第一冗余数据块与所述第二冗余数据块逐一进行异或预算,生成第三冗余数据块,获得由所述第三冗余数据块组成的第三冗余数据时,具体执行以下操作:In one embodiment, the processor 1001 performs XOR budgeting on the first redundant data block and the second redundant data block one by one, generates a third redundant data block, and obtains the first redundant data block from the second redundant data block. When the third redundant data is composed of three redundant data blocks, the following operations are specifically performed:
按照预设顺序分别给所述第一冗余数据块和所述第二冗余数据块进行编号;Number the first redundant data block and the second redundant data block respectively according to a preset sequence;
对编号相同的所述第一冗余数据块和所述第二冗余数据块逐一进行异或运算,生成对应编号的第三冗余数据块;The first redundant data block and the second redundant data block with the same number are XORed one by one to generate the third redundant data block of the corresponding number;
将所述第三冗余数据块按照所述预设顺序组成第三冗余数据。The third redundant data blocks are formed into third redundant data according to the preset sequence.
在一个实施例中,所述处理器1001在执行将所述数据冗余备份方法时,还执行以下操作:In one embodiment, when the processor 1001 executes the data redundancy backup method, the processor 1001 further executes the following operations:
当目标可用区中的损坏数据块的损坏数量小于或等于损坏阈值时,基于所述目标可用区中的冗余数据块对所述损坏数据块进行数据修复处理以获取修复数据块,采用所述修复数据块与所述损坏数据块进行替换;When the damage quantity of the damaged data blocks in the target availability zone is less than or equal to the damage threshold, perform data repair processing on the damaged data blocks based on the redundant data blocks in the target availability zone to obtain the repaired data blocks, using the Replace the repaired data block with the damaged data block;
当目标可用区中的损坏数据块的损坏数量大于损坏阈值时,基于第一剩余可用区和第二剩余可用区中的冗余数据块对所述损坏数据块进行数据修复处理以获取修复数据块,采用所述修复数据块与所述损坏数据块进行替换;When the damaged number of damaged data blocks in the target availability zone is greater than the damage threshold, perform data repair processing on the damaged data blocks based on the redundant data blocks in the first remaining availability zone and the second remaining availability zone to obtain repaired data blocks , using the repaired data block to replace the damaged data block;
所述目标可用区为所述可用区集合中的任一可用区,所述第一剩余可用区和第二剩余可用区为所述可用区集合中除所述目标可用区外的其它两个可用区,所述剩余数据块为所述目标可用区中除所述损坏数据块外的其他冗余数据块,所述冗余数据块为所述第一冗余数据块、所述第二冗余数据块或所述第三冗余数据块。The target availability zone is any availability zone in the availability zone set, and the first remaining availability zone and the second remaining availability zone are the other two available zones in the availability zone set except the target availability zone area, the remaining data blocks are other redundant data blocks in the target availability zone except the damaged data block, and the redundant data blocks are the first redundant data block, the second redundant data block data block or the third redundant data block.
在一个实施例中,所述处理器1001在执行当目标可用区中的损坏数据块的损坏数量小于或等于损坏阈值时,基于所述目标可用区中的冗余数据块对所述损坏数据块进行数据修复处理以获取修复数据块,采用所述修复数据块与所述损坏数据块进行替换时,具体执行以下操作:In one embodiment, when the processor 1001 executes, when the number of corrupted data blocks in the target availability zone is less than or equal to a corruption threshold, the corrupted data blocks are determined based on redundant data blocks in the target availability zone. Perform data repair processing to obtain a repaired data block, and when using the repaired data block to replace the damaged data block, specifically perform the following operations:
获取目标可用区中校验数据块的校验数量和原始数据块的原始数量;Obtain the number of parity data blocks and the original number of original data blocks in the target availability zone;
当所述目标可用区中的所述损坏数据块的损坏数量小于或等于所述校验数量时,在所述目标 可用区中获取所述原始数量指示的剩余数据块;When the damaged quantity of the damaged data blocks in the target availability zone is less than or equal to the check quantity, obtain the remaining data blocks indicated by the original quantity in the target availability zone;
基于所述剩余数据块对所述损坏数据块进行数据修复处理,以获取修复数据块,采用所述修复数据块对所述损坏数据块进行替换。Perform data repair processing on the damaged data block based on the remaining data blocks to obtain a repaired data block, and use the repaired data block to replace the damaged data block.
在一个实施例中,所述处理器1001在执行当目标可用区中的损坏数据块的损坏数量大于损坏阈值时,基于第一剩余可用区和第二剩余可用区中的冗余数据块对所述损坏数据块进行数据修复处理以获取修复数据块,采用所述修复数据块与所述损坏数据块进行替换时,具体执行以下操作:In one embodiment, the processor 1001 performs, when the damaged quantity of the damaged data blocks in the target availability zone is greater than the damage threshold, the pair of all data blocks based on the redundant data blocks in the first remaining availability zone and the second remaining availability zone. Perform data repair processing on the damaged data block to obtain a repaired data block, and when using the repaired data block to replace the damaged data block, specifically perform the following operations:
获取目标可用区中的校验数据块的校验数量;Get the number of checksums of checksum blocks in the target availability zone;
当所述目标可用区中的所述损坏数据块的损坏数量大于所述校验数量时,获取所述损坏数据块的目标编号;When the damage quantity of the damaged data block in the target availability zone is greater than the check quantity, obtain the target number of the damaged data block;
分别在第一剩余可用区和第二剩余可用区中获取所述目标编号对应的冗余数据块;Obtaining redundant data blocks corresponding to the target numbers in the first remaining available zone and the second remaining available zone respectively;
基于所述目标编号对应的冗余数据块对所述损坏数据块进行数据修复处理,以获取修复数据块,采用所述修复数据块对所述损坏数据块进行替换。Perform data repair processing on the damaged data block based on the redundant data block corresponding to the target number to obtain a repaired data block, and use the repaired data block to replace the damaged data block.
在一个实施例中,所述处理器1001在执行分别在第一剩余可用区和第二剩余可用区中获取所述目标编号对应的冗余数据块时,具体执行以下操作:In one embodiment, when the processor 1001 acquires the redundant data blocks corresponding to the target numbers in the first remaining availability zone and the second remaining availability zone respectively, the processor 1001 specifically performs the following operations:
获取第一剩余可用区中所述目标编号对应的第一目标数据块,和第二剩余可用区中所述目标编号对应的第二目标数据块;Obtain the first target data block corresponding to the target number in the first remaining available zone, and the second target data block corresponding to the target number in the second remaining available zone;
所述基于所述目标编号对应的冗余数据块对所述损坏数据块进行数据修复处理,以获取修复数据块,采用所述修复数据块对所述损坏数据块进行替换,包括:Performing data repair processing on the damaged data block based on the redundant data block corresponding to the target number to obtain a repaired data block, and using the repaired data block to replace the damaged data block, including:
对编号相同的所述第一目标数据块和所述第二目标数据块逐一进行异或运算,生成对应编号的修复数据块,采用所述修复数据块对所述损坏数据块进行替换。The XOR operation is performed on the first target data block and the second target data block with the same number one by one to generate a repair data block with a corresponding number, and the damaged data block is replaced by the repair data block.
在本实施例中,将原始数据拆分为第一数据段和第二数据段,分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据,并且加入补充数据块使得第一冗余数据块的数量与第二冗余数据块的数量相同,再逐一对第一冗余数据块和第二冗余数据块进行异或运算,生成相同数量的第三冗余数据块,使得异或运算更加方便快捷。通过先将原始数据进行拆分,再分别对拆分后的数据进行纠删码运算,使得第一数据段和第二数据段中损坏数据较少时可以利用其所在可用区内的数据进行修复,节省了数据修复成本,单个可用区可自行进行数据修复处理,提高了数据冗余备份装置的可靠性,并且通过异或运算的方式生成第三冗余数据,是的冗余备份数据的生成更加方便简单、快捷。In this embodiment, the original data is split into a first data segment and a second data segment, and erasure code operations are performed on the first data segment and the second data segment respectively to obtain the first redundant data and The second redundant data is added, and the supplementary data blocks are added so that the number of the first redundant data blocks is the same as the number of the second redundant data blocks, and then the first redundant data block and the second redundant data block are XORed one by one operation to generate the same number of third redundant data blocks, which makes the XOR operation more convenient and fast. By first splitting the original data, and then performing erasure code operation on the split data, the first data segment and the second data segment can be repaired by using the data in the availability zone when there is less damaged data. , saves the cost of data restoration, a single availability zone can perform data restoration processing by itself, improves the reliability of the data redundant backup device, and generates third redundant data through XOR operation, which is the generation of redundant backup data. More convenient, simple and fast.
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程,是可以通过计算机程序来指令相关的硬件来完成,所述的程序可存储于一计算机可读取存储介质中,该程序在执行时,可包括如上述各方法的实施例的流程。其中,所述的存储介质可为磁碟、光盘、只读存储记忆体或随机存储记忆体等。Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. During execution, the processes of the embodiments of the above-mentioned methods may be included. Wherein, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, or a random storage memory, and the like.
以上所揭露的仅为本申请较佳实施例而已,当然不能以此来限定本申请之权利范围,因此依本申请权利要求所作的等同变化,仍属本申请所涵盖的范围。The above disclosures are only the preferred embodiments of the present application, and of course, the scope of the rights of the present application cannot be limited by this. Therefore, equivalent changes made according to the claims of the present application are still within the scope of the present application.

Claims (20)

  1. 一种数据冗余备份方法,其特征在于,所述方法包括:A data redundancy backup method, characterized in that the method comprises:
    将原始数据拆分为第一数据段和第二数据段;split the original data into a first data segment and a second data segment;
    分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据;Perform erasure coding operations on the first data segment and the second data segment respectively to obtain first redundant data and second redundant data;
    基于所述第一冗余数据和所述第二冗余数据之间的逻辑关系生成第三冗余数据;generating third redundant data based on the logical relationship between the first redundant data and the second redundant data;
    将所述第一冗余数据、所述第二冗余数据和所述第三冗余数据分别保存在可用区集合中不同的可用区中。The first redundant data, the second redundant data and the third redundant data are respectively stored in different availability zones in the set of availability zones.
  2. 根据权利要求1所述的方法,其特征在于,所述将原始数据拆分为第一数据段和第二数据段,包括:The method according to claim 1, wherein the splitting the original data into a first data segment and a second data segment comprises:
    将原始数据均分为至少两个原始数据块;Divide the original data into at least two original data blocks equally;
    将所述原始数据拆分为由a个原始数据块组成的第一数据段和由b个原始数据块组成的第二数据段,所述a和所述b均为正整数。The original data is divided into a first data segment consisting of a original data blocks and a second data segment consisting of b original data blocks, where a and b are both positive integers.
  3. 根据权利要求2所述的方法,其特征在于,所述分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据,包括:The method according to claim 2, wherein the performing an erasure code operation on the first data segment and the second data segment respectively to obtain the first redundant data and the second redundant data, comprising: :
    对所述第一数据段进行纠删码运算,获得m个第一校验数据块,基于a个原始数据块和m个第一校验数据块生成a+m个第一冗余数据块,所述m为不大于a的正整数;Erasure code operation is performed on the first data segment to obtain m first check data blocks, and a+m first redundant data blocks are generated based on a original data blocks and m first check data blocks, The m is a positive integer not greater than a;
    对所述第二数据段进行纠删码运算,获得n个第二校验数据块,基于a个原始数据块和m个第一校验数据块生成b+n个第二冗余数据块,所述n为不大于b的正整数;Perform erasure coding operation on the second data segment to obtain n second check data blocks, and generate b+n second redundant data blocks based on a original data blocks and m first check data blocks, The n is a positive integer not greater than b;
    当a+m与b+n相等时,基于所述第一冗余数据块生成第一冗余数据,并基于所述第二冗余数据块生成第二冗余数据;When a+m is equal to b+n, generating first redundant data based on the first redundant data block, and generating second redundant data based on the second redundant data block;
    当a+m与b+n不相等时,在所述第一冗余数据块或所述第二冗余数据块中添加补充数据块,使得在添加后所述第一冗余数据块和所述第二冗余数据块的数量相同,基于所述第一冗余数据块生成第一冗余数据,并基于所述第二冗余数据块生成第二冗余数据。When a+m is not equal to b+n, a supplementary data block is added to the first redundant data block or the second redundant data block, so that after the addition, the first redundant data block and all redundant data blocks are added. The number of the second redundant data blocks is the same, the first redundant data is generated based on the first redundant data blocks, and the second redundant data is generated based on the second redundant data blocks.
  4. 根据权利要求3所述的方法,其特征在于,所述当a+m与b+n不相等时,在所述第一冗余数据块或所述第二冗余数据块中添加补充数据块,使得在添加后所述第一冗余数据块和所述第二冗余数据块的数量相同,基于所述第一冗余数据块生成第一冗余数据,并基于所述第二冗余数据块生成第二冗余数据,包括:The method according to claim 3, wherein when a+m and b+n are not equal, adding a supplementary data block to the first redundant data block or the second redundant data block , so that the number of the first redundant data block and the second redundant data block is the same after adding, first redundant data is generated based on the first redundant data block, and based on the second redundant data The data block generates second redundant data including:
    当a+m小于b+n时,将至少一个预设内容的补充数据块添加至所述第一冗余数据块中,使得所述第一冗余数据块的数量增加至b+n个;When a+m is less than b+n, adding at least one supplementary data block of preset content to the first redundant data block, so that the number of the first redundant data blocks is increased to b+n;
    当a+m大于b+n时,将至少一个预设内容的补充数据块添加至所述第二冗余数据块中,使得所述第二冗余数据块的数量增加至a+m个。When a+m is greater than b+n, at least one supplementary data block of preset content is added to the second redundant data block, so that the number of the second redundant data blocks is increased to a+m.
  5. 根据权利要求3所述的方法,其特征在于,所述基于所述第一冗余数据和所述第二冗余数据之间的逻辑关系生成第三冗余数据,包括:The method according to claim 3, wherein the generating third redundant data based on the logical relationship between the first redundant data and the second redundant data comprises:
    将所述第一冗余数据块与所述第二冗余数据块逐一进行异或运算,生成第三冗余数据块,获得由所述第三冗余数据块组成的第三冗余数据。Perform an exclusive OR operation on the first redundant data block and the second redundant data block one by one to generate a third redundant data block, and obtain third redundant data composed of the third redundant data block.
  6. 根据权利要求5所述的方法,其特征在于,所述将所述第一冗余数据块与所述第二冗余数据块逐一进行异或预算,生成第三冗余数据块,获得由所述第三冗余数据块组成的第三冗余数据,包括:The method according to claim 5, wherein the XOR budget is performed on the first redundant data block and the second redundant data block one by one to generate a third redundant data block, and the obtained The third redundant data composed of the third redundant data block includes:
    按照预设顺序分别给所述第一冗余数据块和所述第二冗余数据块进行编号;Number the first redundant data block and the second redundant data block respectively according to a preset sequence;
    对编号相同的所述第一冗余数据块和所述第二冗余数据块逐一进行异或运算,生成对应编号的第三冗余数据块;The first redundant data block and the second redundant data block with the same number are XORed one by one to generate the third redundant data block of the corresponding number;
    将所述第三冗余数据块按照所述预设顺序组成第三冗余数据。The third redundant data blocks are formed into third redundant data according to the preset sequence.
  7. 根据权利要求1或6所述的方法,其特征在于,所述方法还包括:The method according to claim 1 or 6, wherein the method further comprises:
    当目标可用区中的损坏数据块的损坏数量小于或等于损坏阈值时,基于所述目标可用区中的冗余数据块对所述损坏数据块进行数据修复处理以获取修复数据块,采用所述修复数据块与所述损坏数据块进行替换;When the damage quantity of the damaged data blocks in the target availability zone is less than or equal to the damage threshold, perform data repair processing on the damaged data blocks based on the redundant data blocks in the target availability zone to obtain the repaired data blocks, using the Replace the repaired data block with the damaged data block;
    当目标可用区中的损坏数据块的损坏数量大于损坏阈值时,基于第一剩余可用区和第二剩余可用区中的冗余数据块对所述损坏数据块进行数据修复处理以获取修复数据块,采用所述修复数据块与所述损坏数据块进行替换;When the damaged number of damaged data blocks in the target availability zone is greater than the damage threshold, perform data repair processing on the damaged data blocks based on the redundant data blocks in the first remaining availability zone and the second remaining availability zone to obtain repaired data blocks , using the repaired data block to replace the damaged data block;
    所述目标可用区为所述可用区集合中的任一可用区,所述第一剩余可用区和第二剩余可用区为所述可用区集合中除所述目标可用区外的其它两个可用区,所述剩余数据块为所述目标可用区中除所述损坏数据块外的其他冗余数据块,所述冗余数据块为所述第一冗余数据块、所述第二冗余数据块或所述第三冗余数据块。The target availability zone is any availability zone in the availability zone set, and the first remaining availability zone and the second remaining availability zone are the other two available zones in the availability zone set except the target availability zone area, the remaining data blocks are other redundant data blocks in the target availability zone except the damaged data block, and the redundant data blocks are the first redundant data block, the second redundant data block data block or the third redundant data block.
  8. 根据权利要求7所述的方法,其特征在于,所述当目标可用区中的损坏数据块的损坏数量小于或等于损坏阈值时,基于所述目标可用区中的冗余数据块对所述损坏数据块进行数据修复处理以获取修复数据块,采用所述修复数据块与所述损坏数据块进行替换,包括:The method according to claim 7, characterized in that, when the damage quantity of the damaged data blocks in the target availability zone is less than or equal to a damage threshold, the damage is determined based on redundant data blocks in the target availability zone. The data block is subjected to data repair processing to obtain a repaired data block, and the repaired data block is used to replace the damaged data block, including:
    获取目标可用区中校验数据块的校验数量和原始数据块的原始数量;Obtain the number of parity data blocks and the original number of original data blocks in the target availability zone;
    当所述目标可用区中的所述损坏数据块的损坏数量小于或等于所述校验数量时,在所述目标可用区中获取所述原始数量指示的剩余数据块;When the damaged quantity of the damaged data blocks in the target availability zone is less than or equal to the check quantity, obtain the remaining data blocks indicated by the original quantity in the target availability zone;
    基于所述剩余数据块对所述损坏数据块进行数据修复处理,以获取修复数据块,采用所述修复数据块对所述损坏数据块进行替换。Perform data repair processing on the damaged data block based on the remaining data blocks to obtain a repaired data block, and use the repaired data block to replace the damaged data block.
  9. 根据权利要求7所述的方法,其特征在于,所述当目标可用区中的损坏数据块的损坏数量 大于损坏阈值时,基于第一剩余可用区和第二剩余可用区中的冗余数据块对所述损坏数据块进行数据修复处理以获取修复数据块,采用所述修复数据块与所述损坏数据块进行替换,包括:The method according to claim 7, wherein when the number of damaged data blocks in the target availability zone is greater than a damage threshold, the method is based on redundant data blocks in the first remaining availability zone and the second remaining availability zone Perform data repair processing on the damaged data block to obtain a repaired data block, and use the repaired data block to replace the damaged data block, including:
    获取目标可用区中的校验数据块的校验数量;Get the number of checksums of checksum blocks in the target availability zone;
    当所述目标可用区中的所述损坏数据块的损坏数量大于所述校验数量时,获取所述损坏数据块的目标编号;When the damage quantity of the damaged data block in the target availability zone is greater than the check quantity, obtain the target number of the damaged data block;
    分别在第一剩余可用区和第二剩余可用区中获取所述目标编号对应的冗余数据块;Obtaining redundant data blocks corresponding to the target numbers in the first remaining available zone and the second remaining available zone respectively;
    基于所述目标编号对应的冗余数据块对所述损坏数据块进行数据修复处理,以获取修复数据块,采用所述修复数据块对所述损坏数据块进行替换。Perform data repair processing on the damaged data block based on the redundant data block corresponding to the target number to obtain a repaired data block, and use the repaired data block to replace the damaged data block.
  10. 根据权利要求9所述的方法,其特征在于,所述分别在第一剩余可用区和第二剩余可用区中获取所述目标编号对应的冗余数据块,包括:The method according to claim 9, wherein the obtaining the redundant data blocks corresponding to the target numbers in the first remaining availability zone and the second remaining availability zone respectively comprises:
    获取第一剩余可用区中所述目标编号对应的第一目标数据块,和第二剩余可用区中所述目标编号对应的第二目标数据块;Obtain the first target data block corresponding to the target number in the first remaining available zone, and the second target data block corresponding to the target number in the second remaining available zone;
    所述基于所述目标编号对应的冗余数据块对所述损坏数据块进行数据修复处理,以获取修复数据块,采用所述修复数据块对所述损坏数据块进行替换,包括:Performing data repair processing on the damaged data block based on the redundant data block corresponding to the target number to obtain a repaired data block, and using the repaired data block to replace the damaged data block, including:
    对编号相同的所述第一目标数据块和所述第二目标数据块逐一进行异或运算,生成对应编号的修复数据块,采用所述修复数据块对所述损坏数据块进行替换。The XOR operation is performed on the first target data block and the second target data block with the same number one by one to generate a repair data block with a corresponding number, and the damaged data block is replaced by the repair data block.
  11. 一种数据冗余备份装置,其特征在于,所述装置包括:A data redundancy backup device, characterized in that the device comprises:
    数据拆分模块,用于将原始数据拆分为第一数据段和第二数据段;a data splitting module for splitting the original data into a first data segment and a second data segment;
    纠删码运算模块,用于分别对所述第一数据段和所述第二数据段进行纠删码运算,得到第一冗余数据和第二冗余数据;an erasure code operation module, configured to perform an erasure code operation on the first data segment and the second data segment respectively to obtain first redundant data and second redundant data;
    逻辑运算模块,用于基于所述第一冗余数据和所述第二冗余数据之间的逻辑关系生成第三冗余数据;a logic operation module, configured to generate third redundant data based on the logical relationship between the first redundant data and the second redundant data;
    数据保存模块,用于将所述第一冗余数据、所述第二冗余数据和所述第三冗余数据分别保存在可用区集合中不同的可用区中。A data saving module, configured to save the first redundant data, the second redundant data and the third redundant data in different availability zones in the availability zone set, respectively.
  12. 根据权利要求11所述的装置,其特征在于,所述数据拆分模块包括:The device according to claim 11, wherein the data splitting module comprises:
    第一拆分单元,用于将原始数据均分为至少两个原始数据块;a first splitting unit, configured to equally divide the original data into at least two original data blocks;
    第二拆分单元,用于将所述原始数据拆分为由a个原始数据块组成的第一数据段和由b个原始数据块组成的第二数据段,所述a和所述b均为正整数。a second splitting unit, configured to split the original data into a first data segment consisting of a original data blocks and a second data segment consisting of b original data blocks, wherein the a and the b are both is a positive integer.
  13. 根据权利要求12所述的装置,其特征在于,所述纠删码运算模块包括:The apparatus according to claim 12, wherein the erasure correction code operation module comprises:
    第一运算单元,用于对所述第一数据段进行纠删码运算,获得m个第一校验数据块,基于a个原始数据块和m个第一校验数据块生成a+m个第一冗余数据块,所述m为不大于a的正整数;a first operation unit, configured to perform an erasure code operation on the first data segment, obtain m first check data blocks, and generate a+m pieces based on a original data blocks and m first check data blocks the first redundant data block, the m is a positive integer not greater than a;
    第二运算单元,用于对所述第二数据段进行纠删码运算,获得n个第二校验数据块,基于a 个原始数据块和m个第一校验数据块生成b+n个第二冗余数据块,所述n为不大于b的正整数;The second operation unit is configured to perform an erasure code operation on the second data segment, obtain n second check data blocks, and generate b+n pieces of data based on a original data blocks and m first check data blocks the second redundant data block, the n is a positive integer not greater than b;
    第一数据生成单元,用于当a+m与b+n相等时,基于所述第一冗余数据块生成第一冗余数据,并基于所述第二冗余数据块生成第二冗余数据;a first data generating unit, configured to generate first redundant data based on the first redundant data block and generate second redundant data based on the second redundant data block when a+m is equal to b+n data;
    第二数据生成单元,用于当a+m与b+n不相等时,在所述第一冗余数据块或所述第二冗余数据块中添加补充数据块,使得在添加后所述第一冗余数据块和所述第二冗余数据块的数量相同,基于所述第一冗余数据块生成第一冗余数据,并基于所述第二冗余数据块生成第二冗余数据。A second data generating unit, configured to add a supplementary data block to the first redundant data block or the second redundant data block when a+m is not equal to b+n, so that after adding, the The number of the first redundant data block and the second redundant data block is the same, the first redundant data is generated based on the first redundant data block, and the second redundant data is generated based on the second redundant data block data.
  14. 根据权利要求13所述的装置,其特征在于,所述第二数据生成单元,具体用于:The device according to claim 13, wherein the second data generating unit is specifically configured to:
    当a+m小于b+n时,将至少一个预设内容的补充数据块添加至所述第一冗余数据块中,使得所述第一冗余数据块的数量增加至b+n个;When a+m is less than b+n, adding at least one supplementary data block of preset content to the first redundant data block, so that the number of the first redundant data blocks is increased to b+n;
    当a+m大于b+n时,将至少一个预设内容的补充数据块添加至所述第二冗余数据块中,使得所述第二冗余数据块的数量增加至a+m个。When a+m is greater than b+n, at least one supplementary data block of preset content is added to the second redundant data block, so that the number of the second redundant data blocks is increased to a+m.
  15. 根据权利要求14所述的装置,其特征在于,所述逻辑运算模块,具体用于:The device according to claim 14, wherein the logic operation module is specifically used for:
    将所述第一冗余数据块与所述第二冗余数据块逐一进行异或运算,生成第三冗余数据块,获得由所述第三冗余数据块组成的第三冗余数据。Perform an exclusive OR operation on the first redundant data block and the second redundant data block one by one to generate a third redundant data block, and obtain third redundant data composed of the third redundant data block.
  16. 根据权利要求15所述的装置,其特征在于,所述逻辑运算模块包括:The device according to claim 15, wherein the logic operation module comprises:
    编号单元,用于按照预设顺序分别给所述第一冗余数据块和所述第二冗余数据块进行编号;a numbering unit, configured to respectively number the first redundant data block and the second redundant data block according to a preset sequence;
    异或运算单元,用于对编号相同的所述第一冗余数据块和所述第二冗余数据块逐一进行异或运算,生成对应编号的第三冗余数据块;an XOR operation unit, configured to perform an XOR operation on the first redundant data block and the second redundant data block with the same number one by one, to generate a third redundant data block corresponding to the number;
    第三数据生成单元,用于将所述第三冗余数据块按照所述预设顺序组成第三冗余数据。A third data generating unit, configured to form third redundant data from the third redundant data blocks according to the preset sequence.
  17. 根据权利要求11或16所述的装置,其特征在于,所述装置还包括:The device according to claim 11 or 16, wherein the device further comprises:
    第一数据修复模块,用于当目标可用区中的损坏数据块的损坏数量小于或等于损坏阈值时,基于所述目标可用区中的冗余数据块对所述损坏数据块进行数据修复处理以获取修复数据块,采用所述修复数据块与所述损坏数据块进行替换;a first data repair module, configured to perform data repair processing on the damaged data block based on the redundant data block in the target availability zone when the damage quantity of the damaged data block in the target availability zone is less than or equal to the damage threshold, so as to obtaining a repaired data block, and using the repaired data block to replace the damaged data block;
    第二数据修复模块,用于当目标可用区中的损坏数据块的损坏数量大于损坏阈值时,基于第一剩余可用区和第二剩余可用区中的冗余数据块对所述损坏数据块进行数据修复处理以获取修复数据块,采用所述修复数据块与所述损坏数据块进行替换;The second data repair module is configured to, when the damaged quantity of the damaged data blocks in the target availability zone is greater than the damage threshold, perform a repair on the damaged data blocks based on the redundant data blocks in the first remaining availability zone and the second remaining availability zone Data repair processing to obtain a repaired data block, and use the repaired data block to replace the damaged data block;
    所述目标可用区为所述可用区集合中的任一可用区,所述第一剩余可用区和第二剩余可用区为所述可用区集合中除所述目标可用区外的其它两个可用区,所述剩余数据块为所述目标可用区中除所述损坏数据块外的其他冗余数据块,所述冗余数据块为所述第一冗余数据块、所述第二冗余数据块或所述第三冗余数据块。The target availability zone is any availability zone in the availability zone set, and the first remaining availability zone and the second remaining availability zone are the other two available zones in the availability zone set except the target availability zone area, the remaining data blocks are other redundant data blocks in the target availability zone except the damaged data block, and the redundant data blocks are the first redundant data block, the second redundant data block data block or the third redundant data block.
  18. 根据权利要求17所述的装置,其特征在于,所述第一数据修复模块包括:The apparatus according to claim 17, wherein the first data repair module comprises:
    第一数量获取单元,用于获取目标可用区中校验数据块的校验数量和原始数据块的原始数量;a first quantity obtaining unit, used for obtaining the check quantity of the check data block and the original quantity of the original data block in the target availability zone;
    第一数据块获取单元,用于当所述目标可用区中的所述损坏数据块的损坏数量小于或等于所述校验数量时,在所述目标可用区获取所述原始数量指示的剩余数据块;a first data block obtaining unit, configured to obtain the remaining data indicated by the original quantity in the target availability zone when the damaged quantity of the damaged data blocks in the target availability zone is less than or equal to the check quantity piece;
    第一替换单元,用于基于所述剩余数据块对所述损坏数据块进行数据修复处理,以获取修复数据块,采用所述修复数据块对所述损坏数据块进行替换。A first replacement unit, configured to perform data repair processing on the damaged data block based on the remaining data block to obtain a repaired data block, and use the repaired data block to replace the damaged data block.
  19. 一种计算机存储介质,其特征在于,所述计算机存储介质存储有多条指令,所述指令适于由处理器加载并执行如权利要求1~10任意一项的方法步骤。A computer storage medium, characterized in that the computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method steps of any one of claims 1-10.
  20. 一种电子设备,其特征在于,包括:处理器和存储器;其中,所述存储器存储有计算机程序,所述计算机程序适于由所述处理器加载并执行如权利要求1~10任意一项的方法步骤。An electronic device, characterized by comprising: a processor and a memory; wherein, the memory stores a computer program, and the computer program is adapted to be loaded by the processor and execute the method according to any one of claims 1 to 10 method steps.
PCT/CN2021/082263 2021-03-23 2021-03-23 Data redundancy backup method and apparatus, storage medium, and electronic device WO2022198429A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202180090137.9A CN116710899A (en) 2021-03-23 2021-03-23 Data redundancy backup method and device, storage medium and electronic equipment
PCT/CN2021/082263 WO2022198429A1 (en) 2021-03-23 2021-03-23 Data redundancy backup method and apparatus, storage medium, and electronic device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/082263 WO2022198429A1 (en) 2021-03-23 2021-03-23 Data redundancy backup method and apparatus, storage medium, and electronic device

Publications (1)

Publication Number Publication Date
WO2022198429A1 true WO2022198429A1 (en) 2022-09-29

Family

ID=83396218

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/082263 WO2022198429A1 (en) 2021-03-23 2021-03-23 Data redundancy backup method and apparatus, storage medium, and electronic device

Country Status (2)

Country Link
CN (1) CN116710899A (en)
WO (1) WO2022198429A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117240873A (en) * 2023-11-08 2023-12-15 阿里云计算有限公司 Cloud storage system, data reading and writing method, device and storage medium

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB1318250A (en) * 1969-10-29 1973-05-23 Honeywell Inc Encoding detecting and or correcting of data
US20030033572A1 (en) * 2001-08-09 2003-02-13 Walton John K. Memory system and method of using same
CN103559102A (en) * 2013-10-22 2014-02-05 北京航空航天大学 Data redundancy processing method and device and distributed storage system
CN104932953A (en) * 2015-06-04 2015-09-23 华为技术有限公司 Data distribution method, data storage method, and relevant device and system
CN108564972A (en) * 2018-03-14 2018-09-21 广东紫晶信息存储技术股份有限公司 A kind of redundancy for data of optical disk storage restores code error correction method
CN109582646A (en) * 2018-09-14 2019-04-05 唐学军 A kind of redundancy and restoration methods and system for file declustering
CN110489269A (en) * 2019-08-16 2019-11-22 西安紫光国芯半导体有限公司 Coding-decoding method, coding decoder and the processor of three bit-errors are corrected in detection
CN111444044A (en) * 2020-03-27 2020-07-24 苏州链原信息科技有限公司 Method for data redundancy backup check, electronic device and computer storage medium
CN112416660A (en) * 2021-01-21 2021-02-26 南京群顶科技有限公司 Erasure technology-based large file distributed backup method and device

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB1318250A (en) * 1969-10-29 1973-05-23 Honeywell Inc Encoding detecting and or correcting of data
US20030033572A1 (en) * 2001-08-09 2003-02-13 Walton John K. Memory system and method of using same
CN103559102A (en) * 2013-10-22 2014-02-05 北京航空航天大学 Data redundancy processing method and device and distributed storage system
CN104932953A (en) * 2015-06-04 2015-09-23 华为技术有限公司 Data distribution method, data storage method, and relevant device and system
CN108564972A (en) * 2018-03-14 2018-09-21 广东紫晶信息存储技术股份有限公司 A kind of redundancy for data of optical disk storage restores code error correction method
CN109582646A (en) * 2018-09-14 2019-04-05 唐学军 A kind of redundancy and restoration methods and system for file declustering
CN110489269A (en) * 2019-08-16 2019-11-22 西安紫光国芯半导体有限公司 Coding-decoding method, coding decoder and the processor of three bit-errors are corrected in detection
CN111444044A (en) * 2020-03-27 2020-07-24 苏州链原信息科技有限公司 Method for data redundancy backup check, electronic device and computer storage medium
CN112416660A (en) * 2021-01-21 2021-02-26 南京群顶科技有限公司 Erasure technology-based large file distributed backup method and device

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117240873A (en) * 2023-11-08 2023-12-15 阿里云计算有限公司 Cloud storage system, data reading and writing method, device and storage medium
CN117240873B (en) * 2023-11-08 2024-03-29 阿里云计算有限公司 Cloud storage system, data reading and writing method, device and storage medium

Also Published As

Publication number Publication date
CN116710899A (en) 2023-09-05

Similar Documents

Publication Publication Date Title
US10936417B2 (en) Multi-stage slice recovery in a dispersed storage network
US9817597B2 (en) Using temporary write locations for increased power efficiency
US10241864B2 (en) Expanding information dispersal algorithm width without rebuilding through imposter slices
US10013191B2 (en) Encoding data for storage in a dispersed storage network
US10073645B2 (en) Initiating rebuild actions from DS processing unit errors
US9760286B2 (en) Adaptive dispersed storage network (DSN) and system
CN111782152A (en) Data storage method, data recovery device, server and storage medium
US11741125B2 (en) Storage network for rebuilding failed slices
CN114385409A (en) Encoding method based on erasure code, distributed system, device and storage medium
US20220253357A1 (en) Batch Rebuilding a Set of Encoded Data Slices
WO2022198429A1 (en) Data redundancy backup method and apparatus, storage medium, and electronic device
US10331519B2 (en) Application of secret sharing schemes at multiple levels of a dispersed storage network
US10506045B2 (en) Memory access using deterministic function and secure seed
US10534668B2 (en) Accessing data in a dispersed storage network
US10067998B2 (en) Distributed sync list
US10528425B2 (en) Transitioning to an optimized data storage approach in a dispersed storage network
US10838814B2 (en) Allocating rebuilding queue entries in a dispersed storage network
US10372380B2 (en) Asserting integrity with a verifiable codec
US20180076953A1 (en) Systematic secret sharing
US11113008B1 (en) Data restoration using partially encoded slice requests
US10972541B1 (en) Priority encoded data slice retention
US10922179B2 (en) Post rebuild verification
US20180081546A1 (en) Modifying information dispersal algorithm (ida) thresholds by splitting existing slices
US20190266148A1 (en) Slice migration in a dispersed storage network

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21932061

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202180090137.9

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21932061

Country of ref document: EP

Kind code of ref document: A1