CN114879904B - Data storage erasure method, device, equipment and readable storage medium - Google Patents
Data storage erasure method, device, equipment and readable storage medium Download PDFInfo
- Publication number
- CN114879904B CN114879904B CN202210428509.0A CN202210428509A CN114879904B CN 114879904 B CN114879904 B CN 114879904B CN 202210428509 A CN202210428509 A CN 202210428509A CN 114879904 B CN114879904 B CN 114879904B
- Authority
- CN
- China
- Prior art keywords
- row
- erasure
- matrix
- data block
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 44
- 238000000034 method Methods 0.000 title claims abstract description 40
- 239000011159 matrix material Substances 0.000 claims abstract description 373
- 238000012937 correction Methods 0.000 claims description 35
- 238000004364 calculation method Methods 0.000 claims description 21
- 238000010276 construction Methods 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 9
- 230000000694 effects Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 6
- 238000011084 recovery Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000002441 reversible effect Effects 0.000 description 2
- JXASPPWQHFOWPL-UHFFFAOYSA-N Tamarixin Natural products C1=C(O)C(OC)=CC=C1C1=C(OC2C(C(O)C(O)C(CO)O2)O)C(=O)C2=C(O)C=C(O)C=C2O1 JXASPPWQHFOWPL-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Signal Processing For Digital Recording And Reproducing (AREA)
Abstract
The invention discloses a data storage erasure method, which comprises the following steps: dividing a data set in each disk into two data blocks; selecting each first data block from partial disks, selecting each second data block from the rest disks, and constructing a first erasure vandermonde matrix; dividing the preset line number by three to obtain a second numerical value; a third numerical value is obtained by utilizing a second numerical value obtained by multiplying the preset line number by two; constructing a first third numerical row except a unit matrix in the second erasure vandermonde matrix by utilizing each residual data block; selecting partial data blocks from the selected data blocks of each round, and constructing a second numeric line by utilizing the rest data blocks and the selected data blocks of each round; constructing a residual row by utilizing each residual data block and each residual data block in each round of selected data blocks; and performing data erasure by using the first erasure vandermonde matrix and/or the second erasure vandermonde matrix. The invention improves the data erasure efficiency. The invention also discloses a device, equipment and a storage medium, which have corresponding technical effects.
Description
Technical Field
The present invention relates to the field of storage technologies, and in particular, to a data storage erasure method, apparatus, device, and computer readable storage medium.
Background
In the face of the storage requirement of mass data, the distributed storage gradually replaces the dominant position of unified storage with the advantages of low cost, good expandability and the like. The distributed storage system mostly uses a disk with relatively low cost as a storage node, the reliability of the storage node is not high, and one distributed storage system usually comprises a plurality of nodes, and the system often has the condition of node failure due to software and hardware faults, human errors and the like. In order to improve the data reliability of the distributed storage system, the data collection node is ensured to reconstruct an original file with higher probability, and a certain amount of redundancy is required to be additionally stored on the basis of storing the original data, so that the system can still normally operate under the condition that part of nodes fail, and the data collection node can still realize decoding recovery on the original file.
An Erasure Code (Erasure Code) belongs to a forward error correction technology in the coding theory, and is introduced into the storage field because the Erasure Code technology has a better effect in preventing data loss. Erasure codes are a variety of, and RS codes (Reed-Solomon codes) used in a distributed environment are more commonly used in practical storage systems. The RS code is associated with two parameters k and r. Given two positive integers k and r, an RS code encodes k data blocks into r additional parity blocks. The manner in which r check blocks are encoded based on the vandermonde matrix or the cauchy matrix is called RS erasure code encoded by the vandermonde matrix. The core concept of erasure codes is to construct a reversible coding matrix to generate the check data, and the inverse matrix can be calculated to recover the original data.
The common calculation of the RS erasure code inverse matrix adopts a Gaussian elimination method, and the common solution is suitable for the inversion of any reversible matrix, and a large amount of redundant operation is introduced although the calculation is regularized. In the data erasure process, the uncorrupted data in the disk needs to be integrally called, the data volume to be called is large, the data call time is long, and the data erasure efficiency is low.
In summary, how to effectively solve the problems of large data volume, long data call time, low data erasure efficiency and the like of the existing data storage erasure method is an urgent need of those skilled in the art.
Disclosure of Invention
The invention aims to provide a data storage erasure method, which greatly reduces the data calling amount during data erasure, reduces the time consumption of data calling and improves the data erasure efficiency; it is another object of the present invention to provide a data storage erasure apparatus, device and computer readable storage medium.
In order to solve the technical problems, the invention provides the following technical scheme:
a data storage erasure method, comprising:
dividing a data set in each disk into two data blocks to obtain first data blocks and second data blocks;
Selecting each first data block from a first number of disks, selecting each second data block from the rest of the disks to obtain each round of selected data blocks, and constructing a first erasure vandermonde matrix according to the total number of the disks and each round of selected data blocks;
obtaining a preset matrix line number, dividing the preset matrix line number by three, and rounding downwards to obtain a second numerical value, and calculating a difference value by using the preset matrix line number and the second numerical value which is doubled to obtain a third numerical value;
Constructing a first line to a third numerical line of an original vandermonde matrix in a second erasure vandermonde matrix by using each remaining first data block and each remaining second data block except each round of selected data blocks in each magnetic disk;
Performing two-round selection on each one-round selection data block to obtain each two-round selection data block, and constructing the next row to the fourth value row of the third value row of the original vandermonde matrix in the second erasure correction vandermonde matrix by using each remaining first data block, each remaining second data block and each two-round selection data block; wherein the fourth value is the sum of the third value and the second value;
Constructing the next row of the fourth numerical row of the original vandermonde matrix in the second erasure-correction vandermonde matrix to the preset matrix row of rows by utilizing each remaining first data block, each remaining second data block and each data block except each two-round selection data block in each round selection data block;
When the data block is lost, performing data erasure by using the first erasure vandermonde matrix and/or the second erasure vandermonde matrix.
In one embodiment of the present invention, constructing the first row to the third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix by using each remaining first data block and each remaining second data block in each disk except each round of the selected data blocks includes:
and constructing a first line to a third numerical line of an original vandermonde matrix in the second erasure vandermonde matrix by taking each remaining first data block and each remaining second data block as the base of each line parameter.
In one embodiment of the present invention, selecting each of the first data blocks from the first numerical disk includes:
dividing the total number of the magnetic disks by downward rounding to obtain the first numerical value;
selecting each first data block from the first numerical disk;
Performing two-round selection on each of the one-round selection data blocks to obtain each of the two-round selection data blocks, wherein the two-round selection data blocks comprise:
dividing the first numerical value by two, and rounding downwards to obtain a fifth numerical value;
and selecting the fifth numerical value of the first data blocks from each round of selected data blocks, and selecting the difference value between the first numerical value and the fifth numerical value of the second data blocks to obtain each round of selected data blocks.
In one embodiment of the present invention, constructing a next row to a fourth value row of the third value row of the original vandermonde matrix in the second erasure-correcting vandermonde matrix using each of the remaining first data blocks, each of the remaining second data blocks, and each of the two-round selection data blocks includes:
Constructing a next row of the third numerical row to a fourth numerical row of the original vandermonde matrix by taking each remaining first data block and each remaining second data block as the base of each row parameter, and obtaining a first factor from the next row of the third numerical row to the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix;
constructing a first row to a second numerical row of an original vandermonde matrix by taking each two-wheel selected data block as a base of each row parameter, and obtaining a second factor from a next row to a fourth numerical row of a third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix;
performing exclusive OR calculation on the first factor and the second factor in each row to obtain a next row of the third value row to the fourth value row of the original vandermonde matrix in the second erasure vandermonde matrix; wherein the difference between the index of the parameter contained in the first factor and the index of the parameter contained in the second factor in each row is the third value.
In one embodiment of the present invention, constructing a row from the next row of the fourth numerical row of the original vandermonde matrix to the preset matrix row of the second erasure vandermonde matrix using each of the remaining first data blocks, each of the remaining second data blocks, and each of the data blocks other than each of the two-round selected data blocks in each of the one-round selected data blocks includes:
Constructing the next row of the fourth numerical value row of the original vandermonde matrix to the preset matrix row by taking each remaining first data block and each remaining second data block as the base of each row parameter, and obtaining a third factor of the next row of the fourth numerical value row of the original vandermonde matrix to the preset matrix row of the second erasure vandermonde matrix;
Constructing a first row to a second numerical row of an original vandermonde matrix by taking each data block except each two-wheel selected data block in each one-wheel selected data block as a base of each row parameter, and obtaining a fourth factor from a next row to a fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix;
Performing exclusive OR calculation on the third factor and the fourth factor in each row to obtain the rows from the next row of the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix to the preset matrix row; wherein the difference between the index of the parameter contained in the third factor and the index of the parameter contained in the fourth factor in each row is the fourth value.
In a specific embodiment of the present invention, when there is a loss of a data block, performing a data erasure operation using the first erasure vandermonde matrix and/or the second erasure vandermonde matrix, including:
And when the number of the disks with the data block loss does not exceed the second numerical value, performing data erasure operation by using the second erasure vandermonde matrix.
In a specific embodiment of the present invention, when the number of disks with data block loss does not exceed the second value, performing data erasure operation by using the second erasure vandermonde matrix, including:
when the lost data block is the first data block and the second data block in the same disk, performing data erasure by using the first row of the original vandermonde matrix and the next row of the third numerical row in the second erasure vandermonde matrix, or performing data erasure by using the first row of the original vandermonde matrix and the next row of the fifth numerical row in the second erasure vandermonde matrix.
A data storage erasure apparatus comprising:
the data block dividing module is used for dividing the data set in each disk into two data blocks to obtain each first data block and each second data block;
The matrix construction module is used for selecting each first data block from the first number of magnetic disks, selecting each second data block from the rest of magnetic disks to obtain each round of selected data blocks, and constructing a first erasure vandermonde matrix according to the total number of magnetic disks and each round of selected data blocks;
The numerical value calculation module is used for obtaining preset matrix line numbers, dividing the preset matrix line numbers by three to obtain a second numerical value by rounding downwards, and calculating a difference value by using the preset matrix line numbers and the second numerical value which is doubled to obtain a third numerical value;
A first matrix row construction module, configured to construct a first row to a third numerical row of an original vandermonde matrix in a second erasure-correction vandermonde matrix by using each remaining first data block and each remaining second data block in each disk except each round of selected data blocks;
A second matrix row construction module, configured to perform two-round selection on each of the one-round selection data blocks to obtain each of the two-round selection data blocks, and construct a next row to a fourth value row of the third value row of the original vandermonde matrix in the second erasure vandermonde matrix by using each of the remaining first data blocks, each of the remaining second data blocks, and each of the two-round selection data blocks; wherein the fourth value is the sum of the third value and the second value;
a third matrix row construction module, configured to construct a row from a next row of the fourth numerical row of the original vandermonde matrix in the second erasure-correction vandermonde matrix to the preset matrix row by using each remaining first data block, each remaining second data block, and each data block except each two-round selected data block in each one-round selected data block;
and the data erasure correction module is used for carrying out data erasure correction operation by utilizing the first erasure correction vandermonde matrix and/or the second erasure correction vandermonde matrix when the data block is lost.
A data storage erasure apparatus comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data storage erasure method as described above when executing the computer program.
A computer readable storage medium having stored thereon a computer program which when executed by a processor implements the steps of a data storage erasure method as described above.
The data storage erasure method provided by the invention divides a data set in each disk into two data blocks to obtain each first data block and each second data block; selecting each first data block from the first number of disks, selecting each second data block from the rest of the disks to obtain each round of selected data blocks, and constructing a first erasure vandermonde matrix according to the total number of the disks and each round of selected data blocks; obtaining preset matrix line numbers, dividing the preset matrix line numbers by three-way downward rounding to obtain second numerical values, and performing difference calculation by using the preset matrix line numbers and the second numerical values which are doubled to obtain third numerical values; constructing a first numerical line to a third numerical line of an original vandermonde matrix in a second erasure vandermonde matrix by utilizing each remaining first data block and each remaining second data block except each round of selected data blocks in each magnetic disk; performing two-round selection on each round of selected data blocks to obtain each round of selected data blocks, and constructing the next row to the fourth row of the third value row of the original vandermonde matrix in the second erasure correction vandermonde matrix by utilizing each remaining first data block, each remaining second data block and each round of selected data blocks; wherein the fourth value is the sum of the third value and the second value; constructing the next row of the fourth numerical row of the original vandermonde matrix in the second erasure correction vandermonde matrix to the first preset matrix row of rows by utilizing each remaining first data block, each remaining second data block and each data block except each two rounds of selected data blocks in each round of selected data blocks; when the data block is lost, performing data erasure by using the first erasure vandermonde matrix and/or the second erasure vandermonde matrix.
According to the technical scheme, the data set in each disk is divided into two data blocks, and the erasure vandermonde matrix is constructed through the effective selection operation of the data blocks, so that when the erasure vandermonde matrix obtained through construction is utilized for data erasure, only part of the data blocks in the disk are required to be called according to the current data loss condition, the data calling quantity during data erasure is greatly reduced, the data calling time is shortened, and the data erasure efficiency is improved.
Correspondingly, the invention further provides a data storage erasure correction device, equipment and a computer readable storage medium corresponding to the data storage erasure correction method, which have the technical effects and are not repeated herein.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of one implementation of a data storage erasure method in an embodiment of the present invention;
FIG. 2 is a flow chart of another implementation of a data storage erasure method in accordance with an embodiment of the present invention;
FIG. 3 is a block diagram illustrating a data storage erasure apparatus in accordance with an embodiment of the present invention;
FIG. 4 is a block diagram illustrating a data storage erasure apparatus in accordance with an embodiment of the present invention;
fig. 5 is a schematic diagram of a specific structure of a data storage erasure apparatus according to this embodiment.
Detailed Description
In order to better understand the aspects of the present invention, the present invention will be described in further detail with reference to the accompanying drawings and detailed description. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, fig. 1 is a flowchart illustrating an implementation of a data storage erasure method according to an embodiment of the present invention, the method may include the steps of:
s101: dividing the data set in each disk into two data blocks to obtain each first data block and each second data block.
The distributed storage system comprises a plurality of disk storage nodes, and the data set in each disk is divided into two data blocks, such as a data block and b data block, so as to obtain each first data block and each second data block, wherein k data blocks are assumed to be total, each first data block is respectively represented by b1, b2, … and bk, and each second data block is respectively represented by a1, a2, … and ak.
S102: and selecting each first data block from the first number of disks, selecting each second data block from the rest of the disks to obtain each round of selected data blocks, and constructing a first erasure vandermonde matrix according to the total number of the disks and each round of selected data blocks.
After dividing a data set in each disk into two data blocks to obtain first data blocks and second data blocks, selecting the first data blocks from the first numerical value disks, selecting the second data blocks from the rest disks to obtain one-round selected data blocks, and constructing a first erasure vandermonde matrix according to the total number of the disks and the one-round selected data blocks. Assuming that the total number of disks is 10, the first value is 4, and any four selected first data blocks may be b1, b2, …, b10, for example, b1, b2, b3, b5, then each second data block is selected from the remaining disks as a4, a6, a7, a8, a9, a10.
S103: obtaining the preset matrix line number, dividing the preset matrix line number by three-way downward rounding to obtain a second numerical value, and calculating a difference value by using the preset matrix line number and the second numerical value which is doubled to obtain a third numerical value.
And presetting the number of matrix rows, if the number of matrix rows is set to be r, dividing the preset number of matrix rows by three-way downward rounding to obtain a second numerical value, and calculating the difference value by using the preset number of matrix rows and the second numerical value which is doubled to obtain a third numerical value. Assuming that the number of rows of the matrix is 10, dividing 10 by three to round down to obtain 3, wherein 3 is the second value, 10-2*3 =4, and the third value is 4.
S104: and constructing a first numerical line to a third numerical line of the original vandermonde matrix in the second erasure vandermonde matrix by utilizing each remaining first data block and each remaining second data block except each round of selected data blocks in each magnetic disk.
After one round of selection is carried out on the first data block and the second data block in each disk, each remaining first data block and each remaining second data block except for each round of selection data block in each disk are utilized to construct a first numerical row to a third numerical row of an original vandermonde matrix in the second erasure vandermonde matrix.
Taking the examples in step S102 and step S103, when the selected first data blocks are b1, b2, b3, b5, and the selected second data blocks are a4, a6, a7, a8, a9, a10, each remaining first data block except for one round of selected data block in each disk is b4, b6, b7, b8, b9, b10, and each remaining second data block is a1, a2, a3, a5. The first row to the fourth row of the original vandermonde matrix in the second erasure vandermonde matrix are constructed by using a1, a2, a3, b4, a5, b6, b7, b8, b9 and b 10.
S105: performing two-round selection on each round of selection data blocks to obtain each round of selection data blocks, and constructing the next row to the fourth row of the third value row of the original vandermonde matrix in the second erasure correction vandermonde matrix by utilizing each remaining first data block, each remaining second data block and each round of selection data blocks; the fourth value is the sum of the third value and the second value.
After one-round selection is carried out on the first data block and the second data block in each magnetic disk, two-round selection is carried out on each one-round selection data block, each two-round selection data block is obtained, each remaining first data block, each remaining second data block and each two-round selection data block are utilized to construct the next row to the fourth numerical row of the third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix, and the fourth numerical is the sum of the third numerical and the second numerical.
After receiving the examples in steps S102 to S104, two rounds of selection are performed on each round of selection data blocks b1, b2, b3, a4, b5, a6, a7, a8, a9, a10, assuming that the two rounds of selection data blocks are b1, a4, b5, a6, respectively. The fifth to seventh rows are constructed using a1, a2, a3, b4, a5, b6, b7, b8, b9, b10 and b1, a4, b5, a6.
S106: and constructing the next row of the fourth numerical row of the original vandermonde matrix in the second erasure correction vandermonde matrix to the first preset matrix row of rows by utilizing each remaining first data block, each remaining second data block and each data block except each two rounds of selected data blocks in each round of selected data blocks.
After two rounds of selection are carried out on each round of selection data block, the next row of the fourth value row of the original vandermonde matrix in the second erasure correction vandermonde matrix is constructed by utilizing each remaining first data block, each remaining second data block and each data block except for each round of selection data block in each round of selection data block.
For example, in the following steps S102 to S105, when the two-round selection data blocks are b1, a4, b5, a6, each data block except the two-round selection data block is b2, b3, a7, a8, a9, a10. Eighth to tenth rows are constructed using a1, a2, a3, b4, a5, b6, b7, b8, b9, b10 and b2, b3, a7, a8, a9, a10.
S107: when the data block is lost, performing data erasure by using the first erasure vandermonde matrix and/or the second erasure vandermonde matrix.
When it is determined that there is a loss of the data block, performing data erasure using the first erasure vandermonde matrix and/or the second erasure vandermonde matrix. Namely, according to the loss quantity of the data blocks, adaptively selecting whether to use only the second erasure vandermonde matrix for data erasure or to use the first erasure vandermonde matrix and the second erasure vandermonde matrix for data erasure together.
According to the technical scheme, the data set in each disk is divided into two data blocks, and the erasure vandermonde matrix is constructed through the effective selection operation of the data blocks, so that when the erasure vandermonde matrix obtained through construction is utilized for data erasure, only part of the data blocks in the disk are required to be called according to the current data loss condition, the data calling quantity during data erasure is greatly reduced, the data calling time is shortened, and the data erasure efficiency is improved.
It should be noted that, based on the above embodiments, the embodiments of the present invention further provide corresponding improvements. The following embodiments relate to the same steps as those in the above embodiments or the steps corresponding to the steps may be referred to each other, and the corresponding beneficial effects may also be referred to each other, which will not be described in detail in the following modified embodiments.
Referring to fig. 2, fig. 2 is a flowchart illustrating another implementation of a data storage erasure method according to an embodiment of the present invention, the method may include the steps of:
s201: dividing the data set in each disk into two data blocks to obtain each first data block and each second data block.
S202: dividing the total number of disks by the downward rounding to obtain a first value.
And obtaining the total number k of the magnetic disks, dividing the total number k of the magnetic disks by downward rounding to obtain a first numerical value. When k is even, the first value is k/2, and when k is even, the first value is (k-1)/2.
Taking the example of the above embodiment, assume that the total number of disks is 10, dividing 10 by rounding down to 5, and the first value is 5.
S203: and selecting each first data block from the first number of disks, selecting each second data block from the rest of the disks to obtain each round of selected data blocks, and constructing a first erasure vandermonde matrix according to the total number of the disks and each round of selected data blocks.
Taking the example of step S202, when k is even, assuming that each selected data block is b2, b4, b6, b8, …, bk, each second data block is selected from the remaining disks, each two data blocks is a1, a3, a5, a7, …, ak-1, and each round of selected data blocks is obtained, namely a1, b2, a3, b4, a5, b6, a7, b8, …, ak-1, bk, and the first erasure matrix is constructed as follows:
wherein I represents a unit array of k.
When k is even, each round of selecting data blocks, namely a1, b2, a3, b4, a5, b6, a7, b8, …, bk-1, ak, and constructing a first erasure vandermonde matrix:
thereby obtaining a first erasure vandermonde matrix a 1 when the total number of disks is even and a first erasure vandermonde matrix a 2 when the total number of disks is odd, respectively.
By setting the first value to be the total number of disks divided by the downward rounding, it is ensured that the number of the first data blocks and the second data blocks used for constructing the first erasure vandermonde matrix is almost equal, i.e., when the total number of disks is even, the number of the first data blocks and the second data blocks constructing the first erasure vandermonde matrix is equal, and when the total number of disks is odd, the number of the first data blocks and the second data blocks constructing the first erasure vandermonde matrix differ by one.
S204: obtaining the preset matrix line number, dividing the preset matrix line number by three-way downward rounding to obtain a second numerical value, and calculating a difference value by using the preset matrix line number and the second numerical value which is doubled to obtain a third numerical value.
Obtaining the preset matrix line number, dividing the preset matrix line number by three-way downward rounding to obtain a second numerical value, setting the second numerical value as t,Calculating a difference value by using the preset matrix line number and the second double value to obtain a third value, wherein the third value is s,/>
S205: and constructing first to third numerical lines of the original vandermonde matrix in the second erasure vandermonde matrix by taking each remaining first data block and each remaining second data block as the base of each line parameter.
And constructing first to third numerical lines of the original vandermonde matrix in the second erasure vandermonde matrix by taking each remaining first data block and each remaining second data block as the base of each line parameter. When k is even, the first row to the third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix are:
When k is an odd number, the first row to the third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix are:
And obtaining the first row to the third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix when the total number of the magnetic disks is even or odd respectively.
S206: dividing the first value by two and rounding down to obtain a fifth value.
After dividing the total number of disks by two, rounding down to obtain a first value, dividing the first value by two, rounding down to obtain a fifth value.
S207: and selecting a fifth numerical value first data block from each round of selected data blocks, and selecting a difference value second data block between the first numerical value and the fifth numerical value to obtain each round of selected data blocks.
And selecting a fifth numerical value first data block from each round of selected data blocks, and selecting a difference value second data block between the first numerical value and the fifth numerical value to obtain each round of selected data blocks.
When k is even, each two rounds of selecting data blocks are a 1,b2,...,bk/2-1,ak/2 respectively; when k is odd, each two rounds of selecting data blocks are a 1,b2,...,a(k-1)/2-1,b(k-1)/2 respectively.
S208: constructing a next row to a fourth value row of a third value row of an original vandermonde matrix by taking each remaining first data block and each remaining second data block as the base of each row parameter, and obtaining first factors of the next row to the fourth value row of the third value row of the original vandermonde matrix in the second erasure vandermonde matrix; the fourth value is the sum of the third value and the second value.
When k is even, each remaining first data block and each remaining second data block are b1, a2, b3, a4, b5, a6, b7, a8, …, bk-1, ak, and a first factor from a next row to a fourth value row of a third value row of an original vandermonde matrix in the second erasure vandermonde matrix is:
When k is odd, each remaining first data block and each remaining second data block are b1, a2, b3, a4, b5, a6, b7, a8, …, ak-1, bk, and a first factor from a next row to a fourth value row of a third value row of an original vandermonde matrix in the second erasure vandermonde matrix is:
and when the total number of the magnetic disks is even or odd, respectively, obtaining a first factor from the next row of the third numerical row to the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix.
S209: and constructing a first row to a second numerical row of the original vandermonde matrix by taking each two rounds of selected data blocks as the base of each row parameter, and obtaining second factors from the next row to the fourth numerical row of the third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix.
When k is even, the second factors from the next row to the fourth row of the third value row of the original vandermonde matrix in the second erasure vandermonde matrix are:
When k is odd, the second factors from the next row to the fourth row of the third value row of the original vandermonde matrix in the second erasure vandermonde matrix are:
And when the total number of the magnetic disks is even or odd, obtaining a second factor from the next row of the third numerical row to the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix.
S210: performing exclusive OR calculation on the first factor and the second factor in each row to obtain the next row to the fourth numerical row of the third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix; wherein the difference between the index of the parameter contained in the first factor and the index of the parameter contained in the second factor in each row is a third value.
When k is even, performing exclusive OR calculation on the first factor and the second factor in each row to obtain a next row to a fourth row of the third value row of the original vandermonde matrix in the second erasure vandermonde matrix:
When k is an odd number, performing exclusive OR calculation on the first factor and the second factor in each row to obtain a next row to a fourth numerical row of the third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix:
And when the total number of the magnetic disks is even or odd, the next row to the fourth row of the third value row of the original vandermonde matrix in the second erasure vandermonde matrix is obtained respectively.
S211: and constructing the next row of the fourth numerical row of the original vandermonde matrix to the row of the first preset matrix by taking each remaining first data block and each remaining second data block as the base of each row parameter, and obtaining a third factor from the next row of the fourth numerical row of the original vandermonde matrix to the row of the first preset matrix in the second erasure vandermonde matrix.
When k is even, constructing the next row of the fourth numerical row of the original vandermonde matrix to the row of the first preset matrix by taking each remaining first data block and each remaining second data block as the base of each row parameter, and obtaining a third factor from the next row of the fourth numerical row of the original vandermonde matrix to the row of the first preset matrix in the second erasure vandermonde matrix:
When k is an odd number, constructing a next row of a fourth numerical row of the original vandermonde matrix to a first preset matrix row by taking each remaining first data block and each remaining second data block as the base of each row parameter, and obtaining a third factor from the next row of the fourth numerical row of the original vandermonde matrix to the first preset matrix row in the second erasure vandermonde matrix:
And when the total number of the magnetic disks is even or odd, obtaining a third factor from the next row of the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix to the row of the first preset matrix row.
S212: and constructing a first row to a second numerical row of the original vandermonde matrix by taking each data block except each two-round selected data block in each round selected data block as a base of each row parameter, and obtaining a fourth factor from the next row to the preset matrix row of the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix.
When k is even, each data block except for each two-round selected data block in each round selected data block is used as a base of each line parameter to construct a first line to a second numerical line of the original vandermonde matrix, and a fourth factor from the next line of the fourth numerical line of the original vandermonde matrix in the second erasure vandermonde matrix to the line of the preset matrix is obtained:
/>
When k is an odd number, each data block except for each two-round selected data block in each round selected data block is used as a base number of each line parameter to construct a first line to a second numerical line of the original vandermonde matrix, and a fourth factor from the next line of the fourth numerical line of the original vandermonde matrix in the second erasure vandermonde matrix to the line of the preset matrix is obtained:
And when the total number of the magnetic disks is even or odd, obtaining a fourth factor from the next row of the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix to the row of the first preset matrix row.
S213: performing exclusive OR calculation on the third factor and the fourth factor in each row to obtain the rows from the next row of the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix to the row of the first preset matrix; wherein the difference between the index of the parameter included in the third factor and the index of the parameter included in the fourth factor in each row is a fourth value.
When k is even, performing exclusive OR calculation on the third factor and the fourth factor in each row to obtain the rows from the next row to the row of the preset matrix row of the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix:
When k is an odd number, performing exclusive OR calculation on the third factor and the fourth factor in each row to obtain the rows from the next row to the row of the preset matrix row of the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix:
And when the total number of the magnetic disks is even or odd, respectively, the next row of the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix is obtained to the row of the first preset matrix.
When k is even, the second erasure vandermonde matrix is:
when k is odd, the second erasure vandermonde matrix is:
Thereby obtaining a second erasure vandermonde matrix when the total number of disks is even or odd, respectively.
The fifth numerical value is obtained by dividing the first numerical value by two and rounding downwards, a first data block of the fifth numerical value is selected from each round of selected data blocks, and a second data block of the difference value between the first numerical value and the fifth numerical value is selected, so that the number of the first data blocks and the number of the second data blocks from the next row of the third numerical value row to the fourth numerical value row for constructing the second erasure vandermonde matrix are ensured to be more similar to the number of the first data blocks and the number of the second data blocks from the next row of the fourth numerical value row to the first preset matrix row for constructing the second erasure vandermonde matrix.
S214: and when the number of the disks with the data block loss does not exceed a second numerical value, performing data erasure operation by using the second erasure vandermonde matrix.
And when the number of the disks with the data block loss does not exceed a second numerical value, performing data erasure operation by using the second erasure vandermonde matrix.
When the number of damaged magnetic disks is not more thanWhen this is the case, call to row k+1 in B1 or B2 to/>Data of a row so that this/>, can be recoveredThe portion of each disk that is located in B1 or B2. And substituting the repaired B part data into the k+s+t+1 to the r row, so that a vandermonde matrix related to the A part data is obtained, and the data related to the A matrix part loss in the damaged data block can be recovered according to an RS erasure recovery method, so that all the lost data is recovered.
If not exceedIf a data block in a disk is lost, if k-l data and l backup redundant data (row 1 to row 1 of the vandermonde matrix) remain to be recovered according to RS erasure, then the two data stripes a1, a2, …, ak and b1, b2, …, bk together need 2*k data to be recovered. However, with the data erasure method according to the embodiment of the present invention, only 1.5 x (k-l) +2*l data blocks are required for recovery, because/>The required data blocks will not exceed 5/3 x k <2 x k.
In one embodiment of the present invention, step S214 may include the steps of:
When the lost data block is the first data block and the second data block in the same disk, performing data erasure by using the next row of the first row and the third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix, or performing data erasure by using the next row of the first row and the fifth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix.
When only one block of data is corrupted, e.g., a1, B1 is corrupted, the data of row k+1 in either B1 or B2, i.e., f 1(b1,a2,...,bk-1,ak) or B 1,a2,...,ak-1,bk, is first called. At this time, the data B1 may be restored first, and then the data B 1,a2,…,bk-1,ak or B 1,a2,…,ak-1,bk may be substituted into the k+s+1 line of B1 or B2, that isOr (b)Then f 1(a1,b2,…,bk/2-1,ak/2) or f 1(a1,b2,…,a(k-1)/2-1,b(k-1)/2) can be obtained. A1 can be found by recalling either data b 2,…,bk/2-1,ak/2 or b 2,…,a(k-1)/2-1,b(k-1)/2.
The main data overhead of RS erasure is reflected on the original data which is not destroyed, according to the existing RS erasure practice, if any one of the two data stripes a1, a2, …, ak and b1, b2, …, bk is lost, we need to use the remaining k-1 data and 1 backup redundant data (i.e. line 1 of the vandermonde matrix) for recovery, then the two stripes need 2*k data in total for recovery. However, by using the data erasure method of the embodiment of the present invention, ai and bi can be recovered only by calling about 75% of the data blocks, i.e., 1.5 x (k-1) +2 data blocks can be recovered, thereby greatly reducing the data call amount during data erasure, reducing the data call time consumption, and improving the data erasure efficiency.
When the number of damaged magnetic disks exceedsWhen we use the remaining data blocks, we can still recover the data blocks in the corrupted r disks. The corrupted data blocks are all among the original data, i.e., a1, a2, …, ak and b1, b2, …, bk corrupted data blocks in r disks. Let them be a1, a2, …, ar and b1, b2, …, br. We know whether to call matrix A1 or matrix A2, we can restore data a 1,b2,...,ak-1, b or a 1,b2,...,bk-1,ak according to RS erasure principles. After the data is restored, the function f1(a1,b2,...,bk/2-1,ak/2),f2(a1,b2,…,bk/2-1,ak/2),…,ft(a1,b2,…,bk/2-1,ak/2),f1(bk/2+1,ak/2+2,...,ak-1,bk),f2(bk/2+1,ak/2+2,...,ak-1,bk),…,ft(bk/2+1,ak/2+2,...,ak-1,bk) or f1(a1,b2,...,a(k-1)/2-1,b(k-1)/2),f2(a1,b2,...,a(k-1)/2-1,b(k-1)/2),…,ft(a1,b2,...,a(k-1)/2-1,b(k-1)/2),f1(a(k+1)/2,b(k+1)/2+1,...,bk-1,ak),f2(a(k+1)/2,b(k+1)/2+1,...,bk-1,ak),…,ft(a(k+1)/2,b(k+1)/2+1,...,bk-1,ak). that brings it into the latter half of the latter k+s+1 to k+r rows of the matrix B1 or B2 eliminates the latter half of the latter k+s+1 to k+r rows, and then B1 and B2 become/>And/>At this time, the original data b 1,a2,...,bk-1,ak or b 1,a2,...,ak-1,bk can be restored according to the RS erasure. So far, the data blocks in all the damaged r disks are restored.
Even if it exceedsThe loss of each data block can still recover all data by using any k data blocks, so that the decoding speed and the data overhead which are the same as those of the RS erasure can still be achieved in the worst case. And, for the actual working environment, 99.75% are the cases of single code block errors (i.e. single disk failures). Therefore, the data storage erasure method provided by the embodiment of the invention can greatly reduce the effective data overhead in the actual use scene.
Corresponding to the above method embodiment, the present invention further provides a data storage erasure correction device, where the data storage erasure correction device described below and the data storage erasure correction method described above can be referred to correspondingly.
Referring to fig. 3, fig. 3 is a block diagram illustrating a data storage erasure apparatus according to an embodiment of the present invention, the apparatus may include:
the data block dividing module 31 is configured to divide a data set in each disk into two data blocks, so as to obtain each first data block and each second data block;
the matrix construction module 32 is configured to select each first data block from the first number of disks, select each second data block from the remaining disks, obtain each round of selected data blocks, and construct a first erasure vandermonde matrix according to the total number of disks and each round of selected data blocks;
The numerical value calculation module 33 is configured to obtain a preset matrix line number, divide the preset matrix line number by three-way lower rounding to obtain a second numerical value, and calculate a difference value by using the preset matrix line number and the second numerical value which is twice to obtain a third numerical value;
a first matrix row construction module 34, configured to construct a first row to a third numerical row of an original vandermonde matrix in the second erasure-correction vandermonde matrix by using each remaining first data block and each remaining second data block except for each round of selection data blocks in each disk;
A second matrix row construction module 35, configured to perform two-round selection on each round of selection data blocks to obtain each round of selection data blocks, and construct a next row to a fourth value row of a third value row of an original vandermonde matrix in the second erasure-correction vandermonde matrix by using each remaining first data block, each remaining second data block, and each round of selection data block; wherein the fourth value is the sum of the third value and the second value;
A third matrix row construction module 36, configured to construct a row from a next row of a fourth numerical row of the original vandermonde matrix in the second erasure-correction vandermonde matrix to a first preset matrix row of rows using each remaining first data block, each remaining second data block, and each data block except each two-round selected data block in each one-round selected data block;
The data erasure module 37 is configured to perform data erasure operations using the first erasure vandermonde matrix and/or the second erasure vandermonde matrix when there is a loss of the data block.
According to the technical scheme, the data set in each disk is divided into two data blocks, and the erasure vandermonde matrix is constructed through the effective selection operation of the data blocks, so that when the erasure vandermonde matrix obtained through construction is utilized for data erasure, only part of the data blocks in the disk are required to be called according to the current data loss condition, the data calling quantity during data erasure is greatly reduced, the data calling time is shortened, and the data erasure efficiency is improved.
In a specific embodiment of the present invention, the first matrix row construction module 34 is specifically a module that uses each remaining first data block and each remaining second data block as the base of each row parameter to construct the first row to the third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix.
In one embodiment of the present invention, the matrix construction module 32 is specifically a module that divides the total number of disks by a first numerical value obtained by rounding down, and selects each first data block from the first numerical value disks;
the second matrix row construction module 35 is specifically a module for dividing the first value by two to obtain a fifth value, selecting a fifth value from the first data blocks selected in each round, and selecting a second data block different from the first value and the fifth value to obtain each second data block selected in each round.
In one embodiment of the present invention, the second matrix row construction module 35 includes:
A first factor obtaining sub-module, configured to construct a next row to a fourth value row of a third value row of the original vandermonde matrix with each remaining first data block and each remaining second data block as a base of each row parameter, and obtain first factors of the next row to the fourth value row of the third value row of the original vandermonde matrix in the second erasure vandermonde matrix;
A second factor obtaining sub-module, configured to construct a first row to a second numerical row of the original vandermonde matrix by using each two rounds of selected data blocks as a base of each row parameter, and obtain a second factor of a next row to a fourth numerical row of a third numerical row of the original vandermonde matrix in the second erasure correction vandermonde matrix;
The first exclusive-or calculation sub-module is used for exclusive-or calculating the first factor and the second factor in each row to obtain the next row to the fourth numerical row of the third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix; wherein the difference between the index of the parameter contained in the first factor and the index of the parameter contained in the second factor in each row is a third value.
In one embodiment of the present invention, the third matrix row construction module 36 includes:
A third factor obtaining sub-module, configured to construct a line from a next line of a fourth numerical line of the original vandermonde matrix to a line of a preset matrix by using each remaining first data block and each remaining second data block as a base of each line parameter, so as to obtain a third factor from the next line of the fourth numerical line of the original vandermonde matrix in the second erasure vandermonde matrix to the line of the preset matrix;
A fourth factor obtaining sub-module, configured to construct a first row to a second numerical row of the original vandermonde matrix by using each data block except for each two-round selected data block in each round selected data block as a base of each row parameter, so as to obtain a fourth factor from a next row to a preset matrix row of a fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix;
The second exclusive-or calculation sub-module is used for exclusive-or calculating the third factor and the fourth factor in each row to obtain the rows from the next row of the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix to the row of the preset matrix; wherein the difference between the index of the parameter included in the third factor and the index of the parameter included in the fourth factor in each row is a fourth value.
In one embodiment of the present invention, the data erasure correction module 37 is specifically a module that performs a data erasure correction operation using the second erasure correction vandermonde matrix when the number of disks having the data block missing does not exceed the second value.
In a specific embodiment of the present invention, the data erasure module 37 is specifically a module that performs data erasure using the next line of the first line and the third numerical line of the original vandermonde matrix in the second erasure vandermonde matrix or performs data erasure using the next line of the first line and the fifth numerical line of the original vandermonde matrix in the second erasure vandermonde matrix when the lost data block is the first data block and the second data block in the same disk.
Corresponding to the above method embodiment, referring to fig. 4, fig. 4 is a schematic diagram of a data storage erasure apparatus provided by the present invention, where the apparatus may include:
A memory 332 for storing a computer program;
A processor 322 for implementing the steps of the data storage erasure method of the above-described method embodiments when executing a computer program.
Specifically, referring to fig. 5, fig. 5 is a schematic diagram of a specific structure of a data storage erasure apparatus according to the present embodiment, where the data storage erasure apparatus may have relatively large differences due to different configurations or performances, and may include a processor (central processing units, CPU) 322 (e.g., one or more processors) and a memory 332, where the memory 332 stores one or more computer applications 342 or data 344. Wherein the memory 332 may be transient storage or persistent storage. The program stored in memory 332 may include one or more modules (not shown), each of which may include a series of instruction operations in the data processing apparatus. Still further, processor 322 may be configured to communicate with memory 332 to execute a series of instruction operations in memory 332 on data storage erasure device 301.
Data storage erasure device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input/output interfaces 358, and/or one or more operating systems 341.
The steps in the data storage erasure method described above may be implemented by the structure of the data storage erasure device.
Corresponding to the above method embodiments, the present invention also provides a computer readable storage medium having a computer program stored thereon, which when executed by a processor, performs the steps of:
Dividing a data set in each disk into two data blocks to obtain first data blocks and second data blocks; selecting each first data block from the first number of disks, selecting each second data block from the rest of the disks to obtain each round of selected data blocks, and constructing a first erasure vandermonde matrix according to the total number of the disks and each round of selected data blocks; obtaining preset matrix line numbers, dividing the preset matrix line numbers by three-way downward rounding to obtain second numerical values, and performing difference calculation by using the preset matrix line numbers and the second numerical values which are doubled to obtain third numerical values; constructing a first numerical line to a third numerical line of an original vandermonde matrix in a second erasure vandermonde matrix by utilizing each remaining first data block and each remaining second data block except each round of selected data blocks in each magnetic disk; performing two-round selection on each round of selected data blocks to obtain each round of selected data blocks, and constructing the next row to the fourth row of the third value row of the original vandermonde matrix in the second erasure correction vandermonde matrix by utilizing each remaining first data block, each remaining second data block and each round of selected data blocks; wherein the fourth value is the sum of the third value and the second value; constructing the next row of the fourth numerical row of the original vandermonde matrix in the second erasure correction vandermonde matrix to the first preset matrix row of rows by utilizing each remaining first data block, each remaining second data block and each data block except each two rounds of selected data blocks in each round of selected data blocks; when the data block is lost, performing data erasure by using the first erasure vandermonde matrix and/or the second erasure vandermonde matrix.
The computer readable storage medium may include: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
For the description of the computer-readable storage medium provided by the present invention, refer to the above method embodiments, and the disclosure is not repeated here.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. The apparatus, device and computer readable storage medium of the embodiments are described more simply because they correspond to the methods of the embodiments, and the description thereof will be given with reference to the method section.
The principles and embodiments of the present invention have been described herein with reference to specific examples, but the description of the examples above is only for aiding in understanding the technical solution of the present invention and its core ideas. It should be noted that it will be apparent to those skilled in the art that various modifications and adaptations of the invention can be made without departing from the principles of the invention and these modifications and adaptations are intended to be within the scope of the invention as defined in the following claims.
Claims (8)
1. A data storage erasure method, comprising:
dividing a data set in each disk into two data blocks to obtain first data blocks and second data blocks;
Dividing the total number of the magnetic disks by a downward rounding to obtain a first numerical value;
selecting each first data block from the first numerical disk;
Selecting each second data block from the rest of the magnetic disks to obtain each round of selected data blocks, and constructing a first erasure vandermonde matrix according to the total number of the magnetic disks and each round of selected data blocks;
obtaining a preset matrix line number, dividing the preset matrix line number by three, and rounding downwards to obtain a second numerical value, and calculating a difference value by using the preset matrix line number and the second numerical value which is doubled to obtain a third numerical value;
Constructing a first line to a third numerical line of an original vandermonde matrix in a second erasure vandermonde matrix by using each remaining first data block and each remaining second data block except each round of selected data blocks in each magnetic disk;
dividing the first numerical value by two, and rounding downwards to obtain a fifth numerical value;
Selecting the fifth numerical value from the first data blocks, and selecting the difference value between the first numerical value and the fifth numerical value from the second data blocks to obtain two-round selected data blocks;
Constructing a next row to a fourth value row of the third value row of the original vandermonde matrix in the second erasure correction vandermonde matrix by using each remaining first data block, each remaining second data block and each two-round selection data block; wherein the fourth value is the sum of the third value and the second value;
Constructing a next row of the third numerical row to a fourth numerical row of the original vandermonde matrix by taking each remaining first data block and each remaining second data block as the base of each row parameter, and obtaining a first factor from the next row of the third numerical row to the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix;
constructing a first row to a second numerical row of an original vandermonde matrix by taking each two-wheel selected data block as a base of each row parameter, and obtaining a second factor from a next row to a fourth numerical row of a third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix;
Performing exclusive OR calculation on the first factor and the second factor in each row to obtain a next row of the third value row to the fourth value row of the original vandermonde matrix in the second erasure vandermonde matrix; wherein the difference between the index of the parameter contained in the first factor and the index of the parameter contained in the second factor in each row is the third value;
When the data block is lost, performing data erasure by using the first erasure vandermonde matrix and/or the second erasure vandermonde matrix.
2. The data storage erasure method of claim 1 wherein constructing the first row of the original vandermonde matrix in the second erasure vandermonde matrix to the third numerical row using each remaining first data block and each remaining second data block in each of the disks except for each of the one round of selected data blocks comprises:
and constructing a first line to a third numerical line of an original vandermonde matrix in the second erasure vandermonde matrix by taking each remaining first data block and each remaining second data block as the base of each line parameter.
3. The data storage erasure method of claim 1 wherein constructing a next row of the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix to a first row of the preset matrix rows using each of the remaining first data blocks, each of the remaining second data blocks, and each of the one-round selected data blocks except for each of the two-round selected data blocks comprises:
Constructing the next row of the fourth numerical value row of the original vandermonde matrix to the preset matrix row by taking each remaining first data block and each remaining second data block as the base of each row parameter, and obtaining a third factor of the next row of the fourth numerical value row of the original vandermonde matrix to the preset matrix row of the second erasure vandermonde matrix;
Constructing a first row to a second numerical row of an original vandermonde matrix by taking each data block except each two-wheel selected data block in each one-wheel selected data block as a base of each row parameter, and obtaining a fourth factor from a next row to a fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix;
Performing exclusive OR calculation on the third factor and the fourth factor in each row to obtain the rows from the next row of the fourth numerical row of the original vandermonde matrix in the second erasure vandermonde matrix to the preset matrix row; wherein the difference between the index of the parameter contained in the third factor and the index of the parameter contained in the fourth factor in each row is the fourth value.
4. A data storage erasure method according to any of claims 1 to 3, wherein when there is a loss of a data block, performing a data erasure operation using the first erasure vandermonde matrix and/or the second erasure vandermonde matrix, comprising:
And when the number of the disks with the data block loss does not exceed the second numerical value, performing data erasure operation by using the second erasure vandermonde matrix.
5. The data storage erasure method according to claim 4, wherein when the number of disks having data block loss does not exceed the second value, performing data erasure operation using the second erasure vandermonde matrix, comprising:
when the lost data block is the first data block and the second data block in the same disk, performing data erasure by using the first row of the original vandermonde matrix and the next row of the third numerical row in the second erasure vandermonde matrix, or performing data erasure by using the first row of the original vandermonde matrix and the next row of the fifth numerical row in the second erasure vandermonde matrix.
6. A data storage erasure apparatus comprising:
the data block dividing module is used for dividing the data set in each disk into two data blocks to obtain each first data block and each second data block;
The matrix construction module is used for dividing the total number of the magnetic disks by the downward rounding to obtain a first numerical value, and selecting each first data block from the magnetic disks with the first numerical value;
Selecting each second data block from the rest of the magnetic disks to obtain each round of selected data blocks, and constructing a first erasure vandermonde matrix according to the total number of the magnetic disks and each round of selected data blocks;
The numerical value calculation module is used for obtaining preset matrix line numbers, dividing the preset matrix line numbers by three to obtain a second numerical value by rounding downwards, and calculating a difference value by using the preset matrix line numbers and the second numerical value which is doubled to obtain a third numerical value;
A first matrix row construction module, configured to construct a first row to a third numerical row of an original vandermonde matrix in a second erasure-correction vandermonde matrix by using each remaining first data block and each remaining second data block in each disk except each round of selected data blocks;
The second matrix row construction module is used for dividing the first numerical value by two to obtain a fifth numerical value by downward rounding, selecting a fifth numerical value from the first data blocks selected in each round, and selecting a second data block with a difference value between the first numerical value and the fifth numerical value to obtain each second round of selected data blocks;
The second matrix row construction module comprises a first factor obtaining sub-module, a second factor obtaining sub-module and a first exclusive OR calculating sub-module, wherein the first factor obtaining sub-module is used for constructing the next row to the fourth numerical row of the third numerical row of the original vandermonde matrix by taking each remaining first data block and each remaining second data block as the base of each row parameter to obtain the first factors of the next row to the fourth numerical row of the third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix;
the second factor obtaining submodule is used for constructing a first row to a second numerical row of the original vandermonde matrix by taking each two rounds of selected data blocks as the base of each row parameter to obtain a second factor of a next row to a fourth numerical row of a third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix;
the first exclusive-or calculation sub-module is used for respectively carrying out exclusive-or calculation on the first factor and the second factor in each row to obtain the next row to the fourth numerical row of the third numerical row of the original vandermonde matrix in the second erasure vandermonde matrix; wherein the difference between the index of the parameter contained in the first factor and the index of the parameter contained in the second factor in each row is a third value; wherein the fourth value is the sum of the third value and the second value;
a third matrix row construction module, configured to construct a row from a next row of the fourth numerical row of the original vandermonde matrix in the second erasure-correction vandermonde matrix to the preset matrix row by using each remaining first data block, each remaining second data block, and each data block except each two-round selected data block in each one-round selected data block;
and the data erasure correction module is used for carrying out data erasure correction operation by utilizing the first erasure correction vandermonde matrix and/or the second erasure correction vandermonde matrix when the data block is lost.
7. A data storage erasure apparatus comprising:
a memory for storing a computer program;
A processor for implementing the steps of the data storage erasure method according to any of claims 1 to 5 when executing the computer program.
8. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the data storage erasure method according to any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210428509.0A CN114879904B (en) | 2022-04-22 | 2022-04-22 | Data storage erasure method, device, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210428509.0A CN114879904B (en) | 2022-04-22 | 2022-04-22 | Data storage erasure method, device, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114879904A CN114879904A (en) | 2022-08-09 |
| CN114879904B true CN114879904B (en) | 2024-05-03 |
Family
ID=82672056
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210428509.0A Active CN114879904B (en) | 2022-04-22 | 2022-04-22 | Data storage erasure method, device, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114879904B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3054602A1 (en) * | 2015-02-09 | 2016-08-10 | Thomson Licensing | Product-matrix regenerating codes |
| CN111404554A (en) * | 2019-09-29 | 2020-07-10 | 杭州海康威视系统技术有限公司 | Method and device for generating coding matrix of erasure code and electronic equipment |
| CN111697976A (en) * | 2020-05-28 | 2020-09-22 | 苏州浪潮智能科技有限公司 | RS erasure correcting quick decoding method and system based on distributed storage |
| CN111858142A (en) * | 2020-07-24 | 2020-10-30 | 山东云海国创云计算装备产业创新中心有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN112052115A (en) * | 2020-09-28 | 2020-12-08 | 山东云海国创云计算装备产业创新中心有限公司 | Data storage erasure method, device, equipment and computer readable storage medium |
| WO2021115021A1 (en) * | 2019-12-11 | 2021-06-17 | 华为技术有限公司 | Data processing method and apparatus, terminal device and computer-readable storage medium |
| CN114281270A (en) * | 2022-03-03 | 2022-04-05 | 山东云海国创云计算装备产业创新中心有限公司 | Data storage method, system, equipment and medium |
-
2022
- 2022-04-22 CN CN202210428509.0A patent/CN114879904B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3054602A1 (en) * | 2015-02-09 | 2016-08-10 | Thomson Licensing | Product-matrix regenerating codes |
| CN111404554A (en) * | 2019-09-29 | 2020-07-10 | 杭州海康威视系统技术有限公司 | Method and device for generating coding matrix of erasure code and electronic equipment |
| WO2021115021A1 (en) * | 2019-12-11 | 2021-06-17 | 华为技术有限公司 | Data processing method and apparatus, terminal device and computer-readable storage medium |
| CN111697976A (en) * | 2020-05-28 | 2020-09-22 | 苏州浪潮智能科技有限公司 | RS erasure correcting quick decoding method and system based on distributed storage |
| CN111858142A (en) * | 2020-07-24 | 2020-10-30 | 山东云海国创云计算装备产业创新中心有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN112052115A (en) * | 2020-09-28 | 2020-12-08 | 山东云海国创云计算装备产业创新中心有限公司 | Data storage erasure method, device, equipment and computer readable storage medium |
| CN114281270A (en) * | 2022-03-03 | 2022-04-05 | 山东云海国创云计算装备产业创新中心有限公司 | Data storage method, system, equipment and medium |
Non-Patent Citations (2)
| Title |
|---|
| 王禹 ; 赵跃龙 ; 侯昉.分布式存储系统最小带宽再生码研究.小型微型计算机系统.2012,(08),全文. * |
| 程志洪 ; 龚志勇 ; .范德蒙阵列纠删码在嵌入式系统中的应用.无线电通信技术.2012,(03),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114879904A (en) | 2022-08-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9647698B2 (en) | Method for encoding MSR (minimum-storage regenerating) codes and repairing storage nodes | |
| EP2278718B1 (en) | Decoding of chain reaction codes through inactivation | |
| US11531593B2 (en) | Data encoding, decoding and recovering method for a distributed storage system | |
| WO2014153716A1 (en) | Methods for encoding minimum bandwidth regenerating code and repairing storage node | |
| CN114153651B (en) | Data encoding method, device, equipment and medium | |
| CN113505019B (en) | Erasure code data and verification recovery method, device and equipment and readable medium | |
| US11303302B2 (en) | Erasure code calculation method | |
| CN114281270B (en) | Data storage method, system, equipment and medium | |
| US11500725B2 (en) | Methods for data recovery of a distributed storage system and storage medium thereof | |
| CN112799875B (en) | Method, system, device and medium for verification recovery based on Gaussian elimination | |
| CN112000512A (en) | Data restoration method and related device | |
| CN111858169B (en) | Data recovery method, system and related components | |
| WO2024098647A1 (en) | Check-code recovery method and system, electronic device and storage medium | |
| CN111682874A (en) | Data recovery method, system, equipment and readable storage medium | |
| CN110895497A (en) | Method and device for reducing erasure code repair in distributed storage | |
| CN114879904B (en) | Data storage erasure method, device, equipment and readable storage medium | |
| CN110990375B (en) | Method for constructing heterogeneous partial repeat codes based on adjusting matrix | |
| CN106911793B (en) | I/O optimized distributed storage data repair method | |
| WO2017041233A1 (en) | Encoding and storage node repairing method for functional-repair regenerating code | |
| CN115113816A (en) | Erasure code data processing system, method, computer device and medium | |
| CN114996047A (en) | Data storage method, system, device and medium | |
| CN112286449B (en) | RS erasure processing equipment and distributed storage system | |
| CN114244376B (en) | Data encoding method, system, device and medium | |
| CN110780813A (en) | Distributed storage system based on subspace codes in binary domain | |
| CN117591336A (en) | Method and system for decentralizing data redundancy storage based on erasure codes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |