CN115098295A - Data local recovery method, equipment and storage medium - Google Patents
Data local recovery method, equipment and storage medium Download PDFInfo
- Publication number
- CN115098295A CN115098295A CN202210764325.1A CN202210764325A CN115098295A CN 115098295 A CN115098295 A CN 115098295A CN 202210764325 A CN202210764325 A CN 202210764325A CN 115098295 A CN115098295 A CN 115098295A
- Authority
- CN
- China
- Prior art keywords
- data
- matrix
- block
- blocks
- local
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1008—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices
- G06F11/1048—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices using arrangements adapted for a specific error detection or correction feature
- G06F11/1056—Updating check bits on partial write, i.e. read/modify/write
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1479—Generic software techniques for error detection or fault masking
- G06F11/1489—Generic software techniques for error detection or fault masking through recovery blocks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Error Detection And Correction (AREA)
- Techniques For Improving Reliability Of Storages (AREA)
Abstract
The embodiment of the application provides a data local recovery method, equipment and a storage medium. The global check block and the local check block in any data group of the data set are obtained by encoding the data blocks, and when any data block in the data group is lost, the data block can be recovered according to the rest data blocks, the global check block and the local check block. When a partial parity block is lost, it can be recovered from the data blocks in the data group. The local check block comprises coding information of the global check block, and when the global check block is lost, the global check block can be recovered according to the data block and the local check block, so that local recovery of any object block in the data group is realized. The sum of the local check blocks of the data set is linearly independent of the global check blocks of the data set, and when the number of the lost object blocks in the data set is greater than that of the global check blocks, the lost object blocks can be recovered, so that the fault tolerance of the data local recovery method is further improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method, a device, and a storage medium for local data recovery.
Background
Erasure codes (Erasure codes) are a coding fault tolerant technique for recovering original data when parts of the data are lost. Erasure codes occupy less storage space while ensuring reliability, and are thus widely used in distributed storage systems. Compared with the traditional copy type data recovery method, the erasure code has the problem of reading and amplifying when reconstructing data.
Currently, there is a method for Local data recovery based on Local self-recovery erasure codes (LRCs). However, the existing local repair scheme still cannot perform local recovery on the global parity chunks under the condition of high fault-tolerant capability. Therefore, a new solution is yet to be proposed.
Disclosure of Invention
Aspects of the present application provide a data local recovery method, device and storage medium, which are used to perform local recovery on a global check block and improve the fault tolerance of the data local recovery method.
An embodiment of the present application further provides a data local recovery method, including: determining a plurality of data groups contained in the data set, wherein an object block in any data group comprises: the system comprises a data block, a global check block and a local check block; responding to a recovery request for any target object block in the data group, and recovering the target object block according to the remaining readable object blocks in the data group; in the data group, the global check block and the local check block are obtained by encoding the data block, and the local check block includes encoding information of the global check block; and the sum of the local check blocks in the data set is linearly independent of the global check block in the data set.
Optionally, the method further comprises: responding to a recovery request for a first number of object blocks in the data set, and recovering the first number of object blocks according to the remaining readable object blocks in the data set; wherein the first number is greater than the number of global parity chunks in the dataset.
Optionally, in response to a recovery request for any target object block in the data group, according to the remaining readable object blocks in the data group, the method further includes: determining a first matrix of r +1 dimension, wherein r is a positive integer; correspondingly generating r derivative rows according to any r data rows in the first matrix; the r derived rows are used for encoding to obtain r local check blocks, and any derived row contains elements in a data row used for generating the derived row; replacing the data rows except the any r data rows in the first matrix by the r derived rows to obtain a second matrix; an identity matrix is connected to the rear side of the second matrix in a side mode to obtain a coding matrix; and coding the data blocks in the data set by adopting the coding matrix to obtain a global check block and local check blocks corresponding to the data groups respectively.
Optionally, any r +1 column in the coding matrix is a column full rank matrix.
Optionally, the first matrix is a cauchy matrix.
Optionally, correspondingly generating r derivative rows according to any r data rows in the first matrix, including: adding additional items to the elements of different columns in the r data rows according to the corresponding relation between the elements of different columns in the r data rows and a data group during matrix calculation to obtain r derived rows; wherein, in the same derivative row, the additional item is added to the element of the column corresponding to any data set; in different derivative lines, the data sets corresponding to the elements added with the additional items are different.
Optionally, adding additional items to the elements in different columns in the r data rows according to the corresponding relationship between the elements in different columns in the r data rows and the data group during matrix calculation, including: for the element of the jth column in the ith data row in the r data rows, determining an additional item of the element of the jth column in the ith data row according to the element of the jth column in the r +1 row of the first matrix; and adding the additional item to the element of the jth column in the ith data row to obtain a derivative row corresponding to the ith data row, wherein i and j are positive integers.
Optionally, the summation result of the elements in the r derived rows is equal to the summation result of the elements in the r +1 th row in the first matrix; wherein the summation comprises an exclusive-or operation of the elements.
In the local data recovery method provided in the embodiment of the present application, when there is a recovery request for any target object block in any data group in a data set, a target object block may be recovered according to remaining readable object blocks in the data group. Any target object block can be a data block, a global parity block or a local parity block in the data group. The global check block and the local check block are obtained by encoding the data blocks, and when any data block in the data group is lost, the data block can be recovered according to the rest data blocks, the global check block and the local check block. The local check block comprises coding information of the global check block, and the global check block can be recovered according to the data block and the local check block when the global check block is lost. When the local check block is lost, the local check block can be recovered according to the data block in the data group. Thus, local recovery for any object block within the data set is achieved. The sum of the local check blocks in the data set is linearly independent of the global check block in the data set, so that the lost object blocks can be recovered when the number of the lost object blocks in the data set is greater than the number of the global check blocks. Furthermore, the fault tolerance of the data local recovery method can be improved while local recovery of any object block in the group is realized.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a schematic flowchart of a data local recovery method according to an exemplary embodiment of the present application;
FIG. 2 is a schematic diagram of a global parity chunk and a local parity chunk corresponding to r data packets;
fig. 3 is a schematic diagram of a global parity chunk and a local parity chunk corresponding to 2 data packets;
fig. 4 is a schematic diagram of a global parity chunk and a local parity chunk corresponding to 3 data packets;
fig. 5 is a schematic structural diagram of a server according to an exemplary embodiment of the present application.
Detailed Description
To make the objects, technical solutions and advantages of the present application more clear, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the examples of the present invention and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, and "a" and "an" generally include at least two, but do not exclude at least one, unless the context clearly dictates otherwise.
It should be understood that the term "and/or" as used herein is merely one type of association that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.
It is also noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a good or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such good or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a commodity or system that includes the element.
Currently, there are some methods for Local data recovery based on Local self-recovery erasure codes (LRCs). In the method, the data is divided into a plurality of groups (groups), the data in the groups can generate local check blocks, and when the number of the lost data blocks in the groups does not exceed the number of the local check blocks in the groups, the lost data blocks can be recovered through other data blocks in the groups, so that the flow rate of the data blocks in the recovery process is reduced.
In a typical method, a global parity block P is computed from all data blocks D; and the local check block L is obtained by the operation of the data blocks in the group. The local check block L can be recovered in a local mode. However, the global parity chunks do not have local recovery property, i.e. all data chunks D need to be read to recover the global parity chunks P.
In another typical method, the global parity block P is obtained by calculating all data blocks D; and the local check block L is obtained by the operation of the data blocks in the group. In contrast, when constructing the coding matrix, it is necessary to make the sum of all local parity check blocks consistent with the sum of all global parity check blocks. Furthermore, when the global parity chunks are lost, the global parity chunks can be recovered through the local parity chunks without reading all the data chunks D. However, this approach sacrifices the fault tolerance of the LRC, i.e. can only tolerate any r block losses, i.e.: when the number of lost blocks is less than or equal to r, the lost blocks may be recovered. Wherein r is the number of the global check blocks, and r is a positive integer.
In some embodiments of the present application, a solution is provided, and the technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.
Fig. 1 is a flowchart illustrating a method for local data recovery according to an exemplary embodiment of the present application, where the method may include the steps shown in fig. 1:
The execution subject of this embodiment may be a server storing data, and the server may be a conventional server, a cloud server, an elastic computing instance on the cloud, or a virtualized data center, and the like, which is not limited in this embodiment. The server runs a data storage application and is used for storing and managing the received data.
In this embodiment, the number storage application on the server may perform local repair on the lost data based on the erasure code, so as to reduce the cost required for data repair.
Wherein the data set comprises a plurality of data groups for performing local calculations in the event of a data failure recovery. The number of data blocks included in each data group may be divided according to requirements, and this embodiment is not limited. Wherein each data group may contain a plurality of object blocks, and the plurality of object blocks may include: data chunks, global parity chunks, and local parity chunks.
Any target object block to be restored may be any one of a data block, a global parity block, or a local parity block. When any object block in the data group is lost, the lost object block can be recovered through the rest of the data blocks.
The global check block is obtained by encoding all data blocks (namely, the global data block) in the data set, and can be used for recovering the global data block when any global data block is lost. That is, for any data group, when any data block in the data group is lost, local recovery of the lost data block can be realized through the global data block in the data group. The local check block in any data group is obtained by encoding according to the data block in the data group and can be designed through the encoding process of the local check block, so that the local check block contains the encoding information of the global check block. And when the global check block corresponding to the data group is lost, the global check block can be recovered through the data block in the data group in which the global check block is located and the local check block.
When the data set comprises a plurality of data groups, the plurality of data groups can correspond to a plurality of global check blocks, and the sum of a plurality of local check blocks is linearly independent of the global check blocks in the data set. That is, the sum of the local parity chunks can be regarded as a new global parity chunk, and the new global parity chunk is linearly independent of the original global parity chunks in the data set. Therefore, when the data set originally has r global check blocks, the sum of the local check blocks can be used as the (r +1) th global check block, so that when any (r +1) object blocks in the data set are lost, the remaining object blocks are adopted to perform data recovery on the (r +1) object blocks.
On this basis, the server may optionally restore a first number of object blocks in the data set based on the remaining readable object blocks in the data set in response to a restore request for the first number of object blocks. Wherein the first number is greater than the number of global parity chunks in the dataset. Typically, the first number is the number of global parity chunks plus one. That is, when r global check blocks are provided in the data set, it can be tolerated that any r +1 object blocks are lost in the data set, and the remaining object blocks are used to perform data recovery on the r +1 object blocks.
In this embodiment, when there is a recovery request for any target object block in any data group in the data set, the target object block may be recovered according to the remaining readable object blocks in the data group. Any target object block can be a data block, a global parity block or a local parity block in the data group. The global check block and the local check block are obtained by encoding the data block, and when any data block in the data group is lost, the data block can be recovered according to the rest data block, the global check block and the local check block. The local check block comprises coding information of the global check block, and the global check block can be recovered according to the data block and the local check block when the global check block is lost. When the local check block is lost, the local check block can be recovered according to the data block in the data group. Thus, local recovery for any object block within the data set is achieved. The sum of the local check blocks in the data set is linearly independent of the global check block in the data set, so that the lost object blocks can be recovered when the number of the lost object blocks in the data set is greater than that of the global check blocks. Furthermore, the fault tolerance of the data local recovery method can be improved while local recovery of any object block in the group is realized.
Before data recovery is performed, the server may generate a global parity chunk and a local parity chunk based on the erasure code principle. The global check block and the local check block are obtained by encoding the data blocks in the data set through the encoding matrix. An alternative generation of the coding matrix will be described below.
Assuming that the data set includes k data blocks, the data set is divided into g data groups, and assuming that the number of the global check blocks and the number of the local check blocks are both r, k is g × r. When each data group contains one global check block, g is r.
In some alternative embodiments, the server may determine a first matrix in the r +1 dimension; the number of rows of the first matrix is r +1 rows, and the number of columns is k columns. From the first matrix, an encoding matrix may be constructed. The coding matrix is used for describing the process of obtaining the check block by coding in a matrix form.
In the process of constructing the encoding matrix, r derivative rows can be correspondingly generated according to any r data rows in the first matrix, wherein the r derivative rows are used for encoding to obtain r local check blocks, and any derivative row contains elements in the data rows used for generating the derivative rows, so that the local check blocks obtained by encoding contain encoding information of the global check blocks.
After r derivative rows are determined, the r derivative rows can be used to replace data rows in the first matrix except for the any r data rows, and a second matrix is obtained. The matrix to be described is defined by "first" and "second", which are only used for convenience of description and distinction, and do not limit the matrix otherwise. After the second matrix is determined, the identity matrix is connected to the back of the second matrix in a side mode to obtain the coding matrix. The elements on the diagonal (called main diagonal) from the top left corner to the bottom right corner of the unit matrix are all 1, the other elements are all 0, and the unit matrix is a full rank matrix. In this embodiment, an element of 1 in the identity matrix is used to represent the parity block to be encoded. In the encoding matrix, the element 1 located in the same row as the original data row in the first matrix is used for calculating a global check block, and the element 1 located in the same row as the derived data row is used for calculating a local check block.
After the coding matrix is constructed, the coding matrix can be used for coding the data blocks in the data set to obtain the global check block and the local check blocks corresponding to the plurality of data sets. This will be specifically described below.
Optionally, in this embodiment, the target of encoding may be to generate r +1 linearly independent global parity chunks. In the r +1 global check blocks, one global check block is obtained by summing the r local check blocks. Before generating the derivative rows, the first matrix may be designed so that after the unit matrices corresponding to r +1 global check blocks are added to the rear side of the first matrix to obtain a new matrix, a matrix formed by any r +1 column taken out from the new matrix is a column full rank matrix, so that any r +1 column in the new matrix is linearly independent. Therefore, when any r +1 block is lost in the k data blocks and the r +1 global check blocks, the data blocks can be repaired through the residual blocks.
In some embodiments, the first matrix may be implemented as a Cauchy matrix (Cauchy matrix), where a Cauchy matrix of dimensions r x g is followed by an identity matrix of dimensions r x r, such that any r +1 columns form a matrix that is a column full rank matrix.
In the first matrix of r +1 rows, any r data rows may be used to calculate r global parity chunks, and the remaining 1 data row may be used to calculate respective local parity chunks of g data groups. Alternatively, r derivative rows may be generated from the remaining 1 data row, the r derivative rows corresponding to the g data sets. Based on the r data rows and the r derivative rows, an adjusted second matrix is obtained.
In order to ensure that the unit matrix connected to the second coding matrix and the unit matrix connected to the first coding matrix have the same linear independent characteristic, the sum of r derived rows in the second coding matrix can be the same as the sum of the remaining 1 data row in the first coding matrix.
Optionally, the r derivative rows may be obtained by adding additional entries to elements in different columns of the r data rows according to a corresponding relationship between elements in different columns of the r data rows in the first matrix and the data set during matrix calculation. Wherein, in the same derivative row, the additional item is added to the element of the column corresponding to any data set; in different derivative lines, the data sets corresponding to the elements added with the additional items are different. Wherein the additional term is such that the sum of the elements in the r derivative rows is the same as the sum of the remaining 1 data row.
For example, the element corresponding to data group g1 in a certain data row may be locally adjusted to obtain the derived data row corresponding to data group g 1. For another example, the element corresponding to data set g2 in another data row may be locally adjusted to obtain a derived data row corresponding to data set g 2. The r derived data rows are respectively used for calculating local check blocks corresponding to the g data groups to obtain r local check blocks.
In the following embodiments, for convenience of description and calculation, r global parity chunks are calculated using rows 1 to r in the first matrix, and r local parity chunks are calculated using row r + 1. In generating r derivative rows, additional entries may be added on a per row basis in the first matrix, resulting in r derivative rows.
Wherein the additional items satisfy the following conditions:
1. in the same derivative line, an additional item is added to an element corresponding to any data set; in different derivative lines, the data sets corresponding to the elements added with the additional items are different.
2. The result of summing the elements in the r derived rows is equal to the result of summing the elements in the r +1 th row in the first matrix; wherein the summation is an exclusive or operation.
For example, for a first data group consisting of 1 st to g-th data, an additional item may be added to an element corresponding to the first data group in the first row in the first matrix, and the other elements may be left unchanged to obtain a second row in the second matrix. The second row may be used to encode a local parity block corresponding to the first data set. For another example, for a second data group consisting of g +1 to 2 g-th data, an additional item may be added to an element corresponding to the second data group in the second row in the first matrix, and the other elements are unchanged to obtain a fourth row in the second matrix. The fourth row may be used to encode a partial parity block corresponding to the second data set. By analogy, for the g-th data group consisting of (r-1) g + 1-rg data, additional items can be added to the element corresponding to the g-th data group in the 2 r-1-th row in the first matrix, and the other elements are unchanged to obtain the 2 r-th row in the second matrix. The 2r row may be used to encode a local parity block corresponding to the g data group.
The following will be further explained in conjunction with a specific calculation process.
Determining a first matrix of dimension (r +1) × k:
[a ij :1≤i≤r+1,1≤j≤k] (r+1)*k
the expansion is as follows:
rows 1-r in the first matrix can be used for encoding to generate r global check blocks, and row r +1 is used for encoding to generate r local check blocks. Based on the first matrix, a second matrix may be generated in dimensions 2r x k. The second matrix comprises 1 st to r th rows and r derivative rows in the first matrix, wherein the derivative rows are distributed in the 2 nd row, the 4 th row, … and the 2r th row in the second matrix. When encoding is performed, a 2r × k unit matrix is added to the rear side of the second matrix to obtain an encoding matrix H, and any r +1 column in the encoding matrix H is a row full rank matrix.
Wherein, d j Is an additional term for local adjustment of rows 1 to r in the first matrix to obtain r derivative rows.
Wherein d is 1 ～d g Added to the element corresponding to the first data set in row 1 of the first matrix. Wherein d is g+1 ～d 2g Added to the element corresponding to the 2 nd data set in row 2 of the first matrix. d (r-1)g+1 ～d rg Added to the element corresponding to the g-th data group in the r-th row in the first matrix. That is, it is satisfied that in the same derivative line, an additional item is added to an element corresponding to any one data group; in different derivative lines, the data sets corresponding to the elements added with the additional items are different.
Alternatively, in order to make the result of summing the elements in the r derived rows equal to the result of summing the elements in the r +1 th row in the first matrix, the summing operation can be designed as an exclusive or operation, and the corresponding additional term for the column is calculated from the elements of the same column in the r data rows.
Optionally, taking the element in the jth column of the ith data row of the r data rows as an example, the additional item of the element in the jth column of the ith data row may be determined according to the element in the jth column of the r +1 row of the first matrix. And adding the additional item to the element of the jth column in the ith data row to obtain a derivative row corresponding to the ith data row. Additional item d in ith row and jth column j The calculation process of (2) is as follows:
and adding corresponding additional items to corresponding elements of different data groups in the first matrix to generate r derived rows corresponding to the local check block. In the second matrix, the summation (exclusive or) operation of the second row, the fourth row, … and the 2r row results in:
h 2 +h 4 +...+h 2r ＝a r+1,1 +a r+1,2 ,...,a r+1,rg
that is, the summation result is equal to the summation result of the elements in the r +1 th row in the first matrix. Therefore, the sum of the local check blocks calculated by the r derived rows and the r global check blocks have a linear independent relation. The sum of the r derived line coded local check blocks can be regarded as a new global check block, and the global check block is linearly independent from the r global check blocks, so that the loss of any r +1 data blocks can be tolerated.
And performing matrix calculation on the coding matrix H constructed by the method and a k-dimensional matrix D formed by k data blocks, and making H x D equal to 0 to obtain a global check block and a local check block. Namely:
based on the above equation, P can be determined 1 、P 2 、P 3 …P r As a global parity chunk. Wherein L is 1 、L 2 、L 3 …L r Corresponding to the derivative row, it can be used as a local check block.
Converting the above formula to a global check block P i Can be expressed as:
wherein, the local check block L i Expressed as:
wherein, k data blocks, r global check blocks and r local check blocks can be divided into r data groups, namely:
G 1 ＝{D 1 ,D 2 ,...,D g ,L 1 ,P 1 }，
G 2 ＝{D g+1 ,D g+2 ,...,D 2g ,L 2 ,P 2 }
…
G r ＝{D (r-1)g+1 ,D (r-1)g+2 ,...,D rg ,L r ,P r }
when a single check block is lost in any data group, data recovery can be performed through other g +1 data blocks in the group. Wherein, the local check block L 1 、L 2 、…、L r Added to be regarded as a new global parity check block P r+1 Due to L 1 、L 2 、…、L r And P 1 、P 2 、…P r Have a linear independent relationship between them, so P 1 、P 2 、P 3 、…、P r 、P r+1 Are linearly independent of each other. Therefore, when D 1 、D 2 、D 3 、…、D k 、P 1 、P 2 、…、P r 、L 1 、L 2 、…、L r When any r +1 data blocks are lost, the lost data blocks can be recovered according to the remaining data blocks.
Based on the embodiment, the coding mode with high reliability, high performance and flexible parameters is realized. The high reliability means that data can be recovered when any r +1 block is lost as a result of the encoding. High performance means that when any single block in a packet is lost, recovery can be performed through a local data block, and read amplification in the data recovery process is reduced. The flexible parameters mean that the number of the data blocks can be completely divided by the number of the global check blocks, and when the number of the data blocks changes, an accurate coding matrix structure can be provided, so that the coding requirements under multiple conditions are met.
To describe the embodiments of the present application in further detail, the following will explain an example in which r is implemented as 2 and k is implemented as 4.
Suppose that the data set contains data blocks D 1 、D 2 、D 3 And D 4 . Grouping the data sets to obtain a data set containing D 1 、D 2 Data group G of 1 And comprises D 3 、D 4 Data group G of 2 。
Determining a first matrix:
summing the rows in the first matrix to obtain d j ：
Adopting the first matrix and the result obtained by calculation of formula 1 to construct a second matrix, and connecting a unit matrix behind the second matrix to obtain an encoding matrix:
based on the above formula 2-1, the sum of the 2 nd row and the 4 th row in the second matrix is calculated to obtain:
h 2 +h 4 ＝a 1,1 +d 1 +a 1,2 +d 2 +a 1,3 +a 1,4 +a 2,1 +a 2,2 +a 2,3 +d 3 +a 2,4 +d 4
＝a 1,1 +(a 1,1 +a 2,1 +a 3,1 )+a 1,2 +(a 1,2 +a 2,2 +a 3,2 )+a 1,3 +a 1,4 +a 2,1 +a 2,2 +
+a 2,3 +(a 1,3 +a 2,3 +a 3,3 )+a 2,4 +(a 1,4 +a 2,4 +a 3,4 )
＝a 3,1 +a 3,2 +a 3,3 +a 3,4
that is, the sum of the 2 nd row and the 3 rd row in the second matrix is equal to the sum of the 3 rd row in the first matrix, so that any 3 columns in the coding matrix satisfy the linear independence characteristic.
Next, data block D is used 1 、D 2 、D 3 And D 4 And performing matrix operation with the coding matrix to obtain a global check block and a local check block:
wherein, the global check block P i Expressed as:
wherein, the local check block L i Expressed as:
based on the above equations 2-2 and 2-3, the global parity chunk P can be obtained 1 、P 2 And a local check block L 1 、L 2 。
The above data block D 1 、D 2 、D 3 And D 4 Global check block P 1 、P 2 And a local check block L 1 、L 2 It can be divided into two data groups: g 1 ＝{D 1 ,D 2 ,L 1 ,P 1 }，G 2 ＝{D 3 ,D 4 ,L 2 ,P 2 }。
When a single check block is lost in any data group, data recovery can be performed through other 3 data blocks in the group. For example, when D 1 When lost, can pass through D 2 、L 1 、P 1 Recovery D 1 。
Wherein, the local check block L 1 、L 2 Added as a new global parity chunk P 3 Due to L 1 、L 2 Sum and P 1 、P 2 Have a linear independent relationship between them, so P 1 、P 2 、P 3 Are linearly independent of each other. Therefore, when D 1 、D 2 、D 3 、D 4 、P 1 、P 2 、L 1 、L 2 When any 3 data blocks are lost, the lost data blocks can be recovered according to the rest data blocks.
In the following, an example will be described in which r is 3 and k is 9.
Suppose that the data set contains data blocks D 1 、D 2 、D 3 、D 4 、D 5 、D 6 、D 7 、D 8 、D 9 . Grouping the data sets to obtain a data set containing D 1 、D 2 、D 3 Data group G of 1 Comprising D 4 、D 5 、D 6 Data group G of 2 And comprises D 7 、D 8 、D 9 Data group G of 3 。
Determining a first matrix:
summing the rows in the first matrix to obtain d j ：
Adopting the first matrix and the result obtained by calculation of formula 1 to construct a second matrix, and connecting a unit matrix behind the second matrix to obtain an encoding matrix:
based on the above formula 3-1, the sum of the second row, the fourth row and the sixth row in the second matrix is calculated to obtain:
h 2 +h 4 +h 6 ＝a 4,1 +a 4,2 +a 4,3 +a 4,4 +a 4,4
that is, the sum of the 2 nd, 4 th and 6 th rows in the second matrix is equal to the sum of the 3 rd row in the first matrix, so that any 3 columns in the coding matrix satisfy the linear independence characteristic.
Next, data block D is used 1 、D 2 、D 3 、D 4 、D 5 、D 6 、D 7 、D 8 、D 9 And performing matrix operation with the coding matrix to obtain a global check block and a local check block:
wherein, the global check block P i Expressed as:
wherein, the local check block L i Expressed as:
based on the above equations 3-2 and 3-3, a global parity chunk P can be obtained 1 、P 2 、P 3 And a local check block L 1 、L 2 、L 3 。
The above data block D 1 、D 2 、D 3 、D 4 、D 5 、D 6 、D 7 、D 8 、D 9 Global parity block P 1 、P 2 、P 3 And a local check block L 1 、L 2 、L 3 It can be divided into three data groups: g 1 ＝{D 1 ,D 2 ,D 3 ,L 1 ,P 1 }，G 2 ＝{D 4 ,D 5 ,D 6 ,L 2 ,P 2 }，G 3 ＝{D 7 ,D 8 ,D 9 ,L 3 ,P 3 }。
When a single check block is lost in any data group, data recovery can be performed through other 4 data blocks in the group. For example, when D 1 When lost, can pass through D 2 、D 3 、L 1 、P 1 Recovery of D 1 。
Wherein, the local check block L 1 、L 2 、L 3 Added to be regarded as a new global parity check block P 4 Due to L 1 、L 2 、L 3 Sum and P 1 、P 2 、P 3 Have a linear independent relationship between them, so P 1 、P 2 、P 3 、P 4 Are linearly independent of each other. Therefore, when D 1 、D 2 、D 3 、D 4 、D 5 、D 6 、D 7 、D 8 、D 9 、P 1 、P 2 、P 3 、L 1 、L 2 、L 3 When any 4 data blocks are lost, the lost data blocks can be recovered according to the remaining data blocks.
It should be noted that, the executing subjects of the steps of the method provided in the foregoing embodiments may be the same device, or different devices may also be used as the executing subjects of the method. For example, the execution subjects of steps 101 to 104 may be device a; for another example, the execution subject of steps 101 and 102 may be device a, and the execution subject of step 103 may be device B; and so on.
In addition, in some of the flows described in the above embodiments and the drawings, a plurality of operations are included in a specific order, but it should be clearly understood that the operations may be executed out of the order presented herein or in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are merely used for distinguishing different operations, and the sequence numbers do not represent any execution order per se. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first", "second", etc. in this document are used for distinguishing different messages, devices, modules, etc., and do not represent a sequential order, nor limit the types of "first" and "second" to be different.
Fig. 5 is a schematic structural diagram of a server provided in an exemplary embodiment of the present application, which is applicable to the information processing system provided in the foregoing embodiment. As shown in fig. 5, the server includes: memory 501, processor 502, and communication component 503.
The memory 501 is used for storing a computer program and may be configured to store other various data to support operations on the server. Examples of such data include instructions for any application or method operating on the server.
A processor 502 coupled to the memory 501 for executing the computer program in the memory 501 for: determining a plurality of data groups contained in the data set, wherein the object block in any data group comprises: the system comprises a data block, a global check block and a local check block; responding to a recovery request for any target object block in the data set, and recovering the target object block according to the remaining readable object blocks in the data set; in the data group, the global check block and the local check block are obtained by encoding the data block, and the local check block includes encoding information of the global check block; the sum of the local check blocks in the data set is linearly independent of the global check block in the data set.
Optionally, the processor 502 is further configured to: responding to a recovery request for a first number of object blocks in the data set, and recovering the first number of object blocks according to the remaining readable object blocks in the data set; wherein the first number is greater than the number of global parity chunks in the dataset.
Optionally, the processor 502, before responding to a recovery request for any target object block in the data set, is further configured to: determining a first matrix of r +1 dimension, wherein r is a positive integer; correspondingly generating r derivative rows according to any r data rows in the first matrix; the r derived rows are used for encoding to obtain r local check blocks, and any derived row contains elements in a data row used for generating the derived row; replacing the data rows except the any r data rows in the first matrix by the r derived rows to obtain a second matrix; an identity matrix is connected to the rear side of the second matrix in a side mode to obtain a coding matrix; and coding the data blocks in the data set by adopting the coding matrix to obtain a global check block and local check blocks corresponding to the data groups respectively.
Optionally, any r +1 column in the coding matrix is a column full rank matrix.
Optionally, the first matrix is a cauchy matrix.
Optionally, when the processor 502 correspondingly generates r derivative rows according to any r data rows in the first matrix, the processor is specifically configured to: adding additional items to the elements of different columns in the r data rows according to the corresponding relation between the elements of different columns in the r data rows and a data group during matrix calculation to obtain r derived rows; wherein, in the same derivative row, the additional item is added to the element of the column corresponding to any data set; in different derivative lines, the data sets corresponding to the elements added with the additional items are different.
Optionally, when, according to the correspondence between the elements in different columns of the r data rows and the data group during matrix calculation, the processor 502 adds additional items to the elements in different columns of the r data rows, specifically configured to: for the j-th element in the ith data row in the r data rows, determining an additional item of the j-th element in the ith data row according to the j-th element in the r +1 row of the first matrix; and adding the additional item to the element of the jth column in the ith data row to obtain a derivative row corresponding to the ith data row, wherein i and j are positive integers.
Optionally, the summation result of the elements in the r derived rows is equal to the summation result of the elements in the r +1 th row in the first matrix; wherein the summation comprises an exclusive-or operation of the elements.
Further, as shown in fig. 5, the server further includes: power supply components 504, and the like. Only some of the components are schematically shown in fig. 5, and it is not meant that the server includes only the components shown in fig. 5.
The memory 501 may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
Wherein the communication component 503 is configured to facilitate communication between the device in which the communication component is located and other devices in a wired or wireless manner. The device in which the communication component is located may access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component may be implemented based on Near Field Communication (NFC) technology, Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
The power supply 504 is configured to provide power to various components of the device in which the power supply is located. The power components may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device in which the power component is located.
In this embodiment, when there is a recovery request for any target object block in any data group in the data set, the target object block may be recovered according to the remaining readable object blocks in the data group. Any target object block can be a data block, a global parity block or a local parity block in the data group. The global check block and the local check block are obtained by encoding the data blocks, and when any data block in the data group is lost, the data block can be recovered according to the rest data blocks, the global check block and the local check block. The local check block comprises coding information of the global check block, and the global check block can be recovered according to the data block and the local check block when the global check block is lost. When the local check block is lost, the local check block can be recovered according to the data block in the data group. Thus, local recovery for any object block within the data set is achieved. The sum of the local check blocks in the data set is linearly independent of the global check block in the data set, so that the lost object blocks can be recovered when the number of the lost object blocks in the data set is greater than the number of the global check blocks. Furthermore, the fault tolerance of the data local recovery method can be improved while local recovery of any object block in the group is realized.
Accordingly, the present application further provides a computer readable storage medium storing a computer program, where the computer program is capable of implementing the steps that can be executed by the server in the foregoing method embodiments when executed.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A method for local recovery of data, comprising:
determining a plurality of data groups contained in the data set, wherein an object block in any data group comprises: the system comprises a data block, a global check block and a local check block;
responding to a recovery request for any target object block in the data set, and recovering the target object block according to the remaining readable object blocks in the data set;
in the data group, the global check block and the local check block are obtained by encoding the data block, and the local check block includes encoding information of the global check block; and the sum of the local check blocks in the data set is linearly independent of the global check block in the data set.
2. The method of claim 1, further comprising:
responding to a recovery request for a first number of object blocks in the data set, and recovering the first number of object blocks according to the remaining readable object blocks in the data set;
wherein the first number is greater than the number of global parity chunks in the dataset.
3. The method of claim 2, wherein in response to a recovery request for any target object block in the data set, further comprising, in accordance with a remainder of the readable object blocks in the data set, prior to:
determining a first matrix of r +1 dimension, wherein r is a positive integer;
correspondingly generating r derivative rows according to any r data rows in the first matrix; the r derived rows are used for encoding to obtain r local check blocks, and any derived row contains elements in a data row used for generating the derived row;
replacing the data rows except the any r data rows in the first matrix by the r derived rows to obtain a second matrix;
connecting an identity matrix at the back of the second matrix to obtain a coding matrix;
and coding the data blocks in the data set by adopting the coding matrix to obtain a global check block and local check blocks corresponding to the data groups respectively.
4. The method of claim 3, wherein any r +1 column in the coding matrix is a column full rank matrix.
5. The method of claim 3 or 4, wherein the first matrix is a Cauchy matrix.
6. The method of claim 3, wherein generating r derivative rows according to any r data rows in the first matrix comprises:
adding additional items to the elements of different columns in the r data rows according to the corresponding relation between the elements of different columns in the r data rows and a data group during matrix calculation to obtain r derived rows;
wherein, in the same derivative row, the additional item is added to the element of the column corresponding to any data set; in different derivative lines, the data sets corresponding to the elements added with the additional items are different.
7. The method of claim 5, wherein adding additional terms to the elements in different columns of the r data rows based on their correspondence to the data sets during matrix computation comprises:
for the element of the jth column in the ith data row in the r data rows, determining an additional item of the element of the jth column in the ith data row according to the element of the jth column in the r +1 row of the first matrix;
and adding the additional item to the element of the jth column in the ith data row to obtain a derivative row corresponding to the ith data row, wherein i and j are positive integers.
8. The method of claim 7, wherein the summation of the elements in the r derived rows is equal to the summation of the elements in the r +1 th row in the first matrix; wherein the summation comprises an exclusive-or operation of the elements.
9. A server, comprising: a memory and a processor;
the memory is to store one or more computer instructions;
the processor is to execute the one or more computer instructions to: performing the steps of the method of any one of claims 1-8.
10. A computer-readable storage medium storing a computer program, wherein the computer program is capable of implementing the local data recovery method according to any one of claims 1 to 8 when executed by a processor.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210764325.1A CN115098295A (en) | 2022-06-29 | 2022-06-29 | Data local recovery method, equipment and storage medium |
| PCT/CN2023/102218 WO2024001974A1 (en) | 2022-06-29 | 2023-06-25 | Local recovery method and device for data, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210764325.1A CN115098295A (en) | 2022-06-29 | 2022-06-29 | Data local recovery method, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115098295A true CN115098295A (en) | 2022-09-23 |
Family
ID=83295604
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210764325.1A Pending CN115098295A (en) | 2022-06-29 | 2022-06-29 | Data local recovery method, equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN115098295A (en) |
| WO (1) | WO2024001974A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115454712A (en) * | 2022-11-11 | 2022-12-09 | 苏州浪潮智能科技有限公司 | Check code recovery method, system, electronic equipment and storage medium |
| WO2024001974A1 (en) * | 2022-06-29 | 2024-01-04 | 阿里巴巴（中国）有限公司 | Local recovery method and device for data, and storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102109015B1 (en) * | 2018-10-23 | 2020-05-11 | 네이버 주식회사 | Method and system for storing data locally repairable and efficient multiple encoding |
| CN113296695A (en) * | 2021-02-08 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Writing method and device of erasure code data in multi-AZ environment |
| CN113687975B (en) * | 2021-07-14 | 2023-08-29 | 重庆大学 | Data processing method, device, equipment and storage medium |
| CN114385409A (en) * | 2021-12-21 | 2022-04-22 | 阿里巴巴（中国）有限公司 | Encoding method based on erasure code, distributed system, device and storage medium |
| CN115098295A (en) * | 2022-06-29 | 2022-09-23 | 阿里巴巴（中国）有限公司 | Data local recovery method, equipment and storage medium |
-
2022
- 2022-06-29 CN CN202210764325.1A patent/CN115098295A/en active Pending
-
2023
- 2023-06-25 WO PCT/CN2023/102218 patent/WO2024001974A1/en unknown
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024001974A1 (en) * | 2022-06-29 | 2024-01-04 | 阿里巴巴（中国）有限公司 | Local recovery method and device for data, and storage medium |
| CN115454712A (en) * | 2022-11-11 | 2022-12-09 | 苏州浪潮智能科技有限公司 | Check code recovery method, system, electronic equipment and storage medium |
| CN115454712B (en) * | 2022-11-11 | 2023-02-28 | 苏州浪潮智能科技有限公司 | Check code recovery method, system, electronic equipment and storage medium |
| WO2024098647A1 (en) * | 2022-11-11 | 2024-05-16 | 苏州元脑智能科技有限公司 | Check-code recovery method and system, electronic device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2024001974A1 (en) | 2024-01-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104657405B (en) | Method and system for the reliability mechanisms based on priority of filing data | |
| CN115098295A (en) | Data local recovery method, equipment and storage medium | |
| US11531593B2 (en) | Data encoding, decoding and recovering method for a distributed storage system | |
| CN107086870B (en) | Repair the MDS array code coding and coding/decoding method of more piece point failure | |
| US20160378624A1 (en) | Erasure coding of data within a group of storage units based on connection characteristics | |
| US20200319973A1 (en) | Layered error correction encoding for large scale distributed object storage system | |
| CN116501553B (en) | Data recovery method, device, system, electronic equipment and storage medium | |
| CN115080303B (en) | Encoding method, decoding method, device and medium for RAID6 disk array | |
| US20200250034A1 (en) | Data storage methods and systems | |
| CN106484559A (en) | A kind of building method of check matrix and the building method of horizontal array correcting and eleting codes | |
| WO2024001494A1 (en) | Data storage method, single-node server, and device | |
| CN111176880B (en) | Disk allocation method, device and readable storage medium | |
| CN111459710A (en) | Erasure code memory recovery method, device and memory system capable of sensing heat degree and risk | |
| CN108614749B (en) | Data processing method and device | |
| CN114048061A (en) | Check block generation method and device | |
| US20200336157A1 (en) | Systematic and xor-based coding technique for distributed storage systems | |
| WO2023082629A1 (en) | Data storage method and apparatus, electronic device, and storage medium | |
| CN111224747A (en) | Coding method capable of reducing repair bandwidth and disk reading overhead and repair method thereof | |
| CN114691414A (en) | Check block generation method and data recovery method | |
| Yongmei et al. | Large LDPC codes for big data storage | |
| CN107463462B (en) | Data restoration method and data restoration device | |
| CN112015325B (en) | Method for generating decoding matrix, decoding method and corresponding device | |
| CN111984443A (en) | Encoding method, decoding method and corresponding devices in distributed system environment | |
| US20210006269A1 (en) | Techniques for improved erasure coding in distributed storage systems | |
| CN108352845A (en) | Method for being encoded to storage data and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |