CN110750382A - Minimum storage regeneration code coding method and system for improving data repair performance - Google Patents

Minimum storage regeneration code coding method and system for improving data repair performance Download PDF

Info

Publication number
CN110750382A
CN110750382A CN201910880818.XA CN201910880818A CN110750382A CN 110750382 A CN110750382 A CN 110750382A CN 201910880818 A CN201910880818 A CN 201910880818A CN 110750382 A CN110750382 A CN 110750382A
Authority
CN
China
Prior art keywords
data
matrix
block
check
repair
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910880818.XA
Other languages
Chinese (zh)
Other versions
CN110750382B (en
Inventor
冯丹
叶柳青
胡燏翀
魏学亮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huazhong University of Science and Technology
Original Assignee
Huazhong University of Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huazhong University of Science and Technology filed Critical Huazhong University of Science and Technology
Priority to CN201910880818.XA priority Critical patent/CN110750382B/en
Publication of CN110750382A publication Critical patent/CN110750382A/en
Application granted granted Critical
Publication of CN110750382B publication Critical patent/CN110750382B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1076—Parity data used in redundant arrays of independent storages, e.g. in RAID systems
    • H—ELECTRICITY
    • H03—ELECTRONIC CIRCUITRY
    • H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
    • H03M13/00—Coding, decoding or code conversion, for error detection or error correction; Coding theory basic assumptions; Coding bounds; Error probability evaluation methods; Channel models; Simulation or testing of codes
    • H03M13/03—Error detection or forward error correction by redundancy in data representation, i.e. code words containing more digits than the source words
    • H03M13/05—Error detection or forward error correction by redundancy in data representation, i.e. code words containing more digits than the source words using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits
    • H03M13/13—Linear codes
    • H03M13/15—Cyclic codes, i.e. cyclic shifts of codewords produce other codewords, e.g. codes defined by a generator polynomial, Bose-Chaudhuri-Hocquenghem [BCH] codes
    • H03M13/151—Cyclic codes, i.e. cyclic shifts of codewords produce other codewords, e.g. codes defined by a generator polynomial, Bose-Chaudhuri-Hocquenghem [BCH] codes using error location or error correction polynomials
    • H03M13/154—Error and erasure correction, e.g. by using the error and erasure locator or Forney polynomial

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Algebra (AREA)
  • Pure & Applied Mathematics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Quality & Reliability (AREA)
  • General Engineering & Computer Science (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)

Abstract

The invention discloses a minimum storage regeneration code coding method and a minimum storage regeneration code coding system for improving data restoration performance, which belong to the field of computer storage, and comprise the steps of equally dividing original data into k data blocks, equally dividing each data block into α data fragments, equally dividing each check block into α check fragments, coding k multiplied by α data fragments after determining a generating matrix for data coding to obtain each check fragment in each check block, respectively storing the data blocks and the check blocks onto different storage nodes after coding is completed, regularly checking whether a failure block exists, if the total number of the failure blocks is larger than the number m of coding blocks, restoring the data block which is least failed, if only one failure data block exists, requesting a data amount of 1/(d-k +1) from d effective storage nodes which are congested to restore the failure data block, and otherwise, requesting a complete block from m effective storage nodes to restore the failure block.

Description

Minimum storage regeneration code coding method and system for improving data repair performance
Technical Field
The invention belongs to the field of computer storage, and particularly relates to a minimum storage regeneration code encoding method and system for improving data restoration performance.
Background
Erasure codes are a coding method for ensuring data redundancy, which first splits original data into data blocks of equal size, and then codes the data blocks into check blocks. When several data blocks or check blocks are lost, erasure coding techniques can ensure that the original data can still be recovered. The technology is widely applied to a distributed storage system and a cloud storage system to improve the reliability of the system and prevent data inaccessibility caused by disk failure or data loss and the like.
In the conventional erasure coding method, original data with a size of M is equally divided into k data blocks: d0、D1、…、Dk-1After encoding a data block, m encoded blocks are generated: p0、P1、…、Pm-1Thus, n blocks are obtained, and the data blocks and the coding blocks are stored on n different storage nodes. A storage node is a logical abstraction of a storage device, which may be either a disk or a storage server. Erasure coding techniques generally have optimal storage efficiency or Maximum Distance Separable (MDS) properties compared to conventional replica techniques, and can provide comparable system reliability at lower storage overhead. However, erasure codes need to read and transmit data on multiple disks during data recovery, and occupy a large amount of storage resources and network resources, so that they are inferior to copy strategies in terms of performance. Taking (n, k) reed-solomon encoding as an example, as long as the data is repairable, k entire blocks of data blocks or check blocks are required for reconstruction, and the required data amount is M.
The minimum storage regeneration code is a special erasure code, which divides the coding block into finer granularity, and then can realize the optimal repair bandwidth by selectively selecting some segments which can be reused to participate in repair. In addition to the parameters n and k in the conventional erasure code method, the minimum stored regeneration code has a parameter d to represent the number of valid blocks other than the failed block through the network connection at the time of repair. When a single point fails, most of the existing minimum storage regeneration codes are all valid blocks (d is n-1) except for the failed block through network connection, and 1/m of data volume is downloaded from each block according to a certain rule to repair the damaged data, so that the repair bandwidth can be effectively reduced. But in a complex network environment it is impractical to require the participation of all active nodes for each repair. Moreover, the data repair delay depends on the time when the last node returns the data request, and when the system is unstable and a congested node appears or the node is solicited by other tasks to delay sending the data required for repair, the data repair performance of the system is greatly reduced. Generally, when the minimum storage regeneration code is used, the data repair performance needs to be improved in an unstable network environment.
Disclosure of Invention
In view of the shortcomings and needs in the art, the present invention provides a method and system for encoding a minimum stored regeneration code for improving data repair performance, which aims to improve data repair performance in an unstable network environment.
To achieve the above object, according to a first aspect of the present invention, there is provided a minimum stored regeneration code encoding method for improving data repair performance, comprising:
equally dividing the original data into k data blocks D0～Dk-1Equally dividing each data block into α data fragments to obtain k multiplied by α data fragments, equally dividing each check block into α check fragments in m check blocks generated by encoding to generate m multiplied by α check fragments in total;
after a generating matrix G for data coding is determined, coding k multiplied by α data fragments according to the generating matrix G to obtain each check fragment in m check blocks;
regularly checking whether the blocks on each storage node fail;
if the failed blocks exist and the total number of the failed blocks is more than m, the repair is failed and the repair is finished; if only one failed data block exists, requesting data from d effective storage nodes with the least congestion, and downloading data amount of 1/(d-k +1) from each requested node to repair the failed data block; under other conditions, requesting data from m effective storage nodes, and downloading complete blocks from each requested node to repair failed blocks;
wherein n is k + m, k is not less than d and not more than n-1, the value of α is related to d, and k is not less than 2.
In the invention, when data restoration is carried out on a single failure data block, the number d of effective nodes connected through a network can be flexibly set in the range of [ k, n-1] according to the network congestion condition; specifically, under a stable network environment, the invention can construct the minimum storage regeneration code with d being n-1 so as to obtain smaller repair bandwidth than the traditional erasure code; in an unstable network environment, the invention can construct the minimum storage regeneration code with k being more than or equal to d being less than n-1 to adapt to the load condition of the sensing node, and select to request data from the relatively non-congested node, thereby not only obtaining smaller repair bandwidth overhead than the traditional erasure code, but also avoiding requesting data from the congested node, and finally obtaining smaller repair time delay than the minimum storage regeneration code with d being equal to n-1. In general, the present invention can improve data repair performance in an unstable network environment.
Further, the air conditioner is provided with a fan,
Figure BDA0002205838250000031
wherein the content of the first and second substances,indicating rounding up.
The invention is based on
Figure BDA0002205838250000033
Determining the fragmentation number α of the data block and the check block can match the selected valid node number d when the fragmentation condition is invalid with a single data block, thereby improving the subsequent data repair effect.
Further, determining a generator matrix G for data encoding includes:
for any check block PtDetermining all data fragments participating in generating check block PtIn each check fragment Pt,0～Pt,α-1So as to obtain a coding position matrix L, and determining a coding coefficient matrix V corresponding to the data slices participating in codingt(ii) a According to
Figure BDA0002205838250000034
Get the check Block PtCorresponding generator matrix block Gt；
The coding position matrixes corresponding to the check blocks are the same;
combining the generated matrix blocks corresponding to the check blocks to obtain a generated matrix as follows:
Figure BDA0002205838250000041
wherein, omicron represents the Hadamard product, t represents the number of the check block, and t is more than or equal to 0 and less than or equal to m-1.
In the invention, the coding position matrixes corresponding to the check blocks are the same, namely for each check block, all the data fragments are the same as the data fragments for generating the check fragments in the check block, and when d is less than n-1, data can be requested to more than d nodes.
Further, the method for acquiring the encoding position matrix L includes:
(S1) initializing a target matrix of α rows and k × α columns so that all elements in the target matrix are 0, wherein each row in the target matrix corresponds to a check block PtEach column corresponds to one data fragment in all the data fragments;
(S2) setting the j-th in the object matrix1Line, i-th1×α+j1The elements of the column being non-zero values to convert the ith1J in each data block1Data slicing
Figure BDA0002205838250000042
Adding into the mixture;
(S3) setting variables
Figure BDA0002205838250000043
(S4) if i is k, the process proceeds to step (S8); otherwise, updating the value of the variable i according to i-i +1 and setting the variable
Figure BDA0002205838250000044
r＝i mod(d-k+1)，
Figure BDA0002205838250000045
Figure BDA0002205838250000046
j is-1, and the column sequence number S of the data fragment is positioned according to the variable r and runi＝r*run；
(S5) if j is step, the process proceeds to step (S4); otherwise, updating the value of the variable j according to j ═ j +1, and setting the variable tmp ═ 1;
(S6) if tmp is run, proceeding to step (S5); otherwise, updating the value of the variable tmp according to tmp +1, and calculating the variable pos as j (d-k +1) run + tmp after setting the variable ex as-1;
(S7) if ex is d-k, the process proceeds to step (S6); otherwise, updating the value of the variable ex according to the ex +1, and setting the pos + ex dis row and the S th row in the target matrixiThe elements of the column are non-zero values to slice the ith pos + ex dis data in the ith data block into Di,pos+ex*disAdding the above-mentioned material, and repeating the step (S7);
(S8) determining the target matrix as an encoding position matrix L;
wherein i is more than or equal to 01≤k-1，0≤j1≤α-1，
Figure BDA0002205838250000051
Meaning rounding up and mod denotes a modulo operation.
The invention uses the coding position matrix L determined by the method to ensure that all data fragments participate in coding to obtain the check block, and simultaneously, adds partial information in the data fragments as redundant information to the check information part, so that more data information can be obtained by only reading less check information during repair, thereby reducing the repair bandwidth.
Further, the coefficient matrix V is encodedtThe obtaining method comprises the following steps:
searching all linearly independent coding coefficient matrixes of a linear equation set obtained by randomly selecting k blocks by an enumeration method so as to obtain a coding coefficient matrix set;
selecting a coding coefficient matrix from the coding coefficient matrix set, so that when a single failure data block is repaired, linear equations obtained by requesting data from d effective storage nodes with the least congestion are linearly independent; determining the selected coding coefficient matrix as a coding coefficient matrix Vt。
The invention determines the coding coefficient matrix V by the methodtThe linear equation system obtained by acquiring any k blocks in n blocks is ensured to be linearly independent (equivalent to matrix reversibility), so that the Maximum Distance Separable (MDS) property can be achieved; at the same time, the coding coefficient matrix V determined by the above methodtAnd the matrix obtained after the selected single data block repairing scheme is reversible is also ensured.
Further, if there is only one failed data block, requesting data from the least congested d valid storage nodes, and downloading a data amount of 1/(d-k +1) from each requested node to repair the failed data block, respectively, includes:
determining the fragment position information participating in repairing the failed data block in the valid block of the requested node to determine a data fragment sequence number seq1 and a check fragment sequence number seq2 participating in repairing the failed data block;
the data fragments and the check fragments which participate in the repair are combined into a matrix M in sequence1Combining the data fragments participating in repair and the data fragments to be repaired in the failure data block into a matrix M in sequence2；
Selecting from the generator matrix G according to the sequence number seq2 of the check fragmentCorresponding rows are combined in sequence to obtain a matrix Mf(ii) a Determining the sequence number seq3 of the data fragment to be repaired according to the number of the data block to be repaired, and determining the sequence number seq3 of the slave matrix M according to the sequence number seq1 of the data fragment and the sequence number seq3 of the data fragmentfCorresponding columns are selected and combined in sequence to obtain a matrix Mf′；
According to matrix Mf' construction matrix MtSo that Mt·M2＝M1(ii) a Matrix MfIs a matrix MtOne matrix block of;
for matrix MtAfter inversion, a repair matrix M is obtainedr(ii) a Using a repair matrix MrSum matrix M1And repairing each data fragment to be repaired, thereby completing the repair of the failed data block.
Further, determining the fragment location information participating in repairing the failed data block in the valid block located in the requested node includes:
(T1) initialize the set R as an empty set and set the variables
Figure BDA0002205838250000061
r＝f mod(d-k+1)，
Figure BDA0002205838250000062
Sf＝r*run，j＝-1；
(T2) if j is step, the process proceeds to step (T4); otherwise, updating the value of the variable j according to j ═ j +1, and setting the variable tmp ═ 1;
(T3) if tmp is run, proceeding to step (T2); otherwise, setting a variable pos (j) (d-k +1) run + tmp, updating the value of the variable tmp according to the tmp (tmp +1), and calculating a slicing position Ps (S)f+ pos and adding into the set R to indicate the Ps-th fragment in the valid block to participate in repairing the failed data block, and then proceeding to the step (T3);
(T4) determining the fragment position recorded in the set R as the fragment position information in the valid block, which is involved in repairing the failed data block;
where f denotes the number of the invalid data block.
Further, requesting data from m valid storage nodes, and downloading complete blocks from each requested node to repair failed blocks respectively, includes:
after generating the unit matrix of k × α rows and k × α columns, the unit matrix is combined with the generator matrix G to form a matrix M of (k + M) × α rows and k × α columns such that M · MD＝MN；MDMatrix composed of all data fragments combined in sequence, MNA matrix formed by combining all the data fragments and the check fragments in sequence;
according to the number of the requested node, determining the fragment serial number for repairing the failure block, selecting corresponding rows from the matrix M, and combining in sequence to obtain the matrix MtSo that Mt·MD＝MV；MVCombining all fragments for repairing the failure block into a matrix in sequence;
for matrix MtAfter inversion, a repair matrix M is obtainedr(ii) a Using a repair matrix MrSum matrix MVAnd repairing the data fragments in each failure block so as to finish repairing the failure blocks.
According to a second aspect of the present invention, there is provided a system comprising a computer readable storage medium for storing an executable program and a processor;
the processor is used for reading an executable program stored in a computer readable storage medium and executing the minimum storage regeneration code encoding method for improving the data repair performance provided by the first aspect of the invention.
Generally, by the above technical solution conceived by the present invention, the following beneficial effects can be obtained:
(1) according to the minimum storage regeneration code coding method for improving the data restoration performance, when data restoration is carried out on a single failure data block, the number d of effective nodes connected through a network can be flexibly set within the range of [ k, n-1] according to the network congestion condition; specifically, under a stable network environment, the invention can construct the minimum storage regeneration code with d being n-1 so as to obtain smaller repair bandwidth than the traditional erasure code; in an unstable environment, the invention can construct the minimum storage regeneration code with k being less than or equal to d < n-1 to adapt to the load condition of the sensing node, and select to request data from the relatively non-congested node, thereby not only obtaining smaller repair bandwidth overhead than the traditional erasure code, but also avoiding requesting data from the congested node, and finally obtaining smaller repair time delay than the minimum storage regeneration code with d being equal to n-1. In general, the present invention can improve data repair performance in an unstable network environment.
(2) According to the minimum storage regeneration code coding method for improving the data repair performance, the coding position matrixes corresponding to all check blocks are the same, when d is less than n-1, data can be requested from more than d nodes, and because the data position information transmitted by the nodes is the same, the data can be repaired only by waiting for the return of d data before, and the waiting of data information with high delay is avoided.
(3) According to the minimum storage regeneration code coding method for improving the data restoration performance, the constructed coding position matrix L ensures that all data fragments participate in coding to obtain the check block, and meanwhile, part of information in the data fragments is added to the check information part as redundant information, so that when restoration is carried out, more data information can be obtained by only reading less check information, and the restoration bandwidth can be reduced.
Drawings
FIG. 1 is a flowchart of a minimum stored regeneration code encoding method for improving data repair performance according to an embodiment of the present invention;
FIG. 2 is a flowchart of a method for determining a code location matrix according to an embodiment of the present invention;
fig. 3 is a flowchart of a method for determining fragment location information participating in repairing a failed data block when repairing a single data block in an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
In order to improve the data repair performance in an unstable network environment, the present invention provides a minimum storage regeneration code encoding method for improving the data repair performance, as shown in fig. 1, including:
equally dividing the original data into k data blocks D0～Dk-1Equally dividing each data block into α data fragments to obtain k multiplied by α data fragments, equally dividing each check block into α check fragments in m check blocks generated by encoding to generate m multiplied by α check fragments, filling with 0 if the data is insufficient in the fragmentation process, and completing the data fragmentation and fragmentation operation through the step;
after determining a generating matrix G for data coding, coding k multiplied by α data fragments according to the generating matrix G to obtain each check fragment in m check blocks, and after the coding is finished, respectively storing the k data blocks and the m check blocks on n different storage nodes;
regularly checking whether the blocks on each storage node fail; by this step, the operation of checking the state of the data block can be completed;
if the failed blocks exist and the total number of the failed blocks is more than m, the repair is failed and the repair is finished; if only one failed data block exists, requesting data from d effective storage nodes with the least congestion, and downloading data amount of 1/(d-k +1) from each requested node to repair the failed data block; under other conditions, requesting data from m effective storage nodes, and downloading complete blocks from each requested node to repair failed blocks; through the step, a corresponding failed block repair scheme can be determined according to the specific failure condition, so that the steps of failed block repair division and failure repair are completed;
wherein n is k + m, k is not less than d and not more than n-1, the value of α is related to d, and k is not less than 2.
According to the minimum storage regeneration code coding method for improving the data repair performance, when data repair is carried out on a single failure data block, the number d of effective nodes connected through a network can be flexibly set within the range of [ k, n-1] according to the network congestion condition, and specifically, under the stable network environment, a minimum storage regeneration code with d being n-1 can be constructed so as to obtain a repair bandwidth smaller than that of a traditional erasure code; in an unstable network environment, a minimum storage regeneration code with k being more than or equal to d < n-1 can be constructed to adapt to the load condition of a sensing node, and data is selected to be requested to a node which is relatively not congested, so that repair bandwidth overhead smaller than that of a traditional erasure code can be obtained, data can be prevented from being requested to a congested node, and repair time delay smaller than that of the minimum storage regeneration code with d being equal to n-1 is finally obtained. In summary, the above-mentioned minimum storage regeneration code encoding method for improving data repair performance can improve data repair performance in an unstable network environment.
Late binding technology (late binding protocol) does not need to know the congestion condition of nodes in the system, and can effectively avoid waiting for other nodes with high delay to return messages by requesting data from more than d nodes and starting to repair the data when the first d nodes are obtained. Dynamic data access technology (dynamic data access technology) mentioned in a paper (EC-Store: weighting the Gap between Store and latency Distributed implementation Systems, International Conference on Distributed computing Systems (ICDCS), 2018) judges the corresponding delay condition and whether the nodes in the network are blocked or not by modeling the system and acquiring the corresponding network and load information. In the present invention, the request of data to a node having a high latency can be avoided by the above-described technique, and it should be understood that other methods of avoiding the request of data to a node having a high latency can be applied to the present invention.
Optionally, in the above minimum storage regeneration code encoding method for improving data repair performance, the number of the data blocks and the number of the check blocks are divided into two
Figure BDA0002205838250000111
Wherein the content of the first and second substances,
Figure BDA0002205838250000112
represents rounding up;
the present embodiment is based on
Figure BDA0002205838250000113
Determining the fragmentation number α of the data block and the check block can match the selected valid node number d when the fragmentation condition is invalid with a single data block, thereby improving the subsequent data repair effect.
In an optional embodiment, in the method for encoding a minimum stored reproduction code for improving data repair performance, determining a generator matrix G for data encoding includes:
for any check block PtDetermining all data fragments participating in generating check block PtIn each check fragment Pt,0～Pt,α-1So as to obtain a coding position matrix L, and determining a coding coefficient matrix V corresponding to the data slices participating in codingt(ii) a According to Gt＝VtObtaining an o-L check block PtCorresponding generator matrix block Gt；
The coding position matrixes corresponding to the check blocks are the same;
combining the generated matrix blocks corresponding to the check blocks to obtain a generated matrix as follows:
Figure BDA0002205838250000114
wherein a represents a Hadamard product, t represents the number of the check block, and t is greater than or equal to 0 and less than or equal to m-1;
by enabling the coding position matrixes corresponding to the check blocks to be the same, namely for each check block, all data fragments are the same as the data fragments for generating the check fragments in the check blocks, when d is less than n-1, data can be requested to more than d nodes, and because the data position information transmitted by the nodes is the same, the data can be repaired only by waiting for the previous d data to return, so that the waiting for the data information with high delay is avoided;
as shown in fig. 2, when determining the generator matrix G, the method for acquiring the encoding position matrix L includes:
(S1) initializing a target matrix of α rows and k × α columns so that all elements in the target matrix are 0, wherein each row in the target matrix corresponds to a check block PtEach column corresponds to one data fragment in all the data fragments;
(S2) setting the j-th in the object matrix1Line, i-th1×α+j1The elements of the column being non-zero values to convert the ith1J in each data block1Data slicing
Figure BDA0002205838250000121
Adding into the mixture;
(S3) setting variables
Figure BDA0002205838250000122
(S4) if i is k, the process proceeds to step (S8); otherwise, updating the value of the variable i according to i-i +1 and setting the variable
Figure BDA0002205838250000123
r＝imod(d-k+1)，
Figure BDA0002205838250000124
Figure BDA0002205838250000125
j is-1, and the column sequence number S of the data fragment is positioned according to the variable r and runi＝r*run；
(S5) if j is step, the process proceeds to step (S4); otherwise, updating the value of the variable j according to j ═ j +1, and setting the variable tmp ═ 1;
(S6) if tmp is run, proceeding to step (S5); otherwise, updating the value of the variable tmp according to tmp +1, and calculating the variable pos as j (d-k +1) run + tmp after setting the variable ex as-1;
(S7) if ex is d-k, the process proceeds to step (S6); otherwise, updating the value of the variable ex according to the ex +1, and setting the pos + ex dis row and the S th row in the target matrixiThe elements of the column are non-zero values to set the ith in the ith data blockpos + ex dis data slices Di,pos+ex*disAdding the above-mentioned material, and repeating the step (S7);
(S8) determining the target matrix as an encoding position matrix L;
wherein i is more than or equal to 01≤k-1，0≤j1≤α-1，
Figure BDA0002205838250000126
Represents rounding up, mod represents a modulo operation;
by the coding position matrix L determined by the method, all data fragments are ensured to participate in coding to obtain the check block, and meanwhile, part of information in the data fragments is added to the check information part as redundant information, so that more data information can be obtained by only reading less check information during repair, and the repair bandwidth can be reduced;
after determining the coding position matrix L, the coefficient matrix V is codedtThe obtaining method comprises the following steps:
searching all linearly independent coding coefficient matrixes of a linear equation set obtained by randomly selecting k blocks by an enumeration method so as to obtain a coding coefficient matrix set;
selecting a coding coefficient matrix from the coding coefficient matrix set, so that when a single failure data block is repaired, linear equations obtained by requesting data from d effective storage nodes with the least congestion are linearly independent; determining the selected coding coefficient matrix as a coding coefficient matrix Vt；
Determining a coding coefficient matrix V by the above methodtThe linear equation system obtained by acquiring any k blocks in n blocks is ensured to be linearly independent (equivalent to matrix reversibility), so that the Maximum Distance Separable (MDS) property can be achieved; at the same time, the coding coefficient matrix V determined by the above methodtAnd the matrix obtained after the selected single data block repairing scheme is reversible is also ensured.
In an optional embodiment, in the above method for encoding a minimum storage regeneration code to improve data repair performance, if there is only one failed data block, the method requests data from d least congested valid storage nodes, and downloads a data amount of 1/(d-k +1) from each requested node to repair the failed data block, including:
determining the fragment position information participating in repairing the failed data block in the valid block of the requested node to determine a data fragment sequence number seq1 and a check fragment sequence number seq2 participating in repairing the failed data block;
the data fragments and the check fragments which participate in the repair are combined into a matrix M in sequence1Combining the data fragments participating in repair and the data fragments to be repaired in the failure data block into a matrix M in sequence2；
Selecting corresponding rows from the generator matrix G according to the sequence number seq2 of the check fragment, and combining the rows in sequence to obtain a matrix Mf(ii) a Determining the sequence number seq3 of the data fragment to be repaired according to the number of the data block to be repaired, and determining the sequence number seq3 of the slave matrix M according to the sequence number seq1 of the data fragment and the sequence number seq3 of the data fragmentfCorresponding columns are selected and combined in sequence to obtain a matrix Mf′；
According to matrix Mf' construction matrix MtSo that Mt·M2＝M1(ii) a Matrix MfIs a matrix MtOne matrix block of;
for matrix MtAfter inversion, a repair matrix M is obtainedr(ii) a Using a repair matrix MrSum matrix M1Repairing each data fragment to be repaired, thereby completing the repair of the failed data block;
specifically, as shown in fig. 3, determining the fragment location information of the valid block located in the requested node, which participates in repairing the failed data block, includes:
(T1) initialize the set R as an empty set and set the variablesr＝f mod(d-k+1)，
Figure BDA0002205838250000142
Sf＝r*run，j＝-1；
(T2) if j is step, the process proceeds to step (T4); otherwise, updating the value of the variable j according to j ═ j +1, and setting the variable tmp ═ 1;
(T3) if tmp is run, proceeding to step (T2); otherwise, setting a variable pos (j) (d-k +1) run + tmp, updating the value of the variable tmp according to the tmp (tmp +1), and calculating a slicing position Ps (S)f+ pos and adding into the set R to indicate the Ps-th fragment in the valid block to participate in repairing the failed data block, and then proceeding to the step (T3);
(T4) determining the fragment position recorded in the set R as the fragment position information in the valid block, which is involved in repairing the failed data block;
where f denotes the number of the invalid data block.
In an optional embodiment, in the above method for encoding a minimum storage regeneration code for improving data repair performance, requesting data from m valid storage nodes, and downloading complete blocks from each requested node to repair a failed block respectively includes:
after generating the unit matrix of k × α rows and k × α columns, the unit matrix is combined with the generator matrix G to form a matrix M of (k + M) × α rows and k × α columns such that M · MD＝MN；MDMatrix composed of all data fragments combined in sequence, MNA matrix formed by combining all the data fragments and the check fragments in sequence;
according to the number of the requested node, determining the fragment serial number for repairing the failure block, selecting corresponding rows from the matrix M, and combining in sequence to obtain the matrix MtSo that Mt·MD＝MV；MVCombining all fragments for repairing the failure block into a matrix in sequence;
for matrix MtAfter inversion, a repair matrix M is obtainedr(ii) a Using a repair matrix MrSum matrix MVAnd repairing the data fragments in each failure block so as to finish repairing the failure blocks.
The invention also provides a system comprising a computer-readable storage medium and a processor, the computer-readable storage medium for storing an executable program;
the processor is used for reading an executable program stored in a computer readable storage medium and executing the minimum storage regeneration code encoding method for improving the data repair performance provided by the first aspect of the invention.
Application example:
the minimum storage regeneration code encoding method for improving data repair performance is further explained below with reference to a specific example.
Operation GF (2) in the finite field8) In the following, m is set to 3, k to 3, and d to 4 according to the network conditions, and accordingly, n is set to k + m to 6,
Figure BDA0002205838250000151
for original data with a data size of 30MB, the following operations are performed:
(1) data blocking:
equally dividing original data with the data size of 30MB into 3 data blocks D of 10MBi(i is 0,1,2), and then 3 data blocks are stored in 3 data nodes N, respectively0，N1，N2Divide each data chunk equally into 4 data fragments according to α, and assign sequence numbers to all data fragments, where data chunk DiThe serial number of the jth data fragment in (1) is i multiplied by 4+ j;
three check blocks P to be generated0、P1、P2Each check block in the check block is equally divided into 4 check fragments;
(2) and (3) data encoding:
according to the method flow shown in fig. 2, the encoding position matrix is determined as follows:
Figure BDA0002205838250000161
each row corresponds to one check fragment in the check block, and each column corresponds to one data fragment in all the data fragments; the coding position matrixes corresponding to the check blocks are the same;
and randomly searching the coding coefficient matrix meeting the MDS property in the coefficient matrix V according to an enumeration method to obtain the coding coefficient matrix corresponding to each check block, so that a generation matrix G can be obtained as follows:
Figure BDA0002205838250000162
the data fragments are encoded according to the generator matrix G, and encoded check block information P can be obtained0、P1、P2Comprises the following steps:
Figure BDA0002205838250000171
storing the three check blocks into different data nodes N respectively0，N1，N2N on three data nodes3，N4，N5；
(3) Checking the data block state:
checking whether the blocks on each data node are in error or lost regularly and sequentially, and if so, turning to the step (4); otherwise, not processing;
(4) repairing and dividing according to the failure blocks:
acquiring the numbers of all failed data blocks and check blocks according to the state check, and generating a lost block set; when the number of the failure blocks exceeds m to 3, indicating that the data is completely lost and cannot be recovered outside the repairable failure range, failing to repair, and ending the repair; if only one failed data block exists, executing step (5.2) to request data from the d least congested effective storage nodes, and downloading data amount of 1/(d-k +1) from each requested node to repair the failed data block; in other cases, namely in the recoverable failure range, when the failed check block exists or the number of the failed data blocks is more than 1, executing the step (5.1) to request data from m effective storage nodes and download the complete block from each requested node to recover the failed block;
(5) and (3) failure repair:
(5.1) conventional repair:
for example, there is a failure set of 0,1 after the status checking step, i.e., there are two failed data blocks D0And D1Randomly selecting 3 effective storage nodes to request dataHere, the selection block {2,3,4} is selected to complete data repair;
first, an identity matrix I of 12 rows and 12 columns is generated, and then combined with a generator matrix G to form a matrix of 24 rows and 24 columns
Figure BDA0002205838250000181
So that M.MD＝MN；MDMatrix composed of all data fragments combined in sequence, MNA matrix formed by combining all the data fragments and the check fragments in sequence;
according to the number of the requested node, determining the fragmentation sequence numbers {8,9,10,11, 12,13,14,15,16,17,18,19} of the failed blocks, selecting corresponding rows from the matrix M, and combining the rows in sequence to obtain the matrix MtSo that Mt·MD＝MV；MVCombining all fragments for repairing the failure block into a matrix in sequence; wherein the content of the first and second substances,
Figure BDA0002205838250000182
for matrix MtAfter inversion, a repair matrix M is obtainedr：
Figure BDA0002205838250000191
Using a repair matrix MrSum matrix MVRepairing the data fragments in each failure block so as to complete the repair of the failure blocks;
due to MD＝Mr·MVBy repairing the matrix MrSum matrix MVThe lost data block can be repaired by the product of the following steps:
(5.2) single data block repair:
for example, there is a failure set 2 after the status check step, i.e., there is only one failureData block D of2Optionally selecting 4 nodes from all the node numbers {0,1,3,4,5} to participate in repair, where the selected node number is {0,1,3,4 };
according to the method flow shown in fig. 3, it can be obtained that the position of the partition participating in the repair in each block in the block is {0,2}, and therefore, the data partition participating in the repair of the failed data block is specifically { D }0,0D0,2,D1,0,D1,2And the corresponding data fragment sequence number seq1 ═ 0,2,4,6, and the check fragment participating in repairing the failed data block is { P }0,0,P0,2,P1,0,P1,2The corresponding check fragment sequence number seq2 ═ 0,2,4,6 };
the data fragments and the check fragments which participate in the repair are combined into a matrix M in sequence1Combining the data fragments participating in repair and the data fragments to be repaired in the failure data block into a matrix M in sequence2；
Selecting corresponding rows from the generator matrix G according to the sequence number seq2 of the check fragment, and combining the rows in sequence to obtain a matrix Mf：
Figure BDA0002205838250000201
Determining the data fragment sequence number seq3 to be repaired to be {8,9,10,11} according to the number 2 of the data block to be repaired, and selecting the matrix M according to the data fragment sequence number seq1 and the data fragment sequence number seq3fTo select a corresponding column, i.e. from the matrix MfSelected 0,2,4,6, 8,9,10 and 11 columns, and then combined in sequence to obtain a matrix Mf′：
Figure BDA0002205838250000202
According to matrix Mf' construction matrix MtSo that Mt·M2＝M1(ii) a Matrix MfIs a matrix MtOne matrix block of; wherein the content of the first and second substances,
Figure BDA0002205838250000211
for matrix MtAfter inversion, a repair matrix M is obtainedr：
Figure BDA0002205838250000212
Using a repair matrix MrSum matrix M1Repairing each data fragment to be repaired, thereby completing the repair of the failed data block;
due to M2＝Mr·M1By repairing the matrix MrSum matrix M1The lost data block can be repaired by the product of the following steps:
it will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims (9)

1. A method for encoding a minimal stored regeneration code for improving data repair performance, comprising:
equally dividing the original data into k data blocks D0～Dk-1Equally dividing each data block into α data fragments to obtain k multiplied by α data fragments, equally dividing each check block into α check fragments in m check blocks generated by encoding to generate m multiplied by α check fragments in total;
after a generating matrix G for data coding is determined, coding k multiplied by α data fragments according to the generating matrix G to obtain each check fragment in m check blocks;
regularly checking whether the blocks on each storage node fail;
if the failed blocks exist and the total number of the failed blocks is more than m, the repair is failed and the repair is finished; if only one failed data block exists, requesting data from d effective storage nodes with the least congestion, and downloading data amount of 1/(d-k +1) from each requested node to repair the failed data block; under other conditions, requesting data from m effective storage nodes, and downloading complete blocks from each requested node to repair failed blocks;
wherein n is k + m, k is not less than d and not more than n-1, the value of α is related to d, and k is not less than 2.
2. The minimum stored regeneration code encoding method for improving data repair performance as claimed in claim 1,
wherein the content of the first and second substances,
Figure FDA0002205838240000012
indicating rounding up.
3. The minimum stored regenerative code encoding method for improving data repair performance according to claim 1 or 2, wherein determining a generator matrix G for data encoding comprises:
for any check block PtDetermining all data fragments participating in generating check block PtIn each check fragment Pt,0～Pt,α-1So as to obtain a coding position matrix L, and determining a coding coefficient matrix V corresponding to the data slices participating in codingt(ii) a According to
Figure FDA0002205838240000021
Get the check Block PtCorresponding generator matrix block Gt；
The coding position matrixes corresponding to the check blocks are the same;
combining the generated matrix blocks corresponding to the check blocks to obtain a generated matrix as follows:
Figure FDA0002205838240000022
wherein the content of the first and second substances,
Figure FDA0002205838240000023
representing the Hadamard product, t represents the number of the check block, and t is more than or equal to 0 and less than or equal to m-1.
4. The minimum stored regenerating code encoding method for improving data repair performance according to claim 3, wherein the method for obtaining the encoding position matrix L includes:
(S1) initializing a target matrix of α rows and k × α columns such that all elements therein are 0, wherein each row in the target matrix corresponds to a parity check block PtEach column corresponds to one data fragment in all the data fragments;
(S2) setting the j-th in the object matrix1Line, i-th1×α+j1The elements of the column being non-zero values to convert the ith1J in each data block1Data slicing
Figure FDA0002205838240000024
Adding into the mixture;
(S3) setting variables
Figure FDA0002205838240000025
i＝-1；
(S4) if i is k, the process proceeds to step (S8); otherwise, updating the value of the variable i according to i-i +1 and setting the variabler＝i mod(d-k+1)，
Figure FDA0002205838240000027
Figure FDA0002205838240000028
j is-1, and the column sequence number S of the data fragment is positioned according to the variable r and runi＝r*run；
(S5) if j is step, the process proceeds to step (S4); otherwise, updating the value of the variable j according to j ═ j +1, and setting the variable tmp ═ 1;
(S6) if tmp is run, proceeding to step (S5); otherwise, updating the value of the variable tmp according to tmp +1, and calculating the variable pos as j (d-k +1) run + tmp after setting the variable ex as-1;
(S7) if ex is d-k, the process proceeds to step (S6); otherwise, updating the value of the variable ex according to the ex +1, and setting the pos + ex dis row and the S th row in the target matrixiThe elements of the column are non-zero values to slice the ith pos + ex dis data in the ith data block into Di,pos+ex*disAdding the above-mentioned material, and repeating the step (S7);
(S8) determining the target matrix as an encoding position matrix L;
wherein i is more than or equal to 01≤k-1，0≤j1≤α-1，Meaning rounding up and mod denotes a modulo operation.
5. The minimum-stored-regeneration-code encoding method for improving data repair performance of claim 4, wherein the encoding coefficient matrix VtThe obtaining method comprises the following steps:
searching all linearly independent coding coefficient matrixes of a linear equation set obtained by randomly selecting k blocks by an enumeration method so as to obtain a coding coefficient matrix set;
selecting a coding coefficient matrix from the coding coefficient matrix set, so that when a single failure data block is repaired, linear equations obtained by requesting data from d effective storage nodes with the least congestion are linearly independent; determining the selected coding coefficient matrix as a coding coefficient matrix Vt。
6. The minimum storage regeneration code encoding method for improving data repair performance as claimed in claim 3, wherein if there is only one failed data block, requesting data from d least congested valid storage nodes, downloading a data amount of 1/(d-k +1) from each requested node to repair the failed data block respectively, comprises:
determining the fragment position information participating in repairing the failed data block in the valid block of the requested node to determine a data fragment sequence number seq1 and a check fragment sequence number seq2 participating in repairing the failed data block;
the data fragments and the check fragments which participate in the repair are combined into a matrix M in sequence1Combining the data fragments participating in the repair and the data fragments to be repaired in the failure data block into an array M in sequence2；
Selecting corresponding rows from the generator matrix G according to the sequence number seq2 of the check fragment, and combining the rows in sequence to obtain a matrix Mf(ii) a Determining the sequence number seq3 of the data fragment to be repaired according to the number of the data block to be repaired, and determining the sequence number seq3 of the slave matrix M according to the sequence number seq1 of the data fragment and the sequence number seq3 of the data fragmentfCorresponding columns are selected and combined in sequence to obtain a matrix Mf′；
According to matrix Mf' construction matrix MtSo that Mt·M2＝M1(ii) a Matrix MfIs a matrix MtOne matrix block of;
for matrix MtAfter inversion, a repair matrix M is obtainedr(ii) a Using a repair matrix MrSum matrix M1And repairing each data fragment to be repaired, thereby completing the repair of the failed data block.
7. The method of claim 6, wherein determining fragmentation location information associated with repairing failed data blocks in valid blocks located in a requested node comprises:
(T1) initialize the set R as an empty set and set the variables
Figure FDA0002205838240000041
r＝f mod(d-k+1)，
Figure FDA0002205838240000042
Sf＝r*run，j＝-1；
(T2) if j is step, the process proceeds to step (T4); otherwise, updating the value of the variable j according to j ═ j +1, and setting the variable tmp ═ 1;
(T3) if tmp is run, proceeding to step (T2); otherwise, setting a variable pos (j) (d-k +1) run + tmp, updating the value of the variable tmp according to the tmp (tmp +1), and calculating a slicing position Ps (S)f+ pos and adding into the set R to indicate the Ps-th fragment in the valid block to participate in repairing the failed data block, and then proceeding to the step (T3);
(T4) determining the fragment position recorded in the set R as the fragment position information in the valid block, which is involved in repairing the failed data block;
where f denotes the number of the invalid data block.
8. The method of claim 3, wherein requesting data from m active storage nodes, downloading complete blocks from each requested node to repair failed blocks, respectively, comprises:
after generating the unit matrix of k × α rows and k × α columns, the unit matrix is combined with the generator matrix G to form a matrix M of (k + M) × α rows and k × α columns such that M · MD＝MN；MDMatrix composed of all data fragments combined in sequence, MNA matrix formed by combining all the data fragments and the check fragments in sequence;
according to the number of the requested node, determining the fragment serial number for repairing the failure block, selecting corresponding rows from the matrix M, and combining in sequence to obtain the matrix MtSo that Mt·MD＝MV；MVCombining all fragments for repairing the failure block into a matrix in sequence;
for matrix MtAfter inversion, a repair matrix M is obtainedr(ii) a Benefit toUsing a repair matrix MrSum matrix MVAnd repairing the data fragments in each failure block so as to finish repairing the failure blocks.
9. A system comprising a computer-readable storage medium and a processor, wherein the computer-readable storage medium is configured to store an executable program;
the processor is used for reading an executable program stored in the computer readable storage medium and executing the minimum storage regeneration code encoding method for improving data repair performance of any one of claims 1 to 8.
CN201910880818.XA 2019-09-18 2019-09-18 Minimum storage regeneration code coding method and system for improving data repair performance Active CN110750382B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910880818.XA CN110750382B (en) 2019-09-18 2019-09-18 Minimum storage regeneration code coding method and system for improving data repair performance

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910880818.XA CN110750382B (en) 2019-09-18 2019-09-18 Minimum storage regeneration code coding method and system for improving data repair performance

Publications (2)

Publication Number Publication Date
CN110750382A true CN110750382A (en) 2020-02-04
CN110750382B CN110750382B (en) 2020-10-30

Family

ID=69276641

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910880818.XA Active CN110750382B (en) 2019-09-18 2019-09-18 Minimum storage regeneration code coding method and system for improving data repair performance

Country Status (1)

Country Link
CN (1) CN110750382B (en)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111447044A (en) * 2020-03-10 2020-07-24 深圳市大数据研究院 Distributed storage method and transmission decoding method
CN111475839A (en) * 2020-04-06 2020-07-31 华中科技大学 Redundant data coding method and storage medium for untrusted environment
CN111585581A (en) * 2020-05-14 2020-08-25 成都信息工程大学 Coding method based on binary domain operation and supporting any code distance
CN111585582A (en) * 2020-05-14 2020-08-25 成都信息工程大学 Coding method based on array operation and freely determined code distance
CN112713964A (en) * 2020-12-22 2021-04-27 潍柴动力股份有限公司 Data verification acceleration method and device, computer equipment and storage medium
CN112732203A (en) * 2021-03-31 2021-04-30 中南大学 Regeneration code construction method, file reconstruction method and node repair method
CN113315753A (en) * 2021-04-25 2021-08-27 国网浙江省电力有限公司电力科学研究院 Block data credibility recovery method based on coding technology
CN114237967A (en) * 2022-02-22 2022-03-25 阿里云计算有限公司 Data reconstruction method and device
CN114595092A (en) * 2022-04-28 2022-06-07 阿里云计算有限公司 Distributed storage system, data reconstruction method, device and storage medium
WO2024037010A1 (en) * 2022-08-17 2024-02-22 华为技术有限公司 Data storage method and apparatus, and storage medium
CN117632041A (en) * 2024-01-25 2024-03-01 国家计算机网络与信息安全管理中心天津分中心 Distributed storage method and device based on regenerated codes and electronic equipment

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103688515A (en) * 2013-03-26 2014-03-26 北京大学深圳研究生院 Method for encoding minimum bandwidth regeneration codes and repairing storage nodes
CN103688514A (en) * 2013-02-26 2014-03-26 北京大学深圳研究生院 Coding method for minimum storage regeneration codes and method for restoring of storage nodes
CN105260259A (en) * 2015-09-16 2016-01-20 长安大学 System minimum storage regeneration code based local repair encoding method
US20160026543A1 (en) * 2014-07-24 2016-01-28 At&T Intellectual Property I, L.P. Distributed Storage of Data
CN105721611A (en) * 2016-04-15 2016-06-29 西南交通大学 General method for generating minimal storage regenerating code with maximum distance separable storage code
CN106776129A (en) * 2016-12-01 2017-05-31 陕西尚品信息科技有限公司 A kind of restorative procedure of the multinode data file based on minimum memory regeneration code
CN107395207A (en) * 2017-07-12 2017-11-24 东莞理工学院 The MDS array codes coding and restorative procedure of more fault-tolerances
CN108810063A (en) * 2017-11-21 2018-11-13 北京大学深圳研究生院 Secure distribution and restorative procedure, the system and medium of data under a kind of cloudy storage environment
CN108923960A (en) * 2018-06-20 2018-11-30 华中科技大学 A kind of memory node restorative procedure for assisting regeneration code based on agency
CN109257050A (en) * 2018-08-09 2019-01-22 东莞理工学院 A kind of reparation binary code generator matrix building method and restorative procedure
CN109257049A (en) * 2018-08-09 2019-01-22 东莞理工学院 A kind of building method and restorative procedure for repairing binary array code check matrix
US20190140667A1 (en) * 2017-11-03 2019-05-09 Board Of Trustees Of Michigan State University Optimal construction of regenerating code through rate-matching

Patent Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103688514A (en) * 2013-02-26 2014-03-26 北京大学深圳研究生院 Coding method for minimum storage regeneration codes and method for restoring of storage nodes
CN103688515A (en) * 2013-03-26 2014-03-26 北京大学深圳研究生院 Method for encoding minimum bandwidth regeneration codes and repairing storage nodes
WO2014153716A1 (en) * 2013-03-26 2014-10-02 北京大学深圳研究生院 Methods for encoding minimum bandwidth regenerating code and repairing storage node
US20160026543A1 (en) * 2014-07-24 2016-01-28 At&T Intellectual Property I, L.P. Distributed Storage of Data
CN105260259A (en) * 2015-09-16 2016-01-20 长安大学 System minimum storage regeneration code based local repair encoding method
CN105721611A (en) * 2016-04-15 2016-06-29 西南交通大学 General method for generating minimal storage regenerating code with maximum distance separable storage code
CN106776129A (en) * 2016-12-01 2017-05-31 陕西尚品信息科技有限公司 A kind of restorative procedure of the multinode data file based on minimum memory regeneration code
CN107395207A (en) * 2017-07-12 2017-11-24 东莞理工学院 The MDS array codes coding and restorative procedure of more fault-tolerances
US20190140667A1 (en) * 2017-11-03 2019-05-09 Board Of Trustees Of Michigan State University Optimal construction of regenerating code through rate-matching
CN108810063A (en) * 2017-11-21 2018-11-13 北京大学深圳研究生院 Secure distribution and restorative procedure, the system and medium of data under a kind of cloudy storage environment
CN108923960A (en) * 2018-06-20 2018-11-30 华中科技大学 A kind of memory node restorative procedure for assisting regeneration code based on agency
CN109257050A (en) * 2018-08-09 2019-01-22 东莞理工学院 A kind of reparation binary code generator matrix building method and restorative procedure
CN109257049A (en) * 2018-08-09 2019-01-22 东莞理工学院 A kind of building method and restorative procedure for repairing binary array code check matrix

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
MICHAEL ABEBE: "EC-Store:Bridging the Gap between Storage and Latency in Distributed Erasure Coded Systems", 《INTERNATIONAL CONFERENCE ON DISTRIBUTED COMPUTING SYSTEMS(ICDCS)》 *

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111447044B (en) * 2020-03-10 2022-12-09 深圳市大数据研究院 Distributed storage method and transmission decoding method
CN111447044A (en) * 2020-03-10 2020-07-24 深圳市大数据研究院 Distributed storage method and transmission decoding method
CN111475839A (en) * 2020-04-06 2020-07-31 华中科技大学 Redundant data coding method and storage medium for untrusted environment
CN111475839B (en) * 2020-04-06 2023-04-18 华中科技大学 Redundant data coding method and storage medium for untrusted environment
CN111585581A (en) * 2020-05-14 2020-08-25 成都信息工程大学 Coding method based on binary domain operation and supporting any code distance
CN111585582A (en) * 2020-05-14 2020-08-25 成都信息工程大学 Coding method based on array operation and freely determined code distance
CN111585582B (en) * 2020-05-14 2023-04-07 成都信息工程大学 Coding method based on array operation and freely determined code distance
CN111585581B (en) * 2020-05-14 2023-04-07 成都信息工程大学 Coding method based on binary domain operation and supporting any code distance
CN112713964A (en) * 2020-12-22 2021-04-27 潍柴动力股份有限公司 Data verification acceleration method and device, computer equipment and storage medium
CN112713964B (en) * 2020-12-22 2022-08-05 潍柴动力股份有限公司 Data verification acceleration method and device, computer equipment and storage medium
CN112732203B (en) * 2021-03-31 2021-06-22 中南大学 Regeneration code construction method, file reconstruction method and node repair method
CN112732203A (en) * 2021-03-31 2021-04-30 中南大学 Regeneration code construction method, file reconstruction method and node repair method
CN113315753A (en) * 2021-04-25 2021-08-27 国网浙江省电力有限公司电力科学研究院 Block data credibility recovery method based on coding technology
CN114237967A (en) * 2022-02-22 2022-03-25 阿里云计算有限公司 Data reconstruction method and device
CN114595092A (en) * 2022-04-28 2022-06-07 阿里云计算有限公司 Distributed storage system, data reconstruction method, device and storage medium
CN114595092B (en) * 2022-04-28 2022-09-20 阿里云计算有限公司 Distributed storage system, data reconstruction method, device and storage medium
WO2024037010A1 (en) * 2022-08-17 2024-02-22 华为技术有限公司 Data storage method and apparatus, and storage medium
CN117632041A (en) * 2024-01-25 2024-03-01 国家计算机网络与信息安全管理中心天津分中心 Distributed storage method and device based on regenerated codes and electronic equipment

Also Published As

Publication number Publication date
CN110750382B (en) 2020-10-30

Similar Documents

Publication Publication Date Title
CN110750382B (en) Minimum storage regeneration code coding method and system for improving data repair performance
RU2501072C2 (en) Distributed storage of recoverable data
CN108540520B (en) Partial repeated code based locality repairing coding and node fault repairing method
CN109643258B (en) Multi-node repair using high-rate minimal storage erase code
US8171102B2 (en) Smart access to a dispersed data storage network
US8364891B2 (en) Storage assignment technique for scalable and fault tolerant storage system
CN110427156B (en) Partition-based MBR (Membrane biological reactor) parallel reading method
CN107729536B (en) Data storage method and device
CN106484559A (en) A kind of building method of check matrix and the building method of horizontal array correcting and eleting codes
JP2018508073A (en) Data removal, allocation and reconstruction
US10346066B2 (en) Efficient erasure coding of large data objects
CN113901069B (en) Data storage method and device of distributed database
US20070276999A1 (en) Adaptive grouping in object raid
US9489254B1 (en) Verification of erasure encoded fragments
CN108762978B (en) Grouping construction method of local part repeated cyclic code
CN110419029B (en) Method for partially updating data content in distributed storage network
US9552254B1 (en) Verification of erasure encoded fragments
CN111224747A (en) Coding method capable of reducing repair bandwidth and disk reading overhead and repair method thereof
CN113157715B (en) Erasure code data center rack collaborative updating method
Yongmei et al. Large LDPC codes for big data storage
CN115344214A (en) Data reading and writing method, device, server and computer readable storage medium
CN104572987A (en) Method and system for improving storage efficiency of simple regenerating codes by compression
CN112463435A (en) Local repair method based on data block access frequency
US20230179423A1 (en) Method and apparatus for storing blockchain transaction data and distributed storage system using the same
US11165440B1 (en) Systems and methods for geometric partitioning in object storage

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant