WO2018112980A1 - 存储控制器、数据处理芯片及数据处理方法 - Google Patents

存储控制器、数据处理芯片及数据处理方法 Download PDF

Info

Publication number
WO2018112980A1
WO2018112980A1 PCT/CN2016/111930 CN2016111930W WO2018112980A1 WO 2018112980 A1 WO2018112980 A1 WO 2018112980A1 CN 2016111930 W CN2016111930 W CN 2016111930W WO 2018112980 A1 WO2018112980 A1 WO 2018112980A1
Authority
WO
WIPO (PCT)
Prior art keywords
check
chunk
column
data
check matrix
Prior art date
Application number
PCT/CN2016/111930
Other languages
English (en)
French (fr)
Inventor
曾雁星
沈建强
王工艺
张进毅
吕温
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP16894803.2A priority Critical patent/EP3364541B1/en
Priority to CN201680089699.0A priority patent/CN110089035B/zh
Priority to PCT/CN2016/111930 priority patent/WO2018112980A1/zh
Priority to US15/977,794 priority patent/US10210044B2/en
Publication of WO2018112980A1 publication Critical patent/WO2018112980A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1076—Parity data used in redundant arrays of independent storages, e.g. in RAID systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1008—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices
    • G06F11/1012—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices using codes or arrangements adapted for a specific type of error
    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B20/00—Signal processing not specific to the method of recording or reproducing; Circuits therefor
    • G11B20/10—Digital recording or reproducing
    • G11B20/18—Error detection or correction; Testing, e.g. of drop-outs
    • G11B20/1833—Error detection or correction; Testing, e.g. of drop-outs by adding special lists or symbols to the coded information
    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11C—STATIC STORES
    • G11C29/00—Checking stores for correct operation ; Subsequent repair; Testing stores during standby or offline operation
    • G11C29/70—Masking faults in memories by using spares or by reconfiguring
    • G11C29/88—Masking faults in memories by using spares or by reconfiguring with partially good memories
    • H—ELECTRICITY
    • H03—ELECTRONIC CIRCUITRY
    • H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
    • H03M13/00—Coding, decoding or code conversion, for error detection or error correction; Coding theory basic assumptions; Coding bounds; Error probability evaluation methods; Channel models; Simulation or testing of codes
    • H03M13/03—Error detection or forward error correction by redundancy in data representation, i.e. code words containing more digits than the source words
    • H03M13/05—Error detection or forward error correction by redundancy in data representation, i.e. code words containing more digits than the source words using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits
    • H03M13/11—Error detection or forward error correction by redundancy in data representation, i.e. code words containing more digits than the source words using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits using multiple parity bits
    • H—ELECTRICITY
    • H03—ELECTRONIC CIRCUITRY
    • H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
    • H03M13/00—Coding, decoding or code conversion, for error detection or error correction; Coding theory basic assumptions; Coding bounds; Error probability evaluation methods; Channel models; Simulation or testing of codes
    • H03M13/37—Decoding methods or techniques, not specific to the particular type of coding provided for in groups H03M13/03 - H03M13/35
    • H03M13/373—Decoding methods or techniques, not specific to the particular type of coding provided for in groups H03M13/03 - H03M13/35 with erasure correction and erasure determination, e.g. for packet loss recovery or setting of erasures for the decoding of Reed-Solomon codes
    • H—ELECTRICITY
    • H03—ELECTRONIC CIRCUITRY
    • H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
    • H03M13/00—Coding, decoding or code conversion, for error detection or error correction; Coding theory basic assumptions; Coding bounds; Error probability evaluation methods; Channel models; Simulation or testing of codes
    • H03M13/37—Decoding methods or techniques, not specific to the particular type of coding provided for in groups H03M13/03 - H03M13/35
    • H03M13/3761—Decoding methods or techniques, not specific to the particular type of coding provided for in groups H03M13/03 - H03M13/35 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes

Definitions

  • the present application relates to the field of storage technologies, and in particular, to a storage controller, a data processing chip, and a data processing method.
  • the storage system in a large-scale storage scenario includes a plurality of storage media and a storage controller, and the storage medium may be a hard disk (English: hard disk drive, HDD) or a solid state drive (English: solid state drive, abbreviated: SSD) or two.
  • the composition of the combination The client sends the data to be written to the storage controller through the communication network, and the data to be written by the storage controller is processed and stored in the storage medium.
  • the existing storage system generally adopts redundant arrays of independent disks (RAID) technology, and the core of the RAID technology is the encoding of the erasure code. And decoding technology.
  • the application provides a storage controller to improve the encoding and decoding efficiency of the erasure code.
  • a first aspect of the present application provides a memory controller including a processor, a memory, and a communication interface.
  • the processor acquires K data chunk chunks to be encoded through the communication interface, and caches the K data chunks into the memory, and each data chunk includes R data encoding blocks.
  • R+1 is a prime number and R+1>K.
  • the processor continuously receives the data to be written by the client through the communication interface and caches the data into the memory. After the preset amount of data to be written is buffered in the memory, the processor divides the preset number of data to be written into K blocks of data to be encoded, and each data chunk is divided into R data. Encoding block.
  • the processor is further configured to execute the code in the memory to: read the K data chunks stored in the memory, according to a check matrix and the K data chunks A first check chunk and a second check chunk are generated, and each check chunk includes R check code blocks.
  • the check matrix has 2*R rows, and the (k-1)*R+1th column to the k*Rth column in the check matrix are the kth data chunk of the K data chunks. a chunk column set, K ⁇ k ⁇ 1, wherein the K*R+1 column to the (K+1)*R column in the check matrix is a chunk column set corresponding to the first check chunk, the school The (K+1)*R+1th column to the (K+2)*R column in the matrix are the chunk column set of the second check chunk.
  • the check matrix is obtained by performing a standard check matrix H or performing N times of replacement operations by the standard check matrix H, N ⁇ 1, and the swapping operation refers to swapping any two chunk column sets; the standard check matrix
  • the remaining coordinates are 0, 2 * R ⁇ i ⁇ 1, (K + 2) * R ⁇ j ⁇ 1,
  • the first storage form is a matrix with 2*R rows and (K+2)*R columns. Since each row of the check matrix represents an XOR equation, the check matrix represents 2*R XOR equations. Therefore, the second storage form of the check matrix is 2*R XOR equations. Through the 2*R XOR equations, a matrix with 2*R rows and (K+2)*R columns can also be obtained.
  • the (k-1)*R+1th column to the k*Rth column in the check matrix respectively correspond to the K data chunks R data encoding blocks of k data chunks
  • the Kth column from the K*R+1 column to the (K+1)*R column in the check matrix respectively correspond to the R check code blocks of the first check chunk
  • the (K+1)*R+1th column to the (K+2)*R column in the check matrix respectively correspond to R check code blocks of the second check chunk. That is, each column in the check matrix corresponds to one data coding block or one check coding block.
  • the D row of the check matrix has three coordinates of 1, and the Dth acts on any row of the check matrix. These three coordinates of 1 correspond to three coding blocks. Performing an exclusive OR operation on any two coding blocks of the three coding blocks corresponding to the coordinates of 1 in the D row of the parity check matrix may be obtained.
  • the coded block that does not participate in the current exclusive OR operation among the three coding blocks corresponding to the coordinates of 1 in the Dth row of the parity check matrix That is, if three coordinates of a row of the check matrix correspond to the coding block 1, the coding block 2, and the coding block 3, respectively, any two coding codes of the coding block 1, the coding block 2, and the coding block 3 An exclusive OR operation between blocks can result in another code block.
  • the processor is configured to generate a first check chunk and a second check chunk according to the check matrix, Encoding begins with the starting code line.
  • the starting code activity is only one row with a coordinate of 1 in the 2*R column corresponding to the first check chunk and the second check chunk.
  • each row of the check matrix corresponds to an exclusive OR equation, and the encoding starts, only K*R data encoding blocks are known, so the encoding can only start from the starting encoding row.
  • the check matrix has a total of 4 start code lines.
  • the check coding block 1 is obtained. If the check code block 1 has only one coordinate in the column corresponding to the check matrix, the encoding process at the start of the first start code line is completed. If the check code block 1 has 2 coordinates in the column corresponding to the check matrix, then jump to the row in which the two coordinates are located has not been used to generate the check code block. The XOR equation corresponding to the row to which the jump is performed is obtained, and the check code block 2 is obtained. The same processing as that of the check code block 1 is performed on the check code block 2.
  • the encoding process at the start of the first start code line is completed. If the check code block 2 has 2 coordinates in the column corresponding to the check matrix, then jump to the row in which the two coordinates are located has not been used to generate the check code block. Executing the exclusive-OR equation corresponding to the row to which the jump is performed, obtaining the check coding block 3, performing the same processing of the check coding block 1 on the check coding block 2, and so on, until the coding process starting from the first start coding line carry out. For the four initial coding lines of the check matrix, the coding process of the first start coding line is performed, and after the coding process starting from the four initial coding lines is completed, 2*R checksums are obtained. Encoding block.
  • the processor acquires the first check according to the check matrix After the chunk and the second check chunk, the K data chunk, the first check chunk, and the second check chunk are respectively stored in the storage of the storage controller through the communication interface
  • the system is K+2 storage media. Generally, different chunks are stored in different storage media.
  • each R check code block forms a check chunk.
  • the two check chunks and K data chunks form a chunk group, and each chunk in a chunk group is When it is stored in a different storage medium, so that the storage medium is damaged, the chunk stored in the damaged storage medium can be recovered by the undamaged chunk in the chunk group.
  • the K data chunk, the first parity chunk, and the second parity chunk are stored in the After the K+2 storage media of the storage system in which the storage controller is located, if the storage medium is damaged in the K+2 storage media, the processor is configured according to the check matrix and the K+2 The data chunk stored on the undamaged storage medium in the storage medium and at least one of the first check chunk and the second check chunk are restored to the damaged storage medium.
  • the damaged data chunk is restored according to the K-1 data chunks and the 2 check chunks stored on the remaining K+1 storage media. If the checksum chunk is stored on the damaged storage medium, the damaged check chunk is restored according to the K data chunks and one check chunk stored on the remaining K+1 storage media.
  • each data chunk that is not damaged and each parity chunk that is not damaged are used, it is not required to use each data encoding block and each of the undamaged data of each data chunk that is not damaged.
  • Checking each check code block of the chunk, which data coding block and which check code block are used to recover the damaged chunk, and which rows correspond to the corresponding column in the check matrix according to the damaged chunk The XOR equation is determined.
  • the storage controller provided by the first aspect of the present application reduces the recovery overhead and improves the efficiency of subsequent recovery loss chunks.
  • a second aspect of the present application provides a data processing chip, including a circuit and a read/write interface, where the circuit is configured to acquire K chunks of chunks of data to be encoded through the read/write interface, and each data chunk includes R Data encoding blocks, R+1 is a prime number and R+1>K; the circuit is further configured to generate a first check chunk and a second check chunk according to the check matrix and the K data chunk, each school The check chunk includes R check code blocks; wherein the check matrix has 2*R rows, and the (k-1)*R+1th column to the kthth Rth column in the check matrix are the K Chunk column of the kth data chunk in the data chunk a set, K ⁇ k ⁇ 1, wherein the K*R+1th column to the (K+1)*R column in the check matrix is a chunk column set that should be the first check chunk, the check matrix The (K+1)*R+1th column to the (K+2)*R column are the chunk column set of the second check chunk; the check matrix is the standard check
  • the third aspect of the present application provides a data processing method.
  • the data execution method is executed, including:
  • each data chunk includes R data encoding blocks, R+1 is a prime number and R+1>K;
  • each check chunk including R check code blocks
  • the check matrix has 2*R rows, and the (k-1)*R+1th column to the k*Rth column in the check matrix are the kth data chunk of the K data chunks. a chunk column set, K ⁇ k ⁇ 1, wherein the K*R+1 column to the (K+1)*R column in the check matrix is a chunk column set corresponding to the first check chunk, the school The (K+1)*R+1th column to the (K+2)*R column in the matrix are the chunk column set of the second check chunk;
  • the check matrix is obtained by performing a standard check matrix H or performing N times of replacement operations by the standard check matrix H, N ⁇ 1, and the swapping operation refers to swapping any two chunk column sets; the standard check matrix
  • the remaining coordinates are 0, 2 * R ⁇ i ⁇ 1, (K + 2) * R ⁇ j ⁇ 1,
  • the (k-1)*R+1th column to the k*Rth column in the check matrix respectively correspond to the K data chunks R data encoding blocks of k data chunks, wherein the Kth column from the K*R+1 column to the (K+1)*R column in the check matrix respectively correspond to the R check code blocks of the first check chunk And the (K+1)*R+1th column to the (K+2)*R column in the check matrix respectively correspond to R check code blocks of the second check chunk;
  • the D row of the check matrix has three coordinates of 1, the Dth acts as any row of the check matrix, and the three codes corresponding to the coordinates of the D row of the check matrix
  • An exclusive OR operation is performed on any two coding blocks in the block to obtain a coding block that does not participate in the current exclusive OR operation among the three coding blocks corresponding to the coordinates of 1 in the Dth row of the parity check matrix.
  • starting coding in the process of generating the first check chunk and the second check chunk according to the check matrix, starting coding The line starts coding.
  • the starting code activity is only one row with a coordinate of 1 in the 2*R column corresponding to the first check chunk and the second check chunk.
  • the check coding block 1 is obtained. If the check code block 1 has only one coordinate in the column corresponding to the check matrix, the encoding process at the start of the first start code line is completed. If the check code block 1 has 2 coordinates in the column corresponding to the check matrix, then jump to the row in which the two coordinates are located has not been used to generate the check code block. The XOR equation corresponding to the row to which the jump is performed is obtained, and the check code block 2 is obtained. The same processing as that of the check code block 1 is performed on the check code block 2.
  • the check code block 2 has only one coordinate in the column corresponding to the check matrix, the encoding process at the start of the first start code line is completed. If the check code block 2 has 2 coordinates in the column corresponding to the check matrix, then jump to the row in which the two coordinates are located has not been used to generate the check code block. Executing the exclusive-OR equation corresponding to the row to which the jump is performed, obtaining the check coding block 3, performing the same processing of the check coding block 1 on the check coding block 2, and so on, until the coding process starting from the first start coding line carry out. Performing the first of the above for the four initial encoded lines of the check matrix After the encoding process of the first encoding line, after the encoding process starting from the four initial encoding lines is completed, 2*R parity encoding blocks are obtained.
  • the method further includes: the K data chunks, the first checksum The chunk and the second check chunk are respectively stored in K+2 storage media of the storage system where the storage controller executing the data processing method is located.
  • the method further includes: when the storage device of the storage system where the storage controller is located, the storage medium is damaged. And storing, according to the check matrix and the data chunk stored on the undamaged storage medium in the K+2 storage medium of the storage system where the storage controller is located, the first check chunk and the second checksum At least one of the chunks recovers the damaged storage medium.
  • the data processing method provided by the third aspect of the present application reduces the recovery overhead by improving in the encoding process, and improves the efficiency of the subsequent recovery loss chunk.
  • the fourth aspect of the present application provides a storage medium, where the program is stored, and when the program is run by the computing device, the computing device performs the data processing method provided by any of the foregoing third or third aspects.
  • the storage medium includes, but is not limited to, a read only memory, a random access memory, a flash memory, an HDD, or an SSD.
  • a fifth aspect of the present application provides a computer program product, comprising: program instructions, when executed by a computer, the computer executes data provided by any of the foregoing third or third aspects Approach.
  • the computer program product can be a software installation package, and if the data processing method provided by any of the foregoing third aspect or the third aspect is required, the computer program product can be downloaded and executed on the computing device. Program product.
  • 1-1 is a schematic structural diagram of a storage system according to an embodiment of the present application.
  • FIG. 1-2 is a schematic structural diagram of another storage system according to an embodiment of the present application.
  • FIG. 2 is a schematic structural diagram of another storage system according to an embodiment of the present application.
  • FIG. 3 is a schematic structural diagram of a check matrix according to an embodiment of the present application.
  • 4-1 is a schematic structural diagram of another check matrix according to an embodiment of the present application.
  • 4-2 is a schematic structural diagram of another check matrix according to an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of still another check matrix according to an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of a storage controller according to an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of another storage controller according to an embodiment of the present disclosure.
  • FIG. 8 is a schematic structural diagram of another storage controller according to an embodiment of the present disclosure.
  • FIG. 9 is a schematic structural diagram of another storage controller according to an embodiment of the present disclosure.
  • FIG. 10 is a schematic structural diagram of a data processing chip according to an embodiment of the present application.
  • an exclusive OR operation between two code blocks refers to the exclusive OR operation of each bit data of two code blocks.
  • the first bit of the coding block 1 is XORed with the first bit of the coding block 2 to obtain the first bit of the coding block 3, and so on, until the last bit of the coding block is XORed with the last bit of the coding block 2, The last bit of the coded block 3 is obtained.
  • the coding block 3 is obtained by performing an exclusive OR operation between the coding block 1 and the coding block 2.
  • the recovery overhead is a parameter for recovering the access overhead of the storage medium required for the damaged storage medium in the case where any of the K+2 storage media storing the chunk group data is damaged.
  • the recovery overhead is equal to the ratio of the size of the coded block read from the uncorrupted storage medium to the size of all data-encoded blocks in the chunk group when the damaged storage medium is recovered. Therefore, the smaller the recovery overhead, the shorter the recovery time required in the event of storage media corruption.
  • the definition of the chunk group will be explained in detail below.
  • mod is the remainder function, that is, A mod B indicates that A is divided by B.
  • Figure 1-1 and Figure 1-2 show two different architecture storage systems.
  • the storage system in Figure 1-1 is also referred to as a storage array, and the storage controller and storage medium are both disposed inside the storage array.
  • 1-2 are distributed storage systems including a plurality of storage nodes, each of which may actually be a server. At least one storage node of the storage system includes a storage controller, each storage node includes a storage medium, and each storage node establishes a communication connection through the communication network.
  • the storage controller in the storage array of Figure 1-1 processes only the data to be written that the client sends to the storage array.
  • Each of the storage controllers in Figure 1-2 can receive the data to be written by the client and encode and decode the erasure code.
  • the data acquired by a storage controller can be stored not only in the storage medium of the storage node where the storage controller is located, but also in the storage medium of other storage nodes through the communication network to implement distributed storage. Since a plurality of storage controllers may work in parallel in a distributed storage system, each of the plurality of storage controllers is responsible for one storage node group in the storage system, and each storage node group includes at least A storage node.
  • the storage controller in a storage node group is responsible for receiving the data to be written sent by the client, encoding it, and storing it in the storage node of the storage node group.
  • the memory controller hereinafter may refer to any of the memory controllers of FIG. 1-1 or 1-2, which is used for encoding and decoding of erasure codes.
  • the storage controller continuously receives the data to be written sent by the client, and after receiving a preset amount of data to be written, the storage controller sets the preset quantity.
  • the data to be written is divided into K chunks of data to be encoded (English: chunk), each chunk of data is divided into R data encoding blocks, and encoding methods based on the K*R data encoding blocks and erasure codes are used. Two checksums are generated, and each check chunk includes R check code blocks.
  • Each K data chunk and the check chunk generated by the K data chunks form a chunk group (English: group).
  • the size of each chunk can be set as needed, for example, 512Byte, 4k Byte, 8k Byte, 32k Byte, and the like.
  • the storage controller After a chunk group is generated, the storage controller stores each chunk in the chunk group into an SSD.
  • the storage medium used by the storage system is similar to HDD or other types of devices.
  • the storage controller stores each chunk in a chunk group into the corresponding SSD. After that, continue to form the data to be written sent by the client into another chunk group and store it in a similar manner.
  • Each chunk is divided into R code blocks for storage in the SSD.
  • a coding block corresponding to a data chunk is referred to as a data coding block
  • a coding block corresponding to a check chunk is referred to as a check coding block.
  • the storage addresses (which may be physical storage addresses or logical storage addresses) of the R code blocks may be discontinuous.
  • each code block in a chunk group has the same size.
  • R and K need to meet the following conditions: R+1 is a prime number and R+1>K, and R and K are positive integers.
  • R is generally a configuration parameter of the storage system
  • K is generally a parameter set by the user.
  • the storage controller has a coding and decoding method for the erasure code under different K and R configurations.
  • Each check code block is obtained by XORing two code blocks from two different chunks.
  • the two chunks can be a data chunk and a check chunk, or both chunks are data chunks.
  • the storage controller determines, by using a check matrix preset in the storage controller, which of the two code blocks in each of the 2*R check code blocks is calculated by the check matrix in the storage controller. get.
  • the memory controller can also know which two coding block operations can be used for each coding block of the chunk stored on the damaged SSD through the check matrix.
  • the number of rows of the check matrix is 2*R and the number of columns is (K+2)*R.
  • Each column of the check matrix corresponds to one code block, and each row corresponds to an XOR equation.
  • X-Y refers to the Yth coding block of the data chunk X, which is hereinafter referred to as a data coding block X-Y, K ⁇ X ⁇ 1, and R ⁇ Y ⁇ 1.
  • check chunk P and check chunk Q The two check chunks are called check chunk P and check chunk Q respectively, so PY refers to the Yth code block of the check chunk P, which is hereinafter referred to as check code block PY, and QY refers to The Yth code block of the chunk Q is checked, which is hereinafter referred to as a check code block QY, and R ⁇ Y ⁇ 1.
  • the R columns corresponding to each chunk in the check matrix are collectively referred to as a chunk column set. Therefore, in a check matrix with a row number of 2*R and a column number of (K+2)*R, there is a total of K+2. Chunk column set Hehe.
  • the first to the Rth columns of the check matrix belong to the chunk column set corresponding to the data chunk 1, and the R+1th to 2Rth columns of the check matrix belong to the chunk column set corresponding to the data chunk 2, and so on, and the check
  • the K*R+1 to (K+1)*R columns of the matrix belong to the chunk column set corresponding to the check chunk P, and the (K+1)*R to (K+2)* of the check matrix
  • the R column belongs to the chunk column set corresponding to the check chunk Q.
  • Each row of the check matrix has three coordinates of 1, indicating that XOR operation is performed between any two of the three code blocks corresponding to the three coordinates to obtain another code block.
  • the first row of the check matrix indicates that among the data encoding block 2-R, the data encoding block K-3, and the check encoding block QR, an exclusive OR operation between any two can obtain no participation.
  • the code block of the XOR operation It should be noted that FIG. 3 is only an exemplary structure of the check matrix.
  • the check matrix applied in the embodiment of the present application may be a standard check matrix H, or may be obtained after performing N times of replacement operations by the standard check matrix H, where N ⁇ 1.
  • a swap operation refers to swapping any two chunk column sets in a matrix with 2*R rows and columns (K+2)*R. Since the standard check matrix H actually provides 2*R XOR equations, each XOR equation is used to derive 1 check code block, so the matrix obtained after performing N swap operations on the standard check matrix H, It is still possible to derive 2*R check code blocks.
  • the standard matrix H meets the following conditions: 2*R ⁇ i ⁇ 1, (K+2)*R ⁇ j ⁇ 1 in the following formula:
  • 4-2 is a check matrix obtained by translating a chunk column set corresponding to the data chunk 1 of the standard matrix H and a chunk column set corresponding to the data chunk 2. It can be seen that the contents of the first column to the fourth column of the standard matrix H provided in FIG. 4-1 are switched to the fifth column to the eighth column of the parity check matrix provided in FIG. 4-2, and FIG. 4-1 provides Standard The contents of the fifth column to the eighth column of the matrix H are switched to the first column to the fourth column of the parity check matrix provided in Fig. 4-2. In the check matrix provided in FIG. 4-2, the first column to the fourth column still correspond to the four data encoding blocks of the data chunk 1, and the fifth column to the eighth column still correspond to the four data encoding blocks of the data chunk 5.
  • encoding is performed from the starting coded line in the check matrix.
  • each row of the check matrix corresponds to an exclusive OR equation, and the encoding starts, only K*R data encoding blocks are known, so the encoding can only start from the starting encoding row.
  • the encoding process for the starting code line 1 is as follows:
  • Step 1 Performs an exclusive OR operation on the start code line 1 to obtain the check code block 1.
  • Step 2 If the check code block 1 only participates in an exclusive OR equation, that is, the column in which the check code block 1 is located has only one coordinate of 1, the code start of the start code line 1 ends.
  • Step 3 If the check code block 1 participates in two exclusive OR equations, that is, the column in which the check code block 1 is located has two coordinates of 1. Then, according to the two exclusive OR equations in which the check code block 1 participates, the line in which the exclusive OR equation not used in step 2 is encoded is obtained, and the check code block 2 is obtained.
  • Step 2 or step 3 is continued for the check coding block 2, that is, if the check code block 2 only participates in an exclusive OR equation, that is, the column in which the check code block 2 is located has only one coordinate of 1, the start code The encoding starting at line 1 ends. If the check coding block 2 participates in two exclusive OR equations, that is, the column in which the check code block 2 is located has two coordinates of 1, the two exclusive OR equations according to the check code block 2 are not used. The row in which the XOR equation is located is encoded to obtain a check code block 3.
  • Step 2 or step 3 is continued for the check code block 3. And so on, until after a certain encoding, the obtained check code block only participates in one XOR equation, the code starting from the start code line 1 ends.
  • the starting code line 1 is any one of the four start code lines in a check matrix.
  • the coding process of the first coding line 1 is performed to obtain 2*R parity coding blocks, and the storage controller acquires all data chunks and checksums. Chunk.
  • the check code block Q-4 participates only in one XOR equation, so the coding starting from the 3rd line ends.
  • the check code block P-3 participates only in one XOR equation, so the coding starting from the fourth line ends.
  • the check code block Q-1 participates only in one XOR equation, so the code starting from the 7th line ends.
  • the check code block P-2 participates only in one XOR equation, so the code starting from the 8th line ends.
  • the storage controller So far, the encoding of the four initial coding lines is finished, and the check coding block P-1 to the check coding block Q-4 have all been encoded, so the storage controller generates the data chunk 1 and the data chunk2 corresponding to the data. Chunk group.
  • the storage controller acquires the chunk group corresponding to the data chunk to be stored and stores it in K+2 SSDs. If the SSD in the K+2 SSDs is damaged, the erasure code is needed.
  • the decoding method restores the data chunk or check chunk stored on the damaged SSD. Specifically, the damaged SSD may be detected by the storage controller, or the storage controller is notified that there is SSD damage in the K+2 SSDs.
  • code block 1 is any of the R code blocks in the corrupted chunk:
  • code block 1 The remaining two coded blocks are XORed to obtain code block 1.
  • R-2 code blocks participate in two XOR equations.
  • Each code block participating in two XOR equations can use any of the two XOR equations it participates in during the recovery process. Therefore, the actual decoding method can be 2 R-2 .
  • the data encoding block 1-1 only participates in the XOR equation corresponding to the second row of the parity check matrix:
  • the data encoding block 1-2 participates in the XOR line corresponding to the third row of the parity check matrix and the eighth row of the check matrix:
  • Data encoding block 1-2 data encoding block 2-1XOR check encoding block Q-4;
  • Data Encoding Block 1-2 Data Encoding Block 2-3XOR Check Encoding Block P-4.
  • the data encoding block 1-3 participates in the XOR equation corresponding to the fourth row of the parity check matrix and the seventh row of the check matrix:
  • Data encoding block 1-3 data encoding block 2-2XOR check encoding block P-1;
  • Data Encoding Block 1-3 Data Encoding Block 2-4 XOR Check Encoding Block Q-1.
  • Block 1-4 Check Code Block P-1XOR Check Code Block Q-2.
  • the data encoding block 1-2 is restored using the exclusive OR equation corresponding to the third row of the parity check matrix
  • the data encoding block 1-3 is restored using the exclusive OR equation corresponding to the fourth row of the check matrix.
  • Decoding method 2 The recovery of the data encoding block 1-2 adopts the exclusive OR equation corresponding to the eighth row of the parity check matrix, and the recovery of the data encoding block 1-3 adopts the exclusive OR equation corresponding to the fourth row of the check matrix, :
  • Decoding method 3 The recovery of the data encoding block 1-2 adopts the exclusive OR equation corresponding to the third row of the parity check matrix, and the recovery of the data encoding block 1-3 adopts the exclusive OR equation corresponding to the seventh row of the check matrix, :
  • Decoding method 4 The recovery of the data encoding block 1-2 adopts the exclusive OR equation corresponding to the eighth row of the parity check matrix, and the recovery of the data encoding block 1-3 adopts the exclusive OR equation corresponding to the seventh row of the check matrix, :
  • the storage controller can preferably use the decoding method 2 to complete the recovery of the data chunk 2 to improve the recovery efficiency.
  • the storage controller can store multiple decoding methods with the least recovery overhead, and the decoding methods with the smallest recovery overheads correspond to the check matrices under different K and R values.
  • the above describes a scene in which only one SSD is corrupted in K+2 SSDs. If two SSDs in K+2 SSDs are damaged, the decoding method is similar to the above encoding method, and the chunks stored on the two SSDs that are about to be damaged are also damaged. It is regarded as a checksum chunk, and the chunks stored on the undamaged K SSDs are regarded as data chunks. Decoding is performed according to the check matrix and the chunk stored on the undamaged K SSDs to obtain the checksum stored on the damaged 2 SSDs.
  • FIG. 6 provides a memory controller 200 that can be used in the memory system shown in FIG. 1-1 or FIG. 1-2.
  • the memory controller 200 includes a bus 202, a processor 204, a memory 208, and a communication interface 206.
  • the processor 204, the memory 208, and the communication interface 206 communicate via a bus 202.
  • the processor 204 can be a central processing unit (English: central processing unit, abbreviation: CPU).
  • the memory 208 may include a volatile memory (English: volatile memory) (English: random access memory, abbreviation: RAM).
  • the memory 208 may also include a non-volatile memory such as a read-only memory (ROM), a flash memory, an HDD or an SSD.
  • the communication interface 206 includes a network interface and a storage medium read/write interface, which are respectively used to acquire data to be written sent by the client and write the obtained chunk group into the storage medium.
  • the memory 208 stores an encoding program and K data chunks.
  • the processor 204 reads the encoding program and the K data chunks from the memory 208 to execute the foregoing encoding process to generate a chunk group, and stores the chunks in the chunk group into different storage through the communication interface 206.
  • the medium In the medium.
  • the memory 208 stores a decoding program and a coding block required in the recovery process.
  • the processor 204 reads the decoding program and the coding block required to recover the damaged storage medium from the memory 208 to perform the aforementioned decoding process to restore the damaged storage medium. Stored chunks.
  • the encoding program and the decoding program can be combined into one program.
  • the check matrix is stored in the memory 208 in a plurality of ways, and may be stored directly in the form of a matrix or in the form of 2*R exclusive OR equations. And the 2*R XOR equations are merged with the encoding program and the decoding program.
  • the processor 204 executes the encoding process, and after accessing the check matrix, after determining the starting encoding line, each row of the corresponding exclusive OR operation of the check matrix is executed, the processor 204 The check matrix is accessed again to perform an exclusive OR operation corresponding to another row of the check matrix until the exclusive OR operation of each row of the check matrix is completed.
  • the decoding process is similar to the encoding process.
  • the memory 208 may not store the check matrix, but store 2*R XOR equations directly in the encoding program and the decoding program.
  • the encoding program directly instructs execution of step 1.1, step 2.1-2.3, step 3.1 and step 4.1-4.3, without storing the check matrix and in the encoding process.
  • the row access check matrix determines which two code blocks need to be XORed in each step.
  • the storage controller provided above reduces the recovery overhead and improves the efficiency of subsequent recovery of lost chunks.
  • the memory controller 400 includes a bus 402, a processor 404, a memory 408, a data processing chip 410, and a communication interface 406.
  • the processor 404, the memory 408, and the communication interface 406 communicate via a bus 402.
  • the processor 404 can be a CPU.
  • Memory 408 can include volatile memory.
  • Memory 408 can also include non-volatile memory.
  • the communication interface 406 includes a network interface and a storage medium read/write interface, which are respectively used to acquire data to be written sent by the client and store the chunk group obtained after the encoding into the storage medium.
  • the data processing chip 410 can be implemented by a circuit, which can be an application-specific integrated circuit (ASIC) or a programmable logic device (abbreviated as PLD).
  • the above PLD may be a complex programmable logic device (English: complex programmable logic device, abbreviation: CPLD), a field programmable gate array (English: field programmable gate array, abbreviated: FPGA), general array logic (English: general array logic, Abbreviation: GAL) or any combination thereof.
  • the data processing chip 410 may specifically include an address unit 4102, an operation unit 4104, a storage unit 4106, and a read/write interface 4108.
  • the location unit 4102, the operation unit 4104, and the storage unit 4106 can be actually integrated into one circuit.
  • the read/write interface 4108 is connected to the bus 402 for acquiring the data encoding block stored in the memory 408 through the bus 402 and storing it in the storage unit 4106 in the scenario where the data processing chip 410 performs encoding, and encoding the encoded code obtained after encoding.
  • the block is sent to the memory 208 via the bus 402 for storage.
  • the storage controller 200 stores the chunk group in the storage medium.
  • the read/write interface 4108 is further configured to acquire, in the scenario that the data processing chip 410 performs decoding, the code block required in the recovery process through the bus 402 and store it in the storage unit 4106, and send the restored code block to the memory through the bus 402. 208.
  • the function of the location unit 4102 is similar to the check matrix.
  • the location unit 4102 indicates which two blocks of the storage unit 4106 should be XORed in the process of performing an exclusive OR operation by the operation unit 4104, so that the operation unit 4104 is stored. A corresponding coded block is obtained in unit 4106 to perform an exclusive OR operation.
  • the operation unit 4104 acquires two coding blocks that need to perform an exclusive OR operation in the process of one exclusive OR operation from the storage unit 4106, and stores the obtained coding block in the storage unit 4106 after performing an exclusive OR operation, and then executes the next time. XOR operation.
  • the storage controller provided by the present application effectively reduces the recovery overhead and improves the efficiency of the subsequent recovery loss chunk.
  • the methods described in connection with the present disclosure can be implemented by a processor executing software instructions.
  • the software instructions can be composed of corresponding software modules, which can be stored in RAM, flash memory, ROM, erasable programmable read only memory (English: erasable programmable read only memory, abbreviation: EPROM), electrically erasable Programming an audio-only memory (English: electrically erasable programmable read only memory, EEPROM), HDD, SSD, optical disc, or any other form of storage medium known in the art.
  • the functions described herein may be implemented in hardware or software.
  • the functions may be stored in a computer readable medium or transmitted as one or more instructions or code on a computer readable medium.
  • a storage medium may be any available media that can be accessed by a general purpose or special purpose computer.

Abstract

一种存储控制器，该存储控制器运行时，根据校验矩阵对从客户端获取的待编码的K个数据大块chunk进行编码，以生成2个校验chunk，以便后续如果有任一chunk损坏的情况下，可以通过该校验矩阵和未损坏的chunk恢复损坏的chunk。该存储控制器提升了恢复损坏的chunk的效率。

Description

存储控制器、数据处理芯片及数据处理方法 技术领域
本申请涉及存储技术领域，尤其涉及一种存储控制器，数据处理芯片以及数据处理方法。
背景技术
大规模存储场景中的存储系统包括多个存储介质和存储控制器，存储介质可以由硬盘(英文：hard disk drive，缩写：HDD)或固态硬盘(英文：solid state drive，缩写：SSD)或两者的组合构成。客户端通过通信网络，将待写入数据发送至存储控制器，存储控制器对待写入的数据进行处理并存入存储介质中。现有的存储系统一般采用了独立磁盘构成的具有冗余能力的阵列(英文：redundant arrays of independent disks，缩写：RAID)技术，而RAID技术的核心就是纠删码(英文：erasure code)的编码和解码技术。
现有的纠删码的编码和解码技术效率较低。
发明内容
本申请提供了一种存储控制器，以提升纠删码的编码和解码效率。
本申请的第一方面提供了一种存储控制器，包括了处理器、存储器和通信接口。该存储控制器运行时，该处理器通过该通信接口获取待编码的K个数据大块chunk，并将所述K个数据chunk缓存入所述存储器，每个数据chunk包括R个数据编码块，R+1为素数且R+1>K。
该处理器通过该通信接口持续接收客户端发来的待写入数据并缓存入该存储器。该存储器中缓存了预设数量大小的待写入数据后，该处理器将所述预设数量的待写入数据分成K个待编码的数据大块，每个数据chunk被分为R个数据编码块。
随后，该处理器，还用于执行所述存储器中的代码执行以下操作：读取所述存储器中存储的所述K个数据chunk，根据校验矩阵和所述K个数据chunk 生成第一校验chunk和第二校验chunk，每个校验chunk包括R个校验编码块。
其中，所述校验矩阵有2*R行，所述校验矩阵中第(k-1)*R+1列至第k*R列为所述K个数据chunk中第k个数据chunk的chunk列集合，K≥k≥1，所述校验矩阵中第K*R+1列至第(K+1)*R列为对应所述第一校验chunk的chunk列集合，所述校验矩阵中第(K+1)*R+1列至第(K+2)*R列为所述第二校验chunk的chunk列集合。
所述校验矩阵为标准校验矩阵H或由标准校验矩阵H执行N次调换操作后得到，N≥1，所述调换操作指将任意两个chunk列集合调换；所述标准校验矩阵H中除以下坐标为1外，其余坐标均为0，2*R≥i≥1，(K+2)*R≥j≥1，
如果i<j，则
H[i+1][j*R+(R-j+i)mod R+1]
H[R+i+1][(j+1)*R-(R-j+i)mod R]
如果i>j，则
H[i+1][j*R+(R-1-j+i)mod R+1]
H[R+i+1][(j+1)*R-(R-1-j+i)mod R]。
所述校验矩阵在所述存储器中的存储形态可以有两种。第一种存储形态为一个行数为2*R，列数为(K+2)*R的矩阵。由于校验矩阵的每一行代表了一个异或方程，因此该校验矩阵代表了2*R个异或方程，因此，所述校验矩阵的第二种存储形态为2*R个异或方程，通过这2*R个异或方程也可以获得一个行数为2*R，列数为(K+2)*R的矩阵。
结合第一方面，在第一方面的第一种实现方式中，所述校验矩阵中第(k-1)*R+1列至第k*R列分别对应所述K个数据chunk中第k个数据chunk的R个数据编码块，所述校验矩阵中第K*R+1列至第(K+1)*R列分别对应所述第一校验chunk的R个校验编码块，所述校验矩阵中第(K+1)*R+1列至第(K+2)*R列分别对应所述第二校验chunk的R个校验编码块。也即，所述校验矩阵中的每一列，对应一个数据编码块或者一个校验编码块。
所述校验矩阵的第D行有3个坐标为1，所述第D行为所述校验矩阵的任一行。这3个为1的坐标对应了3个编码块。对所述校验矩阵的第D行中为1的坐标对应的3个编码块中的任意2个编码块进行一次异或运算可以得 到所述校验矩阵的第D行中为1的坐标对应的3个编码块中未参与本次异或运算的编码块。也即如果所述校验矩阵的某一行的3个为1的坐标分别对应了编码块1、编码块2和编码块3，则编码块1、编码块2和编码块3中任意两个编码块之间进行异或运算可以得到另一个编码块。
结合第一方面的第一种实现方式，在第一方面的第二种实现方式中，所述处理器在根据所述校验矩阵生成第一校验chunk和第二校验chunk的过程中，从起始编码行开始编码。该起始编码行为第一校验chunk和第二校验chunk对应的2*R列中仅有1个坐标为1的行。
由于校验矩阵的每一行对应一个异或方程，而编码开始时，已知的仅有K*R个数据编码块，因此编码只能从起始编码行开始。所述校验矩阵一共有4个起始编码行。
完成第一起始编码行对应的异或方程后(第一起始编码行为4个起始编码行之任一)，获取了校验编码块1。如果校验编码块1在所述校验矩阵对应的列中只有1个坐标为1，则第一起始编码行起始的编码过程完成。如果校验编码块1在所述校验矩阵对应的列中有2个坐标为1，则跳转到这2个坐标所在的行中还未被用于生成校验编码块的行。执行跳转到的行对应的异或方程，获取了校验编码块2。对校验编码块2执行校验编码块1相同的处理。如果校验编码块2在所述校验矩阵对应的列中只有1个坐标为1，则第一起始编码行起始的编码过程完成。如果校验编码块2在所述校验矩阵对应的列中有2个坐标为1，则跳转到这2个坐标所在的行中还未被用于生成校验编码块的行。执行跳转到的行对应的异或方程，获取了校验编码块3，对校验编码块2执行校验编码块1相同的处理，依次类推，直至第一起始编码行起始的编码过程完成。对所述校验矩阵的4个起始编码行，均执行上述第一起始编码行的编码过程，4个起始编码行起始的编码过程均完成后，则获取了2*R个校验编码块。
结合第一方面或第一方面的第一种或第二种实现方式，在第一方面的第三种实现方式中，所述处理器根据所述校验矩阵，获取了所述第一校验chunk和所述第二校验chunk后，通过所述通信接口将所述K个数据chunk、所述第一校验chunk和所述第二校验chunk分别存入所述存储控制器所在的存储系统的K+2个存储介质中。一般不同的chunk存入不同的存储介质中。
获取2*R个校验编码块后，每R个校验编码块形成一个校验chunk，这2个校验chunk和K个数据chunk组成了一个chunk group，一个chunk group中的每个chunk被存入不同的存储介质中，以便后续有存储介质损坏的时候，可以通过该chunk group中未损坏的chunk来恢复损坏的存储介质上存储的chunk。
结合第一方面的第三种实现方式，在第一方面的第四种实现方式中，在所述K个数据chunk、所述第一校验chunk和所述第二校验chunk被存入所述存储控制器所在的存储系统的K+2个存储介质中后，如果所述K+2个存储介质中有存储介质损坏，则所述处理器根据所述校验矩阵和所述K+2存储介质中未损坏的存储介质上存储的数据chunk和所述第一校验chunk和所述第二校验chunk中的至少一个，恢复所述损坏的存储介质。
如果损坏的存储介质上存储的是数据chunk，则根据其余K+1个存储介质上存储的K-1个数据chunk和2个校验chunk恢复损坏的数据chunk。如果损坏的存储介质上存储的是校验chunk，则根据其余K+1个存储介质上存储的K个数据chunk和1个校验chunk，恢复损坏的校验chunk。
恢复过程中，虽然要使用到未损坏的每个数据chunk和未损坏的每个校验chunk，但并不需要使用到未损坏的每个数据chunk的每个数据编码块和未损坏的每个校验chunk的每个校验编码块，具体采用哪些数据编码块和哪些校验编码块来恢复损坏的chunk，需要根据损坏的chunk在所述校验矩阵中对应的列参与了哪几行对应的异或方程决定。
由于解码过程与编码过程强耦合，通过在编码过程中的改进，本申请第一方面提供的存储控制器降低了恢复开销，提升了后续恢复损失的chunk时的效率。
本申请的第二方面提供了一种数据处理芯片，包括电路和读写接口；所述电路用于，通过所述读写接口获取待编码的K个数据大块chunk，每个数据chunk包括R个数据编码块，R+1为素数且R+1>K；所述电路还用于，根据校验矩阵和所述K数据chunk生成第一校验chunk和第二校验chunk，每个校验chunk包括R个校验编码块；其中，所述校验矩阵有2*R行，所述校验矩阵中第(k-1)*R+1列至第k*R列为所述K个数据chunk中第k个数据chunk的chunk列 集合，K≥k≥1，所述校验矩阵中第K*R+1列至第(K+1)*R列为应所述第一校验chunk的chunk列集合，所述校验矩阵中第(K+1)*R+1列至第(K+2)*R列为所述第二校验chunk的chunk列集合；所述校验矩阵为标准校验矩阵H或由标准校验矩阵H执行N次调换操作后得到，N≥1，所述调换操作指示将任意两个chunk列集合调换；所述标准校验矩阵H中除以下坐标为1外，其余坐标均为0，2*R≥i≥1，(K+2)*R≥j≥1，
如果i<j，则
H[i+1][j*R+(R-j+i)mod R+1]
H[R+i+1][(j+1)*R-(R-j+i)mod R]
如果i>j，则
H[i+1][j*R+(R-1-j+i)mod R+1]
H[R+i+1][(j+1)*R-(R-1-j+i)mod R]。
所述第二方面的具体实现方式及取得的技术效果与所述第一方面的各实现方式类似，在此不再赘述。
本申请的第三方面提供了一种数据处理方法，第一方面中提供的存储控制器和第二方面中提供的数据处理芯片工作时，执行该数据执行方法，包括：
获取待编码的K个数据大块chunk并缓存所述K个数据chunk，每个数据chunk包括R个数据编码块，R+1为素数且R+1>K；
根据校验矩阵和所述K个数据chunk生成第一校验chunk和第二校验chunk，每个校验chunk包括R个校验编码块；
其中，所述校验矩阵有2*R行，所述校验矩阵中第(k-1)*R+1列至第k*R列为所述K个数据chunk中第k个数据chunk的chunk列集合，K≥k≥1，所述校验矩阵中第K*R+1列至第(K+1)*R列为对应所述第一校验chunk的chunk列集合，所述校验矩阵中第(K+1)*R+1列至第(K+2)*R列为所述第二校验chunk的chunk列集合；
所述校验矩阵为标准校验矩阵H或由标准校验矩阵H执行N次调换操作后得到，N≥1，所述调换操作指将任意两个chunk列集合调换；所述标准校验矩阵H中除以下坐标为1外，其余坐标均为0，2*R≥i≥1，(K+2)*R≥j≥1，
如果i<j，则
H[i+1][j*R+(R-j+i)mod R+1]
H[R+i+1][(j+1)*R-(R-j+i)mod R]
如果i>j，则
H[i+1][j*R+(R-1-j+i)mod R+1]
H[R+i+1][(j+1)*R-(R-1-j+i)mod R]。
结合第三方面，在第三方面的第一种实现方式中，所述校验矩阵中第(k-1)*R+1列至第k*R列分别对应所述K个数据chunk中第k个数据chunk的R个数据编码块，所述校验矩阵中第K*R+1列至第(K+1)*R列分别对应所述第一校验chunk的R个校验编码块，所述校验矩阵中第(K+1)*R+1列至第(K+2)*R列分别对应所述第二校验chunk的R个校验编码块；
所述校验矩阵的第D行有3个坐标为1，所述第D行为所述校验矩阵的任一行，对所述校验矩阵的第D行中为1的坐标对应的3个编码块中的任意2个编码块进行一次异或运算可以得到所述校验矩阵的第D行中为1的坐标对应的3个编码块中未参与本次异或运算的编码块。
结合第三方面的第一种实现方式，在第三方面的第二种实现方式中，在根据所述校验矩阵生成第一校验chunk和第二校验chunk的过程中，从起始编码行开始编码。该起始编码行为第一校验chunk和第二校验chunk对应的2*R列中仅有1个坐标为1的行。
完成第一起始编码行对应的异或方程后(第一起始编码行为4个起始编码行之任一)，获取了校验编码块1。如果校验编码块1在所述校验矩阵对应的列中只有1个坐标为1，则第一起始编码行起始的编码过程完成。如果校验编码块1在所述校验矩阵对应的列中有2个坐标为1，则跳转到这2个坐标所在的行中还未被用于生成校验编码块的行。执行跳转到的行对应的异或方程，获取了校验编码块2。对校验编码块2执行校验编码块1相同的处理。如果校验编码块2在所述校验矩阵对应的列中只有1个坐标为1，则第一起始编码行起始的编码过程完成。如果校验编码块2在所述校验矩阵对应的列中有2个坐标为1，则跳转到这2个坐标所在的行中还未被用于生成校验编码块的行。执行跳转到的行对应的异或方程，获取了校验编码块3，对校验编码块2执行校验编码块1相同的处理，依次类推，直至第一起始编码行起始的编码过程完成。对所述校验矩阵的4个起始编码行，均执行上述第一起 始编码行的编码过程，4个起始编码行起始的编码过程均完成后，则获取了2*R个校验编码块。
结合第三方面或第三方面的第一种或第二种实现方式，在第三方面的第三种实现方式中，该方法还包括：将所述K个数据chunk、所述第一校验chunk和所述第二校验chunk分别存入执行该数据处理方法的存储控制器所在的存储系统的K+2个存储介质中。
结合第三方面的第三种实现方式，在第三方面的第四种实现方式中，该方法还包括：当所述存储控制器所在的存储系统的K+2个存储介质中有存储介质损坏时，根据所述校验矩阵和所述存储控制器所在的存储系统的K+2存储介质中未损坏的存储介质上存储的数据chunk和所述第一校验chunk和所述第二校验chunk中的至少一个，恢复所述损坏的存储介质。
本申请第三方面提供的数据处理方法通过在编码过程中的改进，降低了恢复开销，提升了后续恢复损失的chunk时的效率。
本申请第四方面提供了一种存储介质，该存储介质中存储了程序，该程序被计算设备运行时，该计算设备执行前述第三方面或第三方面的任一实现方式提供的数据处理方法。该存储介质包括但不限于只读存储器，随机访问存储器，快闪存储器、HDD或SSD。
本申请第五方面提供了一种计算机程序产品，该计算机程序产品包括程序指令，当该计算机程序产品被计算机执行时，该计算机执行前述第三方面或第三方面的任一实现方式提供的数据处理方法。该计算机程序产品可以为一个软件安装包，在需要使用前述第三方面或第三方面的任一实现方式提供的数据处理方法的情况下，可以下载该计算机程序产品并在计算设备上执行该计算机程序产品。
附图说明
为了更清楚地说明本申请实施例的技术方法，下面将对实施例中所需要使用的附图作以简单地介绍。
图1-1为本申请实施例提供的存储系统的组织结构示意图；
图1-2为本申请实施例提供的另一存储系统的组织结构示意图；
图2为本申请实施例提供的又一存储系统的组织结构示意图；
图3为本申请实施例提供的校验矩阵的结构示意图；
图4-1为本申请实施例提供的另一校验矩阵的结构示意图；
图4-2为本申请实施例提供的又一校验矩阵的结构示意图；
图5为本申请实施例提供的又一校验矩阵的结构示意图；
图6为本申请实施例提供的存储控制器的组织结构示意图；
图7为本申请实施例提供的另一存储控制器的组织结构示意图；
图8为本申请实施例提供的又一存储控制器的组织结构示意图；
图9为本申请实施例提供的又一存储控制器的组织结构示意图；
图10为本申请实施例提供的数据处理芯片的组织结构示意图。
具体实施方式
下面结合本申请实施例中的附图，对本申请实施例中的技术方法进行描述。
本申请中各个“第一”、“第二”、“第n”之间不具有逻辑或时序上的依赖关系。
贯穿本说明书，两个编码块之间的异或运算(英文：exclusive OR，缩写：XOR)，指代两个编码块的每一bit数据依次进行异或运算。例如编码块1的第1bit与编码块2的第1bit进行异或运算，得到编码块3的第1bit，依次类推，直至编码块的最后一个bit与编码块2的最后一个bit进行异或运算，得到编码块3的最后一个bit。则，编码块3由编码块1和编码块2之间进行异或运算得到。
贯穿本说明书，恢复开销为衡量存储chunk group的数据的K+2个存储介质中任一存储介质损坏的情况下，恢复损坏的存储介质所需的对存储介质的访问开销的参数。恢复开销等于恢复损坏的存储介质时从未损坏的存储介质读取的编码块的大小与该chunk group中全部数据编码块的大小之比。因此，恢复开销越小，说明了在有存储介质损坏的情况下，所需的恢复时间越短。chunk group的定义，将在下文详细说明。
贯穿本说明书，mod为求余函数，也即A mod B指示A除以B得到的 余数，A和B均为整数，例如4mod 3＝1。
本申请实施例所应用的架构
如图1-1和图1-2介绍了两种不同架构的存储系统。图1-1中的存储系统也称为存储阵列，存储控制器和存储介质均设置于存储阵列内部。图1-2为分布式的存储系统，该存储系统包括多个存储节点，每个存储节点实际可以为服务器。该存储系统的至少一个存储节点包括存储控制器，每个存储节点均包括存储介质，各个存储节点通过通信网络建立通信连接。
图1-1的存储阵列中的存储控制器仅对客户端发往该存储阵列的待写入数据进行处理。图1-2中的每个存储控制器均可以接收客户端发来的待写入数据并对其进行纠删码的编码和解码处理。一个存储控制器编码后获取的数据不仅可以被存入该存储控制器所在的存储节点的存储介质，还可以通过通信网络发往其他存储节点的存储介质，以实现分布式的存储。由于分布式的存储系统中，可能存在多个存储控制器并行工作，因此这多个存储控制器中的每个存储控制器负责存储系统中的一个存储节点组，每个存储节点组中包括至少一个存储节点。一个存储节点组中的存储控制器负责接收客户端发来的待写入数据，对其进行编码后存入该存储节点组的存储节点中。下文中的存储控制器，可以指代图1-1或图1-2中的任一存储控制器，该存储控制器用于纠删码的编码和解码。
如图2所示，在存储系统运行过程中，存储控制器持续接收客户端发来的待写入数据，接收到预设数量大小的待写入数据后，存储控制器将所述预设数量的待写入数据分成K个待编码的数据大块(英文：chunk)，每个数据chunk被分为R个数据编码块，并根据这K*R个数据编码块和纠删码的编码方法，生成2个校验chunk，每个校验chunk包括R个校验编码块。每K个数据chunk和通过这K个数据chunk生成的校验chunk组成一个chunk组(英文：group)。其中，每个chunk的大小可以根据需要进行设置，例如为512Byte、4k Byte、8k Byte、32k Byte等。
生成一个chunk group后，存储控制器将该chunk group中的每个chunk存入一个SSD中，存储系统采用的存储介质为HDD或其他种类的设备的情况与之类似。存储控制器将一个chunk group中每个chunk存入对应的SSD 中之后，继续将客户端发来的待写入数据形成另一个chunk group并采用类似的方式存储。
每个chunk在SSD中被分为R个编码块进行存储。图2中，将数据chunk对应的编码块称为数据编码块，将校验chunk对应的编码块称为校验编码块。虽然每个chunk的全部R个编码块都存储于同一个SSD，但这R个编码块的存储地址(可以是物理存储地址或者逻辑存储地址)可以不连续。一般一个chunk group中的每个编码块的大小相同。R和K需要符合以下条件：R+1为素数且R+1>K，R、K均为正整数。R一般为存储系统的配置参数，K一般为用户设置的参数，为了应对用户的不同需求，存储控制器具有应对不同K和R的配置下的纠删码的编码和解码方法。
如果任何一个SSD损坏了，那么需要用到损坏的SSD上的chunk所属的chunk group上的其余chunk来恢复损坏的SSD上的chunk，恢复过程需要使用纠删码的解码方法。
每个校验编码块，是由2个来自2个不同chunk的编码块进行异或运算得到的，这两个chunk可以为一个数据chunk和一个校验chunk，或者这两个chunk均为数据chunk。存储控制器在生成校验chunk的过程中通过预设于存储控制器中的校验矩阵来确定这2*R个校验编码块中的每个校验编码块分别由哪2个编码块运算得到。
由于异或运算的特性，生成一个校验编码块的2个编码块和该校验编码块之中，任意2个编码块进行异或运算都可以得到剩余的1个编码块。因此，当任一SSD损坏时，存储控制器通过该校验矩阵也可以得知该损坏的SSD上存储的chunk的每个编码块可以通过哪2个编码块运算得出。
该校验矩阵的行数为2*R且列数为(K+2)*R。校验矩阵的每一列对应一个编码块，每一行对应一个异或方程。如图3所示，X-Y指代数据chunk X的第Y个编码块，后文称之为数据编码块X-Y，K≥X≥1，R≥Y≥1。而两个校验chunk分别称之为校验chunk P和校验chunk Q，因此P-Y指代校验chunk P的第Y个编码块，后文称之为校验编码块P-Y，而Q-Y指代校验chunk Q的第Y个编码块，后文称之为校验编码块Q-Y，R≥Y≥1。
校验矩阵中每一个chunk对应的R列，合称为一个chunk列集合，因此一个行数为2*R且列数为(K+2)*R的校验矩阵中，一共有K+2个chunk列集 合。该校验矩阵的第1至第R列属于数据chunk 1对应的chunk列集合，该校验矩阵的第R+1至第2R列属于数据chunk 2对应的chunk列集合，依次类推，该校验矩阵的第K*R+1至第(K+1)*R列属于校验chunk P对应的chunk列集合，该校验矩阵的第(K+1)*R至第(K+2)*R列属于校验chunk Q对应的chunk列集合。
校验矩阵的每一行有3个坐标为1，指示这3个坐标对应的3个编码块中任意2个之间进行异或运算可以得到另一个编码块。如图3中，校验矩阵第1行指示：数据编码块2-R、数据编码块K-3、校验编码块Q-R三者之中，任意两者之间进行异或运算可以获得没参与异或运算的编码块。需要说明的是，图3仅为示例性的展示校验矩阵的结构。
本申请实施例所应用的校验矩阵
本申请实施例所应用的校验矩阵，可以为标准校验矩阵H，或由标准校验矩阵H执行N次调换操作后得到，N≥1。一次调换操作指，将一个行数为2*R且列数为(K+2)*R的矩阵中任意两个chunk列集合调换。由于标准校验矩阵H实际提供了2*R个异或方程，每个异或方程用于得出1个校验编码块，因此对标准校验矩阵H执行N次调换操作后得到的矩阵，仍然可以得出2*R个校验编码块。
该标准矩阵H符合以下条件，下式中2*R≥i≥1，(K+2)*R≥j≥1：
如果i<j，则
H[i+1][j*R+(R-j+i)mod R+1]＝1          式1
H[R+i+1][(j+1)*R-(R-j+i)mod R]＝1          式2
如果i>j，则
H[i+1][j*R+(R-1-j+i)mod R+1]＝1         式3
H[R+i+1][(j+1)*R-(R-1-j+i)mod R]＝1        式4
除式1至式4指示的坐标外，标准校验矩阵H中其余坐标均为0。
如图4-1为K＝2，R＝4时的标准矩阵H。图4-2为将标准矩阵H的数据chunk 1对应的chunk列集合和数据chunk 2对应的chunk列集合调换后得到的校验矩阵。可以看到，图4-1提供的标准矩阵H的第1列至第4列的内容被调换到了图4-2提供的校验矩阵的第5列至第8列，而图4-1提供的标准 矩阵H的第5列至第8列的内容被调换到了图4-2提供的校验矩阵的第1列至第4列。图4-2提供的校验矩阵中，第1列至第4列依然对应数据chunk 1的4个数据编码块，第5列至第8列依然对应数据chunk 5的4个数据编码块。
以下，介绍编码过程。
首先，从校验矩阵中的起始编码行进行编码。
起始编码行为校验矩阵中2个校验chunk对应的2*R列中仅有1个坐标为1的行。由于校验矩阵的每一行对应一个异或方程，而编码开始时，已知的仅有K*R个数据编码块，因此编码只能从起始编码行开始。
每个校验矩阵中，有4个起始编码行。这4个起始编码行对应的编码过程互不干扰，实际使用中可以并行执行。
针对起始编码行1的编码过程如下：
step 1：根据起始编码行1进行异或运算得出校验编码块1。
step 2：如果校验编码块1仅参与了一个异或方程，即校验编码块1所在的列仅有一个坐标为1，则起始编码行1起始的编码结束。
step 3：如果校验编码块1参与了两个异或方程，即校验编码块1所在的列有两个坐标为1。则根据校验编码块1参与的两个异或方程中，未在step 2中使用的异或方程所在的行进行编码，得到校验编码块2。
对校验编码块2继续执行step 2或step 3，也即如果校验编码块2仅参与了一个异或方程，即校验编码块2所在的列仅有一个坐标为1，则起始编码行1起始的编码结束。校验编码块2如果参与了两个异或方程，即校验编码块2所在的列有两个坐标为1，则根据校验编码块2参与的两个异或方程中，未在中使用的异或方程所在的行进行编码，得到校验编码块3。
对校验编码块3继续执行step 2或step 3。依此类推，直至某次编码后，得到的校验编码块仅参与了1个异或方程，则该起始编码行1起始的编码结束。
其中，起始编码行1为一个校验矩阵中的4个起始编码行中之任一。
对校验矩阵的4个起始编码行，均执行上述起始编码行1的编码过程即可获取2*R个校验编码块，则存储控制器获取了全部的数据chunk和校验 chunk。
以K＝2、R＝4，且采用标准校验矩阵的情况为例，如图5，第3、4、7、8行中，校验chunk P和校验chunk Q对应的2*R列内仅有1个坐标为1，因此第3、4、7、8行为该校验矩阵的起始编码行。
根据第3行进行编码：
step 1.1数据编码块1-2XOR数据编码块2-1＝校验编码块Q-4。
由标准校验矩阵的第16列可知，校验编码块Q-4仅参与了一个异或方程，因此由第3行起始的编码结束。
根据第4行进行编码：
step 2.1数据编码块1-3XOR数据编码块2-2＝校验编码块P-1。
由标准校验矩阵的第9列可知，校验编码块P-1参与了两个异或方程，分别对应标准校验矩阵的第4行和第6行，因此，接下来根据第6行进行编码：
step 2.2数据编码块1-4XOR校验编码块P-1＝校验编码块Q-2。
由标准校验矩阵的第14列可知，校验编码块Q-2参与了两个异或方程，分别对应标准校验矩阵的第6行和第1行，因此，接下来根据第1行进行编码：
step 2.3数据编码块2-4XOR校验编码块Q-2＝校验编码块P-3。
由标准校验矩阵的第11列可知，校验编码块P-3仅参与了一个异或方程，因此由第4行起始的编码结束。
根据第7行进行编码：
step 3.1数据编码块1-3XOR数据编码块2-4＝校验编码块Q-1。
由标准校验矩阵的第13列可知，校验编码块Q-1仅参与了一个异或方程，因此由第7行起始的编码结束。
根据第8行进行编码：
step 4.1数据编码块1-2XOR数据编码块2-3＝校验编码块P-4。
由标准校验矩阵的第12列可知，校验编码块P-4参与了两个异或方程，分别对应标准校验矩阵的第2行和第8行，因此，接下来根据第2行进行编码：
step 4.2数据编码块1-1XOR校验编码块P-4＝校验编码块Q-3。
由标准校验矩阵的第15列可知，校验编码块Q-3参与了两个异或方程，分别对应标准校验矩阵的第2行和第5行，因此，接下来根据第5行进行编码：
step 4.3数据编码块2-1XOR校验编码块Q-3＝校验编码块P-2。
由标准校验矩阵的第10列可知，校验编码块P-2仅参与了一个异或方程，因此由第8行起始的编码结束。
至此，4个起始编码行的编码均结束了，校验编码块P-1至校验编码块Q-4已经全部被编码得出，因此存储控制器生成了数据chunk 1和数据chunk2对应的chunk group。
以下，介绍解码过程。
通过上述编码过程，存储控制器获取了待存储的数据chunk对应的chunk group并存入K+2个SSD后，如果这K+2个SSD中有SSD损坏了，则需要用到纠删码的解码方法恢复损坏的SSD上存储的数据chunk或校验chunk。具体的，可以由该存储控制器检测出该损坏的SSD，或者该存储控制器被通知这K+2个SSD中有SSD损坏。
如果K+2个SSD中仅有一个SSD损坏，也即chunk group中只有一个chunk需要恢复。对于该损坏的chunk的每个编码块的恢复过程如下，编码块1为损坏的chunk中的R个编码块中之任一：
则根据该校验矩阵，获取编码块1参与的异或方程，并根据编码块1参与的异或方程获取用于恢复编码块1的其余两个编码块；
将其余两个编码块做异或运算，得到编码块1。
由于任一chunk的R个编码块中，有R-2个编码块参与了两个异或方程。每个参与了两个异或方程的编码块在恢复的过程中，可以使用其参与的两个异或方程中的任何一个。因此，实际的解码方法可以有2R-2种。
这2R-2种解码方法虽然都可以完成损坏的chunk的恢复，但由于编码块1的恢复过程中，需要将用于编码块1恢复的两个编码块从SSD中读到存储控制器中，再由存储控制器完成恢复过程。不同的解码方法可能导致恢复全部R个编码块的过程中，需要从SSD中读出的编码块的数量不同，因此对于一个确定的校验矩阵，对于任一chunk损坏的情况下，可以采用一种需要 从SSD中读出编码块的数量最少的解码方法。
仍以K＝2、R＝4，且采用标准校验矩阵的情况下为例，如图4-1。
如果数据chunk 1所在的SSD损坏，则需要恢复数据编码块1-1、数据编码块1-2、数据编码块1-3和数据编码块1-4。
数据编码块1-1仅参与了校验矩阵第2行对应的异或方程：
数据编码块1-1＝校验编码块P-4XOR校验编码块Q-3。
数据编码块1-2参与了校验矩阵第3行和校验矩阵第8行对应的异或方程：
数据编码块1-2＝数据编码块2-1XOR校验编码块Q-4；
数据编码块1-2＝数据编码块2-3XOR校验编码块P-4。
数据编码块1-3参与了校验矩阵第4行和校验矩阵第7行对应的异或方程：
数据编码块1-3＝数据编码块2-2XOR校验编码块P-1；
数据编码块1-3＝数据编码块2-4XOR校验编码块Q-1。
数据编码块1-4仅参与了校验矩阵第6行对应的异或方程：
数据编码块1-4＝校验编码块P-1XOR校验编码块Q-2。
由于数据编码块1-2和数据编码块1-3均参与了两个异或方程，因此一共有22＝4种解码方法。
解码方法1，数据编码块1-2的恢复采用了校验矩阵第3行对应的异或方程，且数据编码块1-3的恢复采用了校验矩阵第4行对应的异或方程，则：
恢复开销＝(校验编码块P-4、校验编码块Q-3、数据编码块2-1、校验编码块Q-4、数据编码块2-2、校验编码块P-1、校验编码块Q-2)/8个数据编码块＝0.875。
解码方法2：数据编码块1-2的恢复采用了校验矩阵第8行对应的异或方程，且数据编码块1-3的恢复采用了校验矩阵第4行对应的异或方程，则：
恢复开销＝(校验编码块P-4、校验编码块Q-3、数据编码块2-3、数据编码块2-2、校验编码块P-1、校验编码块Q-2)/8个数据编码块＝0.75。
解码方法3：数据编码块1-2的恢复采用了校验矩阵第3行对应的异或方程，且数据编码块1-3的恢复采用了校验矩阵第7行对应的异或方程，则：
恢复开销＝(校验编码块P-4、校验编码块Q-3、数据编码块2-1、校验编 码块Q-4、数据编码块2-4、校验编码块Q-1、校验编码块P-1、校验编码块Q-2)/8个数据编码块＝1。
解码方法4：数据编码块1-2的恢复采用了校验矩阵第8行对应的异或方程，且数据编码块1-3的恢复采用了校验矩阵第7行对应的异或方程，则：
恢复开销＝(校验编码块P-4、校验编码块Q-3、数据编码块2-3、校验编码块Q-1、数据编码块2-4、校验编码块P-1、校验编码块Q-2)/8个数据编码块＝0.875。
可以看出，解码方法2的恢复开销最小。因此，针对数据chunk 2损坏的情况下，存储控制器可以优选的采用解码方法2来完成数据chunk 2的恢复，以提升恢复效率。
针对每个chunk损坏的情况下，都有至少一种需要读出编码块的数量最少的解码方法。因此存储控制器内可以存储有多种恢复开销最小的解码方法，不同恢复开销最小的解码方法对应不同K和R取值下的校验矩阵。
以上介绍了K+2个SSD中仅有一个SSD损坏的场景，如果K+2个SSD中有两个SSD损坏，那么解码方法与前述编码方法类似，也即将损坏的2个SSD上存储的chunk视为校验chunk，未损坏的K个SSD上存储的chunk的视为数据chunk。根据所述校验矩阵和未损坏的K个SSD上存储的chunk进行解码，以获取损坏的2个SSD上存储的校验chunk。
纠删码的编码方法和解码方法之间强耦合，例如如果编码的时候，数据编码块1-1＝校验编码块P-4XOR校验编码块Q-3，那么如果校验编码块P-4损坏时，需要采用数据编码块1-1XOR校验编码块Q-3恢复校验编码块P-4，而如果校验编码块Q-3损坏时，需要采用数据编码块1-1XOR校验编码块P-4恢复校验编码块Q-3。因此本申请提供的编码方法有效降低了恢复开销，提升了后续恢复损失的chunk时的效率。
本申请实施例所应用的存储控制器
如图6提供了一种存储控制器200，存储控制器200可以运用于图1-1或图1-2所示的存储系统中。存储控制器200包括总线202、处理器204、存储器208和通信接口206。处理器204、存储器208和通信接口206之间通过总线202通信。
其中，处理器204可以为中央处理器(英文：central processing unit，缩写：CPU)。存储器208可以包括易失性存储器(英文：volatile memory)，例如随机存取存储器(英文：random access memory，缩写：RAM)。存储器208还可以包括非易失性存储器(英文：non-volatile memory)，例如只读存储器(英文：read-only memory，缩写：ROM)，快闪存储器，HDD或SSD。
通信接口206包括网络接口和存储介质读写接口，分别用于获取客户端发来的待写入数据和将编码后获得的chunk group写入存储介质中。
如图7，当存储控制器200在执行编码过程中，存储器208中存储有编码程序以及K个数据chunk。
存储控制器200运行时，处理器204从存储器208中读取编码程序和K个数据chunk，以执行前述编码过程生成chunk group，并通过通信接口206将该chunk group中的各个chunk存入不同存储介质中。
如图8，当存储控制器200在执行解码过程中，存储器208中存储有解码程序以及恢复过程中所需的编码块。
当存储控制器200所在的存储系统的存储介质损坏时，处理器204从存储器208中读取解码程序和恢复损坏的存储介质所需的编码块，以执行前述解码过程，恢复损坏的存储介质上存储的chunk。
编码程序和解码程序可以合并为一个程序。
校验矩阵在存储器208中的存储方式有多种，可以直接以矩阵的形式存储，也可以以2*R个异或方程的形式存储。并且将这2*R个异或方程与编码程序和解码程序融合。
以矩阵的形式存储的情况下，在编码过程中，处理器204执行编码程序，访问校验矩阵确定了起始编码行后，每执行完校验矩阵的一行对应的异或运算，处理器204再次访问校验矩阵以执行校验矩阵另一行对应的异或运算，直至校验矩阵每一行对应的异或运算均执行完毕。解码过程类似与编码过程类似。
对于每一个校验矩阵，都可以有确定的编码过程和解码过程，因此存储器208中也可以不存储校验矩阵，而是直接在编码程序和解码程序中存储2*R个异或方程。例如图5对应的编码方法中，编码程序直接指示执行step 1.1，step 2.1-2.3，step 3.1和step 4.1-4.3，无须存储校验矩阵并且在编码过程中逐 行访问校验矩阵以确定每一个step中需要对哪两个编码块作异或运算。类似的，在解码程序中也可以直接存储恢复开销最小的解码方法，例如K＝2、R＝4，且采用标准校验矩阵的情况下，如果数据chunk 1损坏，为了恢复数据编码块1-1至数据编码块1-4，解码程序直接指示执行以下异或运算：数据编码块1-1＝校验编码块P-4XOR校验编码块Q-3；数据编码块1-2＝数据编码块2-3XOR校验编码块P-4；数据编码块1-3＝数据编码块2-2XOR校验编码块P-1；数据编码块1-4＝校验编码块P-1XOR校验编码块Q-2。
以上提供的存储控制器降低了恢复开销，提升了后续恢复损失的chunk时的效率。
如图9，提供了另一种存储控制器400，存储控制器400可以运用于图1-1或图1-2所示的存储系统中。存储控制器400包括总线402、处理器404、存储器408、数据处理芯片410和通信接口406。处理器404、存储器408和通信接口406之间通过总线402通信。
其中，处理器404可以为CPU。存储器408可以包括易失性存储器。存储器408还可以包括非易失性存储器。
通信接口406包括网络接口和存储介质读写接口，分别用于获取客户端发来的待写入数据和将编码后获得的chunk group存入存储介质。
数据处理芯片410可以通过电路实现，所述电路可以Wie专用集成电路(英文：application-specific integrated circuit，缩写：ASIC)或可编程逻辑器件(英文：programmable logic device，缩写：PLD)。上述PLD可以是复杂可编程逻辑器件(英文：complex programmable logic device，缩写：CPLD)，现场可编程门阵列(英文：field programmable gate array，缩写：FPGA)，通用阵列逻辑(英文：generic array logic,缩写：GAL)或其任意组合。
如图10所示，数据处理芯片410具体可以包括选址单元4102、运算单元4104、存储单元4106和读写接口4108。选址单元4102、运算单元4104、存储单元4106实际可以集成为一个电路。
读写接口4108与总线402相连，用于在数据处理芯片410执行编码的场景下，通过总线402获取存储器408中存储的数据编码块并存入存储单元4106，并将编码后获取的校验编码块通过总线402发往存储器208，以便存 储控制器200将chunk group存入存储介质。读写接口4108还用于在数据处理芯片410执行解码的场景下，通过总线402获取恢复过程中所需的编码块并存入存储单元4106，并将恢复出的编码块通过总线402发往存储器208。
选址单元4102的功能与校验矩阵类似，选址单元4102指示运算单元4104进行一次异或运算的过程中应当将存储单元4106中哪两个编码块进行异或运算，以便运算单元4104从存储单元4106中获取对应的编码块以完成异或运算。
运算单元4104从存储单元4106中获取一次异或运算的过程中需要进行异或运算的两个编码块，执行完一次异或运算后将得到的编码块存入存储单元4106中，接着执行下一次异或运算。
由于纠删码的编码方法和解码方法之间强耦合，因此本申请提供的存储控制器有效降低了恢复开销，提升了后续恢复损失的chunk时的效率。
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。
结合本申请公开内容所描述的方法可以由处理器执行软件指令的方式来实现。软件指令可以由相应的软件模块组成，软件模块可以被存放于RAM、快闪存储器、ROM、可擦除可编程只读存储器(英文：erasable programmable read only memory，缩写：EPROM)、电可擦可编程只读存储器(英文：electrically erasable programmable read only memory，缩写：EEPROM)、HDD、SSD、光盘或者本领域熟知的任何其它形式的存储介质中。
本领域技术人员应该可以意识到，在上述一个或多个示例中，本申请所描述的功能可以用硬件或软件来实现。当使用软件实现时，可以将这些功能存储在计算机可读介质中或者作为计算机可读介质上的一个或多个指令或代码进行传输。存储介质可以是通用或专用计算机能够存取的任何可用介质。
以上该的具体实施方式，对本申请的目的、技术方案和有益效果进行了进一步详细说明，所应理解的是，以上该仅为本申请的具体实施方式而已，并不用于限定本申请的保护范围，凡在本申请的技术方案的基础之上，所做的任何修改、改进等，均应包括在本申请的保护范围之内。

Claims (13)

  1. 一种存储控制器，其特征在于，包括处理器、存储器和通信接口；
    所述处理器，用于通过所述通信接口获取待编码的K个数据大块chunk，并将所述K个数据chunk缓存入所述存储器，每个数据chunk包括R个数据编码块，R+1为素数且R+1>K；
    所述处理器，还用于执行所述存储器中的代码执行以下操作：
    读取所述存储器中存储的所述K个数据chunk，根据校验矩阵和所述K个数据chunk生成第一校验chunk和第二校验chunk，每个校验chunk包括R个校验编码块；
    其中，所述校验矩阵有2*R行，所述校验矩阵中第(k-1)*R+1列至第k*R列为所述K个数据chunk中第k个数据chunk的chunk列集合，K≥k≥1，所述校验矩阵中第K*R+1列至第(K+1)*R列为对应所述第一校验chunk的chunk列集合，所述校验矩阵中第(K+1)*R+1列至第(K+2)*R列为所述第二校验chunk的chunk列集合；
    所述校验矩阵为标准校验矩阵H或由标准校验矩阵H执行N次调换操作后得到，N≥1，所述调换操作指将任意两个chunk列集合调换；所述标准校验矩阵H中除以下坐标为1外，其余坐标均为0，2*R≥i≥1，(K+2)*R≥j≥1，
    如果i<j，则
    H[i+1][j*R+(R-j+i)modR+1]
    H[R+i+1][(j+1)*R-(R-j+i)mod R]
    如果i>j，则
    H[i+1][j*R+(R-1-j+i)modR+1]
    H[R+i+1][(j+1)*R-(R-1-j+i)modR]。
  2. 如权利要求1所述的存储控制器，其特征在于，所述校验矩阵中第(k-1)*R+1列至第k*R列分别对应所述K个数据chunk中第k个数据chunk的R个数据编码块，所述校验矩阵中第K*R+1列至第(K+1)*R列分别对应所述第一校验chunk的R个校验编码块，所述校验矩阵中第(K+1)*R+1列至第(K+2)*R列分别对应所述第二校验chunk的R个校验编码块；
    所述校验矩阵的第D行有3个坐标为1，所述第D行为所述校验矩阵的 任一行，对所述校验矩阵的第D行中为1的坐标对应的3个编码块中的任意2个编码块进行一次异或运算可以得到所述校验矩阵的第D行中为1的坐标对应的3个编码块中未参与本次异或运算的编码块。
  3. 如权利要求1或2所述的存储控制器，其特征在于，所述处理器还用于，通过所述通信接口将所述K个数据chunk、所述第一校验chunk和所述第二校验chunk分别存入所述存储控制器所在的存储系统的K+2个存储介质中。
  4. 如权利要求3所述的存储控制器，其特征在于，所述处理器还用于，当所述存K+2个存储介质中有存储介质损坏时，根据所述校验矩阵和所述K+2存储介质中未损坏的存储介质上存储的数据chunk和所述第一校验chunk和所述第二校验chunk中的至少一个，恢复所述损坏的存储介质。
  5. 一种数据处理芯片，其特征在于，包括电路和读写接口；
    所述电路用于，通过所述读写接口获取待编码的K个数据大块chunk，每个数据chunk包括R个数据编码块，R+1为素数且R+1>K；
    所述电路还用于，根据校验矩阵和所述K数据chunk生成第一校验chunk和第二校验chunk，每个校验chunk包括R个校验编码块；
    其中，所述校验矩阵有2*R行，所述校验矩阵中第(k-1)*R+1列至第k*R列为所述K个数据chunk中第k个数据chunk的chunk列集合，K≥k≥1，所述校验矩阵中第K*R+1列至第(K+1)*R列为应所述第一校验chunk的chunk列集合，所述校验矩阵中第(K+1)*R+1列至第(K+2)*R列为所述第二校验chunk的chunk列集合；
    所述校验矩阵为标准校验矩阵H或由标准校验矩阵H执行N次调换操作后得到，N≥1，所述调换操作指示将任意两个chunk列集合调换；
    所述标准校验矩阵H中除以下坐标为1外，其余坐标均为0，2*R≥i≥1，(K+2)*R≥j≥1，
    如果i<j，则
    H[i+1][j*R+(R-j+i)modR+1]
    H[R+i+1][(j+1)*R-(R-j+i)modR]
    如果i>j，则
    H[i+1][j*R+(R-1-j+i)modR+1]
    H[R+i+1][(j+1)*R-(R-1-j+i)modR]。
  6. 如权利要求5所述的数据处理芯片，其特征在于，所述校验矩阵中第(k-1)*R+1列至第k*R列分别对应所述K个数据chunk中第k个数据chunk的R个数据编码块，所述校验矩阵中第K*R+1列至第(K+1)*R列分别对应所述第一校验chunk的R个校验编码块，所述校验矩阵中第(K+1)*R+1列至第(K+2)*R列分别对应所述第二校验chunk的R个校验编码块；
    所述校验矩阵的第D行有3个坐标为1，所述第D行为所述校验矩阵的任一行，所述电路对所述校验矩阵的第D行中为1的坐标对应的3个编码块中的任意2个编码块进行一次异或运算可以得到所述校验矩阵的第D行中为1的坐标对应的3个编码块中未参与本次异或运算的编码块。
  7. 如权利要求5或6所述的数据处理芯片，其特征在于，所述数据处理芯片运用于存储控制器中；
    所述电路，还用于通过所述读写接口将所述K个数据chunk、所述第一校验chunk和所述第二校验chunk存入所述存储控制器的存储器中，以便所述存储控制器将所述K个数据chunk、所述第一校验chunk和所述第二校验chunk分别存入所述存储控制器所在的存储系统的K+2个存储介质中。
  8. 如权利要求7所述的数据处理芯片，其特征在于，所述电路还用于，当所述K+2个存储介质中有存储介质损坏时，根据所述校验矩阵和所述K+2存储介质中未损坏的存储介质上存储的数据chunk和所述第一校验chunk和所述第二校验chunk中的至少一个，恢复所述损坏的存储介质。
  9. 如权利要求5至8任一所述的数据处理芯片，其特征在于，所述数据处理芯片包括现场可编程门阵列FPGA。
  10. 一种数据处理方法，其特征在于，所述方法适用于存储控制器；所述方法包括：
    获取待编码的K个数据大块chunk并缓存所述K个数据chunk，每个数据chunk包括R个数据编码块，R+1为素数且R+1>K；
    根据校验矩阵和所述K个数据chunk生成第一校验chunk和第二校验chunk，每个校验chunk包括R个校验编码块；
    其中，所述校验矩阵有2*R行，所述校验矩阵中第(k-1)*R+1列至第k*R 列为所述K个数据chunk中第k个数据chunk的chunk列集合，K≥k≥1，所述校验矩阵中第K*R+1列至第(K+1)*R列为对应所述第一校验chunk的chunk列集合，所述校验矩阵中第(K+1)*R+1列至第(K+2)*R列为所述第二校验chunk的chunk列集合；
    所述校验矩阵为标准校验矩阵H或由标准校验矩阵H执行N次调换操作后得到，N≥1，所述调换操作指将任意两个chunk列集合调换；所述标准校验矩阵H中除以下坐标为1外，其余坐标均为0，2*R≥i≥1，(K+2)*R≥j≥1，
    如果i<j，则
    H[i+1][j*R+(R-j+i)modR+1]
    H[R+i+1][(j+1)*R-(R-j+i)modR]
    如果i>j，则
    H[i+1][j*R+(R-1-j+i)modR+1]
    H[R+i+1][(j+1)*R-(R-1-j+i)modR]。
  11. 如权利要求10所述的数据处理方法，其特征在于，所述校验矩阵中第(k-1)*R+1列至第k*R列分别对应所述K个数据chunk中第k个数据chunk的R个数据编码块，所述校验矩阵中第K*R+1列至第(K+1)*R列分别对应所述第一校验chunk的R个校验编码块，所述校验矩阵中第(K+1)*R+1列至第(K+2)*R列分别对应所述第二校验chunk的R个校验编码块；
    所述校验矩阵的第D行有3个坐标为1，所述第D行为所述校验矩阵的任一行，对所述校验矩阵的第D行中为1的坐标对应的3个编码块中的任意2个编码块进行一次异或运算可以得到所述校验矩阵的第D行中为1的坐标对应的3个编码块中未参与本次异或运算的编码块。
  12. 如权利要求10或11所述的数据处理方法，其特征在于，还包括：
    将所述K个数据chunk、所述第一校验chunk和所述第二校验chunk分别存入所述存储控制器所在的存储系统的K+2个存储介质中。
  13. 如权利要求12所述的数据处理方法，其特征在于，还包括：
    当所述K+2个存储介质中有存储介质损坏时，根据所述校验矩阵和所述K+2存储介质中未损坏的存储介质上存储的数据chunk和所述第一校验chunk和所述第二校验chunk中的至少一个，恢复所述损坏的存储介质。
PCT/CN2016/111930 2016-12-24 2016-12-24 存储控制器、数据处理芯片及数据处理方法 WO2018112980A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP16894803.2A EP3364541B1 (en) 2016-12-24 2016-12-24 Storage controller, data processing chip, and data processing method
CN201680089699.0A CN110089035B (zh) 2016-12-24 2016-12-24 存储控制器、数据处理芯片及数据处理方法
PCT/CN2016/111930 WO2018112980A1 (zh) 2016-12-24 2016-12-24 存储控制器、数据处理芯片及数据处理方法
US15/977,794 US10210044B2 (en) 2016-12-24 2018-05-11 Storage controller, data processing chip, and data processing method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2016/111930 WO2018112980A1 (zh) 2016-12-24 2016-12-24 存储控制器、数据处理芯片及数据处理方法

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/977,794 Continuation US10210044B2 (en) 2016-12-24 2018-05-11 Storage controller, data processing chip, and data processing method

Publications (1)

Publication Number Publication Date
WO2018112980A1 true WO2018112980A1 (zh) 2018-06-28

Family

ID=62624263

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/111930 WO2018112980A1 (zh) 2016-12-24 2016-12-24 存储控制器、数据处理芯片及数据处理方法

Country Status (4)

Country Link
US (1) US10210044B2 (zh)
EP (1) EP3364541B1 (zh)
CN (1) CN110089035B (zh)
WO (1) WO2018112980A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021012164A1 (zh) * 2019-07-22 2021-01-28 华为技术有限公司 数据重构的方法、装置、计算机设备、存储介质及系统
CN113505019A (zh) * 2021-05-14 2021-10-15 山东云海国创云计算装备产业创新中心有限公司 一种纠删码数据及校验恢复方法、装置、设备及可读介质

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110071727B (zh) * 2019-04-26 2023-05-05 成都海光集成电路设计有限公司 编码方法、译码方法、纠错方法及装置
CN111090540B (zh) * 2019-06-24 2022-06-14 华为技术有限公司 基于纠删码的数据处理方法与装置
CN111404554B (zh) * 2019-09-29 2023-09-01 杭州海康威视系统技术有限公司 一种纠删码的编码矩阵生成方法、装置及电子设备
CN111930552A (zh) * 2020-06-17 2020-11-13 深圳佰维存储科技股份有限公司 坏块数据的恢复方法、装置、存储介质及电子设备
CN112799875B (zh) * 2020-12-18 2023-01-06 苏州浪潮智能科技有限公司 基于高斯消元进行校验恢复的方法、系统、设备及介质
CN112764673B (zh) * 2020-12-28 2024-03-15 中国测绘科学研究院 高光谱线阵数据存储速率优化方法、装置和存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100251068A1 (en) * 2009-03-27 2010-09-30 Li-Lien Lin Storage controller with encoding/decoding circuit programmable to support different ecc requirements and related method thereof
CN102739346A (zh) * 2011-04-08 2012-10-17 晨星软件研发(深圳)有限公司 用于准循环低密度奇偶校验码的再定址解码器与解码方法
CN104202057A (zh) * 2014-02-12 2014-12-10 中兴通讯股份有限公司 信息处理方法及装置
CN104484126A (zh) * 2014-11-13 2015-04-01 华中科技大学 一种基于纠删码的数据安全删除方法和系统

Family Cites Families (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5579475A (en) 1991-02-11 1996-11-26 International Business Machines Corporation Method and means for encoding and rebuilding the data contents of up to two unavailable DASDS in a DASD array using simple non-recursive diagonal and row parity
US6823425B2 (en) * 2001-10-23 2004-11-23 Ivivity, Inc. System and method for implementing advanced RAID using a set of unique matrices as coefficients
US6993701B2 (en) 2001-12-28 2006-01-31 Network Appliance, Inc. Row-diagonal parity technique for enabling efficient recovery from double failures in a storage array
US7533325B2 (en) 2003-07-14 2009-05-12 International Business Machines Corporation Anamorphic codes
US7870458B2 (en) * 2007-03-14 2011-01-11 Harris Corporation Parallel arrangement of serial concatenated convolutional code decoders with optimized organization of data for efficient use of memory resources
KR101370903B1 (ko) * 2007-03-16 2014-03-10 엘지전자 주식회사 Ｌｄｐｃ 코드를 이용한 부호화 및 복호화 방법
US8677208B2 (en) 2007-09-21 2014-03-18 Hewlett-Packard Development Company, L.P. Generating a parallel recovery plan for a data storage system
US8250427B2 (en) 2008-06-18 2012-08-21 Hewlett-Packard Development Company Selecting erasure codes for a fault tolerant system
US8296515B1 (en) 2009-05-22 2012-10-23 Nvidia Corporation RAID-6 computation system and method
US9058291B2 (en) 2011-02-28 2015-06-16 International Business Machines Corporation Multiple erasure correcting codes for storage arrays
US8595589B2 (en) * 2011-09-30 2013-11-26 Mitsubishi Electric Research Laboratories, Inc. Quasi-cyclic low-density parity-check codes
US8990495B2 (en) 2011-11-15 2015-03-24 Emc Corporation Method and system for storing data in raid memory devices
US8954822B2 (en) * 2011-11-18 2015-02-10 Sandisk Enterprise Ip Llc Data encoder and decoder using memory-specific parity-check matrix
CN102521067B (zh) 2011-12-01 2013-07-24 华中科技大学 优化部分条带写性能的raid-6编码和重构方法
US9495246B2 (en) 2013-01-21 2016-11-15 Kaminario Technologies Ltd. Raid erasure code applied to partitioned stripe
CN103761171B (zh) * 2014-02-11 2017-04-05 中国科学院成都生物研究所 一种针对二进制编码冗余存储系统的低带宽数据重构方法
CN104461781B (zh) 2014-12-01 2017-10-31 华中科技大学 一种基于纠删码的数据块重建方法
WO2016058289A1 (zh) 2015-01-20 2016-04-21 北京大学深圳研究生院 一种能修复多个节点失效的mds纠删码
US9501353B2 (en) 2015-01-28 2016-11-22 Quantum Corporation Erasure code prioritization
CN104866243A (zh) 2015-05-31 2015-08-26 上海交通大学 优化输入输出负载的raid-6横向斜向校验编码及解码方法
CN104850468B (zh) * 2015-05-31 2018-04-27 上海交通大学 基于校验矩阵的纠删码解码方法
US9959168B2 (en) * 2016-03-04 2018-05-01 Sandisk Technologies Llc ECC and RAID-type decoding
US10142419B2 (en) * 2016-03-04 2018-11-27 Sandisk Technologies Llc Erasure correcting coding using data subsets and partial parity symbols

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100251068A1 (en) * 2009-03-27 2010-09-30 Li-Lien Lin Storage controller with encoding/decoding circuit programmable to support different ecc requirements and related method thereof
CN102739346A (zh) * 2011-04-08 2012-10-17 晨星软件研发(深圳)有限公司 用于准循环低密度奇偶校验码的再定址解码器与解码方法
CN104202057A (zh) * 2014-02-12 2014-12-10 中兴通讯股份有限公司 信息处理方法及装置
CN104484126A (zh) * 2014-11-13 2015-04-01 华中科技大学 一种基于纠删码的数据安全删除方法和系统

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021012164A1 (zh) * 2019-07-22 2021-01-28 华为技术有限公司 数据重构的方法、装置、计算机设备、存储介质及系统
CN112543920A (zh) * 2019-07-22 2021-03-23 华为技术有限公司 数据重构的方法、装置、计算机设备、存储介质及系统
CN112543920B (zh) * 2019-07-22 2023-02-10 华为技术有限公司 数据重构的方法、装置、计算机设备、存储介质及系统
CN113505019A (zh) * 2021-05-14 2021-10-15 山东云海国创云计算装备产业创新中心有限公司 一种纠删码数据及校验恢复方法、装置、设备及可读介质
CN113505019B (zh) * 2021-05-14 2024-01-26 山东云海国创云计算装备产业创新中心有限公司 一种纠删码数据及校验恢复方法、装置、设备及可读介质

Also Published As

Publication number Publication date
EP3364541A4 (en) 2018-08-22
US20180260276A1 (en) 2018-09-13
CN110089035B (zh) 2021-01-29
US10210044B2 (en) 2019-02-19
EP3364541A1 (en) 2018-08-22
EP3364541B1 (en) 2019-08-14
CN110089035A (zh) 2019-08-02

Similar Documents

Publication Publication Date Title
WO2018112980A1 (zh) 存储控制器、数据处理芯片及数据处理方法
CN111480148B (zh) 具有对等数据恢复的存储系统
US6883131B2 (en) XOR processing incorporating error correction code data protection
US8327080B1 (en) Write-back cache protection
US9026729B1 (en) Data recovery after triple disk failure
CN103793182B (zh) 用于可扩展存储保护的系统和方法
CN104052576B (zh) 一种云存储下基于纠错码的数据恢复方法
CN111465922B (zh) 具有对等数据清理的存储系统
KR101837318B1 (ko) 메시지-타입 메모리 모듈을 위한 액세스 방법 및 디바이스
US20080126912A1 (en) Storing data redundantly
US20150089282A1 (en) Method of, and apparatus for, layout rectification of erasure encoded storage systems
JP2004227560A (ja) 外部格納サブシステムにおけるパリティー格納方法及びエラーブロック復旧方法
JP7047018B2 (ja) 消去コードベースのデータ処理方法および装置
CN106874140B (zh) 数据存储方法及装置
US10067833B2 (en) Storage system
CN110442298B (zh) 存储设备异常检测方法及装置、分布式存储系统
KR20190038964A (ko) 에러 정정 코드 유닛, 그것의 인코딩 및 디코딩 방법
KR20180059201A (ko) 비휘발성 메모리를 포함하는 raid 시스템
KR20220045343A (ko) 데이터 처리 시스템 내 데이터 전송에서 발생한 에러를 정정하는 장치 및 방법
KR102004928B1 (ko) 데이터 저장 장치 및 그것의 에러 정정 코드 처리 방법
US20190042365A1 (en) Read-optimized lazy erasure coding
CN110431531B (zh) 存储控制器、数据处理芯片及数据处理方法
US20190004942A1 (en) Storage device, its controlling method, and storage system having the storage device
JP5360666B2 (ja) 複数のディスクアレイのｉ／ｏ動作を実行する方法及びシステム
KR20210131058A (ko) 메모리 시스템 내 데이터를 보호하는 장치 및 방법

Legal Events

Date Code Title Description
REEP Request for entry into the european phase

Ref document number: 2016894803

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2016894803

Country of ref document: EP

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16894803

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE