CN113360264A

CN113360264A - Erasure processing system and method and distributed storage system

Info

Publication number: CN113360264A
Application number: CN202110656854.5A
Authority: CN
Inventors: 王明明; 张磊; 吴睿振; 王凛
Original assignee: Shandong Yunhai Guochuang Cloud Computing Equipment Industry Innovation Center Co Ltd
Current assignee: Shandong Yunhai Guochuang Cloud Computing Equipment Industry Innovation Center Co Ltd
Priority date: 2021-06-11
Filing date: 2021-06-11
Publication date: 2021-09-07

Abstract

The application discloses an erasure correcting processing system, an erasure correcting processing method and a distributed storage system, wherein the erasure correcting processing system comprises an erasure correcting processing carrier, a task scheduler and a task processor, the task scheduler and the task processor are arranged on the erasure correcting processing carrier, a task receiving interface is arranged on the task scheduler, and the erasure correcting processing system comprises: the task scheduler is used for receiving the erasure correcting task request by using the task receiving interface and analyzing the erasure correcting task request to obtain data information, matrix information and configuration information; and the task processor is used for calling input data according to the data information, calling an erasure correction matrix according to the matrix information, performing erasure correction calculation according to the input data, the erasure correction matrix and the configuration information to obtain output data, and outputting the output data according to the configuration information. According to the technical scheme disclosed by the application, the task scheduler and the task processor which are arranged on the erasure correcting processing carrier realize erasure correcting processing by using a hardware structure, so that the occupation of the erasure correcting processing on CPU computing resources is reduced, and the expense of the CPU is reduced.

Description

Erasure processing system and method and distributed storage system

Technical Field

The present application relates to the field of distributed storage technologies, and in particular, to an erasure correction processing system and method, and a distributed storage system.

Background

Erasure coding techniques have been introduced into the storage area because of their superior effectiveness in preventing data loss. The erasure codes are of various types, and RS codes (Reed-Solomon codes) applied in a distributed environment are more common in a real storage system.

At present, in distributed storage, for coding and decoding based on RS erasure codes, solutions of open source software such as an ISA (Instruction Set Architecture) of intel (intel), jerasure2.0 (a class library supporting erasure codes in storage applications based on C) are mostly used, that is, software related to RS runs on a Central Processing Unit (CPU) to perform erasure coding and decoding, but this implementation manner may occupy more CPU computing resources and increase CPU overhead.

In summary, how to reduce the occupation of CPU computing resources by erasure correction is a technical problem to be solved urgently by those skilled in the art.

Disclosure of Invention

In view of this, an object of the present application is to provide an erasure correction processing system, an erasure correction processing method, and a distributed storage system, which are used to reduce occupation of CPU computing resources by erasure correction.

In order to achieve the above purpose, the present application provides the following technical solutions:

an erasure correction processing system comprises an erasure correction processing carrier, a task scheduler and a task processor, wherein the task scheduler and the task processor are arranged on the erasure correction processing carrier, the task scheduler is provided with a task receiving interface, and the erasure correction processing system comprises:

the task scheduler is used for receiving the erasure correcting task request by using the task receiving interface and analyzing the erasure correcting task request to obtain data information, matrix information and configuration information;

the task processor is used for calling input data according to the data information, calling an erasure correction matrix according to the matrix information, performing erasure correction calculation according to the input data, the erasure correction matrix and the configuration information to obtain output data, and outputting the output data according to the configuration information.

Preferably, the task processor includes:

the input data scheduling circuit is used for calling the input data according to the data information and sending the input data to a plurality of target processing circuits corresponding to the erasure correcting task requests;

the matrix scheduling circuit is used for calling the erasure correcting matrix according to the matrix information and sending the erasure correcting matrix to the target processing circuit;

the configuration scheduling circuit is used for sending the configuration information to the target processing circuit and the target output data scheduling circuit which is connected with the target processing circuit in a one-to-one correspondence manner;

n processing circuits; the target processing circuit in the processing circuit is used for carrying out erasure correction calculation according to the input data, the erasure correction matrix and the configuration information, and N is more than or equal to 1;

n output data scheduling circuits; and the target output data scheduling circuit in the output data scheduling circuit is used for outputting the output data according to the configuration information.

Preferably, the target output data scheduling circuit is further configured to count the number of times the target processing circuit outputs the calculation result, and send a notification of task completion to the task scheduler when the number of times the target processing circuit outputs the calculation result matches the expected number of times of the task.

Preferably, the task scheduler is further configured to send a task end prompt after receiving a notification of task end.

Preferably, the matrix scheduling circuit is specifically configured to, when the erasure correcting task request is a decoding request, call an original encoding matrix, and calculate, according to the original encoding matrix, a decoding matrix corresponding to the decoding request.

Preferably, data flow control exists between the input data scheduling circuit and the target processing circuit, between the matrix scheduling circuit and the target processing circuit, between the configuration scheduling circuit and the target processing circuit, and between the configuration scheduling circuit and the target output data scheduling circuit.

Preferably, the input data scheduling circuit is specifically configured to send the input data to the target processing circuits in the processing circuits, where the target processing circuits correspond to the erasure correcting task requests according to the numbering sequence of the processing circuits.

Preferably, the task receiving interface is specifically configured to receive the erasure correcting task request from a distributed storage node connected to the erasure correcting processing system.

A distributed storage system comprising an erasure processing system as described in any of the above, distributed storage nodes connected to the erasure processing system.

An erasure processing method, comprising:

a task scheduler arranged on an erasure correction processing carrier receives an erasure correction task request by utilizing a task receiving interface, and analyzes the erasure correction task request to obtain data information, matrix information and configuration information;

and the task processor arranged on the erasure correction processing carrier calls input data according to the data information, calls an erasure correction matrix according to the matrix information, performs erasure correction calculation according to the input data, the erasure correction matrix and the configuration information to obtain output data, and outputs the output data according to the configuration information.

The application provides an erasure correcting processing system, an erasure correcting processing method and a distributed storage system, wherein the erasure correcting processing system comprises an erasure correcting processing carrier, a task scheduler and a task processor, the task scheduler and the task processor are arranged on the erasure correcting processing carrier, a task receiving interface is arranged on the task scheduler, and the erasure correcting processing system comprises: the task scheduler is used for receiving the erasure correcting task request by using the task receiving interface and analyzing the erasure correcting task request to obtain data information, matrix information and configuration information; and the task processor is used for calling input data according to the data information, calling an erasure correction matrix according to the matrix information, performing erasure correction calculation according to the input data, the erasure correction matrix and the configuration information to obtain output data, and outputting the output data according to the configuration information.

According to the technical scheme, the task scheduler and the task processor are arranged on the erasure correction processing carrier, the erasure correction task request is received by using the task receiving interface on the task scheduler, the erasure correction task request is analyzed by using the task scheduler, and erasure correction processing is performed by using the task processor according to information obtained by analysis, so that erasure correction processing is performed by using a hardware structure and is not realized by related software running on a CPU, occupation of computation resources of the CPU by erasure correction processing is reduced, and expenditure of the CPU is reduced.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

Fig. 1 is a schematic structural diagram of an erasure processing system according to an embodiment of the present application;

fig. 2 is a schematic structural diagram of a distributed storage system according to an embodiment of the present application;

fig. 3 is a schematic diagram of an erasure processing system for simultaneously processing encoding and decoding according to an embodiment of the present application;

fig. 4(a) is a schematic diagram of a construction method of the coding matrix in fig. 3 according to an embodiment of the present application;

fig. 4(b) is a schematic diagram of a construction method corresponding to the decoding matrix 1 in fig. 3 according to an embodiment of the present application;

fig. 4(c) is a schematic diagram of a construction method of the decoding matrix 2 in fig. 3 according to an embodiment of the present application;

fig. 4(d) is a schematic diagram of a construction method corresponding to the decoding matrix 3 in fig. 3 according to an embodiment of the present application;

fig. 5 is a flowchart of working data of an erasure processing system according to an embodiment of the present application;

fig. 6 is a schematic diagram illustrating scheduling of three consecutive erasure correcting task requests according to an embodiment of the present application;

fig. 7 is a flowchart of an erasure processing method according to an embodiment of the present application.

Detailed Description

In the face of the storage requirement of mass data, the distributed storage gradually replaces the dominant position of unified storage by the advantages of low cost, good expandability and the like, and has gained more and more attention in the aspects of theoretical research and practical application. On the other hand, a distributed storage system usually comprises a plurality of nodes, and the system often has node failure due to software and hardware failures, human errors and the like. In order to improve the data reliability of the distributed storage system and ensure that the data collection node can realize the reconstruction of the original file with high probability, a certain amount of redundancy needs to be additionally stored on the basis of storing the original data, so that the system can still normally operate under the condition that partial nodes fail, and the data collection node can still realize decoding recovery of the original file. Meanwhile, in order to maintain the reliability of the system, the failed node needs to be repaired in time, so that it is very important to design a good node repair mechanism.

Erasure Code (Erasure Code) belongs to a forward error correction technique in the coding theory, and is applied to the communication field for the first time to solve the problems of loss and loss in data transmission. Erasure codes can effectively reduce storage overhead on the premise of ensuring the same reliability, so erasure code technology is widely applied to various large storage systems and data centers.

For a common RS code applied in a distributed environment, which is related to two parameters k (number of data blocks) and r (number of check blocks), given two positive integers k and r, the RS code encodes k data blocks into r additional check blocks, and the way in which r check blocks are encoded based on a vandermonde matrix or a cauchy matrix is called an RS erasure code using vandermonde matrix or cauchy matrix, specifically, the van der dermonde matrix-based RS erasure code is as follows:

the RS erasure code based on the Cauchy matrix is as follows:

in the two RS erasure codes, the upper k × k matrix corresponds to k original data blocks, and the lower r × k matrix corresponds to a coding matrix, which is obtained by correlating with original data D₁To D_kMultiplying to obtain newly added P₁To P_rThe resulting r check data are encoded. When any data less than r is in error or lost in transmission and error correction is needed, the inverse matrix of the matrix corresponding to the remaining data is used to multiply the data (multiplication in a GF Field), that is, the original data block D is obtained₁To D_k。

With D₁To D_rFor example, decoding is performed after data loss, and the specific implementation process is as follows:

in view of the above, the core of the erasure code is to construct a reversible coding matrix for generating the parity data, and the inverse matrix can be calculated to recover the original data. Common RS erasure codes use the above-described cauchy matrix or vandermonde matrix, which has the advantages that the resulting matrix is definitely reversible, any sub-matrix thereof is also reversible, and the size expansion of the matrix is simple.

D in the above introduction₁To D_k，P₁To P_rThe data unit of (a) may be 8bit, 16bit or other values (referred to herein asSign bit, symbol) depending on the bit width of the multiplication operation, i.e., the finite field definition of the galois field operation. For a task, if the block size is Zbit. For multiplication, if 8-bit galois field operation is used, it is split into (Z/8) independent data and the same matrix for operation, and the operation result is output. The encoded output of the 1 st byte can be found as follows:

calculate for the 1 st byte

Therefore, one obvious characteristic of erasure correction is that each data block and check block are irrelevant between different symbols.

For erasure, currently, the industry mostly uses open source software running on a general-purpose CPU, but this implementation method occupies the computing resources of the CPU and increases the cost of the CPU.

Therefore, the application provides an erasure correction processing system, an erasure correction processing method and a distributed storage system, which are used for reducing the occupation of CPU computing resources by erasure correction.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

Referring to fig. 1, which shows a schematic structural diagram of an erasure correction processing system provided in an embodiment of the present application, an erasure correction processing system provided in an embodiment of the present application may include an erasure correction processing carrier, a task scheduler 1 and a task processor 2 that are disposed on the erasure correction processing carrier, where a task receiving interface is disposed on the task scheduler 1, where:

the task scheduler 1 is used for receiving the erasure correcting task request by using the task receiving interface and analyzing the erasure correcting task request to obtain data information, matrix information and configuration information;

and the task processor 2 is used for calling input data according to the data information, calling an erasure correction matrix according to the matrix information, performing erasure correction calculation according to the input data, the erasure correction matrix and the configuration information to obtain output data, and outputting the output data according to the configuration information.

In this application, the erasure correcting processing system may include an erasure correcting processing carrier, a task scheduler 1 and a task processor 2, where the erasure correcting processing carrier may specifically be a chip or an FPGA (Field Programmable Gate Array), and the task scheduler 1 is provided with a task receiving interface for receiving an erasure correcting task request, and the task receiving interface may specifically be an I2C (Inter-Integrated Circuit), a general interface or a dedicated interface. It should be noted that the erasure correcting task request received by the task receiving interface on the task scheduler 1 may be an encoding request or a decoding request, and for encoding, the number k of data blocks and the check block data r, an encoding matrix, a decoding matrix, and block size information need to be configured; decoding needs to configure data block data k and check block data r, an encoding or decoding matrix, a block size, missing data blocks, and existing check block information, where the sum of the missing data blocks and the check blocks is lbn (lock block number, number of lost blocks), and as long as lbn is less than or equal to r, erasure decoding can recover all missing data blocks and check blocks.

After receiving the erasure correction task request by using the task receiving interface, the task scheduler 1 located on the erasure correction processing carrier may parse the erasure correction task request to obtain data information, matrix information, and configuration information, where the data information may specifically include an erasure correction policy, a block size (where the data block and the check block have the same size, that is, the size of each block is the same), and the like, where the erasure correction policy may specifically be represented as k + r, that is, the number of data blocks + the number of check blocks, for example: 4+2, 6+2, etc., the block size facilitates the erasure correction processing system to know when the current task request ends and when the next task request begins, i.e., to separate the task request from the task request, the matrix information is the matrix information required for erasure correction processing, specifically, the matrix index information (the number k of input data blocks, the number lbn of output data blocks, etc.), so as to determine the specific information of the matrix according to the matrix index information and the type of the required matrix (specifically, the matrix information may be determined according to the erasure correction task request), and the configuration information may include an erasure correction policy and a block size, so as to know how much data is input and output, when the current task request ends, etc., according to the configuration information.

After parsing the erasure correction task request, the task scheduler 1 on the erasure correction processing carrier may send the parsed relevant information to the task processor 2 on the erasure correction processing carrier. After receiving the relevant information, the task processor 2 calls in the input data according to the data information, and calls in the erasure correction matrix according to the matrix information, and specifically, the input data and the erasure correction matrix may be completed through external auxiliary hardware (such as Direct Memory Access (DMA)) or may be called in from a distributed storage node connected to the erasure correction processing system. It should be noted that the erasure matrix mentioned herein may be specifically an original encoding matrix or a decoding matrix required for decoding.

After the input data and the erasure matrix are called, the task processor 2 may perform corresponding calculation by using the input data and the erasure matrix according to the configuration information to calculate output data, and output the calculated output data according to information such as block size in the configuration information, where for the output of the output data, see fig. 2, it shows a schematic structural diagram of the distributed storage system provided in the embodiment of the present application, that is, the erasure processing system may output the output data obtained by responding to the erasure task request to the distributed storage nodes connected thereto, so that the distributed storage nodes store the output data in the corresponding storage pools.

It should be noted that, in the above process, the task processor 2 may split the block according to the sign bit, perform operation on the split independent data, and then recombine the calculation result obtained by the operation according to the configuration information to obtain the final output data.

Through the process, the erasure correction processing is completed through the hardware structures of the task scheduler 1 and the task processor 2 which are arranged on the erasure correction processing carrier, and the erasure correction processing is not required to be realized through open source software running on a CPU, so that compared with the existing erasure correction processing method, the method can reduce the consumption of CPU computing resources and reduce the expense of the CPU.

In addition, it should be noted that the erasure correcting processing system of the present application can be applicable to multiple RS erasure correcting codes, and can process both encoding and decoding, so as to achieve the same hardware structure, and implement all requirements of RS erasure correcting encoding and decoding in a multiplexing manner, and the implementation method thereof is specifically as follows:

referring to fig. 3, a schematic diagram of the erasure correction processing system provided in the embodiment of the present application for processing encoding and decoding at the same time is shown, and the encoding of 4+2, the decoding of 4+2 (

data block

2, 3 needs to be recovered), the decoding of 4+2 (data block 3, parity block 1 needs to be recovered), and the decoding of 4+2 (parity block 1 needs to be recovered) are taken as examples in the drawing for explanation. The erasure correcting hardware (i.e. the above mentioned task processor 2) is responsible for multiplication of matrix and data, and different types of matrices are provided for the erasure correcting hardware to complete encoding or decoding: providing a coding matrix, wherein erasure hardware can complete coding; a decoding matrix is provided and erasure correction hardware can complete the decoding. As shown, the encoding matrix is the original encoding matrix; the decoding matrix 1 is the inverse of the encoding matrix (a 2-row 4-column matrix solved for the missing data blocks 2, 3); the decoding matrix 2 is a new matrix (2 rows and 4 columns) formed by 1 row element (1 row and 4 columns matrix) of the inverse matrix of the lost data block 3 and a matrix (1 row and 4 columns matrix) calculated according to the inverse matrix and 1 row of the coding matrix, and the decoding matrix 3 is the first row of the coding matrix, specifically, see fig. 4(a) -4 (D), wherein fig. 4(a) shows a schematic diagram of a construction method corresponding to the coding matrix in fig. 3, wherein D represents a data block, C represents a check block, B represents a corresponding matrix, the coding process is to find C1 and C2 (i.e., corresponding check block 1 and check block 2 in fig. 3), and the last two rows area in B is the coding matrix; FIG. 4(b) is a schematic diagram of a construction method corresponding to the decoding matrix 1 in FIG. 3, and the corresponding decoding is to find D2 (i.e. the corresponding data block 2 in FIG. 3) and D3 (i.e. the corresponding data block 2 in FIG. 3)Corresponding data block 3) in FIG. 3, B'₁Is the matrix in B after the data corresponding to D2 and D3 are lost, B'₁ ^-1Is B'₁Surfivors represents the remaining blocks, and B'₁ ^-1The second and third rows in (1) are decoding matrices; FIG. 4(C) is a schematic diagram of a construction method of the decoding matrix 2 in FIG. 3, and the corresponding decoding is to find D3 (i.e. the corresponding data block 3 in FIG. 3) and C1 (i.e. the corresponding check block 1, B 'in FIG. 3)'₂Is the matrix in B after the data corresponding to D3 and C1 are lost, B'₂ ^-1Is B'₂Inverse matrix of (1), in B'₂ ^-1*B’₂*D＝B’₂ ^-1Multiplying both sides of Survivors by the row of matrix C1 in the original coding matrix, and marking the result of the dotted frame as a new matrix B ', and B'₂ ^-1Are spliced together to form decoding matrix 2, and decoding matrix 2 and [ D1, D2, D4, C2 [ ]]By calculation, D3 and C1 can be solved simultaneously; fig. 4(d) shows a schematic diagram of a construction method corresponding to the decoding matrix 3 in fig. 3, and the check block 1 is solved by using the first row in the encoding matrix as the decoding matrix 3.

Therefore, by scheduling the matrix (the coding matrix or the decoding matrixes of different types) and the input data (the data block or the survival data block and the survival check block) by the erasure processing system, three typical application scenarios (only part of check blocks are calculated, only the data block is calculated, and the data block and the check block are calculated) are realized by supporting both coding and decoding by hardware.

Referring to fig. 5 and fig. 6, fig. 5 shows a flowchart of working data of an erasure correction processing system provided by the embodiment of the present application, and fig. 6 shows a schematic diagram of scheduling three consecutive erasure correction task requests provided by the embodiment of the present application. In an erasure processing system provided in an embodiment of the present application, the task processor 2 may include:

the input data scheduling circuit 21 is used for calling input data according to the data information and sending the input data to the target processing circuits corresponding to the erasure correcting task requests;

the matrix scheduling circuit 22 is used for calling an erasure correction matrix according to the matrix information and sending the erasure correction matrix to the target processing circuit;

the configuration scheduling circuit 23 is configured to send configuration information to the target processing circuit and the target output data scheduling circuit connected to the target processing circuit in a one-to-one correspondence manner;

n processing circuits 24; a target processing circuit in the processing circuit 24, configured to perform erasure correction calculation according to the input data, the erasure correction matrix, and the configuration information, where N is greater than or equal to 1;

n output data scheduling circuits 25; a target output data scheduling circuit in the output data scheduling circuit 25 for outputting the output data according to the configuration information.

The task processor 2 in this application may comprise input data scheduling circuitry 21, matrix scheduling circuitry 22, configuration scheduling circuitry 23, N processing circuitry 24, N output data scheduling circuitry 25, wherein, the input data scheduling circuit 21, the matrix scheduling circuit 22 and the configuration scheduling circuit 23 are all connected with the task scheduler 1 and the N processing circuits 24, the configuration scheduling circuit 23 is further connected to N output data scheduling circuits 25, the N processing circuits 24 are connected to the N output data scheduling circuits 25 in a one-to-one correspondence, n is an integer greater than 1, the number of the processing circuits 24 and the output data scheduling circuits 25 can be expanded according to the erasure correction processing requirement, and the number k of the data blocks supported by each processing circuit 24 has no upper limit, that is, the erasure correction processing system in the present application can process any erasure correction strategy, and has strong expandability. It should be noted that fig. 1 illustrates an example where N is 6, but the number of N is not limited to 6, and the structure and the processing procedure of N being other positive integers are similar to those in fig. 2, and are not described again here.

After the task scheduler 1 analyzes the erasure correction task request to obtain the relevant information, the data information may be sent to the input data scheduling circuit 21, the input data scheduling circuit 21 calls input data from the outside according to the received data information, and sends the input data to the number of target processing circuits corresponding to the erasure correction task request, that is, selects a corresponding number of target processing circuits from the processing circuits 24 according to an erasure correction policy included in the erasure correction task request and sends the input data, for example, for a 7+3 coding request, at this time, 7 data blocks are sent to 3 target processing circuits (each target processing circuit receives 7 data blocks). It should be noted that, in this embodiment, the data information further includes the target processing circuits to which the input data is sent, that is, the input data is sent to the target processing circuits by using a broadcast principle.

The task scheduler 1 transmits the matrix information obtained by the analysis to the matrix scheduling circuit 22 connected thereto, and the matrix scheduling circuit 22 is responsible for externally tuning in the erasure correction matrix based on the matrix information, specifically, for an encoding request, the encoding matrix can be tuned in from the outside, for a decoding request, the decoding matrix can be tuned in from the outside, and the like. The matrix scheduling circuit 22, after tuning in the erasure correction matrix, transmits the erasure correction matrix to each of the above-mentioned target processing circuits.

While the task scheduler 1 sends the correspondence information to the input data scheduling circuit 21 and the matrix scheduling circuit 22, it may also send the parsed configuration information to the configuration scheduling circuit 23 connected thereto. After receiving the configuration information, the configuration scheduling circuit 23 sends the configuration information to each target processing circuit and the target output data scheduling circuit connected to the target processing circuit in a one-to-one correspondence manner, so that the target processing circuit and the target output data scheduling circuit know when the current task ends according to the configuration information, and the target output data scheduling circuit can output data according to the configuration information.

In the erasure correction processing system, when the target processing circuits corresponding to the erasure correction task request among the N processing circuits 24 receive the relevant information, each target processing circuit performs erasure correction calculation by using the input data and the erasure correction matrix according to the configuration information, respectively, to implement parallel erasure correction calculation. For example: for the processing of the erasure task request 1 in fig. 6, 3 processing circuits 24 in the processing circuits 24 are selected as target processing circuits, the first target processing circuit is used to calculate the parity check block 1, the second target processing circuit is used to calculate the parity check block 2, and the third target processing circuit is used to calculate the parity check block 3; for erasure task request 2 in fig. 6, 2 processing circuits 24 are selected as target processing circuits, and one of the target processing circuits is used to calculate missing data block 1, and the other target processing circuit is used to calculate missing data block 4; for erasure correction task request 3 in fig. 6, the implementation process is similar to the first two, and it selects 6 processing circuits 24 as target processing circuits to perform parallel erasure correction computation, so as to obtain data block 2, data block 4, data block 7, data block 9, check block 1, and check block 5, respectively (the output throughput rate reaches 100%). Compared with the prior art that the erasure correction calculation is carried out on the CPU in a serial mode, the parallel calculation is realized through the plurality of processing circuits 24, the erasure correction calculation efficiency is improved, and the data throughput rate is improved. It should be noted that the processing circuit 24 in the present application may be specifically a galois field multiplication unit to perform erasure correction calculation.

In the erasure correction processing system, a target output data scheduling circuit connected to a target processing circuit among the N output data scheduling circuits 25 receives a calculation result obtained by the target processing circuit performing erasure correction calculation each time, processes the calculation result according to the configuration information to obtain output data, and outputs the output data.

Through the process, the concurrence is realized through the multi-path processing circuit, the throughput rate of processing the erasure correction task is greatly improved due to the hardware structure that the single processing circuit circularly calculates the input data, and the erasure correction processing efficiency and reliability are improved due to the fact that special functions can be realized through the special circuits.

In the erasure correction processing system provided in the embodiment of the present application, the target output data scheduling circuit is further configured to count the number of times that the target processing circuit outputs the calculation result, and send a notification of task completion to the task scheduler 1 when the number of times that the target processing circuit outputs the calculation result is consistent with the expected number of times of the task.

In this application, the target output data scheduling circuit may count the number of times the target processing circuit outputs the calculation result when receiving the calculation result output by the target processing circuit, and when the number of times the target processing circuit outputs the calculation result is consistent with the expected number of times of the task, it indicates that the target processing circuit completes erasure correction calculation corresponding to the erasure correction task request, and at this time, each target output data scheduling circuit may send a notification of completion of the task to the task scheduler 1, so that the task scheduler 1 sends a task response, and the task scheduler 1 is convenient to receive a subsequent task, and the like. Where the task expectation times are specifically equal to the block size/sign bit.

The erasure processing system provided in the embodiment of the present application, the task scheduler 1, is further configured to send a task end prompt after receiving a notification of task end.

After receiving the notification of the end of the task sent by the target output data scheduling circuit, the task scheduler 1 may also send a task end prompt, so that the distributed storage node and/or the user may know that the erasure task request has ended, thereby facilitating taking measures, such as: storing, using, etc. the output data.

The matrix scheduling circuit 22 is specifically configured to call an original encoding matrix when the erasure correcting task request is a decoding request, and calculate a decoding matrix corresponding to the decoding request according to the original encoding matrix.

In this application, when the matrix task request is a decoding request, the matrix scheduling circuit 22 may call in an original encoding matrix (the original encoding matrix is used for calculating each check block during encoding), calculate a corresponding decoding matrix through gaussian elimination according to the original encoding, and send the decoding matrix to the target processing unit. Of course, the matrix scheduling circuit 22 may also directly call the corresponding decoding matrix from the outside to send to the target processing circuit for calculation, which is not limited in this application.

In the implementation, for the case where both the data block and the parity block are lost (specifically, as the case corresponding to the decoding matrix 2 in fig. 3), the matrix scheduling circuit 22 is used to directly obtain the required decoding matrix, so that the target processing circuit can directly calculate the data block and the parity block by using the decoding matrix, and does not need to perform decoding calculation to obtain the lost data block and then encode and calculate the lost parity block as in the prior art, thereby reducing the execution steps and improving the erasure processing efficiency and performance.

In the erasure processing system provided in the embodiment of the present application, data flow control exists between the input data scheduling circuit 21 and the target processing circuit, between the matrix scheduling circuit 22 and the target processing circuit, between the configuration scheduling circuit 23 and the target processing circuit, and between the configuration scheduling circuit 23 and the target output data scheduling circuit.

In the present application, data flow control exists between the input data scheduling circuit 21 and the target processing circuit, between the matrix scheduling circuit 22 and the target processing circuit, between the configuration scheduling circuit 23 and the target processing circuit, and between the configuration scheduling circuit 23 and the target output data scheduling circuit, where the data flow control mentioned here is that data flow between circuits is a handshake mechanism, for example, a next circuit cannot receive data, and a previous unit cannot continue to transmit data, so as to ensure that related data is not lost. Specifically, the data flow control between the input data scheduling circuit 21 and the target processing circuit ensures that input data is not lost, the data flow control between the matrix scheduling circuit 22 and the target processing circuit ensures that erasure matrices are not lost, and the data flow control between the configuration scheduling circuit 23 and the target processing circuit and the data flow control between the configuration scheduling circuit 23 and the target output data scheduling circuit ensure that configuration information is not lost.

Data loss can be prevented as much as possible through data flow control, so that reliability of erasure correction processing is improved.

In the erasure correction processing system provided in the embodiment of the present application, the input data scheduling circuit 21 is specifically configured to send the input data to a number of target processing circuits corresponding to the erasure correction task request in the processing circuit 24 according to the numbering sequence of the processing circuit 24.

When the input data scheduling circuit 21 transmits the input data to the target processing circuits in the processing circuits 24, the input data may be specifically transmitted to the corresponding number of target processing circuits according to the numbering sequence of the processing circuits 24, and specifically, the target processing circuits may be selected in the descending order of the numbering sequence and the input data may be transmitted, so as to prevent confusion of data transmission and improve erasure processing performance.

In the erasure correction processing system provided in the embodiment of the present application, the task receiving interface is specifically configured to receive an erasure correction task request from a distributed storage node connected to the erasure correction processing system.

In the present application, the task receiving interface in the task scheduler 1 may specifically receive erasure correcting task requests from distributed storage nodes connected to the erasure correcting processing system, and specifically as shown in fig. 2, the task receiving interface may sequentially receive a plurality of erasure correcting task requests from distributed storage nodes connected to the erasure correcting processing system, and perform serial processing on the erasure correcting task requests according to the receiving order, and send output data corresponding to task responses to the distributed storage nodes connected to the erasure correcting processing system, and the output data is stored in corresponding storage pools by the distributed storage nodes. In addition, as can be seen from fig. 2, the erasure correction processing system in the present application also supports the pipeline design of input and output, and fully ensures the input.

Of course, the task receiving interface may also receive an erasure correcting task request sent by the user, and the method for the task receiving interface to receive the erasure correcting task request is not limited in this application.

The embodiment of the application further provides a distributed storage system, which may include any one of the erasure correcting processing systems and a distributed storage node connected to the erasure correcting processing system.

The specific structure of the distributed storage system provided by the embodiment of the application can be seen in fig. 2, and the erasure processing system is arranged in the distributed storage system to realize repair, so that the stability of the distributed storage system is improved.

An erasure correction processing method is further provided in the embodiment of the present application, and referring to fig. 7, a flowchart of an erasure correction processing method provided in the embodiment of the present application is shown, where the erasure correction processing method may include:

s71: a task scheduler arranged on an erasure correction processing carrier receives an erasure correction task request by utilizing a task receiving interface, and analyzes the erasure correction task request to obtain data information, matrix information and configuration information;

s72: and the task processor arranged on the erasure correction processing carrier calls in input data according to the data information, calls in an erasure correction matrix according to the matrix information, performs erasure correction calculation according to the input data, the erasure correction matrix and the configuration information to obtain output data, and outputs the output data according to the configuration information.

In an erasure correction processing method provided in an embodiment of the present application, a task processor calls input data according to data information, calls an erasure correction matrix according to matrix information, performs erasure correction calculation according to the input data, the erasure correction matrix, and configuration information to obtain output data, and outputs the output data according to the configuration information, including:

the input data scheduling circuit calls input data according to the data information and sends the input data to a number of target processing circuits corresponding to the erasure correcting task request;

the matrix scheduling circuit is used for calling an erasure correction matrix according to the matrix information and sending the erasure correction matrix to the target processing circuit;

the configuration scheduling circuit sends the configuration information to the target processing circuit and the target output data scheduling circuit which is connected with the target processing circuit in a one-to-one correspondence mode;

a target processing circuit in the N processing circuits carries out erasure correction calculation according to input data, erasure correction matrixes and configuration information, wherein N is more than or equal to 1;

and the target output data scheduling circuit in the N output data scheduling circuits outputs output data according to the configuration information.

The erasure correction processing method provided in the embodiment of the present application further includes:

the target output data scheduling circuit counts the number of times the target processing circuit outputs the calculation result, and when the number of times the target processing circuit outputs the calculation result matches the desired number of times of the task, sends a notification of completion of the task to the task scheduler.

and the task scheduler sends out a task ending prompt after receiving the notification of task ending.

In an erasure correction processing method provided in an embodiment of the present application, when an erasure correction task request is a decoding request, a matrix scheduling circuit calls an erasure correction matrix according to matrix information, including:

the matrix scheduling circuit calls the original coding matrix and calculates a decoding matrix corresponding to the decoding request according to the original coding matrix.

and data flow control is carried out between the input data scheduling circuit and the target processing circuit, between the matrix scheduling circuit and the target processing circuit, between the configuration scheduling circuit and the target processing circuit and between the configuration scheduling circuit and the target output data scheduling circuit.

In an erasure correction processing method provided in an embodiment of the present application, an input data scheduling circuit sends input data to a number of target processing circuits corresponding to an erasure correction task request, including:

and the input data scheduling circuit sends the input data to a plurality of target processing circuits corresponding to the erasure correcting task requests in the processing circuits according to the numbering sequence of the processing circuits.

In an erasure processing method provided in an embodiment of the present application, a task receiving interface receives an erasure task request, including:

the task receiving interface receives an erasure task request from a distributed storage node connected to an erasure processing system.

For a detailed description of a relevant part in the distributed storage system and the erasure correction processing method provided in the embodiment of the present application, reference may be made to a detailed description of a corresponding part in the erasure correction processing system provided in the present application, and details are not described herein again.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include elements inherent in the list. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element. In addition, parts of the above technical solutions provided in the embodiments of the present application, which are consistent with the implementation principles of corresponding technical solutions in the prior art, are not described in detail so as to avoid redundant description.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An erasure correction processing system is characterized by comprising an erasure correction processing carrier, a task scheduler and a task processor, wherein the task scheduler and the task processor are arranged on the erasure correction processing carrier, the task scheduler is provided with a task receiving interface, and the erasure correction processing system comprises:

2. An erasure processing system according to claim 1, wherein the task processor includes:

3. An erasure processing system according to claim 2, wherein the target output data scheduling circuit is further configured to count a number of times the target processing circuit outputs the calculation result, and to send a notification of task end to the task scheduler when the number of times the target processing circuit outputs the calculation result matches a task expected number of times.

4. An erasure processing system according to claim 3, wherein the task scheduler is further configured to issue a task end prompt upon receiving notification of task end.

5. The erasure processing system according to claim 2, wherein the matrix scheduling circuit is specifically configured to, when the erasure task request is a decoding request, call an original encoding matrix, and calculate, according to the original encoding matrix, a decoding matrix corresponding to the decoding request.

6. A erasure processing system according to claim 2, wherein data flow control exists between the input data scheduling circuit and the target processing circuit, between the matrix scheduling circuit and the target processing circuit, between the configuration scheduling circuit and the target processing circuit, and between the configuration scheduling circuit and the target output data scheduling circuit.

7. The erasure processing system of claim 2, wherein the input data scheduling circuit is specifically configured to send the input data to the target processing circuits corresponding to the erasure task requests in the processing circuits according to the numbering sequence of the processing circuits.

8. An erasure processing system according to claim 1, wherein the task receiving interface is specifically configured to receive the erasure task request from a distributed storage node connected to the erasure processing system.

9. A distributed storage system comprising an erasure processing system according to any one of claims 1 through 8, a distributed storage node connected to the erasure processing system.

10. An erasure processing method, comprising: