CN112052115A - Data storage erasure method, device, equipment and computer readable storage medium - Google Patents
Data storage erasure method, device, equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN112052115A CN112052115A CN202011041807.1A CN202011041807A CN112052115A CN 112052115 A CN112052115 A CN 112052115A CN 202011041807 A CN202011041807 A CN 202011041807A CN 112052115 A CN112052115 A CN 112052115A
- Authority
- CN
- China
- Prior art keywords
- target
- erasure
- data
- matrix
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1008—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices
- G06F11/1048—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices using arrangements adapted for a specific error detection or correction feature
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11C—STATIC STORES
- G11C29/00—Checking stores for correct operation ; Subsequent repair; Testing stores during standby or offline operation
- G11C29/04—Detection or location of defective memory elements, e.g. cell constructio details, timing of test signals
- G11C29/08—Functional testing, e.g. testing during refresh, power-on self testing [POST] or distributed testing
- G11C29/12—Built-in arrangements for testing, e.g. built-in self testing [BIST] or interconnection details
- G11C29/38—Response verification devices
- G11C29/42—Response verification devices using error correcting codes [ECC] or parity check
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Error Detection And Correction (AREA)
- Detection And Correction Of Errors (AREA)
Abstract
The invention discloses a data storage erasure method, which comprises the following steps: analyzing the received target data erasure task to obtain a target data block and a target matrix; acquiring a target resource block corresponding to a target data erasure task; and carrying out erasure correction operation on the target data block and the target matrix by using the target resource block based on a time division multiplexing algorithm to obtain an erasure correction result. By applying the technical scheme provided by the embodiment of the invention, the occupation amount of CPU computing resources is greatly reduced, and the resource utilization rate is improved. The invention also discloses a data storage erasure correcting device, equipment and a storage medium, and has corresponding technical effects.
Description
Technical Field
The present invention relates to the field of storage technologies, and in particular, to a data storage erasure method, apparatus, device, and computer-readable storage medium.
Background
In order to improve the data reliability of the distributed storage system and ensure that the data collection node can realize the reconstruction of the original file with high probability, a certain amount of redundancy needs to be additionally stored on the basis of storing the original data, so that the system can still normally operate under the condition that partial nodes fail, and the data collection node can still realize decoding recovery of the original file. Meanwhile, in order to maintain the reliability of the system, the failed nodes need to be repaired in time.
Erasure Code (Erasure Code) belongs to a forward error correction technology in coding theory, and has a good effect in preventing data loss, so that Erasure Code is introduced into the storage field for application.
The existing data storage and erasure correcting mode is that in the process of executing an encoding task or a decoding task, a resource block is respectively allocated to each path of output data buffer, and one data erasure correcting task may occupy a plurality of resource blocks, occupies more computing resources of a CPU, and has a low resource utilization rate.
In summary, how to effectively solve the problems of the existing data storage erasure correction mode, such as more occupation of CPU computing resources and lower resource utilization rate, is a problem that needs to be solved urgently by those skilled in the art at present.
Disclosure of Invention
The invention aims to provide a data storage erasure correcting method which greatly reduces the occupation amount of CPU computing resources and improves the resource utilization rate; another object of the present invention is to provide a data storage erasure correcting apparatus, device and computer readable storage medium.
In order to solve the technical problems, the invention provides the following technical scheme:
a data storage erasure method, comprising:
analyzing the received target data erasure task to obtain a target data block and a target matrix;
acquiring a target resource block corresponding to the target data erasure task;
and carrying out erasure correction operation on the target data block and the target matrix by using the target resource block based on a time division multiplexing algorithm to obtain an erasure correction result.
In a specific embodiment of the present invention, parsing the received target data erasure task to obtain a target data block and a target matrix includes:
analyzing the received target data erasure task to obtain a target erasure task type;
when the target erasure task type is coding, acquiring a target original data block and a target coding matrix;
and when the target erasure correcting task type is decoding, acquiring the target residual data block, the target verification data block and a target decoding matrix.
In a specific embodiment of the present invention, when the number of the target data erasure correcting tasks is multiple, acquiring a target resource block corresponding to the target data erasure correcting task includes:
acquiring each current idle resource block;
performing erasure operation on the target data block and the target matrix by using the target resource block based on a time division multiplexing algorithm, including:
and performing parallel erasure operation on each target data block and each target matrix by using each resource block based on the time division multiplexing algorithm.
In a specific embodiment of the present invention, after parsing the received target data erasure task to obtain a target data block and a target matrix, the method further includes:
acquiring the data size of the target data block;
performing parallel erasure operation on each target data block and each target matrix by using each resource block based on the time division multiplexing algorithm, wherein the parallel erasure operation comprises the following steps:
distributing each resource block to each target data block and each matrix row of each target matrix in a time division multiplexing mode according to the data size;
and carrying out erasure correction operation on each target data block and each matrix row by using each resource block.
A data storage erasure apparatus, comprising:
the task analysis module is used for analyzing the received target data erasure task to obtain a target data block and a target matrix;
a resource block obtaining module, configured to obtain a target resource block corresponding to the target data erasure task;
and the erasure result obtaining module is used for carrying out erasure operation on the target data block and the target matrix by utilizing the target resource block based on a time division multiplexing algorithm to obtain an erasure result.
In a specific embodiment of the present invention, the task parsing module includes:
the task type obtaining submodule is used for analyzing the received target data erasure correcting task to obtain a target erasure correcting task type;
the first input data acquisition submodule is used for acquiring a target original data block and a target coding matrix when the target erasure task type is coding;
and the second input data acquisition submodule is used for acquiring a target residual data block, a target verification data block and a target decoding matrix when the target erasure task type is decoding.
In a specific embodiment of the present invention, the resource block obtaining module is specifically a module for obtaining each currently idle resource block when the number of the target data erasure correcting tasks is multiple;
the erasure result obtaining module is specifically a module for performing parallel erasure operation on each target data block and each target matrix by using each target resource block based on the time division multiplexing algorithm.
In one embodiment of the present invention, the method further comprises:
the data size acquisition module is used for acquiring the data size of the target data block after analyzing the received target data erasure task to obtain a target data block and a target matrix;
the erasure result obtaining module includes:
the resource block distribution submodule is used for distributing each resource block to each target data block and each matrix row of each target matrix in a time division multiplexing mode according to the data size;
and the erasure correction operation submodule is used for carrying out erasure correction operation on each target data block and each matrix row by using each resource block.
A data storage erasure device, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data storage erasure correction method as described above when executing the computer program.
A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the data storage erasure method as set out above.
By applying the method provided by the embodiment of the invention, the received target data erasure task is analyzed to obtain a target data block and a target matrix; acquiring a target resource block corresponding to a target data erasure task; and carrying out erasure correction operation on the target data block and the target matrix by using the target resource block based on a time division multiplexing algorithm to obtain an erasure correction result. By adopting the time division multiplexing algorithm to realize erasure correction operation in a specific target resource block after acquiring the target data block and the target matrix required by erasure correction processing, compared with the existing erasure correction mode of distributing one resource block for each path of output data buffer, the invention greatly reduces the occupation amount of CPU computing resources and improves the resource utilization rate.
Accordingly, embodiments of the present invention further provide a data storage erasure correcting apparatus, a device and a computer-readable storage medium corresponding to the data storage erasure correcting method, which have the above technical effects and are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flowchart illustrating an implementation of a data storage erasure checking method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating another embodiment of a data storage erasure method according to an embodiment of the present invention;
FIG. 3 is a block diagram of the present single hardware processing unit;
FIG. 4 is a schematic diagram of resource partitioning of a hardware processing erasure task in the prior art;
FIG. 5 is a schematic diagram of resource partitioning for processing an erasure task by hardware according to an embodiment of the present invention;
FIG. 6 is a flowchart illustrating another implementation of a data storage erasure checking method according to an embodiment of the present invention;
FIG. 7 is a block diagram illustrating an exemplary data storage erasure correcting apparatus according to an embodiment of the present invention;
fig. 8 is a block diagram of a data storage erasure correcting apparatus according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The first embodiment is as follows:
referring to fig. 1, fig. 1 is a flowchart of an implementation of a data storage erasure checking method according to an embodiment of the present invention, where the method may include the following steps:
s101: and analyzing the received target data erasure task to obtain a target data block and a target matrix.
When data is backed up or pre-stored backup data is lost, a target data erasure task is generated, wherein the target data erasure task comprises a target data block and a target matrix required for erasure correction processing. And sending the target data erasure correcting task to an erasure correcting processing center, receiving the target data erasure correcting task by the erasure correcting processing center, and analyzing the received target data erasure correcting task to obtain a target data block and a target matrix.
Specifically, when the target data erasure task is an encoding task, the target data block includes an original data block, and the target matrix is a matrix written in the matrix queue in advance. When the target data erasure task is a decoding task, the target data block comprises an original data block and a check data block, and the target matrix is a matrix written into a matrix queue in advance.
S102: and acquiring a target resource block corresponding to the target data erasure task.
And allocating corresponding target resource blocks for the target data erasure tasks in advance. And after the target data block and the target matrix are obtained through analysis, a target resource block corresponding to the target data erasure task is obtained.
S103: and carrying out erasure correction operation on the target data block and the target matrix by using the target resource block based on a time division multiplexing algorithm to obtain an erasure correction result.
After a target resource block corresponding to the target data erasure task is obtained, erasure correction operation is performed on the target data block and the target matrix by using the target resource block based on a time division multiplexing algorithm, and an erasure correction result is obtained.
If RS (Reed-Solomon Code) codes are adopted for erasure correction processing, Galois field multiplication operation and Galois field addition operation can be sequentially carried out on a target data block and a first matrix row in a target matrix by utilizing a target resource block in a first time period; carrying out Galois field multiplication operation and Galois field addition operation on a target data block and a second matrix row in the target matrix in sequence by using the target resource block in a second time period; until the Galois field multiplication operation and the Galois field addition operation are carried out on the target data block and each matrix row in the target matrix. By adopting the time division multiplexing algorithm to realize erasure correction operation in a specific target resource block after acquiring the target data block and the target matrix required by erasure correction processing, compared with the existing erasure correction mode of distributing one resource block for each path of output data buffer, the invention greatly reduces the occupation amount of CPU computing resources and improves the resource utilization rate.
By applying the method provided by the embodiment of the invention, the received target data erasure task is analyzed to obtain a target data block and a target matrix; acquiring a target resource block corresponding to a target data erasure task; and carrying out erasure correction operation on the target data block and each matrix row in the target matrix by using the target resource block based on a time division multiplexing algorithm to obtain an erasure correction result. By adopting the time division multiplexing algorithm to realize erasure correction operation in a specific target resource block after acquiring a target data block and a target matrix required by erasure correction processing, compared with the existing erasure correction mode of distributing a hardware processing unit for each path of output data buffer, the invention greatly reduces the occupation amount of CPU computing resources and improves the resource utilization rate.
It should be noted that, based on the first embodiment, the embodiment of the present invention further provides a corresponding improvement scheme. In the following embodiments, steps that are the same as or correspond to those in the first embodiment may be referred to each other, and corresponding advantageous effects may also be referred to each other, which are not described in detail in the following modified embodiments.
Example two:
referring to fig. 2, fig. 2 is a flowchart of another implementation of a data storage erasure checking method according to an embodiment of the present invention, where the method may include the following steps:
s201: and analyzing the received target data erasure task to obtain the type of the target erasure task.
The target data erasure task comprises task type information, and after the target data erasure task is received, the received target data erasure task is analyzed to obtain a target erasure task type. The erasure task type includes encoding and decoding.
S202: and when the target erasure task type is coding, acquiring a target original data block and a target coding matrix.
When encoding, the original data block and the encoding matrix written in the matrix queue in advance are needed. And when the target erasure task type is coding, acquiring a target original data block and a target coding matrix.
S203: and when the target erasure correcting task type is decoding, acquiring a target residual data block, a target verification data block and a target decoding matrix.
When decoding is performed, the surviving remaining data blocks, the surviving check data blocks, and the decoding matrix written in the matrix queue in advance are needed. And when the target erasure correcting task type is decoding, acquiring a target original data block, a target verification data block and a target matrix.
S204: and when the number of the target data erasure tasks is multiple, acquiring each current idle resource block.
When the number of the target data erasure tasks is determined to be multiple, if multiple coding tasks exist currently, or multiple decoding tasks exist currently, or both coding tasks and decoding tasks exist currently, each current idle resource block is obtained.
S205: and performing parallel erasure operation on each target data block and each target matrix by using each resource block based on a time division multiplexing algorithm.
After each current idle resource block is obtained, parallel erasure operation is carried out on each target data block and each target matrix by using each resource block based on a time division multiplexing algorithm. By adopting the mode of multi-channel resource block concurrence and time division multiplexing in a single resource block, the resource utilization rate is improved, and the data throughput rate is further improved. And the number of input and output channels of the hardware structure and the size of a storage unit of the matrix can be expanded to support different data node numbers k and different check node numbers r, so that the expansibility is stronger, and one hardware can support all erasure combinations below the design index k and r. Meanwhile, one hardware realizes the support of both encoding and decoding through the scheduling of the matrix (namely, the encoding matrix or the decoding matrix) and the input data (namely, the data block or the data block and the existence check block).
In a specific example application, referring to fig. 3, fig. 3 is a structural block diagram of the single resource block. As shown in fig. 3, the proposed RS erasure correction resource block takes one maximum supported data node number k as 6 and one maximum supported check node number r as 4 as an example. The module is divided into a control path and a data path, wherein the control path is mainly responsible for scheduling the matrix and controlling data to flow into the multiplication unit; the data path includes data input and output control. The functions of the specific modules are described as follows:
1. a control path:
rows 1 to 4 of the matrix: storing the 1 st to 4 th rows of the matrix, respectively.
And (3) configuring a queue: the configuration information to be processed by the resource block for storing the erasure correction task comprises information such as the number k of data blocks, the number r of check blocks, the size of the blocks and the like.
Matrix queue: storing the encoding or decoding matrix.
A multiple-input-one-output unit: and selecting one path of signal from the output signals according to the selection signal and outputting the selected path of signal.
Resource block scheduling: the matrix in the matrix queue is transferred to the 1 st row to the 4 th row of the matrix by reading the information in the configuration queue; controlling data flow of the data path into the multiplication unit; a tune-in multiplication unit for time division multiplexing all (or part) of the 4-row matrix; and controlling a selection signal to selectively send 1 path of output data into an output data buffer according to the configured r information of the output data.
2. Data path:
and (3) output data buffering: the buffer is used for buffering input and output data.
And (3) reading data control: under the control of the processing unit schedule, the data in the output buffer to be processed is read out. For example, in a 4+2 erasure coding configuration, only the first 4 output buffered data will be read.
A Galois field multiplication unit: the Galois field multiplication is completed.
A Galois field addition unit: galois field addition, i.e., XOR, is accomplished.
An input-multiple-output unit: under the control of the selection signal, one output signal is selected and sent to one of the output paths.
The erasure data stream steps are explained as follows:
the first step is as follows: data is fed into the input buffers 1, 2, 3, 4, 5, 6, and when the number of input nodes is less than 6, only part of the input data is input into the input buffers, for example, for a 2+1 encoding task, only data is input into the input buffers 1, 2, and no data is stored in the other input buffers. For encoding or decoding tasks, the input data type is also different, taking encoding or decoding of k + r as an example: for the encoding task, all k data nodes are input, for the decoding task, taking the example that L data nodes need to be recovered, the input data is divided into the surviving (k-L) data nodes and the surviving L check node data, and the sum of the input data nodes is k.
The second step is that: when monitoring that a task is input in the configuration queue, the processing scheduling unit reads the input task type from the configuration queue, wherein the input task type comprises the encoding or decoding of the task type, the number k of data blocks and the number r of check blocks. The matrix is distributed to the 1 st, 2 nd, 3 rd and 4 th rows of the matrix by scheduling the matrix which has been written into the matrix queue in advance, and if r <4, the matrix is distributed to the 1 st, 2 nd, 3 rd and 4 th rows of the partial matrix. E.g., an encoding scenario where r is 2, the matrix is distributed only to rows 1, 2. Then, through controlling the input selection signal, the control matrix is sent into a Galois multiplication unit in a time division multiplexing mode, specifically, the 1 st row, the 2 nd row, the 1 st row and the 2 nd row of the matrix are circulated; the data in the input buffer matrix are sequentially sent to the Galois multiplication unit by controlling the input data selection signal, if k <6, the data in part of the input data buffer is sent to the Galois multiplication unit, for example, the coding scene of k ═ 4, only the 1 st, 2 nd, 3 th and 4 th input data buffer data need to be sent to the Galois multiplication unit. And switching to the execution of the next erasure correcting task until the matrix corresponding to the task completes data scheduling.
The third step: and completing the multiplication operation of the matrix. The specific task is to complete multiplication of a certain row of the matrix and input data, wherein the input data is a data block in an encoding scene, and the input data is a data block and a check block in a decoding scene. The order of the operation data is that the Galois multiplication unit completes the operation, sends the output multiplication result to the Galois addition unit, and then outputs the addition operation result from the addition unit. The operation is only completed by multiplication corresponding to valid data and addition operation, for example, when k is 4, the multiplication units 1, 2, 3, 4 do operation, the multiplication units 5, 6 do not participate in operation, the multiplication units 5, 6 directly output the result 0, and the XOR result is not affected during addition operation.
The fourth step: the processing scheduling unit determines that objects to be scheduled and output are the first r output queues according to the task types in the configuration queues, the number r of check blocks if coding is performed, determines that the objects to be scheduled and output are the first L output queues according to the number L of data blocks to be recovered if decoding is performed, determines the size of data to be output to each output buffer queue according to the size of an output data block of the coding and decoding task, controls an output data selection signal, sequentially and circularly outputs data to a target queue until all the output data are completely output to the output queues, and then switches to the execution of the next erasure correcting task. For example, for an encoding task with r equal to 3 and a block size equal to 1KByte, outputting a control signal requires sequential control data to store the data in an output buffer according to the following scheduling:
the 1 st output buffer stores the 1 st Byte- > the 2 nd output buffer stores the 2 nd Byte- > the 3 rd output buffer stores the 3 rd Byte- > the 1 st output buffer stores the 4 th Byte in such a way until- > the 1 st output buffer stores the 3070 th Byte- > the 2 nd output buffer stores the 3071 th Byte- > the 3 rd output buffer stores the 3072 th Byte.
The fifth step: the output data buffer provides data to the external output, which returns the data to the host side. For example, for erasure decoding of 4+2, the block size is 1Kbyte, there are data nodes 1 and 2 to be recovered, the output data buffers 1 and 2 will respectively generate data output of 1Kbyte, and the output data buffers 3 and 4 have no data generated during the task execution process.
The resource block in fig. 3 is suitable for multiple RS erasure codes, and can process both coding and decoding, so that the same hardware structure is achieved, and all requirements of RS erasure coding and decoding are met in a multiplexing manner.
With 4+2 (i.e. four target data blocks, two matrix rows, which can be referred to as the following analogy), 3+1, 4+1 three consecutive encoding tasks, see fig. 4, and fig. 4 is a resource partitioning diagram of a hardware processing erasure correcting task in the prior art. Referring to fig. 5, fig. 5 is a schematic diagram of resource partitioning of a hardware processing erasure task in the embodiment of the present invention. In fig. 4, t1 time processes the 4+2 task, t2 time processes the 3+1 task, and t3 time processes the 4+1 task. It can be seen that during time t2, the hardware resource multiplication units 4, 5, 6, 7, 8, the XOR operation unit 2 are in a fully idle state. During time t3, the hardware resource multiplication units 5, 6, 7, 8, the XOR operation unit 2 are in a completely idle state. In FIG. 5, it can be seen that at time t1+ t2, the task processing unit processes 4+2 tasks, at time t1 the processing unit 2 processes 3+1 tasks, and at time t2 the processing unit 2 processes 4+1 tasks. It can be seen that with the same hardware resources as in fig. 4, the same task can be completed only with time t1+ t2, and only the multiplication unit 8 will be idle for time t 2. It can be seen that the resource utilization rate can be improved by adopting the mode of multi-channel resource block concurrence and time division multiplexing in the resource blocks, thereby improving the throughput rate of data.
Example three:
referring to fig. 6, fig. 6 is a flowchart of another implementation of a data storage erasure checking method in an embodiment of the present invention, where the method may include the following steps:
s601: and analyzing the received target data erasure task to obtain the type of the target erasure task.
S602: and when the target erasure task type is coding, acquiring a target original data block and a target coding matrix.
S603: and when the target erasure correcting task type is decoding, acquiring a target residual data block, a target verification data block and a target decoding matrix.
S604: and when the number of the target data erasure tasks is multiple, acquiring each current idle resource block.
S605: and acquiring the data size of the target data block.
S606: and distributing each resource block to each target data block and each matrix row of each target matrix in a time division multiplexing mode according to the data size.
S607: and carrying out erasure correction operation on each target data block and each matrix row by using each resource block.
Corresponding to the above method embodiment, the embodiment of the present invention further provides a data storage erasure correcting apparatus, and the data storage erasure correcting apparatus described below and the data storage erasure correcting method described above may be referred to correspondingly.
Referring to fig. 7, fig. 7 is a block diagram of a data storage erasure correcting apparatus according to an embodiment of the present invention, where the apparatus may include:
the task analysis module 71 is configured to analyze the received target data erasure task to obtain a target data block and a target matrix;
a resource block obtaining module 72, configured to obtain a target resource block corresponding to the target data erasure task;
and an erasure result obtaining module 73, configured to perform erasure operation on the target data block and the target matrix based on a time division multiplexing algorithm by using the target resource block, so as to obtain an erasure result.
By applying the device provided by the embodiment of the invention, the received target data erasure task is analyzed to obtain a target data block and a target matrix; acquiring a target resource block corresponding to a target data erasure task; and carrying out erasure correction operation on the target data block and the target matrix by using the target resource block based on a time division multiplexing algorithm to obtain an erasure correction result. By adopting the time division multiplexing algorithm to realize erasure correction operation in a specific target resource block after acquiring the target data block and the target matrix required by erasure correction processing, compared with the existing erasure correction mode of distributing one resource block for each path of output data buffer, the invention greatly reduces the occupation amount of CPU computing resources and improves the resource utilization rate.
In an embodiment of the present invention, the task parsing module 71 includes:
the task type obtaining submodule is used for analyzing the received target data erasure correcting task to obtain a target erasure correcting task type;
and the first input data acquisition submodule is used for acquiring a target original data block and a target coding matrix when the type of the target erasure task is coding.
And the second input data acquisition submodule is used for acquiring a target residual data block, a target check data block and a target decoding matrix when the target erasure task type is decoding.
In a specific embodiment of the present invention, the resource block obtaining module 72 is specifically a module for obtaining each currently idle resource block when the number of target data erasure correcting tasks is multiple;
the erasure result obtaining module 73 is specifically a module that performs parallel erasure operation on each target data block and each target matrix based on a time division multiplexing algorithm by using each resource block.
In one embodiment of the present invention, the method further comprises:
the data size acquisition module is used for acquiring the data size of the target data block after analyzing the received target data erasure task to obtain the target data block and the target matrix;
the erasure result obtaining module 73 includes:
the resource block distribution submodule is used for distributing each resource block to each target data block and each matrix row of each target matrix in a time division multiplexing mode according to the data size;
and the erasure correction operation submodule is used for carrying out erasure correction operation on each target data block and each matrix row by utilizing each resource block.
Corresponding to the above method embodiment, referring to fig. 8, fig. 8 is a schematic diagram of a data storage erasure correcting device provided by the present invention, where the device may include:
a memory 81 for storing a computer program;
the processor 82, when executing the computer program stored in the memory 81, may implement the following steps:
analyzing the received target data erasure task to obtain a target data block and a target matrix; acquiring a target resource block corresponding to a target data erasure task; and carrying out erasure correction operation on the target data block and the target matrix by using the target resource block based on a time division multiplexing algorithm to obtain an erasure correction result.
For the introduction of the device provided by the present invention, please refer to the above method embodiment, which is not described herein again.
Corresponding to the above method embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the steps of:
analyzing the received target data erasure task to obtain a target data block and a target matrix; acquiring a target resource block corresponding to a target data erasure task; and carrying out erasure correction operation on the target data block and the target matrix by using the target resource block based on a time division multiplexing algorithm to obtain an erasure correction result.
The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
For the introduction of the computer-readable storage medium provided by the present invention, please refer to the above method embodiments, which are not described herein again.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device, the apparatus and the computer-readable storage medium disclosed in the embodiments correspond to the method disclosed in the embodiments, so that the description is simple, and the relevant points can be referred to the description of the method.
The principle and the implementation of the present invention are explained in the present application by using specific examples, and the above description of the embodiments is only used to help understanding the technical solution and the core idea of the present invention. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (10)
1. A data storage erasure method, comprising:
analyzing the received target data erasure task to obtain a target data block and a target matrix;
acquiring a target resource block corresponding to the target data erasure task;
and carrying out erasure correction operation on the target data block and the target matrix by using the target resource block based on a time division multiplexing algorithm to obtain an erasure correction result.
2. The data storage erasure method of claim 1, wherein parsing the received target data erasure task to obtain a target data block and a target matrix comprises:
analyzing the received target data erasure task to obtain a target erasure task type;
when the target erasure task type is coding, acquiring a target original data block and a target coding matrix;
and when the target erasure correcting task type is decoding, acquiring a target residual data block, a target verification data block and a target decoding matrix.
3. The data storage erasure correcting method according to claim 1 or 2, wherein when the number of the target data erasure correcting tasks is multiple, acquiring a target resource block corresponding to the target data erasure correcting task includes:
acquiring each current idle resource block;
performing erasure operation on the target data block and the target matrix by using the target resource block based on a time division multiplexing algorithm, including:
and performing parallel erasure operation on each target data block and each target matrix by using each resource block based on the time division multiplexing algorithm.
4. The data storage erasure method of claim 3, wherein after parsing the received target data erasure task to obtain the target data block and the target matrix, further comprising:
acquiring the data size of the target data block;
performing parallel erasure operation on each target data block and each target matrix by using each resource block based on the time division multiplexing algorithm, wherein the parallel erasure operation comprises the following steps:
distributing each resource block to each target data block and each matrix row of each target matrix in a time division multiplexing mode according to the data size;
and carrying out erasure correction operation on each target data block and each matrix row by using each resource block.
5. A data storage erasure apparatus, comprising:
the task analysis module is used for analyzing the received target data erasure task to obtain a target data block and a target matrix;
a resource block obtaining module, configured to obtain a target resource block corresponding to the target data erasure task;
and the erasure result obtaining module is used for carrying out erasure operation on the target data block and the target matrix by utilizing the target resource block based on a time division multiplexing algorithm to obtain an erasure result.
6. The data storage erasure apparatus of claim 5, wherein the task parsing module comprises:
the task type obtaining submodule is used for analyzing the received target data erasure correcting task to obtain a target erasure correcting task type;
the first input data acquisition submodule is used for acquiring a target original data block and a target coding matrix when the target erasure task type is coding;
and the second input data acquisition submodule is used for acquiring a target residual data block, a target verification data block and a target decoding matrix when the target erasure task type is decoding.
7. The data storage erasure correcting device according to claim 5 or 6, wherein the resource block obtaining module is specifically a module for obtaining each currently idle resource block when the number of the target data erasure correcting tasks is multiple;
the erasure result obtaining module is specifically a module for performing parallel erasure operation on each target data block and each target matrix by using each target resource block based on the time division multiplexing algorithm.
8. The data storage erasure apparatus of claim 7, further comprising:
the data size acquisition module is used for acquiring the data size of the target data block after analyzing the received target data erasure task to obtain a target data block and a target matrix;
the erasure result obtaining module includes:
the resource block distribution submodule is used for distributing each resource block to each target data block and each matrix row of each target matrix in a time division multiplexing mode according to the data size;
and the erasure correction operation submodule is used for carrying out erasure correction operation on each target data block and each matrix row by using each resource block.
9. A data storage erasure device, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data storage erasure method of any one of claims 1 to 4 when executing said computer program.
10. A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the data storage erasure method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011041807.1A CN112052115B (en) | 2020-09-28 | 2020-09-28 | Data storage erasure method, device, equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011041807.1A CN112052115B (en) | 2020-09-28 | 2020-09-28 | Data storage erasure method, device, equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112052115A true CN112052115A (en) | 2020-12-08 |
| CN112052115B CN112052115B (en) | 2022-07-08 |
Family
ID=73606013
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011041807.1A Active CN112052115B (en) | 2020-09-28 | 2020-09-28 | Data storage erasure method, device, equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112052115B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114879904B (en) * | 2022-04-22 | 2024-05-03 | 山东云海国创云计算装备产业创新中心有限公司 | Data storage erasure method, device, equipment and readable storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5216677A (en) * | 1989-09-20 | 1993-06-01 | Matsushita Electric Industrial Co., Ltd. | Data reproducing apparatus |
| CN101040476A (en) * | 2004-06-18 | 2007-09-19 | 高通股份有限公司 | Power control using erasure techniques |
| JP2014049148A (en) * | 2012-08-30 | 2014-03-17 | Renesas Electronics Corp | Semiconductor storage device |
| CN104601179A (en) * | 2014-12-12 | 2015-05-06 | 北京麓柏科技有限公司 | Erasure code coding circuit and decoding circuit and coding and encoding circuit of storage system |
| CN104991740A (en) * | 2015-06-24 | 2015-10-21 | 华中科技大学 | Universal matrix optimization method for accelerating erasure correcting code encoding and decoding processes |
| CN111090540A (en) * | 2019-06-24 | 2020-05-01 | 华为技术有限公司 | Data processing method and device based on erasure codes |
| CN111694692A (en) * | 2020-06-24 | 2020-09-22 | 山东云海国创云计算装备产业创新中心有限公司 | Data storage erasure method, device and equipment and readable storage medium |
| CN111697976A (en) * | 2020-05-28 | 2020-09-22 | 苏州浪潮智能科技有限公司 | RS erasure correcting quick decoding method and system based on distributed storage |
-
2020
- 2020-09-28 CN CN202011041807.1A patent/CN112052115B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5216677A (en) * | 1989-09-20 | 1993-06-01 | Matsushita Electric Industrial Co., Ltd. | Data reproducing apparatus |
| CN101040476A (en) * | 2004-06-18 | 2007-09-19 | 高通股份有限公司 | Power control using erasure techniques |
| JP2014049148A (en) * | 2012-08-30 | 2014-03-17 | Renesas Electronics Corp | Semiconductor storage device |
| CN104601179A (en) * | 2014-12-12 | 2015-05-06 | 北京麓柏科技有限公司 | Erasure code coding circuit and decoding circuit and coding and encoding circuit of storage system |
| CN104991740A (en) * | 2015-06-24 | 2015-10-21 | 华中科技大学 | Universal matrix optimization method for accelerating erasure correcting code encoding and decoding processes |
| CN111090540A (en) * | 2019-06-24 | 2020-05-01 | 华为技术有限公司 | Data processing method and device based on erasure codes |
| CN111697976A (en) * | 2020-05-28 | 2020-09-22 | 苏州浪潮智能科技有限公司 | RS erasure correcting quick decoding method and system based on distributed storage |
| CN111694692A (en) * | 2020-06-24 | 2020-09-22 | 山东云海国创云计算装备产业创新中心有限公司 | Data storage erasure method, device and equipment and readable storage medium |
Non-Patent Citations (2)
| Title |
|---|
| MOHAMMAD ESMAEILZADEH等: "Optimizing completion delay in network coded systems over TDD erasure channels with memory", 《2012 INTERNATIONAL SYMPOSIUM ON COMMUNICATIONS AND INFORMATION TECHNOLOGIES (ISCIT)》 * |
| 李萌: "DVB-H数据链路层FEC以及软切换的算法研究", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114879904B (en) * | 2022-04-22 | 2024-05-03 | 山东云海国创云计算装备产业创新中心有限公司 | Data storage erasure method, device, equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112052115B (en) | 2022-07-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7496785B2 (en) | Enclosure-based raid parity assist | |
| CN112860475B (en) | Method, device, system and medium for recovering check block based on RS erasure code | |
| KR101572038B1 (en) | Correction of errors in a memory array | |
| CN108347306B (en) | Similar local reconstruction code encoding and node fault repairing method in distributed storage system | |
| US11742879B2 (en) | Machine-learning error-correcting code controller | |
| CN105808170B (en) | A kind of RAID6 coding methods that can repair single disk error | |
| US8972815B1 (en) | Recovery of media datagrams | |
| CN112052115B (en) | Data storage erasure method, device, equipment and computer readable storage medium | |
| CN110597654A (en) | System and method for ultra-fast error correction code with parity | |
| KR101621752B1 (en) | Distributed Storage Apparatus using Locally Repairable Fractional Repetition Codes and Method thereof | |
| CN113687975A (en) | Data processing method, device, equipment and storage medium | |
| CN114237971A (en) | Erasure code coding layout method and system based on distributed storage system | |
| JP2013058103A (en) | Video recording and reproducing device and rebuild processing method | |
| CN112104412B (en) | Accelerator suitable for low-orbit satellite broadband communication | |
| US7769966B2 (en) | Apparatus and method for judging validity of transfer data | |
| US20100211736A1 (en) | Method and system for performing i/o operations on disk arrays | |
| CN107885615B (en) | Distributed storage data recovery method and system | |
| CN113504875A (en) | Erasure code system recovery method and system based on multi-level scheduling | |
| CN113721848A (en) | Multi-node scheduling repair method and system based on erasure codes | |
| GB2463252A (en) | A message passing LDPC matrix decoder with parallel sub-decoders scheduled to avoid memory contention problems | |
| CN111984443A (en) | Encoding method, decoding method and corresponding devices in distributed system environment | |
| JP2000138595A (en) | Viterbi decoder | |
| US8190972B2 (en) | Error checking and correction overlap ranges | |
| CN116501553B (en) | Data recovery method, device, system, electronic equipment and storage medium | |
| CN114142970B (en) | Fault-tolerant transmission method for two-dimensional array data high-speed transmission |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |