CN114546708A - Method and device suitable for uniform and distributed storage erasure correction - Google Patents

Method and device suitable for uniform and distributed storage erasure correction Download PDF

Info

Publication number
CN114546708A
CN114546708A CN202210153468.9A CN202210153468A CN114546708A CN 114546708 A CN114546708 A CN 114546708A CN 202210153468 A CN202210153468 A CN 202210153468A CN 114546708 A CN114546708 A CN 114546708A
Authority
CN
China
Prior art keywords
data
determining
encoding
decoding
parameter information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210153468.9A
Other languages
Chinese (zh)
Inventor
吴睿振
王明明
陈静静
王凛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong Yunhai Guochuang Cloud Computing Equipment Industry Innovation Center Co Ltd
Original Assignee
Shandong Yunhai Guochuang Cloud Computing Equipment Industry Innovation Center Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong Yunhai Guochuang Cloud Computing Equipment Industry Innovation Center Co Ltd filed Critical Shandong Yunhai Guochuang Cloud Computing Equipment Industry Innovation Center Co Ltd
Priority to CN202210153468.9A priority Critical patent/CN114546708A/en
Publication of CN114546708A publication Critical patent/CN114546708A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1004—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's to protect a block of data words, e.g. CRC or checksum
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1008—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices
    • G06F11/1064—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices in cache or content addressable memories
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • G06F3/0689—Disk arrays, e.g. RAID, JBOD

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Quality & Reliability (AREA)
  • Computer Security & Cryptography (AREA)
  • Error Detection And Correction (AREA)

Abstract

The invention provides a method, a system, equipment and a storage medium suitable for unified and distributed storage erasure, wherein the method comprises the following steps: in response to receiving input data, determining an encoding and decoding mode and determining parameter information required by encoding and decoding according to the encoding and decoding mode; setting a control function based on hardware requirements to fulfill performance requirements in the hardware; temporarily storing the parameter information and carrying out concurrent control on data; and determining the number and the relation of multiply-add modules based on natural number field operation or Galois field operation based on the multiple, the data quantity and the performance requirement relation of the operation so as to complete corresponding encoding and decoding operation. The invention is configured into the same output format by realizing different algorithms under the scene of uniformly and distributively storing different algorithm requirements, and has strong universality.

Description

Method and device suitable for unified and distributed storage erasure
Technical Field
The present invention relates to the field of distributed systems, and more particularly, to a method, system, device, and storage medium for erasure correction suitable for unified and distributed storage.
Background
Erasure Code (Erasure Code) belongs to a forward error correction technique in the coding theory, and is applied to the communication field for the first time to solve the problems of loss and loss in data transmission. Erasure coding techniques have been introduced into the storage area because of their superior effectiveness in preventing data loss. Erasure codes can effectively reduce storage overhead while ensuring the same reliability, and therefore erasure code technology is widely applied to various large storage systems and data centers, such as, for example, Azure by microsoft, F4 by Facebook, and the like. The application of erasure correction is divided into a uniform system and a distributed system, wherein the uniform system uses the traditional erasure correction code using mode. In the distributed system, a RAID code is used after combining erasure codes with load balancing and other requirements, and RAID (redundant Arrays of Independent disks) is a disk array with redundancy capability, and a disk array is obtained by combining a plurality of Independent disks together, so that a disk group with a large capacity is obtained. By adopting the RAID storage technology, the storage capacity can be greatly improved, the input and output request processing capacity of the system is improved, and the reliability of data is improved by the distributed storage technology of data, a parallel access means and an information redundancy technology.
The RAID mainly uses data striping, data check, and mirroring techniques to obtain higher performance, higher reliability, better fault-tolerance capability, and higher scalability. The strategies and architectures of these three techniques may be applied or combined according to different data application requirements, so RAID may be divided into different levels according to different strategies and architectures: RAID0, 1, 5, 6, 10. The algorithms used by the uniform and distributed storage are all erasure correction algorithms, but have corresponding differences in algorithm details and implementation. Different erasure techniques may use different algorithms, thereby requiring different erasure acceleration cards to perform corresponding erasure operations.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, a system, a computer device, and a computer readable storage medium for erasure correction suitable for uniform and distributed storage, where corresponding firmware in the embodiments of the present invention only needs to adapt to different algorithms, that is, can adapt to a general hardware structure, so as to implement different erasure correction algorithms in different application scenarios, thereby implementing general design in a wider application scenario.
Based on the above object, an aspect of the embodiments of the present invention provides a method for erasure correction suitable for unified and distributed storage, including the following steps: responding to received input data, determining a coding and decoding mode and determining parameter information required by coding and decoding according to the coding and decoding mode; setting a control function based on hardware requirements to fulfill performance requirements in the hardware; temporarily storing the parameter information and carrying out concurrent control on data; and determining the number and the relation of multiply-add modules based on natural number field operation or Galois field operation based on the multiple, the data quantity and the performance requirement relation of the operation so as to complete corresponding encoding and decoding operation.
In some embodiments, the determining the encoding and decoding manner and determining the parameter information required for encoding and decoding according to the encoding and decoding manner includes: and determining the selection of a data address for performing coding and decoding operation based on the requirement of load balancing, and controlling input or output to perform data reading of different addresses according to the selected data address.
In some embodiments, the determining the encoding and decoding manner and determining the parameter information required for encoding and decoding according to the encoding and decoding manner includes: and determining parameter information based on the load balancing data arrangement mode.
In some embodiments, the determining the number and relationship of the multiply-add modules based on natural number domain operation or galois domain operation to complete the corresponding codec operation based on the multiple, data amount and operation performance requirement relationship includes: and reading parameter information from the matrix structure based on the relation between the data and the parameters for operation, and outputting combined data through a multiplication and addition module of natural number field operation or Galois field operation.
In another aspect of the embodiments of the present invention, a system for erasure correction suitable for uniform and distributed storage is provided, including: the determining module is configured to respond to the received input data, determine an encoding and decoding mode and determine parameter information required by encoding and decoding according to the encoding and decoding mode; a setting module configured to set a control function based on hardware requirements to fulfill performance requirements in hardware; the control module is configured to temporarily store the parameter information and perform concurrent control on data; and the operation module is configured to determine the number and the relation of the multiplication and addition modules based on natural number field operation or Galois field operation based on the multiple, the data volume and the performance requirement relation of the operation so as to complete corresponding encoding and decoding operation.
In some embodiments, the determining module is configured to: and determining the selection of data addresses for coding and decoding operation based on the requirement of load balancing, and controlling input or output to read data with different addresses according to the selected data addresses.
In some embodiments, the determining module is configured to: and determining parameter information based on the load balancing data arrangement mode.
In some embodiments, the arithmetic module is configured to: and reading parameter information from the matrix structure based on the relation between the data and the parameters for operation, and outputting combined data through a multiplication and addition module of natural number field operation or Galois field operation.
In another aspect of the embodiments of the present invention, there is also provided a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method as above.
In a further aspect of the embodiments of the present invention, a computer-readable storage medium is also provided, in which a computer program for implementing the above method steps is stored when the computer program is executed by a processor.
The invention has the following beneficial technical effects: under the scene of unified and distributed storage of different algorithm requirements, different algorithms are configured into the same output format, operation is realized through a matrix and PE (multiply-add module based on natural number field operation or Galois field operation), the operation division can be carried out on the basis of the requirements of performance and area to determine the specific PE number so as to achieve different design requirements, and finally, corresponding encoding and decoding work is completed through data combination and tray falling so as to achieve the performance of configurable optimal performance.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
FIG. 1 is a schematic diagram of an embodiment of a method for applying uniform and distributed storage erasure provided by the present invention;
FIG. 2 is a hardware architecture diagram of an embodiment of a system for applying uniform and distributed storage erasure provided by the present invention;
FIG. 3 is a schematic diagram of the operation of the matrix module of the present invention;
FIG. 4 is a schematic diagram of the PE module for encoding and decoding operation according to the present invention;
FIG. 5 is a schematic diagram of an embodiment of a system for applying uniform and distributed storage erasure provided by the present invention;
FIG. 6 is a schematic hardware diagram of an embodiment of a computer device suitable for unified and distributed storage erasure provided by the present invention;
FIG. 7 is a schematic diagram of an embodiment of a computer storage medium adapted for unified and distributed storage erasure provided by the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are only used for convenience of expression and should not be construed as a limitation to the embodiments of the present invention, and no description is given in the following embodiments.
In a first aspect of the embodiments of the present invention, an embodiment of a method applicable to erasure correction in unified and distributed storage is provided. FIG. 1 is a schematic diagram illustrating an embodiment of a method for applying uniform and distributed storage erasure correction provided by the present invention. As shown in fig. 1, the embodiment of the present invention includes the following steps:
s1, responding to the received input data, determining an encoding and decoding mode and determining parameter information required by encoding and decoding according to the encoding and decoding mode;
s2, setting a control function based on the hardware requirement to complete the performance requirement in the hardware;
s3, temporarily storing the parameter information and carrying out concurrent control on data; and
and S4, determining the number and the relationship of multiply-add modules based on natural number field operation or Galois field operation based on the multiple, data amount and performance requirement relationship of the operation to complete the corresponding encoding and decoding operation.
When the erasure algorithm is applied to unified and distributed different operating systems, the algorithm is changed in order to adapt to different scenes and application requirements, and the following simple descriptions are firstly made respectively:
1. unified storage:
common erasure correction algorithms for unified storage include van der mons and cauchy algorithms of RS, where k represents a data amount and m represents a parity amount generated by passing data, and erasure correction under a scene of a larger k and a larger m is usually supported by unified storage. The erasure code encodes the k data blocks into r additional parity blocks. The way that r parity chunks are encoded based on the vandermonde matrix or the cauchy matrix is called RS erasure coding encoded by the vandermonde matrix or the cauchy matrix, and the specific encoding process is as follows.
RS erasure codes based on vandermonde matrices:
Figure BDA0003511375170000051
RS erasure code based on Cauchy matrix:
Figure BDA0003511375170000052
the decoding is data obtained by inverse operation, and the decoding is performed by taking D1 to Dr data loss as an example, and the process is as follows.
The RS erasure code recovery data mode:
Figure BDA0003511375170000061
it is known that the core concept of erasure codes is to construct a reversible coding matrix to generate check data, and the inverse matrix can be calculated to recover the original data.
2. Distributed storage:
the erasure correction algorithm used by distributed storage is called RAID, and RAID0, 1, 5, 6 are common, wherein the algorithms involving multiple error recovery are RAID5 and RAID 6. The conventional algorithm principle of RAID5 uses:
d1+d2+d3+...+dm+p＝0 (1)
the algorithm principle of RAID6 is:
Figure BDA0003511375170000062
the coding and decoding algorithm of RAID is to solve the above relation equation with p as unknown number. The operation here uses galois field operation in storage, so it can be seen that in conventional RAIDs 5 and 6, the relationship of p is:
RAID5：
Figure BDA0003511375170000063
RAID6：
Figure BDA0003511375170000064
in a storage system, in order to reduce the operation complexity and ensure that data does not overflow, the above unified and stored encoding and decoding operations are generally implemented by galois fields. That is, in the hardware implementation, the addition and subtraction are implemented by exclusive or operation, and the multiplication and division are implemented by galois multiplication and division for different galois field polynomials. As can be seen from the above, the unified and distributed storage uses different algorithms for different application environments, and the hardware required to be implemented is different when implementing the hardware accelerator.
FIG. 2 is a diagram of a hardware architecture of an embodiment of a system for applying uniform and distributed storage erasure correction provided by the present invention. In the embodiment of the present invention, the hardware architecture is divided into four modules, i.e., 1, 2, 3, and 4, which are enclosed by dashed lines in fig. 2. In fig. 2, 1 denotes a firmware and i/o control module, 2 denotes a hardware control module, 3 denotes a matrix module, and 4 denotes a PE module for encoding and decoding operations.
And responding to the received input data, determining an encoding and decoding mode and determining parameter information required by encoding and decoding according to the encoding and decoding mode.
The following illustrates the encoding and decoding operation mode and the data mode of the firmware under the unified or distributed mode, respectively:
A. unified memory code
Most of the unified memories use RS codes, and most of them use van der mond or cauchy codes, and the encoding conditions thereof satisfy the description of fig. 1, and here, the matrix output by the firmware to the logic core 3 is parameter information required for encoding, and when k is 3 and m is 2, van der mond codes are used as an example, the output information here is:
Figure BDA0003511375170000071
B. unified memory decoding
The architecture of the present invention can solve the decoding problem at the same time, and correspondingly, taking the same situation in the above a as an example, the matrix output by the firmware to the logic core 3 at this time is the inverse matrix of the encoding matrix, taking the first and second data that have errors and need to be recovered as an example:
Figure BDA0003511375170000072
H-1the generation of (a) involves different inversion algorithms, which are not within the scope of the present invention. Completion H-1After inversion, the data position to be recovered is set at H-1The corresponding parameters in (3) are combined into a matrix, and the matrix is transmitted to the matrix part in the logic core 3 of the hardware through firmware. In the above example, since the first and second data to be restored by an error occur, the parameters of the first and second rows are combined into a matrix after inversion, and the matrix is transmitted to the hardware.
C. Distributed storage encoding
In the case where the distributed storage uses a RAID algorithm, and when k is 3.m is 2, which is required to be encoded in the same environment as described above, based on the definition of the RAID algorithm, a RAID6 algorithm is used, and the encoding matrix generated at this time is based on the RAID6 algorithm in the formula (4):
Figure BDA0003511375170000081
the H matrix corresponds to parameter information of a used coding algorithm after a coding position is determined, and the parameter information is based on the sequence of multiplicands (data information) to form matrix information similar to unified storage.
D. Distributed storage decoding
When decoding, the same RAID decoding algorithm is used, and in the above case, for example, if the first and second data have errors, the parameters may be obtained by substituting different pieces of position information using the same formula (4). Using the same situation as above for example, the corresponding information is:
Figure BDA0003511375170000082
H-1since the calculation method of (2) is a RAID algorithm, the calculation method is the same as the encoding algorithm, and parameters corresponding to different pieces of position information may be substituted. In the above operations, galois fields are often used for operation, the matrix format of data transmission is not changed, the corresponding relationship is not changed, and the parameters corresponding to the operations are changed for different galois fields.
In some embodiments, the determining the encoding and decoding manner and determining the parameter information required for encoding and decoding according to the encoding and decoding manner includes: and determining the selection of a data address for performing coding and decoding operation based on the requirement of load balancing, and controlling input or output to perform data reading of different addresses according to the selected data address. The firmware and input/output control module 1 belongs to a peripheral function module of a hardware structure, and input and output of data are controlled by the name of the module after encoding or decoding is finished. The hardware structure of the invention can use the same hardware structure, and is realized by different algorithms of firmware, and different parameter configurations are carried out, so as to complete the work of coding and decoding. Data input for encoding or decoding work, splitting or combining of data based on input parts, input hardware structure, splitting and combining of data, defined based on requirements of 3 and 4 in the logical core. The data output of the encoding or decoding work is determined based on the specific implementation structure of hardware and the disk dropping mode. The firmware part determines the data address selection for coding and decoding operation and the parameter operation of coding and decoding based on the requirement of load balancing. And after the data address selection is completed, controlling the input or the output to read data of different addresses. After the parameter operation is completed, the corresponding parameter to be operated is input to the part 3. For the unified or distributed coding and decoding requirements, the calculated algorithm and the obtained parameters are different. The control functions are set based on hardware requirements to fulfill performance requirements in the hardware. The hardware control module 2 sets a control function related to a register and the like based on different hardware requirements, so as to meet the requirements in the aspects of performance and the like in hardware.
In some embodiments, the determining the encoding and decoding manner and determining the parameter information required for encoding and decoding according to the encoding and decoding manner includes: and determining parameter information based on the load balancing data arrangement mode. For example, the left-hand rotation is taken to illustrate that the parameters corresponding to the positions of different data bits are different parameters determined by the positions corresponding to the first row.
And temporarily storing the parameter information and carrying out concurrent control on data.
In order to complete the corresponding work of coding or decoding, the matrix module receives the parameter information which is operated by the firmware and combined into a matrix, and the parameter information is stored in the matrix structure of 3. Parameters in the matrix structure participate in the operation of encoding and decoding, and each parameter corresponds to different data to be operated.
The firmware determines the method of dividing the data information based on the parameter information transmitted to the matrix structure, taking formula (5) as an example, at this time, the data to be operated for the data parallel distribution control is the multiple relation data distributed based on 1, 1, 1 and 1, 2, 3. The multiple relation is determined based on performance requirements, the higher the speed requirement is, the higher the multiple relation can be on the premise of divisible distribution, the higher the area requirement is, the lower the multiple relation is, and the comprehensive result of the two is the reference for determining the final multiple relation.
Fig. 3 is a schematic diagram of the matrix module according to the present invention, as shown in fig. 3, if the multiple to be distributed is 2, and the corresponding data to be calculated is d1, d2, and d3, the data d1, d2, and d3 are distributed based on the multiple relation being 2, and are divided into two pieces of data on average, where d1, 0, and d1, 1 are combined to form d1, and the same applies to other data.
And determining the number and the relation of multiply-add modules based on natural number field operation or Galois field operation based on the performance requirement relation of multiple, data quantity and operation to complete corresponding encoding and decoding operation.
In some embodiments, the determining the number and relationship of the multiply-add modules based on natural number domain operation or galois domain operation to complete the corresponding codec operation based on the multiple, data amount and operation performance requirement relationship includes: and reading parameter information from the matrix structure based on the relation between the data and the parameters for operation, and outputting combined data through a multiplication and addition module of natural number field operation or Galois field operation.
After temporary storage of parameters and parallel data distribution control are completed by the module 3, the number and the relation of the PEs are determined based on the multiple, the data volume and the performance requirement relation of operation, and corresponding encoding and decoding operation is completed. Wherein the PE is a multiply-add module based on natural number field operation or Galois field operation. The number of PEs is determined based on performance requirements, for example, in the above-mentioned example, the data amount is 3, the multiple relation is 2, and when the number of rows in the matrix structure is 2, if the pursuit speed is the highest, 2 × 3 × 2 — 12 PEs may be divided, each PE corresponds to one data block, the parameter information is read from the matrix structure based on the relation between the data and the parameter to perform an operation, and then the combined data is output by the PEs, thereby completing the output of the codec.
FIG. 4 is a schematic diagram of the PE module for encoding and decoding operation according to the present invention. The above PE group division is decided based on the number of parameter rows in the matrix structure in pursuit of speed. The number of PEs is determined based on the data size, the multiple relation, and the number of rows in the matrix structure. When the area is sought, the number of PEs can be reduced, and arithmetic multiplexing can be performed based on the similar relationship. And finally, combining and outputting the results obtained by the operation based on the division relation, wherein the output purpose is to perform the landing of the coded and decoded data, and the landing position is determined by the load balancing position information in the step 1.
According to the invention, under the scene of uniformly and distributively storing different algorithm requirements, different algorithms are realized, configured into the same output format, and the operation is realized through the matrix and the PE operation module, the specific PE quantity can be determined based on the requirements of performance and area by the operation division, so that different design requirements are met, and finally, corresponding encoding and decoding work is completed through data combination and disk dropping, so that the performance of configurable optimal performance is achieved.
It should be particularly noted that, the steps in the embodiments of the method for applying uniform and distributed storage erasure correction described above can be mutually intersected, replaced, added, and deleted, so that these methods for applying uniform and distributed storage erasure correction, which are reasonably arranged and combined, should also belong to the protection scope of the present invention, and should not limit the protection scope of the present invention to the embodiments.
In view of the above, a second aspect of the embodiments of the present invention provides a system suitable for uniform and distributed storage erasure. As shown in fig. 5, the system 200 includes the following modules: the determining module is configured to respond to the received input data, determine an encoding and decoding mode and determine parameter information required by encoding and decoding according to the encoding and decoding mode; a setting module configured to set a control function based on hardware requirements to fulfill performance requirements in hardware; the control module is configured to temporarily store the parameter information and perform concurrent control on data; and the operation module is configured to determine the number and the relation of the multiplication and addition modules based on natural number field operation or Galois field operation based on the multiple, the data volume and the performance requirement relation of the operation so as to complete corresponding encoding and decoding operation.
In some embodiments, the determining module is configured to: and determining the selection of a data address for performing coding and decoding operation based on the requirement of load balancing, and controlling input or output to perform data reading of different addresses according to the selected data address.
In some embodiments, the determining module is configured to: and determining parameter information based on the load balancing data arrangement mode.
In some embodiments, the arithmetic module is configured to: and reading parameter information from the matrix structure based on the relation between the data and the parameters for operation, and outputting combined data through a multiplication and addition module of natural number field operation or Galois field operation.
In view of the above object, a third aspect of an embodiment of the present invention provides a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, the instructions being executable by the processor to perform the steps of: s1, responding to the received input data, determining an encoding and decoding mode and determining parameter information required by encoding and decoding according to the encoding and decoding mode; s2, setting a control function based on the hardware requirement to complete the performance requirement in the hardware; s3, temporarily storing the parameter information and carrying out concurrent control on data; and S4, determining the number and relation of multiply-add modules based on natural number field operation or Galois field operation based on the multiple, data amount and performance requirement relation of operation to complete corresponding encoding and decoding operation.
In some embodiments, the determining the encoding and decoding manner and determining the parameter information required for encoding and decoding according to the encoding and decoding manner includes: and determining the selection of a data address for performing coding and decoding operation based on the requirement of load balancing, and controlling input or output to perform data reading of different addresses according to the selected data address.
In some embodiments, the determining the encoding and decoding manner and determining the parameter information required for encoding and decoding according to the encoding and decoding manner includes: and determining parameter information based on the load balancing data arrangement mode.
In some embodiments, the determining the number and relationship of the multiply-add modules based on natural number domain operation or galois domain operation to complete the corresponding codec operation based on the multiple, data amount and operation performance requirement relationship includes: and reading parameter information from the matrix structure based on the relationship between the data and the parameters to perform operation, and outputting combined data through a multiplication and addition module of the natural number domain operation or the Galois field operation.
Fig. 6 is a schematic hardware structure diagram of an embodiment of the computer device adapted to perform unified and distributed storage erasure correction according to the present invention.
Taking the device shown in fig. 6 as an example, the device includes a processor 301 and a memory 302.
The processor 301 and the memory 302 may be connected by a bus or other means, such as the bus connection in fig. 6.
The memory 302 is used as a non-volatile computer-readable storage medium for storing non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the method for applying uniform and distributed storage erasure correction in the embodiments of the present application. The processor 301 executes various functional applications of the server and data processing by running non-volatile software programs, instructions and modules stored in the memory 302, i.e., implements a method suitable for unified and distributed storage erasure correction.
The memory 302 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of methods applicable to uniform and distributed storage erasure correction, and the like. Further, the memory 302 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 302 optionally includes memory located remotely from processor 301, which may be connected to a local module via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
Computer instructions 303 corresponding to one or more methods of applying unified and distributed storage erasure correction are stored in memory 302 and when executed by processor 301 perform the methods of applying unified and distributed storage erasure correction in any of the method embodiments described above.
Any embodiment of a computer device that performs the method for applying unified and distributed storage erasure described above may achieve the same or similar effects as any of the preceding method embodiments corresponding thereto.
The present invention also provides a computer readable storage medium storing a computer program which, when executed by a processor, performs a method for adaptable unified and distributed storage erasure.
FIG. 7 is a schematic diagram of one embodiment of a computer storage medium adapted for uniform and distributed storage erasure correction as described above in connection with the present invention. Taking the computer storage medium as shown in fig. 7 as an example, the computer readable storage medium 401 stores a computer program 402 which, when executed by a processor, performs the method as described above.
Finally, it should be noted that, as one of ordinary skill in the art can appreciate that all or part of the processes in the methods of the above embodiments can be implemented by a computer program to instruct related hardware, and a program suitable for the methods of the uniform and distributed storage erasure correction can be stored in a computer readable storage medium, and when executed, the program can include the processes of the embodiments of the methods described above. The storage medium of the program may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also combinations between technical features in the above embodiments or in different embodiments are possible, and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements and the like that may be made without departing from the spirit or scope of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.

Claims (10)

1. A method for erasure correction suitable for both unified and distributed storage, comprising the steps of:
in response to receiving input data, determining an encoding and decoding mode and determining parameter information required by encoding and decoding according to the encoding and decoding mode;
setting a control function based on hardware requirements to fulfill performance requirements in the hardware;
temporarily storing the parameter information, and carrying out concurrency control on data; and
and determining the number and the relation of multiply-add modules based on natural number field operation or Galois field operation based on the performance requirement relation of multiple, data quantity and operation to complete corresponding encoding and decoding operation.
2. The method of claim 1, wherein the determining the codec mode and determining the parameter information required for the codec according to the codec mode comprises:
and determining the selection of a data address for performing coding and decoding operation based on the requirement of load balancing, and controlling input or output to perform data reading of different addresses according to the selected data address.
3. The method of claim 1, wherein the determining the codec mode and determining the parameter information required for the codec according to the codec mode comprises:
and determining parameter information based on the load balancing data arrangement mode.
4. The method of claim 1, wherein determining the number and relationship of multiply-add modules based on natural number domain operations or galois field operations to perform the corresponding codec operations based on the performance requirement relationship of multiples, data amounts and operations comprises:
and reading parameter information from the matrix structure based on the relation between the data and the parameters for operation, and outputting combined data through a multiplication and addition module of natural number field operation or Galois field operation.
5. A system for accommodating unified and distributed storage erasure, comprising:
the determining module is configured to respond to the received input data, determine an encoding and decoding mode and determine parameter information required by encoding and decoding according to the encoding and decoding mode;
a setting module configured to set a control function based on hardware requirements to fulfill performance requirements in hardware;
the control module is configured to temporarily store the parameter information and perform concurrent control on data; and
and the operation module is configured to determine the number and the relation of the multiplication and addition modules based on natural number field operation or Galois field operation based on the multiple, the data amount and the performance requirement relation of the operation so as to complete corresponding encoding and decoding operation.
6. The system of claim 5, wherein the determination module is configured to:
and determining the selection of a data address for performing coding and decoding operation based on the requirement of load balancing, and controlling input or output to perform data reading of different addresses according to the selected data address.
7. The system of claim 5, wherein the determination module is configured to:
and determining parameter information based on the load balancing data arrangement mode.
8. The system of claim 5, wherein the operation module is configured to:
and reading parameter information from the matrix structure based on the relation between the data and the parameters for operation, and outputting combined data through a multiplication and addition module of natural number field operation or Galois field operation.
9. A computer device, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method of any one of claims 1 to 4.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 4.
CN202210153468.9A 2022-02-18 2022-02-18 Method and device suitable for uniform and distributed storage erasure correction Pending CN114546708A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210153468.9A CN114546708A (en) 2022-02-18 2022-02-18 Method and device suitable for uniform and distributed storage erasure correction

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210153468.9A CN114546708A (en) 2022-02-18 2022-02-18 Method and device suitable for uniform and distributed storage erasure correction

Publications (1)

Publication Number Publication Date
CN114546708A true CN114546708A (en) 2022-05-27

Family

ID=81676209

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210153468.9A Pending CN114546708A (en) 2022-02-18 2022-02-18 Method and device suitable for uniform and distributed storage erasure correction

Country Status (1)

Country Link
CN (1) CN114546708A (en)

Similar Documents

Publication Publication Date Title
US8327080B1 (en) Write-back cache protection
EP3504623B1 (en) Multiple node repair using high rate minimum storage regeneration erasure code
CN112860475B (en) Method, device, system and medium for recovering check block based on RS erasure code
CN111682874B (en) Data recovery method, system, equipment and readable storage medium
WO2023151290A1 (en) Data encoding method and apparatus, device, and medium
CN114090345B (en) Disk array data recovery method, system, storage medium and equipment
CN113687975B (en) Data processing method, device, equipment and storage medium
CN112799875B (en) Method, system, device and medium for verification recovery based on Gaussian elimination
US11249846B2 (en) Erasure code-based data processing method and apparatus
CN115080303B (en) Encoding method, decoding method, device and medium for RAID6 disk array
CN114816837B (en) Erasure code fusion method and system, electronic device and storage medium
US20120198195A1 (en) Data storage system and method
CN104850504A (en) Equation parallel computing method for accelerating XOR based RAID-6 coding/decoding process
CN116266148A (en) End-to-end data protection for in-memory computation/neighbor memory computation
CN114546272A (en) Method, system, apparatus and storage medium for fast universal RAID demotion to RAID5
CN114895842A (en) TP-RAID (transport protocol-redundant array of independent disks) encoding and decoding method, system, equipment and storage medium
CN116501553B (en) Data recovery method, device, system, electronic equipment and storage medium
WO2022007225A1 (en) Data storage method, storage system, storage device, and storage medium
CN114546708A (en) Method and device suitable for uniform and distributed storage erasure correction
CN115113816A (en) Erasure code data processing system, method, computer device and medium
CN110431531B (en) Storage controller, data processing chip and data processing method
CN114756175A (en) Decoding method, system, equipment and medium for disk array
CN107463462B (en) Data restoration method and data restoration device
CN114946144B (en) Low density parity check code encoding method and encoder
CN113986608A (en) Disk array data recovery method, system, storage medium and equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination