CN112052114B - Data storage and recovery method, coder and decoder and coding and decoding system - Google Patents
Data storage and recovery method, coder and decoder and coding and decoding system Download PDFInfo
- Publication number
- CN112052114B CN112052114B CN202010880335.2A CN202010880335A CN112052114B CN 112052114 B CN112052114 B CN 112052114B CN 202010880335 A CN202010880335 A CN 202010880335A CN 112052114 B CN112052114 B CN 112052114B
- Authority
- CN
- China
- Prior art keywords
- virtual machine
- matrix
- machine cluster
- file
- chip
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000013500 data storage Methods 0.000 title claims abstract description 22
- 238000011084 recovery Methods 0.000 title claims abstract description 18
- 239000011159 matrix material Substances 0.000 claims abstract description 87
- 238000005520 cutting process Methods 0.000 claims abstract description 15
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 14
- 238000003860 storage Methods 0.000 claims abstract description 12
- 238000012544 monitoring process Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 10
- 238000004590 computer program Methods 0.000 description 7
- 239000013598 vector Substances 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 239000012634 fragment Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1004—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's to protect a block of data words, e.g. CRC or checksum
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1469—Backup restoration techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data storage and recovery method, a coder-decoder and a coding-decoding system, wherein the data storage method comprises the following steps: cutting an original file to be stored, so as to cut the original file into a plurality of file segments; determining a generation matrix according to the number of preset data blocks and the number of preset check blocks; dividing the generator matrix into a plurality of low-dimensional sub-matrices; the low-dimensional submatrix is adopted to code all file segments in batches based on an erasure code coding algorithm; and carrying out distributed storage on each coded file segment. The data storage and recovery method, the coder and decoder and the coding and decoding system can improve the data processing speed.
Description
Technical Field
The present invention relates to the field of data storage, and more particularly, to a data storage and recovery method, a codec, and a codec system.
Background
In recent years, erasure Coding (EC) has become one of the mainstream schemes for large-scale data storage as an emerging data storage scheme. And through the EC mechanism, the data are stored in different node hard disks according to a certain redundancy proportion after being cut and fragmented, so that the storage space is saved while the data loss is avoided when partial equipment fails, and the problems of data storage reliability and space utilization rate are well solved. The following is a brief description of the mainstream RS (Reed-Solomon) class erasure coding mechanism in EC codes.
Two core parameters m and n in the erasure code are respectively represented by the number of check blocks and the number of original data blocks. The erasure code encoding process is shown in fig. 1. In the encoding process in fig. 1, C represents a check block and D represents an original data block. B represents a generator matrix.
If part of the data is lost, as shown in fig. 2, the cross represents the lost part, 3 data blocks are lost in total, and the row corresponding to the lost data block is removed from the generating matrix for encoding, so that the B matrix becomes a new matrix B' with n×n dimensions. At the same time, the vector formed by C and D is changed from n+m rows to n rows, and in the above process, the correspondence between the new matrix B' and the remaining data block vector Surviors can be obtained, as shown in fig. 3. So far, after losing part of the data block, the target of decoding is to find the original data vector D. At this time, only the inverse matrix (B ') -1 of the matrix B ' needs to be calculated, and the data vector D can be calculated by the formula d= (B ') -1 x Surviors, so as to achieve the purpose of recovering the original data.
The inventor finds that the EC mechanism uses matrix multiplication operation in the encoding and decoding processes, so that although the EC code can greatly save space, extra calculation amount can be generated in the storing and recovering processes, especially for a system with larger data amount to be processed, when the storage node fails to trigger the data recovery, a large number of files are decoded, and at the moment, a large amount of CPU resource consumption is generated, the overall performance of the system is possibly influenced, and normal calculation tasks are dragged. In particular, when data is recovered, the computing node needs to read the data blocks and the check blocks of other nodes, which may generate a peak of network resource occupation in a short time.
The information disclosed in this background section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person of ordinary skill in the art.
Disclosure of Invention
The invention aims to provide a data storage and recovery method, a coder-decoder and a coding-decoding system, which can improve the data processing speed.
To achieve the above object, the present invention provides a data storage method, comprising: cutting an original file to be stored, so as to cut the original file into a plurality of file segments; determining a generation matrix according to the number of preset data blocks and the number of preset check blocks; dividing the generator matrix into a plurality of low-dimensional sub-matrices; the low-dimensional submatrix is adopted to code all file segments in batches based on an erasure code coding algorithm; and carrying out distributed storage on each coded file segment.
In one embodiment of the present invention, cutting an original file to be stored into a plurality of file segments includes: cutting according to a fixed length, and randomly arranging the cut file segments according to the original sequence or the disordered sequence according to the instruction of the control unit.
In one embodiment of the present invention, the matrix operation module in the form of hardware disposed in the FPGA chip or the GPU chip is used for batch encoding.
Based on the same inventive concept, the invention also provides a data recovery method, which comprises the following steps: scanning a file to be restored; according to the scanning condition, removing matrix rows corresponding to the lost data blocks in a generating matrix for erasure code coding to obtain a new matrix; solving an inverse matrix of the new matrix; dividing the inverse matrix into a plurality of low-dimensional sub-matrices; and carrying out batch decoding on each file segment by adopting the low-dimensional submatrix based on the erasure code decoding algorithm.
In one embodiment of the present invention, the matrix operation module in the form of hardware set in the FPGA chip or the GPU chip is used for batch decoding.
Based on the same inventive concept, the invention also provides an encoder which is an independent plug-in unit, and the independent plug-in unit is mutually matched with an FPGA chip or a GPU chip and is used for executing the data storage method, wherein a matrix operation module in a hardware form is arranged in the FPGA chip or the GPU chip, so that the encoder can perform batch encoding.
Based on the same inventive concept, the invention also provides a decoder which is an independent plug-in unit, and the independent plug-in unit is mutually matched with an FPGA chip or a GPU chip and is used for executing the data recovery method, wherein a matrix operation module in a hardware form is arranged in the FPGA chip or the GPU chip, so that the decoder can perform batch decoding.
Based on the same inventive concept, the invention also provides a coder-decoder which is an independent plug-in unit, wherein the independent plug-in unit is matched with the FPGA chip or the GPU chip and is used for executing coding operation and decoding operation. The encoding and decoding device comprises an FPGA chip, a GPU chip, a matrix operation module and an encoding and decoding module, wherein the encoding and decoding device is used for cutting an original file to be stored into a plurality of file segments, determining a generating matrix according to the number of preset data blocks and the number of preset check blocks, dividing the generating matrix into a plurality of low-dimensional submatrices, and the FPGA chip or the GPU chip is internally provided with a matrix operation module in a hardware form and is used for carrying out batch encoding on each file segment by adopting the low-dimensional submatrices based on erasure code encoding algorithm.
The coder-decoder is used for scanning the file to be restored in the decoding operation, and removing the matrix row corresponding to the lost data block in the generating matrix for erasure code coding according to the scanning condition so as to obtain a new matrix; and the FPGA chip or the GPU chip adopts the low-dimensional submatrices to decode all file segments in batches based on an erasure code decoding algorithm.
Based on the same inventive concept, the invention also provides a cloud platform-based coding and decoding system, which comprises: the system comprises a first virtual machine cluster, a second virtual machine cluster and a third virtual machine cluster. The first virtual machine cluster is used for interacting with the cloud platform through an application programming interface and used for executing encoding operation and decoding operation. The second virtual machine cluster is coupled with the first virtual machine cluster and is used for verifying and storing the data chain information file in the encoding and decoding process, and the first virtual machine cluster is also used for sending a stored or fetched instruction to the second virtual machine cluster. The third virtual machine cluster is coupled with the first virtual machine cluster and is used for storing the encoded data block files in a distributed mode, and the first virtual machine cluster is also used for sending a stored or fetched instruction to the third virtual machine cluster. The first virtual machine cluster is provided with the coder-decoder and the FPGA chip or the GPU chip.
In an embodiment of the present invention, the codec system further includes: and the fourth virtual machine cluster is coupled with the first virtual machine cluster, the second virtual machine cluster and the third virtual machine cluster, is used for monitoring and managing the working condition of each virtual machine cluster, and is also used for receiving the requests of expanding, deleting or updating the virtual machine nodes in the virtual machine clusters, which are sent in a Web form.
Compared with the prior art, according to the data storage and recovery method, the codec and the codec system, in the process of encoding and decoding, the matrix is divided into a plurality of low-dimension submatrices, and the submatrices are adopted for batch encoding or batch decoding, so that the operation efficiency can be improved. Preferably, a hardware matrix operation module arranged in an FPGA chip or a GPU chip is adopted to perform batch matrix operation, so that the operation efficiency is greatly improved, the resource consumption of a CPU is reduced, and the system performance is improved. Preferably, in the encoding process, when the file is segmented and sliced, the control unit can realize the switching of the continuous arrangement and the random arrangement of the file fragments, the occupation of network resources can be further reduced and the reading speed can be improved under the continuous arrangement, and the data security can be ensured to a greater extent under the random arrangement.
Drawings
FIG. 1 is an erasure code coding algorithm according to the prior art;
FIG. 2 is a schematic diagram of a conversion of a generated matrix to a new matrix after a missing data block according to the prior art;
FIG. 3 is a correspondence of a new matrix to remaining data block vectors according to the prior art;
FIG. 4 is a schematic diagram of a data storage method according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a data recovery method according to an embodiment of the present invention;
Fig. 6 is a schematic diagram of the composition of a cloud platform-based codec system according to an embodiment of the present invention.
Detailed Description
The following detailed description of embodiments of the invention is, therefore, to be taken in conjunction with the accompanying drawings, and it is to be understood that the scope of the invention is not limited to the specific embodiments.
Throughout the specification and claims, unless explicitly stated otherwise, the term "comprise" or variations thereof such as "comprises" or "comprising", etc. will be understood to include the stated element or component without excluding other elements or components.
In order to overcome the problems in the prior art, the present invention first provides a data storage method, as shown in fig. 4, and the data storage method according to an embodiment includes: step S10 to step S14.
In step S10, file cutting is performed: cutting the original file to be stored, thereby cutting the original file into a plurality of file segments. Preferably, cutting the original file to be stored so as to be cut into a plurality of file segments includes: cutting according to a fixed length, and randomly arranging the cut file segments according to the original sequence or the disordered sequence according to the instruction of the control unit. In this embodiment, by setting the control unit, the random arrangement and the sequential arrangement of the segmented file segments are flexibly switched, so that the occupation of network resources can be further reduced and the reading speed can be improved under continuous arrangement, and the data security can be ensured to a greater extent under sequential arrangement.
In step S11, a generator matrix is determined: and determining a generation matrix according to the number of the preset data blocks and the number of the preset check blocks. Wherein the sub-matrices of the generator matrix have reversible properties, such as a cauchy matrix.
In step S12, matrix segmentation is performed: the generator matrix is partitioned into a plurality of low-dimensional sub-matrices.
Batch coding is performed in step S13: and carrying out batch coding on each file segment by adopting the low-dimensional submatrix based on the erasure code coding algorithm. Preferably, the matrix operation modules in the form of hardware arranged in the FPGA chip or the GPU chip are used for carrying out batch coding, so that the operation efficiency can be greatly improved, the CPU resource consumption is reduced, and the system performance is improved. The dimensions of the submatrices depend on the types of the FPGA chip or the GPU chip, and taking a main-stream NIVIDIA GPU architecture in the market as an example, the special Tensor Core matrix operation module solidifies the 4 x 4fp16/FP32 matrix multiplication and addition operation commonly used in deep learning, and the matrix B can be divided into a group of 4-dimensional submatrices and is used for batch modular operation of the 4*4 matrix, wherein FP in FP16 represents a single-precision floating point number, and 16 represents data length of 16 bits.
File storage is performed in step S14: and carrying out distributed storage on each coded file segment.
Based on the same inventive concept, the invention also provides a data recovery method. As shown in fig. 5, the data recovery method according to an embodiment includes: step S20 to step S24.
The file to be restored is scanned in step S20.
In step S21, according to the scanning situation, the rows corresponding to the lost data blocks in the generated matrix are removed, so as to obtain a new matrix.
In step S22, the new matrix is inverted.
The inverse matrix is partitioned into a plurality of low-dimensional submatrices in step S23.
Batch decoding is performed in step S24: and carrying out batch decoding on each file segment by adopting the low-dimensional submatrix based on the erasure code decoding algorithm. Preferably, the matrix operation module in the form of hardware arranged in the FPGA chip or the GPU chip is adopted for batch decoding.
Based on the same inventive concept, the invention also provides an encoder. In one embodiment, the encoder is a stand-alone plug-in that cooperates with the FPGA chip or the GPU chip to perform the data storage method of the embodiment shown in fig. 4. The encoder is used as an independent plug-in, can be flexibly distributed to specified files in the system for use, and avoids increasing the complexity of the system architecture.
Based on the same inventive concept, the invention also provides a decoder. In one embodiment, the decoder is a stand-alone plug-in that cooperates with the FPGA chip or the GPU chip to perform the data recovery method of the embodiment shown in fig. 5. The decoder is used as an independent plug-in unit, and can be flexibly distributed to specified files in the system for use, so that the complexity of the system architecture is prevented from being increased.
Based on the same inventive concept, the invention also provides a coder-decoder. In one embodiment, the codec is a stand-alone plug-in that is interoperable with the FPGA chip or the GPU chip for performing encoding and decoding operations. The encoding and decoding device comprises an FPGA chip, a GPU chip, a matrix operation module and an encoding and decoding module, wherein the encoding and decoding device is used for cutting an original file to be stored into a plurality of file segments, determining a generating matrix according to the number of preset data blocks and the number of preset check blocks, dividing the generating matrix into a plurality of low-dimensional submatrices, and the FPGA chip or the GPU chip is internally provided with a matrix operation module in a hardware form and is used for carrying out batch encoding on each file segment by adopting the low-dimensional submatrices based on erasure code encoding algorithm. Optionally, in an embodiment, the codec is further configured to store the encoded file segments in a distributed manner.
The method comprises the steps that in the process of decoding, a coder and decoder is used for scanning a file to be restored, removing rows in a generation matrix corresponding to a lost data block according to scanning conditions to obtain a new matrix, solving an inverse matrix of the new matrix, dividing the inverse matrix into a plurality of low-dimensional submatrices, and the FPGA chip or the GPU chip adopts the low-dimensional submatrices to decode all file segments in batches based on erasure code decoding algorithm. The codec can be used as an independent plug-in, and can be flexibly distributed to specified files in the system for use, so that the complexity of the system architecture is prevented from being increased.
Based on the same inventive concept, the present invention also provides a cloud platform-based codec system, as shown in fig. 6, in an embodiment, the codec system includes: a first virtual machine cluster 10, a second virtual machine cluster 11, a third virtual machine cluster 12.
The first virtual machine cluster 10 is used to interact with the cloud platform through an application programming interface 10 a. The first virtual machine cluster 10 is provided with a codec 10b, and further provided with an FPGA chip 10c and/or a GPU chip 10d, for performing encoding operations and decoding operations. When the batch coding or the batch decoding is performed, the batch matrix operation is performed through the matrix operation module in the FPGA chip 10c or the GPU chip 10d, so that the operation efficiency is greatly improved. The number of matrix operation modules for batch matrix calculation is the same as the number of sub-matrices. One or more matrix operation modules may be included in one FPGA chip 10c or GPU chip 10 d. If one FPGA chip 10c or GPU chip 10d includes one matrix operation module, a plurality of FPGA chips 10c or GPU chips 10d are required to complete the batch matrix operation.
Optionally, the first virtual machine cluster 10 is further configured to monitor files that need to be stored in a distributed manner.
The second virtual machine cluster 11 is coupled to the first virtual machine cluster 10, and is configured to verify and store a data chain information file in the encoding and decoding process, and the first virtual machine cluster 10 is further configured to send a command for accessing or fetching to the second virtual machine cluster 11.
A third virtual machine cluster 12 is coupled to the first virtual machine cluster 10 for storing the encoded data block files in a distributed manner, and the first virtual machine cluster 10 is further configured to send a command for accessing or fetching to the third virtual machine cluster 12.
Optionally, the codec system of an embodiment further includes: and a fourth virtual machine cluster 13, which is coupled to the first virtual machine cluster 10, the second virtual machine cluster 11, and the third virtual machine cluster 12, and is configured to monitor and manage the working conditions of each virtual machine cluster, and further is configured to receive a request for expanding, deleting, or updating a virtual machine node in the virtual machine cluster sent in a Web form. The encoding and decoding system according to the embodiment adopts a reasonable software framework, can be better compatible with the current mainstream cloud computing and storage framework, and meets the requirements of easy expansion and integration.
In summary, according to the data storage and recovery method, the codec and the codec system of the present embodiment, in the encoding and decoding process, the matrix is divided into a plurality of low-dimensional submatrices, and batch encoding or batch decoding is performed by using the submatrices, so that the operation efficiency can be improved. Preferably, a hardware matrix operation module arranged in an FPGA chip or a GPU chip is adopted to perform batch matrix operation instead of processing the data by a CPU, so that the operation efficiency is greatly improved, the average performance can be calculated by 15 to 30 times of the processing of the CPU, the energy consumption efficiency can be calculated by 30 to 80 times of the processing of the CPU, the resource consumption of the CPU is greatly reduced, and the system performance is improved. And the coding and decoding processing capacity is greatly improved, so that the scheme can achieve higher number of EC cutting parts, and has higher storage resource utilization rate compared with the original EC code. Preferably, in the encoding process, when the file is segmented and sliced, the control unit can realize the switching of the continuous arrangement and the random arrangement of the file fragments, the occupation of network resources can be further reduced and the reading speed can be improved under the continuous arrangement, and the data security can be ensured to a greater extent under the random arrangement.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The foregoing descriptions of specific exemplary embodiments of the present invention are presented for purposes of illustration and description. It is not intended to limit the invention to the precise form disclosed, and obviously many modifications and variations are possible in light of the above teaching. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application to thereby enable one skilled in the art to make and utilize the invention in various exemplary embodiments and with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims and their equivalents.
Claims (6)
1. A data storage and recovery method is characterized in that,
The data storage method comprises the following steps:
Cutting an original file to be stored into a plurality of file segments, wherein the cutting is performed according to a fixed length, and the cut file segments are arranged according to an original order or a random order according to an instruction of a control unit;
determining a generation matrix according to the number of preset data blocks and the number of preset check blocks;
Dividing the generator matrix into a plurality of low-dimensional sub-matrices;
The low-dimensional submatrix is adopted to carry out batch coding on each file segment based on an erasure code coding algorithm, wherein a matrix operation module in a hardware form arranged in an FPGA chip or a GPU chip is adopted to carry out batch coding;
Carrying out distributed storage on each coded file segment;
The data recovery method comprises the following steps:
scanning a file to be restored;
According to the scanning condition, removing matrix rows corresponding to the lost data blocks in a generating matrix for erasure code coding to obtain a new matrix;
solving an inverse matrix of the new matrix;
dividing the inverse matrix into a plurality of low-dimensional sub-matrices;
And performing batch decoding on each file segment by adopting the low-dimensional submatrix based on an erasure code decoding algorithm.
2. An encoder, wherein the encoder is an independent plug-in unit, and the independent plug-in unit is matched with an FPGA chip or a GPU chip, so as to execute the data storage method according to claim 1, and a matrix operation module in a hardware form is arranged in the FPGA chip or the GPU chip, so that the encoder can perform batch encoding.
3. A decoder, wherein the decoder is an independent plug-in, and the independent plug-in is matched with an FPGA chip or a GPU chip, so as to perform the data recovery method according to claim 1, and a matrix operation module in a hardware form is arranged in the FPGA chip or the GPU chip, so that the decoder can perform batch decoding.
4. A codec is characterized in that the codec is an independent plug-in, and the independent plug-in is matched with an FPGA chip or a GPU chip for executing encoding operation and decoding operation,
Wherein the coder-decoder is used for cutting an original file to be stored into a plurality of file segments in the process of executing coding operation, determining a generating matrix according to the number of preset data blocks and the number of preset check blocks, the generating matrix is divided into a plurality of low-dimensional submatrices, and a matrix operation module in a hardware form is arranged in the FPGA chip or the GPU chip and is used for carrying out batch coding on each file segment by adopting the low-dimensional submatrices based on an erasure code coding algorithm; and
The coder-decoder is used for scanning the file to be restored in the decoding operation, and removing the matrix row corresponding to the lost data block in the generating matrix for erasure code coding according to the scanning condition so as to obtain a new matrix; and the FPGA chip or the GPU chip adopts the low-dimensional submatrices to decode all file segments in batches based on an erasure code decoding algorithm.
5. A cloud platform-based codec system, comprising:
The first virtual machine cluster is used for interacting with the cloud platform through an application programming interface and executing encoding operation and decoding operation;
the second virtual machine cluster is coupled with the first virtual machine cluster and is used for verifying and storing the data chain information file in the encoding and decoding process, and the first virtual machine cluster is also used for sending a stored or fetched instruction to the second virtual machine cluster;
the third virtual machine cluster is coupled with the first virtual machine cluster and is used for storing the encoded data block files in a distributed manner, and the first virtual machine cluster is also used for sending a stored or fetched instruction to the third virtual machine cluster;
Wherein, the first virtual machine cluster is provided with the codec as claimed in claim 4, and is further provided with the FPGA chip or the GPU chip as claimed in claim 4.
6. The cloud platform based codec system of claim 5, wherein said codec system further comprises:
And the fourth virtual machine cluster is coupled with the first virtual machine cluster, the second virtual machine cluster and the third virtual machine cluster, and is used for monitoring and managing the working condition of each virtual machine cluster and receiving the requests of expanding, deleting or updating the virtual machine nodes in the virtual machine clusters, which are sent in a Web form.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010880335.2A CN112052114B (en) | 2020-08-27 | 2020-08-27 | Data storage and recovery method, coder and decoder and coding and decoding system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010880335.2A CN112052114B (en) | 2020-08-27 | 2020-08-27 | Data storage and recovery method, coder and decoder and coding and decoding system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112052114A CN112052114A (en) | 2020-12-08 |
| CN112052114B true CN112052114B (en) | 2024-05-07 |
Family
ID=73600217
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010880335.2A Active CN112052114B (en) | 2020-08-27 | 2020-08-27 | Data storage and recovery method, coder and decoder and coding and decoding system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112052114B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102833040A (en) * | 2012-08-03 | 2012-12-19 | 中兴通讯股份有限公司 | Method and device for decoding, and coding and decoding system |
| CN104991740A (en) * | 2015-06-24 | 2015-10-21 | 华中科技大学 | Universal matrix optimization method for accelerating erasure correcting code encoding and decoding processes |
| CN105007286A (en) * | 2014-04-23 | 2015-10-28 | 苏宁云商集团股份有限公司 | Decoding method, decoding device, and cloud storage method and system |
| CN107656832A (en) * | 2017-09-18 | 2018-02-02 | 华中科技大学 | A kind of correcting and eleting codes method of low data reconstruction expense |
| CN110457161A (en) * | 2019-07-26 | 2019-11-15 | 成都信息工程大学 | A kind of efficiently highly reliable big data storage system, method, computer program |
| CN110502365A (en) * | 2019-07-11 | 2019-11-26 | 平安科技（深圳）有限公司 | The method, apparatus and computer equipment they data storage and restored |
| CN111541512A (en) * | 2020-03-13 | 2020-08-14 | 中国科学院深圳先进技术研究院 | Data processing method, terminal device and readable storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100218037A1 (en) * | 2008-09-16 | 2010-08-26 | File System Labs Llc | Matrix-based Error Correction and Erasure Code Methods and Apparatus and Applications Thereof |
-
2020
- 2020-08-27 CN CN202010880335.2A patent/CN112052114B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102833040A (en) * | 2012-08-03 | 2012-12-19 | 中兴通讯股份有限公司 | Method and device for decoding, and coding and decoding system |
| CN105007286A (en) * | 2014-04-23 | 2015-10-28 | 苏宁云商集团股份有限公司 | Decoding method, decoding device, and cloud storage method and system |
| CN104991740A (en) * | 2015-06-24 | 2015-10-21 | 华中科技大学 | Universal matrix optimization method for accelerating erasure correcting code encoding and decoding processes |
| CN107656832A (en) * | 2017-09-18 | 2018-02-02 | 华中科技大学 | A kind of correcting and eleting codes method of low data reconstruction expense |
| CN110502365A (en) * | 2019-07-11 | 2019-11-26 | 平安科技（深圳）有限公司 | The method, apparatus and computer equipment they data storage and restored |
| CN110457161A (en) * | 2019-07-26 | 2019-11-15 | 成都信息工程大学 | A kind of efficiently highly reliable big data storage system, method, computer program |
| CN111541512A (en) * | 2020-03-13 | 2020-08-14 | 中国科学院深圳先进技术研究院 | Data processing method, terminal device and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112052114A (en) | 2020-12-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP1506621B1 (en) | Decoding of chain reaction codes through inactivation of recovered symbols | |
| CN109643258B (en) | Multi-node repair using high-rate minimal storage erase code | |
| CN110178122B (en) | Data synchronous repair method of distributed storage system and storage medium | |
| US9465692B2 (en) | High reliability erasure code distribution | |
| US8775860B2 (en) | System and method for exact regeneration of a failed node in a distributed storage system | |
| CN108347306B (en) | Similar local reconstruction code encoding and node fault repairing method in distributed storage system | |
| CN106874140B (en) | Data storage method and device | |
| CN113687975B (en) | Data processing method, device, equipment and storage medium | |
| CN114153651B (en) | Data encoding method, device, equipment and medium | |
| CN111682874A (en) | Data recovery method, system, equipment and readable storage medium | |
| CN108762978B (en) | Grouping construction method of local part repeated cyclic code | |
| CN112052114B (en) | Data storage and recovery method, coder and decoder and coding and decoding system | |
| CN115113816A (en) | Erasure code data processing system, method, computer device and medium | |
| CN107665152B (en) | Decoding method of erasure code | |
| CN112860476A (en) | Approximate erasure code coding method and device based on video layered storage | |
| CN106302573B (en) | Method, system and device for processing data by adopting erasure code | |
| CN113157485B (en) | Expansion construction method of partial repetition code | |
| US11165440B1 (en) | Systems and methods for geometric partitioning in object storage | |
| CN112988454B (en) | Expansion part repeated code construction method | |
| CN115793984A (en) | Data storage method and device, computer equipment and storage medium | |
| CN115237665A (en) | Rapid disk repairing method and system for high-density storage server | |
| Wang et al. | A fast repair code based on regular graphs for distributed storage systems | |
| CN116578439A (en) | Repair tree construction method and data repair method based on simulated annealing algorithm | |
| Gao et al. | A Distributed Storage Algorithm Based on Cauchy RS Code |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |