CN111858142A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111858142A CN111858142A CN202010725775.0A CN202010725775A CN111858142A CN 111858142 A CN111858142 A CN 111858142A CN 202010725775 A CN202010725775 A CN 202010725775A CN 111858142 A CN111858142 A CN 111858142A
- Authority
- CN
- China
- Prior art keywords
- data block
- matrix
- inverse matrix
- data blocks
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 27
- 239000011159 matrix material Substances 0.000 claims abstract description 193
- 238000000034 method Methods 0.000 claims abstract description 43
- 238000012545 processing Methods 0.000 claims abstract description 14
- 238000004590 computer program Methods 0.000 claims description 7
- 230000009467 reduction Effects 0.000 claims description 7
- 230000008569 process Effects 0.000 abstract description 13
- 238000010586 diagram Methods 0.000 description 8
- 238000011084 recovery Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 238000012937 correction Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1008—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices
- G06F11/1012—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices using codes or arrangements adapted for a specific type of error
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M13/00—Coding, decoding or code conversion, for error detection or error correction; Coding theory basic assumptions; Coding bounds; Error probability evaluation methods; Channel models; Simulation or testing of codes
- H03M13/61—Aspects and characteristics of methods and arrangements for error correction or error detection, not provided for otherwise
- H03M13/615—Use of computational or mathematical techniques
- H03M13/616—Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Pure & Applied Mathematics (AREA)
- Mathematical Physics (AREA)
- Mathematical Optimization (AREA)
- Mathematical Analysis (AREA)
- Computational Mathematics (AREA)
- Algebra (AREA)
- Computing Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- Error Detection And Correction (AREA)
Abstract
The application discloses a data processing method, a data processing device, an electronic device and a computer readable storage medium, wherein the method comprises the following steps: acquiring an original data block; performing right shift operation on the original data block by using a target Van der Waals matrix so as to encode the original data block to obtain an RS erasure code; wherein the element of the ith row and the jth column in the target Van der Waals matrix is 2(i‑1)(j‑1)(ii) a Storing the encoded data block; wherein the encoded data block includes the original data block and a parity data block. In the application, each element in the target vandermonde matrix is an exponent of 2, so that only the original storage block needs to be shifted based on the target vandermonde matrix in the encoding process, the hardware implementation is simple, and all multiplier operations related to standard vandermonde matrix encoding are omitted. Therefore, the data processing method provided by the application reduces the use of Van der Waals matrix to compileThe computational complexity of the code.
Description
Technical Field
The present application relates to the field of storage technologies, and in particular, to a data processing method and apparatus, an electronic device, and a computer-readable storage medium.
Background
In the face of the storage requirement of mass data, the distributed storage gradually replaces the dominant position of unified storage by the advantages of low cost, good expandability and the like. On one hand, a distributed storage system mostly uses inexpensive disks as storage nodes, and the reliability of each storage node is often not very high, and on the other hand, a distributed storage system usually contains many nodes, and due to software and hardware failures, human errors and other reasons, the system often has node failures. Therefore, in order to improve the data reliability of the distributed storage system and ensure that the data collection node can reconstruct the original file with high probability, a certain amount of redundancy needs to be additionally stored on the basis of storing the original data, so that the system can still normally operate under the condition that partial nodes fail, the data collection node can still decode and recover the original file, and meanwhile, the failed nodes need to be repaired in time.
Erasure codes (Erasure codes) belong to a forward error correction technology in the coding theory, and RS codes (Reed-Solomon codes) applied in a distributed environment are common in an actual storage system. The RS code is related to two parameters k and r, and given two positive integers k and r, encodes k data blocks into r additional check blocks. The method for obtaining r check blocks by encoding based on the vandermonde matrix is called RS erasure code encoded by using the vandermonde matrix, and the specific encoding process is as follows:
the k × k matrix of the upper portion corresponds to k original data blocks, and the r × k matrix of the lower portion corresponds to a vandermonde matrix, which corresponds to the original data D1-DkMultiplying to obtain newly added P1-PrR check data. When any maximum r data are in error or lost in transmission and need to be corrected, the inverse matrix of the matrix corresponding to the residual data and the data phase are usedBy multiplying, the original data block D is obtained1-Dk. For decoding with D1 to Dr data loss as an example, the process is as follows:
for different numbers of original data blocks to be stored, different numbers of check data blocks are required, so that different van der waals matrices are required to be constructed. Under different coding conditions, the combination of the lost storage data blocks is random, namely, a matrix of the random combination is to be inverted during decoding, and the computational complexity is increased for RS erasure decoding and storage data recovery based on the Van der Menu matrix.
Therefore, how to reduce the computational complexity of the van der waals matrix codec is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
An object of the present application is to provide a data processing method, an apparatus, an electronic device, and a computer-readable storage medium, which reduce the computational complexity of encoding and decoding using a vandermonde matrix.
In order to achieve the above object, the present application provides a data processing method, including:
acquiring an original data block;
performing right shift operation on the original data block by using a target Van der Waals matrix so as to encode the original data block to obtain an RS erasure code; wherein the element of the ith row and the jth column in the target Van der Waals matrix is 2(i-1)(j-1)；
Storing the encoded data block; wherein the encoded data block includes the original data block and a parity data block.
Wherein, still include:
when the target original data block is lost, generating a decoding matrix according to the original data block which is not lost and the check data block;
determining the number of data blocks of continuous lost data blocks, and determining an inverse matrix unit corresponding to the number of the data blocks;
performing left shift operation on elements in the inverse matrix unit according to the positions of the continuous lost data blocks to obtain a middle inverse matrix;
and calculating an inverse matrix of the decoding matrix by using the intermediate inverse matrix based on a matrix reduction method, and recovering the target original data block by using the inverse matrix and the encoded data block.
Wherein the determining the inverse matrix unit corresponding to the number of the data blocks includes:
determining an n-order target Van der Monte matrix; wherein n is the number of data blocks;
and utilizing a Lagrange difference method to perform inverse operation on the n-order target Van der Monte matrix to obtain inverse matrix units corresponding to the number of the data blocks.
Wherein, the left shift operation of the elements in the inverse matrix unit according to the positions of the consecutive lost data blocks to obtain a middle inverse matrix comprises:
determining the position distance between the first data block in the continuous lost data blocks and the first data block in the original data blocks;
and shifting the elements except the first column in the inverse matrix unit by the position distance to the left to obtain the middle inverse matrix.
To achieve the above object, the present application provides a data processing apparatus comprising:
the acquisition module is used for acquiring an original data block;
the encoding module is used for carrying out right shift operation on the original data block by utilizing a target Van der Monte matrix so as to encode the original data block to obtain an encoded data block; wherein the element of the ith row and the jth column in the target Van der Waals matrix is 2(i-1)(j-1)；
A storage module for storing the encoded data block; wherein the RS erasure code includes the original data block and a check data block.
Wherein, still include:
the generating module is used for generating a decoding matrix according to the original data block which is not lost and the check data block when the target original data block is lost;
the device comprises a determining module, a judging module and a judging module, wherein the determining module is used for determining the number of data blocks of continuous lost data blocks and determining an inverse matrix unit corresponding to the number of the data blocks;
the shift module is used for carrying out left shift operation on the elements in the inverse matrix unit according to the positions of the continuous lost data blocks to obtain a middle inverse matrix;
and the decoding module is used for calculating an inverse matrix of the decoding matrix by using the intermediate inverse matrix based on a matrix reduction method and recovering the target original data block by using the inverse matrix and the encoded data block.
Wherein the determining module comprises:
a first determining unit for determining the number of data blocks of consecutive lost data blocks;
a second determining unit, configured to determine an n-order target vandermonde matrix; wherein n is the number of data blocks;
and the inversion unit is used for carrying out inversion operation on the n-order target Van der Monte matrix by utilizing a Lagrange difference method to obtain an inverse matrix unit corresponding to the number of the data blocks.
Wherein the shifting module comprises:
a third determining unit, configured to determine a position distance between a first data block in the consecutive lost data blocks and a first data block in the original data blocks;
and the shifting unit is used for shifting the elements except the first column in the inverse matrix unit by the position distance to the left to obtain the middle inverse matrix.
To achieve the above object, the present application provides an electronic device including:
a memory for storing a computer program;
a processor for implementing the steps of the data processing method when executing the computer program.
To achieve the above object, the present application provides a computer-readable storage medium having stored thereon a computer program, which when executed by a processor, implements the steps of the data processing method as described above.
According to the above scheme, the data processing method provided by the application comprises the following steps: acquiring an original data block; performing right shift operation on the original data block by using a target Van der Waals matrix so as to encode the original data block to obtain an RS erasure code; wherein the element of the ith row and the jth column in the target Van der Waals matrix is 2(i-1)(j-1)(ii) a Storing the encoded data block; wherein the encoded data block includes the original data block and a parity data block.
In the application, each element in the target vandermonde matrix is an exponent of 2, so that only the original storage block needs to be shifted based on the target vandermonde matrix in the encoding process, the hardware implementation is simple, and all multiplier operations related to standard vandermonde matrix encoding are omitted. Therefore, the data processing method provided by the application reduces the computational complexity of encoding and decoding by using the Van der Monte matrix. The application also discloses a data processing device, an electronic device and a computer readable storage medium, which can also realize the technical effects.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure without limiting the disclosure. In the drawings:
FIG. 1 is a flow diagram illustrating a method of data processing in accordance with an exemplary embodiment;
FIG. 2 is a flow diagram illustrating another data processing method in accordance with an exemplary embodiment;
fig. 3 is a schematic diagram illustrating a combination of n-2 partial matrices in n-4 according to an exemplary embodiment;
FIG. 4 is a block diagram illustrating a data processing apparatus in accordance with an exemplary embodiment;
FIG. 5 is a block diagram illustrating an electronic device in accordance with an exemplary embodiment.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The embodiment of the application discloses a data processing method, which reduces the computational complexity of coding and decoding by using a Van der Monte matrix.
Referring to fig. 1, a flow chart of a data processing method according to an exemplary embodiment is shown, as shown in fig. 1, including:
s101: acquiring an original data block;
s102: performing right shift operation on the original data block by using a target Van der Waals matrix so as to encode the original data block to obtain an encoded data block; wherein the element of the ith row and the jth column in the target Van der Waals matrix is 2(i -1)(j-1)；
S103: storing the encoded data block; wherein the encoded data block includes the original data block and a parity data block.
In this embodiment, the original data blocks are data blocks to be stored, and the number of check data blocks, that is, the maximum number of data blocks allowed to be lost, is first determined. Secondly, an encoding matrix is constructed by utilizing the Van der Mongolian matrix, wherein the upper half part of the encoding matrix is an identity matrix, and the lower half part of the encoding matrix is the Van der Mongolian matrix. And finally, encoding the original data block by using the encoding matrix to obtain an encoded data block, wherein the encoded data block comprises the original data block and a check data block, the node stores the encoded data block, and when the original data block is lost, the check data block in the encoded data block can be used for data recovery.
For RS erasure codes, k original data blocks are encoded into r additional parity data blocks based on the vandermonde matrix. Taking k as 4 and r as 2 as an example, the encoding process is as follows:
for example, the lost data blocks after transmission are the second data block D2 and the third data block D3. The way to decode the recovered data block is:
the next two rows in the RS erasure operation above are the vandermonde matrices used to recover the data. The Van der Monte matrix is an important displacement matrix, Toeplitz matrices can be realized through construction, and common inversion modes include a Gaussian elimination method and a Lagrange interpolation method. The inverse of the Lagrange interpolation method can obtain:
wherein Z is [ Z ]ij]Expressed is the inverse of the vandermonde matrix of order n, n being the number of data blocks of consecutive lost data blocks, xpAre elements in the vandermonde matrix.
In the distributed storage RS erasure correction, a common erasure correction application environment supports maximum n ═ 4, that is, when four consecutive storage block errors need to be recovered at the same time. When n is 1, the inverse of the vandermonde matrix is the reciprocal of the element, and when n is 2 to n is 4, the inverse of the vandermonde matrix is as follows:
from the above formula, the inverse matrix of the vandermonde matrix is in obvious multiplication relation, so for convenient hardware calculation, the coding matrix of multiplication multiple relation is constructed, the multiplication considering 2 exponents in hardware can be expressed as shift operation, and is simpler than the traditional addition (subtraction) method and multiplication (division) operation, therefore, the multiplication relation selects 2 exponents and considers the Toeplitz characteristic to construct the target vandermonde matrix, in which the element of the ith row and the jth column is 2(i-1)(j-1)Namely:
for example, if k is 4 and r is 3, the coding matrix is:
the encoding process is as follows:
in the embodiment of the application, each element in the target vandermonde matrix is an exponent of 2, so that only the original storage block needs to be shifted based on the target vandermonde matrix in the encoding process, the hardware implementation is simple, and all multiplier operations related to standard vandermonde matrix encoding are omitted. Therefore, the data processing method provided by the embodiment of the application reduces the computational complexity of encoding and decoding by using the vandermonde matrix.
The following describes the decoding process in detail:
referring to fig. 2, a flow diagram of another data processing method according to an exemplary embodiment is shown, as shown in fig. 2, including:
s201: when the target original data block is lost, generating a decoding matrix according to the original data block which is not lost and the check data block;
the purpose of this step is to construct a decoding matrix, first selecting check data blocks for data recovery, the number of which is the same as the number of lost target original data blocks. And then, the original data blocks which are not lost and the selected check data blocks form a decoding matrix in the corresponding rows of the coding matrix.
For example, if k is 4, r is 3, the lost data blocks are D2 and D3, and the check data blocks selected for data recovery are P1 and P2, the decoding matrix is:
s202: determining the number of data blocks of continuous lost data blocks, and determining an inverse matrix unit corresponding to the number of the data blocks;
s203: performing left shift operation on elements in the inverse matrix unit according to the positions of the continuous lost data blocks to obtain a middle inverse matrix;
in a specific implementation, the number of data blocks of consecutive lost data blocks and an inverse matrix unit corresponding to the number of data blocks are determined first, so that the inverse matrix of the decoding matrix is calculated in the subsequent step. Specifically, the inverse matrix unit is an n-order target vandermonde matrix, and n is the number of data blocks in which data blocks are continuously lost. Namely, the step of determining the inverse matrix unit corresponding to the number of the data blocks may include: determining an n-order target Van der Monte matrix; wherein n is the number of data blocks; and utilizing a Lagrange difference method to perform inverse operation on the n-order target Van der Monte matrix to obtain inverse matrix units corresponding to the number of the data blocks.
For example, n is 2 to n is 4, and the corresponding inverse matrix unit is:
it can be known that the constructed encoding matrix presents obvious Toeplitz characteristics during decoding, namely, the Van der Menu inverse matrix of decoding operation only needs to complete half of the operation to obtain the encoding matrix. The case of n being 2-4 is exemplified above, because the case of n represents the case that several consecutive memory blocks are lost and need to be recovered, the case of 4 is less encountered in RS erasure correction practical application, and most of the cases are n-2 and a small number of n-3. For example, when the vandermonde matrix is a matrix with n equal to 4 for dealing with the loss of at most 4 consecutive blocks, most cases of different combinations of n equal to 2 occur, and in this case, it is necessary to deal with the inverse matrix problem of the partial matrix with n equal to 2 in n equal to 4, that is, it is necessary to perform left shift operation on the elements in the inverse matrix unit according to the positions of consecutive lost data blocks to obtain an intermediate inverse matrix.
As shown in fig. 3, when an n-4 encoding is confronted with a case where n-2 contiguous memory blocks are lost, there are 9 different cases of matrix inversion (in the dashed box) of n-2, in which case different 9 combinations of inverse matrices can be realized by simple shift calculation. Namely, the step of performing left shift operation on the elements in the inverse matrix unit according to the positions of the consecutive lost data blocks to obtain an intermediate inverse matrix includes: determining the position distance between the first data block in the continuous lost data blocks and the first data block in the original data blocks; and shifting the elements except the first column in the inverse matrix unit by the position distance to the left to obtain the middle inverse matrix.
Taking the inverse matrix with n-2 as a basic unit, another matrix can be obtained through shift calculation, and the method is as follows:
any combination of the following:
based on the relationship of x when constructing the matrix, xmIs an exponent of 2, so the inverse matrix is a left shift operation based on the base unit of the inverse matrix of n-2 when restoring the stored data block. And similarly, the case that n is 3 can be obtained, when n is greater than 4, only the inverse matrix basic unit greater than 4 needs to be added, and the corresponding actual erasure correction recovery only needs to be left shifted based on the data after the basic unit recovery is completed.
For example, the lost data blocks are D2 and D3, and the inverse matrix unit with n equal to 2 is
D2 is D1, the distance is 1, the 2 nd column of the inverse matrix unit is left shifted by 1 bit to obtain the middle inverse matrix
S204: and calculating an inverse matrix of the decoding matrix by using the intermediate inverse matrix based on a matrix reduction method, and recovering the target original data block by using the inverse matrix and the encoded data block.
In this step, based on the intermediate inverse matrix
Decoding matrix can be calculated based on matrix reduction method
The inverse of (c). The decoding process is as follows:
it can be seen that, in the decoding aspect, the present embodiment firstly needs to store several van der waals inverse matrix units in the case of the largest continuous missing data block, and the inverse matrix units exhibit obvious Toeplitz matrix characteristics through construction, so that the storage requirement is only half of that of a normal matrix. In terms of hardware, decoding is carried out by selecting a stored Van der Mongolian inverse matrix basic unit to carry out operation recovery firstly according to the condition of continuously losing data blocks and then carrying out leftward shift on the basis of the position. In the aspect of decoding, most of operations are omitted, the operation complexity is reduced, the operation speed and the parallelism are improved, the algorithm mode is easy to multiplex, the method is suitable for realizing a hardware circuit, and the blank of a hardware RS decoder is filled.
In the following, a data processing apparatus provided in an embodiment of the present application is introduced, and a data processing apparatus described below and a data processing method described above may be referred to each other.
Referring to fig. 4, a block diagram of a data processing apparatus according to an exemplary embodiment is shown, as shown in fig. 4, including:
an obtaining module 401, configured to obtain an original data block;
an encoding module 402, configured to perform right shift operation on the original data block by using a target vandermonde matrix, so as to encode the original data block to obtain an encoded data block; wherein the element of the ith row and the jth column in the target Van der Waals matrix is 2(i-1)(j-1)；
A storage module 403, configured to store the encoded data block; wherein the RS erasure code includes the original data block and a check data block.
In the embodiment of the application, each element in the target vandermonde matrix is an exponent of 2, so that only the original storage block needs to be shifted based on the target vandermonde matrix in the encoding process, the hardware implementation is simple, and all multiplier operations related to standard vandermonde matrix encoding are omitted. Therefore, the data processing device provided by the embodiment of the application reduces the computational complexity of encoding and decoding by using the vandermonde matrix.
On the basis of the above embodiment, as a preferred implementation, the method further includes:
the generating module is used for generating a decoding matrix according to the original data block which is not lost and the check data block when the target original data block is lost;
the device comprises a determining module, a judging module and a judging module, wherein the determining module is used for determining the number of data blocks of continuous lost data blocks and determining an inverse matrix unit corresponding to the number of the data blocks;
the shift module is used for carrying out left shift operation on the elements in the inverse matrix unit according to the positions of the continuous lost data blocks to obtain a middle inverse matrix;
and the decoding module is used for calculating an inverse matrix of the decoding matrix by using the intermediate inverse matrix based on a matrix reduction method and recovering the target original data block by using the inverse matrix and the encoded data block.
On the basis of the foregoing embodiment, as a preferred implementation, the determining module includes:
a first determining unit for determining the number of data blocks of consecutive lost data blocks;
a second determining unit, configured to determine an n-order target vandermonde matrix; wherein n is the number of data blocks;
and the inversion unit is used for carrying out inversion operation on the n-order target Van der Monte matrix by utilizing a Lagrange difference method to obtain an inverse matrix unit corresponding to the number of the data blocks.
On the basis of the above embodiment, as a preferred implementation, the shift module includes:
a third determining unit, configured to determine a position distance between a first data block in the consecutive lost data blocks and a first data block in the original data blocks;
and the shifting unit is used for shifting the elements except the first column in the inverse matrix unit by the position distance to the left to obtain the middle inverse matrix.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
The present application further provides an electronic device, and referring to fig. 5, a structure diagram of an electronic device 500 provided in an embodiment of the present application may include a processor 11 and a memory 12, as shown in fig. 5. The electronic device 500 may also include one or more of a multimedia component 13, an input/output (I/O) interface 14, and a communication component 15.
The processor 11 is configured to control the overall operation of the electronic device 500, so as to complete all or part of the steps in the data processing method. The memory 12 is used to store various types of data to support operation at the electronic device 500, such data can include, for example, instructions for any application or method operating on the electronic device 500 and application-related data. The Memory 12 may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk or optical disk. The multimedia component 13 may comprise a screen, which may be a touch screen, for example. The I/O interface 14 provides an interface between the processor 11 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 15 is used for wired or wireless communication between the electronic device 500 and other devices. Wireless Communication, such as Wi-Fi, bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so that the corresponding Communication component 15 may include: Wi-Fi module, bluetooth module, NFC module.
In an exemplary embodiment, the electronic Device 500 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components for executing the above-mentioned data Processing method.
In another exemplary embodiment, there is also provided a computer readable storage medium comprising program instructions which, when executed by a processor, implement the steps of the above-described data processing method. For example, the computer readable storage medium may be the memory 12 described above comprising program instructions executable by the processor 11 of the electronic device 500 to perform the data processing method described above.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. A data processing method, comprising:
acquiring an original data block;
performing right shift operation on the original data block by using a target Van der Waals matrix so as to encode the original data block to obtain an RS erasure code; wherein the element of the ith row and the jth column in the target Van der Waals matrix is 2(i-1)(j-1)；
Storing the encoded data block; wherein the encoded data block includes the original data block and a parity data block.
2. The data processing method of claim 1, further comprising:
when the target original data block is lost, generating a decoding matrix according to the original data block which is not lost and the check data block;
determining the number of data blocks of continuous lost data blocks, and determining an inverse matrix unit corresponding to the number of the data blocks;
performing left shift operation on elements in the inverse matrix unit according to the positions of the continuous lost data blocks to obtain a middle inverse matrix;
and calculating an inverse matrix of the decoding matrix by using the intermediate inverse matrix based on a matrix reduction method, and recovering the target original data block by using the inverse matrix and the encoded data block.
3. The data processing method according to claim 2, wherein the determining the inverse matrix unit corresponding to the number of the data blocks comprises:
determining an n-order target Van der Monte matrix; wherein n is the number of data blocks;
and utilizing a Lagrange difference method to perform inverse operation on the n-order target Van der Monte matrix to obtain inverse matrix units corresponding to the number of the data blocks.
4. The data processing method according to claim 2, wherein said left-shifting the elements in the inverse matrix unit according to the positions of the consecutive missing data blocks to obtain an intermediate inverse matrix, comprises:
determining the position distance between the first data block in the continuous lost data blocks and the first data block in the original data blocks;
and shifting the elements except the first column in the inverse matrix unit by the position distance to the left to obtain the middle inverse matrix.
5. A data processing apparatus, comprising:
the acquisition module is used for acquiring an original data block;
the encoding module is used for carrying out right shift operation on the original data block by utilizing a target Van der Monte matrix so as to encode the original data block to obtain an encoded data block; wherein the element of the ith row and the jth column in the target Van der Waals matrix is 2(i-1)(j-1)；
A storage module for storing the encoded data block; wherein the RS erasure code includes the original data block and a check data block.
6. The data processing apparatus of claim 5, further comprising:
the generating module is used for generating a decoding matrix according to the original data block which is not lost and the check data block when the target original data block is lost;
the device comprises a determining module, a judging module and a judging module, wherein the determining module is used for determining the number of data blocks of continuous lost data blocks and determining an inverse matrix unit corresponding to the number of the data blocks;
the shift module is used for carrying out left shift operation on the elements in the inverse matrix unit according to the positions of the continuous lost data blocks to obtain a middle inverse matrix;
and the decoding module is used for calculating an inverse matrix of the decoding matrix by using the intermediate inverse matrix based on a matrix reduction method and recovering the target original data block by using the inverse matrix and the encoded data block.
7. The data processing apparatus of claim 6, wherein the determining module comprises:
a first determining unit for determining the number of data blocks of consecutive lost data blocks;
a second determining unit, configured to determine an n-order target vandermonde matrix; wherein n is the number of data blocks;
and the inversion unit is used for carrying out inversion operation on the n-order target Van der Monte matrix by utilizing a Lagrange difference method to obtain an inverse matrix unit corresponding to the number of the data blocks.
8. The data processing apparatus of claim 6, wherein the shifting module comprises:
a third determining unit, configured to determine a position distance between a first data block in the consecutive lost data blocks and a first data block in the original data blocks;
and the shifting unit is used for shifting the elements except the first column in the inverse matrix unit by the position distance to the left to obtain the middle inverse matrix.
9. An electronic device, comprising:
a memory for storing a computer program;
processor for implementing the steps of the data processing method according to any one of claims 1 to 4 when executing said computer program.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the data processing method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010725775.0A CN111858142A (en) | 2020-07-24 | 2020-07-24 | Data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010725775.0A CN111858142A (en) | 2020-07-24 | 2020-07-24 | Data processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111858142A true CN111858142A (en) | 2020-10-30 |
Family
ID=72950843
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010725775.0A Pending CN111858142A (en) | 2020-07-24 | 2020-07-24 | Data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111858142A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113360264A (en) * | 2021-06-11 | 2021-09-07 | 山东云海国创云计算装备产业创新中心有限公司 | Erasure processing system and method and distributed storage system |
| CN113836482A (en) * | 2021-07-30 | 2021-12-24 | 深圳大学 | Code distributed computing system |
| CN114281270A (en) * | 2022-03-03 | 2022-04-05 | 山东云海国创云计算装备产业创新中心有限公司 | Data storage method, system, equipment and medium |
| CN114442950A (en) * | 2022-01-21 | 2022-05-06 | 山东云海国创云计算装备产业创新中心有限公司 | Data recovery method, system, device and computer readable storage medium |
| CN116028260A (en) * | 2023-01-09 | 2023-04-28 | 海光信息技术股份有限公司 | Data processing method, processing device and storage system |
| CN116092567A (en) * | 2023-01-09 | 2023-05-09 | 海光信息技术股份有限公司 | Data processing method, processing device and storage system |
| CN114879904B (en) * | 2022-04-22 | 2024-05-03 | 山东云海国创云计算装备产业创新中心有限公司 | Data storage erasure method, device, equipment and readable storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109727196A (en) * | 2018-12-26 | 2019-05-07 | 呈像科技(北京)有限公司 | Image interpolation processing method |
| CN111404554A (en) * | 2019-09-29 | 2020-07-10 | 杭州海康威视系统技术有限公司 | Method and device for generating coding matrix of erasure code and electronic equipment |
-
2020
- 2020-07-24 CN CN202010725775.0A patent/CN111858142A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109727196A (en) * | 2018-12-26 | 2019-05-07 | 呈像科技(北京)有限公司 | Image interpolation processing method |
| CN111404554A (en) * | 2019-09-29 | 2020-07-10 | 杭州海康威视系统技术有限公司 | Method and device for generating coding matrix of erasure code and electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| 尹训春: "基于编码技术的网盘存储的研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113360264A (en) * | 2021-06-11 | 2021-09-07 | 山东云海国创云计算装备产业创新中心有限公司 | Erasure processing system and method and distributed storage system |
| CN113836482A (en) * | 2021-07-30 | 2021-12-24 | 深圳大学 | Code distributed computing system |
| CN113836482B (en) * | 2021-07-30 | 2023-12-05 | 深圳大学 | Coding distributed computing system |
| CN114442950A (en) * | 2022-01-21 | 2022-05-06 | 山东云海国创云计算装备产业创新中心有限公司 | Data recovery method, system, device and computer readable storage medium |
| CN114442950B (en) * | 2022-01-21 | 2024-01-23 | 山东云海国创云计算装备产业创新中心有限公司 | Data recovery method, system, device and computer readable storage medium |
| CN114281270A (en) * | 2022-03-03 | 2022-04-05 | 山东云海国创云计算装备产业创新中心有限公司 | Data storage method, system, equipment and medium |
| CN114281270B (en) * | 2022-03-03 | 2022-05-27 | 山东云海国创云计算装备产业创新中心有限公司 | Data storage method, system, equipment and medium |
| CN114879904B (en) * | 2022-04-22 | 2024-05-03 | 山东云海国创云计算装备产业创新中心有限公司 | Data storage erasure method, device, equipment and readable storage medium |
| CN116028260A (en) * | 2023-01-09 | 2023-04-28 | 海光信息技术股份有限公司 | Data processing method, processing device and storage system |
| CN116092567A (en) * | 2023-01-09 | 2023-05-09 | 海光信息技术股份有限公司 | Data processing method, processing device and storage system |
| CN116028260B (en) * | 2023-01-09 | 2024-02-27 | 海光信息技术股份有限公司 | Data processing method, processing device and storage system |
| CN116092567B (en) * | 2023-01-09 | 2024-03-22 | 海光信息技术股份有限公司 | Data processing method, processing device and storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111858142A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113297000B (en) | RAID (redundant array of independent disks) coding circuit and coding method | |
| CN111697976B (en) | RS erasure correcting quick decoding method and system based on distributed storage | |
| CN111682874B (en) | Data recovery method, system, equipment and readable storage medium | |
| CN109873645B (en) | Soft chip hunting restoration for multiple word line failures | |
| CN107852176A (en) | LDPC code encoder and decoder | |
| CN111858169B (en) | Data recovery method, system and related components | |
| CN112799875B (en) | Method, system, device and medium for verification recovery based on Gaussian elimination | |
| CN112000512B (en) | Data restoration method and related device | |
| Hou et al. | A new construction and an efficient decoding method for Rabin-like codes | |
| CN113296999B (en) | RAID6 coding method and coding circuit | |
| CN113297001A (en) | RAID (redundant array of independent disks) coding and decoding method and coding and decoding circuit | |
| CN113505019A (en) | Erasure code data and check recovery method, device, equipment and readable medium | |
| Tang et al. | Fast en/decoding of Reed-Solomon codes for failure recovery | |
| JP7429223B2 (en) | Turbo product code decoding method, device, decoder and computer storage medium | |
| CN107306140B (en) | GEL code word structure coding and decoding method, device and related equipment | |
| KR102475784B1 (en) | One-sub-symbol linear repair schemes | |
| CN103151078B (en) | Error detection and correction code generation method for memory | |
| CN108140016B (en) | Optimizer for determining optimal sequence of operations for matrix-vector multiplication | |
| KR101569637B1 (en) | Method and Apparatus for Non-Iterative Soft-Decision BCH Decoding using Test Syndrome | |
| Chen et al. | A new Zigzag MDS code with optimal encoding and efficient decoding | |
| KR102607761B1 (en) | Method and apparatus for generating a decoding position control signal for decoding using polar codes | |
| CN108471315B (en) | Erasure decoding method and device | |
| CN107463462B (en) | Data restoration method and data restoration device | |
| CN108352845B (en) | Method and device for encoding storage data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201030 |