CN105007286A - Decoding method, decoding device, and cloud storage method and system - Google Patents
Decoding method, decoding device, and cloud storage method and system Download PDFInfo
- Publication number
- CN105007286A CN105007286A CN201410167214.8A CN201410167214A CN105007286A CN 105007286 A CN105007286 A CN 105007286A CN 201410167214 A CN201410167214 A CN 201410167214A CN 105007286 A CN105007286 A CN 105007286A
- Authority
- CN
- China
- Prior art keywords
- data
- decoded
- burst
- decoding
- former
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/004—Arrangements for detecting or preventing errors in the information received by using forward error control
- H04L1/0056—Systems characterized by the type of code used
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The invention provides a decoding method, a decoding device, and a cloud storage method and system, and belongs to the field of cloud storage technology. The decoding method comprises: reading to-be-decoded data from an input buffer zone; transferring the read to-be-decoded data including an original data fragment and a checking data fragment to an output module, and processing the to-be-decoded data in the output module; and selecting the corresponding data fragment according to the number of the data fragments in an original data zone and a to-be-decoded zone of the output module after the processing, performing decoding, and combining the decoded result to whole original data. According to the technical scheme in the invention, parallel processing of decoding and data obtaining can be performed, and the decoding efficiency is greatly improved. The sequence of the to-be-decoded data is not required, and the to-be-decoded including original data and checking data can be processed in the input buffer zone at a random sequence.
Description
Technical field
The present invention relates to cloud technical field of memory, be specifically related to a kind of coding/decoding method and device and cloud storage means and system.
Background technology
Cloud storage refers to by functions such as cluster application, network technology or distributed file systems, various dissimilar memory device a large amount of in network is gathered collaborative work by application software, a system of data storage and Operational Visit function is externally provided jointly.In cloud storage system, file is generally fragmented and is kept in multiple cloud storage server.In addition, when data communication, need the data of communication also can be divided into multiple burst, send the other side to piecewise.
When data storage and communication, in order to solve the problem of reliability, correcting and eleting codes (Erasure Codes, referred to as EC) technology can be adopted, file is encoded, be divided into several initial data bursts and verification burst, carry out respectively storing or communication.Read-Solomon (Reed-Solomon, referred to as RS) correcting and eleting codes is a kind of conventional EC technology, and after original document is encoded by it, sized by point, m identical initial data burst and n verification burst, carry out storing or communication respectively.File is stored or communication recipient, as long as obtain wherein any m burst, namely recovers original or data by decoding process, n burst therefore can be resisted to damage or lose, substantially increasing the reliability of system.Store for computer documents, the memory space utilance of correcting and eleting codes system is m/ (m+n), and far away higher than copy storage mode, therefore, RS correcting and eleting codes exchanges storage capacity for by using computing capability, significantly reduces carrying cost and O&M cost.
Nineteen sixty, Reed (I.S.Reed) and Saloman (G.Solomon) propose a kind of method constructing correcting and eleting codes, use the correcting and eleting codes of the method to be referred to as Reed-Solomon code, are called for short RS code.Correcting and eleting codes based on RS coding techniques structure is then called RS correcting and eleting codes.(n, a k) correcting and eleting codes is that k source data is encoded to n (n>k) individual data, makes by k any in this n data according to original k the source data of equal restructural.The correcting and eleting codes system adopting m burst and n verification burst is exactly (m+n, m) correcting and eleting codes.
The actualizing technology key of RS correcting and eleting codes is searching generator matrix G, and in this matrix, any k arranges equal linear independence.In RS coding techniques, generator matrix can be generalized circular matrix and Cauchy matrix.Use the coding of generalized circular matrix generation Vandermonde code (Vandermond Code), use the coding of Cauchy matrix generation Cauchy's code (CauchyCode).Its coding principle is: initial data D [D1, D2 ..Dm] is multiplied by generator matrix G, produce m initial data D [D1, D2 ..Dm] and n checking data P [P1, P2 ..., Pn].Decoding principle is: in m+n initial data and checking data, obtain m data arbitrarily, form data D' to be decoded, such as [D1 ... Di, P1..Pj].In generator matrix, remove n do not occur those row that data are corresponding, form a new generator matrix G, then calculate the inverse matrix G' of G, last D'*G'=D.In order to make the result of matrix operation remain on real number field, computing changes into and is limited to Galois (Galois) territory G (2^8) or G (2^16) carries out.Concrete operation flow process and characteristic, thoroughly studied already, relevant mathematics books and publication can be found, therefore do not repeat them here.
But, the no matter system of the RS correcting and eleting codes system of generalized circular matrix and Cauchy matrix, decoding has the shortcoming that common, namely just D can be calculated by D'*G'=D, therefore after only collecting an arbitrary m burst, if wherein containing verification burst, corresponding decoding generator matrix G' and data D' to be decoded could determine, system could start decoding.Before system receives m burst, even if know the numbering of this burst, corresponding decoding generator matrix G' can be obtained, but due to data D'=to be decoded [D1 ... Di, P1..Pj], wherein at least lose a fragment data, therefore can not determine D'.After only having acquisition m burst, D' could finally determine, system could start decoding, therefore system decodes performance suffers the slowest that in any m burst.In existing system, network transmission module and decoding computing module are serials, and before collecting m burst, network transmission module is in work, and the computing module free time of decoding; After starting decoding, decoding computing module work and network transmission module is idle, reduces systematic function.
Another shortcoming is, according to RS decoding algorithm, decoding calculating quantity is O (n), is namely directly proportional with verification burst quantity n.Therefore, the original slice contained in data to be decoded is more, and verification burst is fewer, and the performance of decoding is better.Consider the P2P memory model based on public network, for the configuration of m+n burst, client has received m-k original slice and k verification burst, starts to decode.Before decoding is done, t original slice is received again.Obviously, represent during t=k that client have received whole original slice, decoding can be stopped.But during t<k, or needs decoding, according to m-k+t original slice and the decoding of k-t verification burst, the configuration verifying burst than former m-k original slice+k is better.If but termination coding restarts, then the amount of calculation before and time have all slatterned.Because RS decoding is a kind of calculating relatively consuming time, uses less verification burst to reduce amount of calculation when decoding, well improving performance.
Therefore, in the coding/decoding method of prior art based on RS correcting and eleting codes, there is the problem of maximum delay node in m arbitrary node of acquisition of being limited by, and once after decoding and starting, after obtaining again new data fragmentation, system cannot utilize thus reduce the problem of performance.In addition, existing decoding process, have particular requirement to the order of input data, be generally divided into two parts, front portion is former data acquisition system, and rear portion is the set of checking data, and coding/decoding method is dumb.
Summary of the invention
The invention provides a kind of coding/decoding method and device and cloud storage means and system, it can solve decode operation and data acquisition work serial in prior art and carry out, thus reduces the problem of decoding efficiency.
For achieving the above object, on the one hand, provide a kind of coding/decoding method in embodiments of the invention, comprise step:
Read the data to be decoded of input block, by the moving data to be decoded comprising former data fragmentation and checking data burst that reads to output module, and treat decoded data in the output module and process;
According to the quantity of data fragmentation in data field, output module Central Plains after process and region to be decoded, choose corresponding data fragmentation and carry out decode operation, decode operation result is combined into whole former data.
Preferably, in described method, described decoded data for the treatment of in the output module carries out processing specifically comprising:
If read checking data burst, then the data with existing of the corresponding position in the region to be decoded of described checking data burst and output module is carried out xor operation, and using data with existing new as this position for XOR result;
If read former data fragmentation, then described former data fragmentation is moved to the corresponding position of the former data field of output module, described former data fragmentation is multiplied by respectively whole elements of generator matrix corresponding row simultaneously, each result is carried out xor operation with the data with existing of the corresponding position in the region to be decoded of output module respectively, and using data with existing new as this position for XOR result.
Preferably, in described method, the described quantity according to data fragmentation in data field, output module Central Plains and region to be decoded after process, choose corresponding data fragmentation and carry out decode operation and specifically comprise:
The total quantity m of former data fragmentation when if the quantity sum of data fragmentation is not less than coding in data field, output module Central Plains and region to be decoded, then whole m-k obtainable in former data field former data fragmentation and k checking data burst selecting from region to be decoded are decoded.
Preferably, carrying out described in decodes specifically comprises:
According to the position of k the former data fragmentation lost and the position of k according to burst selecting region to be decoded, from generator matrix, choose corresponding element form selection matrix, obtain the inverse matrix of selection matrix as decoding matrix;
The k selected from region to be decoded is formed matrix to be decoded according to burst simultaneously;
Described matrix to be decoded is multiplied with described decoding matrix, obtains k former data fragmentation of described loss.
On the other hand, also provide a kind of cloud storage means in embodiments of the invention simultaneously, comprise step:
Former data are encoded, is divided at least one former data fragmentation and at least one checking data burst, carries out respectively storing and/or communication;
When user will obtain described former data, perform coding/decoding method as above.
Again on the one hand, additionally provide a kind of decoding device in embodiments of the invention, comprising:
Data acquisition module, for reading the data to be decoded of input block, by the moving data to be decoded comprising former data fragmentation and checking data burst that reads to output module, and treats decoded data in the output module and processes;
Data decode module, for the quantity according to data fragmentation in data field, output module Central Plains after process and region to be decoded, chooses corresponding data fragmentation and carries out decode operation, decode operation result is combined into whole former data.
Preferably, described data acquisition module specifically comprises:
Input block module, for receiving and depositing the data to be decoded comprising former data fragmentation and checking data burst;
Judge and read module, for reading the data to be decoded in the module of input block, if read checking data burst, then the data with existing of the corresponding position in the region to be decoded of described checking data burst and output module is carried out xor operation, and using data with existing new as this position for XOR result; If read former data fragmentation, then described former data fragmentation is moved to the corresponding position of the former data field of output module, described former data fragmentation is multiplied by respectively whole elements of generator matrix corresponding row simultaneously, each result is carried out xor operation with the data with existing of the corresponding position in the region to be decoded of output module respectively, and using data with existing new as this position for XOR result.
Preferably, described data decode module also comprises:
Burst chooses module, during the total quantity m of former data fragmentation when being not less than coding for the quantity sum of data fragmentation in former data field in the output module and region to be decoded, whole m-k obtainable in former data field former data fragmentation and the k selected from region to be decoded are decoded according to burst.
Preferably, described data decode module comprises further:
Matrix inversion module, for according to the position of k the former data fragmentation lost and the position of k according to burst selecting region to be decoded, chooses corresponding element and forms selection matrix, obtain the inverse matrix of selection matrix as decoding matrix from generator matrix;
Computing module, is multiplied with described decoding matrix for the matrix to be decoded k selected from region to be decoded formed according to burst, obtains k former data fragmentation of described loss.
On the other hand, also provide a kind of cloud storage system in embodiments of the invention, comprising:
Coding module, for former data being encoded, being divided at least one former data fragmentation and at least one checking data burst, carrying out respectively storing and/or communication; And decoding device as above.
The present invention carries out multistage decoding by the mode processing data in input block one by one, after first data arrives behind input block, can start decoding effort; After data all in input block are all disposed, data recovery operation can be completed at once, make full use of the computing capability of decoding client and can not leave unused, thus can decoding while reception data, substantially increase decoding efficiency.The present invention treats the order not requirement of decoded data, comprises the data to be decoded of former data and checking data, can random order be processed in input block.
Accompanying drawing explanation
Fig. 1 is the schematic flow sheet of the coding/decoding method in one embodiment of the present of invention;
Fig. 2 is the structural representation of the decoding device in one embodiment of the present of invention;
The schematic flow sheet of coding/decoding method of Fig. 3 for providing in a preferred embodiment of the present invention.
Embodiment
For making those skilled in the art understand technical scheme of the present invention better, below in conjunction with the drawings and specific embodiments, the present invention is described in further detail.
In prior art, in coding/decoding method based on RS correcting and eleting codes, obtaining the data identical with former data bulk and forming data to be decoded, after determining corresponding decoding generator matrix, system just starts decoding, the communication of data and decode procedure work in series, restricted to the input sequence of data, generally need first to input former data, then input validation data, coding/decoding method is dumb, and decoding efficiency is low.
In coding/decoding method of the present invention, the calculating operation of decoding starts from first any burst obtained, whole decoding effort can be completed after any m the burst obtained, can obtain m burst according to arbitrary order, no matter they are former data or checking data, and decode operation and data acquisition work can walk abreast and carry out, if before decoding is done, m burst has been exceeded, the former data that system meeting prioritizing selection is wherein all, the least possible use checking data in the data acquisition system of input.
Fig. 1 is the schematic flow sheet of the coding/decoding method in one embodiment of the present of invention, and as shown in Figure 1, this coding/decoding method comprises:
Step S101, read the data to be decoded of input block, by the moving data to be decoded comprising former data fragmentation and checking data burst that reads to output module, and treat decoded data in the output module and process.
Particularly, first the encoding metadata of file is obtained, comprise former data fragmentation number m, checking data burst number n, to build generator matrix Code [m] [n], monitors the data in input block, wherein, according to the characteristic of system storage and the damaged condition of memory node, in input block, both may there is former data fragmentation, also may have checking data burst.If read checking data burst, then the data with existing of the corresponding position in the region to be decoded of described checking data burst and output module is carried out XOR (XOR) and operate, and using data with existing new as this position for XOR result; If read former data fragmentation, then described former data fragmentation is moved to the corresponding position of the former data field of output module, described former data fragmentation is multiplied by respectively whole elements of generator matrix corresponding row simultaneously, each result is carried out xor operation with the data with existing of the corresponding position in the region to be decoded of output module respectively, and using data with existing new as this position for XOR result.
For example, if there are the multiple data in input block, prioritizing selection reads checking data burst (the school inspection data fragmentation Pj as occurred at first in input block) wherein, and by the corresponding position in the region to be decoded of itself and output module (for a jth checking data burst Pj, its relevant position is exactly a jth position in region to be decoded) data with existing carry out xor operation (because of adopt xor operation, the primary data 0 of this position can be considered data with existing when to carry out xor operation for the first time in this position), XOR result becomes the new data with existing in this position (Pj'), when there is no checking data, select former data fragmentation wherein (the former data fragmentation Di as occurred at first in input block), moved to the corresponding position of the former data field of output module (for i-th former data fragmentation Di on the one hand, its relevant position is exactly i-th position of former data field), whole elements of generator matrix corresponding row (i.e. the i-th row) are multiplied by respectively (i.e. Code [i] [j] on the other hand with this former data fragmentation (i.e. Di), j is from 1 to n), and by result respectively with (the i.e. jth position in region to be decoded, relevant position in the region to be decoded of output module, j carries out xor operation and event memory from 1 to data with existing n), the data in process input block are continued, until data are empty in input block after being disposed.
Step S102, quantity according to data fragmentation in data field, output module Central Plains and region to be decoded after process, choose corresponding data fragmentation and carry out decode operation, decode operation result is combined into whole former data.
Particularly, if during the total quantity m of the former data fragmentation when quantity sum of data fragmentation is not less than coding in data field, output module Central Plains and region to be decoded, can decode operation be carried out.If there is m former data fragmentation former data field, then can no longer carry out follow-up matrix multiple work, directly m former data fragmentation is assembled into former data file in order.If former data field is less than the individual former data fragmentation of m (assumed lost k former data fragmentation, k≤n), then last obtainable whole m-k former data fragmentation and the k selected from region to be decoded are decoded according to burst (when to have more than k according to burst in region to be decoded, only random or selective sequential wherein k).
For example, decode procedure is: according to the position of k the former data fragmentation lost and the position of k according to burst selecting region to be decoded, from generator matrix Code [m] [n], choose corresponding element form selection matrix G1 [k] [k] (assuming that lose former data fragmentation Di, selected a region to be decoded jth data fragmentation Pj' simultaneously, then choose the element of the i-th row jth row in generator matrix, form selection matrix thus), obtain the inverse matrix of selection matrix as decoding matrix Decode [k] [k]; The k selected from region to be decoded is formed matrix P'[k to be decoded according to burst] simultaneously, matrix to be decoded is multiplied (according to data composition situation with decoding matrix, can premultiplication or the right side take advantage of), k the former data fragmentation lost can be obtained, it is inserted respectively the corresponding position of the former data field of output module, subsequently whole m former data fragmentation is assembled into former data file in order.
Preferably, if just have any m data burst in input block at the beginning, both comprised former data fragmentation and also comprised checking data burst, and burst quantity no longer increases, then above flow process can do following optimization.Wherein, in buffering area, the order of burst can be out of order, and such as use the decoding cases of 4 former data fragmentations, 2 checking data bursts, input block data may be [D3, P1, D2, P2, D1, D4], also may be [P2, D2, D1, P1, D4, D3] etc.The order of these data is relevant with system acceptance order.
Optimization method: scanning buffer data from the beginning to the end, if checking data burst Pj, by the corresponding position data with existing XOR in the region to be decoded of itself and output module.Namely the data Pj':P of a jth position in the region to be decoded of output module is now obtained by following formula
j' ⊕=P
j, j ∈ 1...n, wherein, the initial value of data Pj' is 0.
If former data fragmentation Di, Di to be moved into the relevant position (i.e. i-th position) of the former data field of output module on the one hand, another aspect Di is multiplied by each Elements C ode [i] [k] of generator matrix corresponding row respectively, and (k is all numberings of checking data burst, in previous example, be and be multiplied by respectively Code [i] [1] and Code [i] [2]), and by result respectively with the corresponding position data with existing XOR in the region to be decoded of output module.Namely the data Pj':P of a jth position in the region to be decoded of output module is now obtained by following formula
j' ⊕=D
i* Code [i] [j], i ∈ 1...m, j ∈ 1...n.
In the coding/decoding method that the present embodiment provides, the calculating operation of decoding system decodes starts from first any burst obtained, by having calculated whole decoding effort after any m the burst obtained, m burst can be obtained according to arbitrary order, no matter they are former data or checking data, and decode operation and data fragmentation acquisition work can walk abreast and carry out.If before decoding is done, in the data acquisition system of input, exceed m burst, the former data fragmentation that system meeting prioritizing selection is wherein all, the least possible use verification burst, thus improve decoding efficiency.
Present invention also offers a kind of decoding processing device, this device is used for realizing above-described embodiment and preferred implementation, has carried out repeating no more of explanation.As used below, term " module " can be realize the software of predetermined function and/or the combination of hardware.Although the device described by following examples preferably realizes with software, hardware, or the realization of the combination of software and hardware also may and conceived and implemented.
Fig. 2 is the structural representation of the decoding device in one embodiment of the present of invention, and as shown in Figure 2, this decoding device comprises: data acquisition module 21 and data decode module 22.Wherein, data acquisition module 21, for reading the data to be decoded of input block, by the moving data to be decoded comprising former data fragmentation and checking data burst that reads to output module, and is treated decoded data in the output module and is processed; Data decode module 22, for the quantity according to data fragmentation in data field, output module Central Plains after process and region to be decoded, is chosen corresponding data fragmentation and is carried out decode operation, decode operation result is combined into whole former data.
Data acquisition module 21 specifically comprises: input block module 211, decoding are set up module 212 and judged and read module 213.
Input block module 211, for depositing data fragmentation to be decoded, both having comprised former data fragmentation, also having comprised checking data burst, and minimum capacity is m, and heap(ed) capacity is m+n.Leave in after data fragmentation enters in input block, after data fragmentation is processed, shift out input block.Data fragmentation in input block according to input sequence process, also can process according to certain rule, such as priority treatment checking data burst, whether preferentially can depend on judgement and read module.
Decoding sets up module 212, the data needed during for setting up decoding, comprises metadata when obtaining former encoded data file, comprise former data fragmentation quantity m, checking data burst quantity n, file size, finally end up whether mend 0 and so on coded message.By these metadata, decoding is set up generator matrix Code [m] [n] during the former data encoding of module creation.Generator matrix is used for generating each check code when encoding by this matrix.
Judge and read module 213, for monitoring input block, judge that whether decode operation is overtime, and whether can enter into next step matrix inversion operation.If there is data fragmentation input block, then preferentially reads checking data burst and process; Otherwise process former data fragmentation.Preferential reading checking data burst herein not necessarily, also can read the data fragmentation of input one by one according to the principle of first in first out and process.The principle of process is: if checking data burst, then the data with existing of itself and corresponding position, output module region to be decoded is carried out xor operation, and using data with existing new as this position for XOR result; If former data fragmentation, then moved to the corresponding position of the former data field of output module, it is multiplied by respectively whole elements of generator matrix corresponding row simultaneously, by each result, the data with existing of corresponding position, region to be decoded carries out xor operation with output module respectively, and using data with existing new as this position for XOR result.
Judge and all data fragmentations in read module 213 circular treatment input block, comprise afterwards at the stylish data fragmentation added of decoding, until input block be empty or overtime (if overtime, then failure of decoding).After input block data fragmentation is disposed and the data fragmentation quantity sum in the former data field of output module and region to be decoded is more than or equal to m, namely enters next step data decode module 22 and carry out decoding process.
Data decode module 22 specifically comprises: matrix inversion module 221, computing module 222 and output module 223.
Matrix inversion module 221, principle uses more former data fragmentation and less data fragmentation to be decoded as far as possible.(amount to m-k) after all being used by former data fragmentation whole in output module, then select the k in output module region to be decoded according to burst (composition decoding matrix P'[k]).According to the position of k the former data fragmentation lost and the position of k according to burst selecting region to be decoded, from generator matrix Code [m] [n], choose corresponding element form selection matrix G1 [k] [k], obtain its inverse matrix as decoding matrix Decode [k] [k] by the classical matrix mode of inverting; Computing module 222, is multiplied matrix to be decoded with decoding matrix, obtains those the former data fragmentations lost.The former data fragmentation calculated is inserted the corresponding position of the former data field of output module; Output module 223, for being assembled into former data file in order by whole m of former data field former data fragmentations and exporting.
Preferably, output module 223 comprises former data field 2231 and region to be decoded 2232 further.Its data field, Central Plains 2231 amounts to m position, respectively corresponding original m former data fragmentation.Each former data fragmentation Di in input block, can be judged with read module 213 by mobile (or copying) corresponding i-th position, former data field 2231 to output module.Region 2232 to be decoded amounts to n position, respectively corresponding original n checking data burst.Each checking data burst Pj in input block, can be judged and carry out xor operation with the corresponding jth position data with existing Pj' in region to be decoded 2232 of read module 213 and output module and result is saved as new Pj'.For each former data fragmentation Di in input block, also can be judged and be multiplied by Code [i] [j] with read module 213, and carry out xor operation with Pj' and result is saved as new Pj'.
Solve in the decoding device that the present embodiment provides in the existing coding/decoding method based on RS correcting and eleting codes and system, the communication of existence and the complete serial of decode procedure, decoding speed is slow, and system is dumb, and the problem that encoding rate is low.Time started due to system decodes is no longer limited by m burst of acquisition, can further improve decoding speed, is specially adapted to the system that network environment is poor, as based on the P2P download environment of public network and slow network.Adopt the present invention in field of data communication, can accomplish that the decoding of data limit is listened on limit, substantially increase decoding efficiency.
Fig. 3 is the flow chart of the fast decoding method based on RS correcting and eleting codes of a preferred embodiment of the invention, and as shown in Figure 3, the method comprises the steps:
Step S302, the metadata information that module obtains file is set up in decoding.
Particularly, metadata information comprise burst quantity information m and n of coding, file size, finally end up whether mend 0 and so on coded message.If former data file stores according to Chunk (large piecemeal), then also comprise burst information and the length of Chunk.By m and n, decoding is set up generator matrix Code [m] [n] when module foundation is encoded originally.Generate n check code by this matrix during coding, meanwhile, also initialization inverse matrix Decode [n] [n], this matrix deposits at most the matrix information of n*n.The countless certificate of current Decode array.If there be m-k former data fragmentation in final input block, only need to use k checking data burst, then inverse matrix only needs Decode [k] [k] store data.
Step S304, judges and read module, checks whether input block is empty.
Particularly, if the data in input block are empty, then perform step S308, if the data in input block are not empty, then perform step S306.
Step S306, if having data in input block, judges and whether has checking data burst in read module inspection input block, if had, then performing step S310; If no, then perform step S312.
As another kind of implementation of the present invention, can affected following correction in this step: first order takes out a data fragmentation in input block, and then judges whether it is checking data burst.If checking data burst, perform step S310, otherwise perform step S312.
Step S308, when input block is empty, judgement and read module check whether output module has m data available burst.
Particularly, data available burst sum is the former data field of output module and the summation of area data burst quantity to be decoded.Be more than or equal to m data burst if had, then perform step S320; If be no more than m, then perform step S316.
Step S310, if there is checking data burst input block, to judge and the data with existing of the corresponding position in the region to be decoded of itself and output module is carried out xor operation by read module, and using data with existing new as this position for XOR result, go to step 304 subsequently.
Step S312, if there is former data fragmentation input block, Di is moved or copies to the corresponding position of the former data field of output module by judgement and read module.Go to step S314.
Step S314, Di is multiplied by Code [i] [j] by judgement and read module respectively, and the data with existing of the corresponding position in the region to be decoded of result and output module is carried out xor operation, and using data with existing new as this position for XOR result, namely perform Pj' ⊕=Di*Code [i] [j] operation, then perform step S304.
Step S316, judges and read module judges that whether decode operation is overtime.
Particularly, if there is no time-out, then perform step S304; If overtime, then perform step S318.
Step S318, decoding is failed because of time-out, judges and read module report decoding failed message.
Now, decode operation terminates.
Step S320, output module has had at least m data burst (comprise the data fragmentation in former data field and region to be decoded).
Reading and judge module choose the former data fragmentation of all former data fields, amount to m-k; Then in region to be decoded, select k according to burst, and form matrix P'[k to be decoded].
Step S322, matrix inversion module, according to the position of k the former data fragmentation lost and the position of k according to burst selecting region to be decoded, from generator matrix Code [m] [n], choose corresponding element form selection matrix G1 [k] [k], obtain the inverse matrix of selection matrix as decoding matrix Decode [k] [k].
Step S324, according to matrix P'[k to be decoded] and decoding matrix D ecode [k] [k], calculate k former data fragmentation of loss.The former data fragmentation lost is inserted the corresponding position of the former data field of output module.Perform step S326.
Step S326, by all data fragmentations in former data field of output module, reverts to original file or complete long data block (Chunk).In former data field, the existing data fragmentation originally put into, also has the obliterated data burst calculated.
Now, whole decoding process is terminated.
It should be noted that, the fast decoding flow process based on RS correcting and eleting codes shown in Fig. 3 just normally realizes principle in order to what the fast coding flow process that the embodiment of the present invention provides was described, also may there will be abnormal conditions during specific implementation, therefore also need to provide abnormality processing, file by block process, by file metadata process, to last mend zero or separately the processing procedures such as assembled are carried out to tail block.
Technical scheme of the present invention can realize downloading data fragmentation limit, limit and decode; During final decoding, the former data fragmentation of all acquisitions can be made full use of, when reducing decoding, verify the quantity of burst.Have nothing to do with the sequence of input data during decoding, data random order can enter decode system, as long as the data fragmentation quantity entered in input block is more than or equal to m, and last all energy successfully decoded.And during decoding, need the matrix carrying out matrix inversion to become k*k from original m*m, k≤n, matrix inversion becomes and is more prone to, and can improve decoding performance.Process and the concrete numerical value of m, n of decoding process have nothing to do, and also have nothing to do with generator matrix kind.Therefore burst quantity m can significantly promote, and improves space utilisation.The invention solves in prior art, in the coding/decoding method based on RS correcting and eleting codes, the decoding calculating quantity of existence is huge, and matrix inversion difficulty, decoding speed is slow, and the problem that system coding rate is low.Meanwhile, the time started of system decodes is no longer limited by m burst of acquisition, further increases decoding speed, is specially adapted to the P2P download environment based on public network and slow network; Or for field of data communication, can decode while reception data, substantially increase decoding efficiency.
Obviously, those skilled in the art should be understood that, above-mentioned of the present invention each module or each step can realize with general calculation element, they can concentrate on single calculation element, or be distributed on network that multiple calculation element forms, alternatively, they can realize with the executable program code of calculation element, thus they storages can be performed within hardware, and in some cases, step shown or described by can performing with the order be different from herein, or they are made into each integrated circuit modules respectively, or the multiple module in them or step are made into single integrated circuit module to realize.Like this, the present invention is not limited to any specific hardware and software combination.The generator matrix that the present invention and RS correcting and eleting codes use has nothing to do, and no matter vandermonde matrix, Cauchy matrix or other any matrix meeting RS correcting and eleting codes character are all applicable.
Be understandable that, the illustrative embodiments that above execution mode is only used to principle of the present invention is described and adopts, but the present invention is not limited thereto.For those skilled in the art, without departing from the spirit and substance in the present invention, can make various modification and improvement, these modification and improvement are also considered as protection scope of the present invention.
Claims (10)
1. a coding/decoding method, is characterized in that, comprises step:
Read the data to be decoded of input block, by the moving data to be decoded comprising former data fragmentation and checking data burst that reads to output module, and treat decoded data in the output module and process;
According to the quantity of data fragmentation in data field, output module Central Plains after process and region to be decoded, choose corresponding data fragmentation and carry out decode operation, decode operation result is combined into whole former data.
2. coding/decoding method according to claim 1, is characterized in that, in described method, described decoded data for the treatment of in the output module carries out processing specifically comprising:
If read checking data burst, then the data with existing of the corresponding position in the region to be decoded of described checking data burst and output module is carried out xor operation, and using data with existing new as this position for XOR result;
If read former data fragmentation, then described former data fragmentation is moved to the corresponding position of the former data field of output module, described former data fragmentation is multiplied by respectively whole elements of generator matrix corresponding row simultaneously, each result is carried out xor operation with the data with existing of the corresponding position in the region to be decoded of output module respectively, and using data with existing new as this position for XOR result.
3. coding/decoding method according to claim 1, is characterized in that, in described method, and the described quantity according to data fragmentation in data field, output module Central Plains and region to be decoded after process, choose corresponding data fragmentation and carry out decode operation and specifically comprise:
The total quantity m of former data fragmentation when if the quantity sum of data fragmentation is not less than coding in data field, output module Central Plains and region to be decoded, then whole m-k obtainable in former data field former data fragmentation and the k selected from region to be decoded are decoded according to burst.
4. coding/decoding method according to claim 3, is characterized in that, described in carry out decoding and specifically comprise:
According to the position of k the former data fragmentation lost and the position of k according to burst selecting region to be decoded, from generator matrix, choose corresponding element form selection matrix, obtain the inverse matrix of selection matrix as decoding matrix;
The k selected from region to be decoded is formed matrix to be decoded according to burst simultaneously;
Described matrix to be decoded is multiplied with described decoding matrix, obtains k former data fragmentation of described loss.
5. a cloud storage means, is characterized in that, described method comprises step:
Former data are encoded, is divided at least one former data fragmentation and at least one checking data burst, carries out respectively storing and/or communication;
When user will obtain described former data, perform the coding/decoding method according to any one of claim 1-4.
6. a decoding device, is characterized in that, comprising:
Data acquisition module, for reading the data to be decoded of input block, by the moving data to be decoded comprising former data fragmentation and checking data burst that reads to output module, and treats decoded data in the output module and processes;
Data decode module, for the quantity according to data fragmentation in data field, output module Central Plains after process and region to be decoded, chooses corresponding data fragmentation and carries out decode operation, decode operation result is combined into whole former data.
7. decoding device according to claim 6, is characterized in that, described data acquisition module specifically comprises:
Input block module, for receiving and depositing the data to be decoded comprising former data fragmentation and checking data burst;
Judge and read module, for reading the data to be decoded in the module of input block, if read checking data burst, then the data with existing of the corresponding position in the region to be decoded of described checking data burst and output module is carried out xor operation, and using data with existing new as this position for XOR result; If read former data fragmentation, then described former data fragmentation is moved to the corresponding position of the former data field of output module, described former data fragmentation is multiplied by respectively whole elements of generator matrix corresponding row simultaneously, each result is carried out xor operation with the data with existing of the corresponding position in the region to be decoded of output module respectively, and using data with existing new as this position for XOR result.
8. decoding device according to claim 6, is characterized in that, described data decode module also comprises:
Burst chooses module, during the total quantity m of former data fragmentation when being not less than coding for the quantity sum of data fragmentation in former data field in the output module and region to be decoded, whole m-k obtainable in former data field former data fragmentation and the k selected from region to be decoded are decoded according to burst.
9. decoding device according to claim 8, is characterized in that, described data decode module comprises further:
Matrix inversion module, for according to the position of k the former data fragmentation lost and the position of k according to burst selecting region to be decoded, chooses corresponding element and forms selection matrix, obtain the inverse matrix of selection matrix as decoding matrix from generator matrix;
Computing module, is multiplied with described decoding matrix for the matrix to be decoded k selected from region to be decoded formed according to burst, obtains k former data fragmentation of described loss.
10. a cloud storage system, is characterized in that, described system comprises:
Coding module, for former data being encoded, being divided at least one former data fragmentation and at least one checking data burst, carrying out respectively storing and/or communication; And the decoding device according to any one of claim 6-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410167214.8A CN105007286B (en) | 2014-04-23 | 2014-04-23 | Coding/decoding method and device and cloud storage method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410167214.8A CN105007286B (en) | 2014-04-23 | 2014-04-23 | Coding/decoding method and device and cloud storage method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105007286A true CN105007286A (en) | 2015-10-28 |
| CN105007286B CN105007286B (en) | 2018-12-28 |
Family
ID=54379807
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410167214.8A Active CN105007286B (en) | 2014-04-23 | 2014-04-23 | Coding/decoding method and device and cloud storage method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105007286B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502365A (en) * | 2019-07-11 | 2019-11-26 | 平安科技（深圳）有限公司 | The method, apparatus and computer equipment they data storage and restored |
| CN112052114A (en) * | 2020-08-27 | 2020-12-08 | 江苏超流信息技术有限公司 | Data storage and recovery method, coder-decoder and coder-decoder system |
| CN112286449A (en) * | 2020-10-16 | 2021-01-29 | 山东云海国创云计算装备产业创新中心有限公司 | RS erasure processing equipment and distributed storage system |
| CN115242774A (en) * | 2022-06-24 | 2022-10-25 | 浪潮通信技术有限公司 | Edge computing gateway configuration transmission method and device |
| CN116560915A (en) * | 2023-07-11 | 2023-08-08 | 北京谷数科技股份有限公司 | Data recovery method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101034894A (en) * | 2007-02-07 | 2007-09-12 | 华为技术有限公司 | Circuit and method for realizing the decoding |
| CN101488104A (en) * | 2009-02-26 | 2009-07-22 | 北京世纪互联宽带数据中心有限公司 | System and method for implementing high-efficiency security memory |
| CN102681791A (en) * | 2012-04-13 | 2012-09-19 | 华中科技大学 | Energy saving storage method of cluster storage system |
| CN102833040A (en) * | 2012-08-03 | 2012-12-19 | 中兴通讯股份有限公司 | Method and device for decoding, and coding and decoding system |
-
2014
- 2014-04-23 CN CN201410167214.8A patent/CN105007286B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101034894A (en) * | 2007-02-07 | 2007-09-12 | 华为技术有限公司 | Circuit and method for realizing the decoding |
| CN101488104A (en) * | 2009-02-26 | 2009-07-22 | 北京世纪互联宽带数据中心有限公司 | System and method for implementing high-efficiency security memory |
| CN102681791A (en) * | 2012-04-13 | 2012-09-19 | 华中科技大学 | Energy saving storage method of cluster storage system |
| CN102833040A (en) * | 2012-08-03 | 2012-12-19 | 中兴通讯股份有限公司 | Method and device for decoding, and coding and decoding system |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502365A (en) * | 2019-07-11 | 2019-11-26 | 平安科技（深圳）有限公司 | The method, apparatus and computer equipment they data storage and restored |
| CN110502365B (en) * | 2019-07-11 | 2024-03-01 | 平安科技（深圳）有限公司 | Data storage and recovery method and device and computer equipment |
| CN112052114A (en) * | 2020-08-27 | 2020-12-08 | 江苏超流信息技术有限公司 | Data storage and recovery method, coder-decoder and coder-decoder system |
| CN112052114B (en) * | 2020-08-27 | 2024-05-07 | 江苏超流信息技术有限公司 | Data storage and recovery method, coder and decoder and coding and decoding system |
| CN112286449A (en) * | 2020-10-16 | 2021-01-29 | 山东云海国创云计算装备产业创新中心有限公司 | RS erasure processing equipment and distributed storage system |
| CN112286449B (en) * | 2020-10-16 | 2023-04-25 | 山东云海国创云计算装备产业创新中心有限公司 | RS erasure processing equipment and distributed storage system |
| CN115242774A (en) * | 2022-06-24 | 2022-10-25 | 浪潮通信技术有限公司 | Edge computing gateway configuration transmission method and device |
| CN116560915A (en) * | 2023-07-11 | 2023-08-08 | 北京谷数科技股份有限公司 | Data recovery method and device, electronic equipment and storage medium |
| CN116560915B (en) * | 2023-07-11 | 2023-09-19 | 北京谷数科技股份有限公司 | Data recovery method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105007286B (en) | 2018-12-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110750382B (en) | Minimum storage regeneration code coding method and system for improving data repair performance | |
| CN105453466B (en) | Rate matching method and device for polarization code | |
| CN101582698B (en) | Protection of data from erasures using subsymbole based | |
| EP3014451B1 (en) | Locally generated simple erasure codes | |
| RU2501072C2 (en) | Distributed storage of recoverable data | |
| US9647698B2 (en) | Method for encoding MSR (minimum-storage regenerating) codes and repairing storage nodes | |
| US9141679B2 (en) | Cloud data storage using redundant encoding | |
| WO2017039795A1 (en) | Systems and methods for verification of code resiliency for data storage | |
| CN105007286A (en) | Decoding method, decoding device, and cloud storage method and system | |
| EP3504623A1 (en) | Multiple node repair using high rate minimum storage regeneration erasure code | |
| CN111149093A (en) | Data coding, decoding and repairing method of distributed storage system | |
| EP3193472A1 (en) | Data packet processing method and device | |
| CN101840377A (en) | Data storage method based on RS (Reed-Solomon) erasure codes | |
| CN113687975B (en) | Data processing method, device, equipment and storage medium | |
| WO2023151290A1 (en) | Data encoding method and apparatus, device, and medium | |
| CN105518996B (en) | A kind of data decoding method based on binary field reed-solomon code | |
| CN113055021A (en) | Decoding method, decoding device, network equipment and storage medium | |
| US11626890B2 (en) | Dynamically variable error correcting code (ECC) system with hybrid rateless reed-solomon ECCs | |
| Wu et al. | DSPR: Secure decentralized storage with proof-of-replication for edge devices | |
| US9450617B2 (en) | Distribution and replication of erasure codes | |
| CN103092716A (en) | Method and device for obtaining erasure code parameters | |
| CN116662063B (en) | Error correction configuration method, error correction method, system, equipment and medium for flash memory | |
| CN106302573B (en) | Method, system and device for processing data by adopting erasure code | |
| US20240232002A9 (en) | Method and apparatus for storing blockchain data based on error correction code | |
| CN103701842A (en) | Data transmitting method, data receiving method, server and client |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 210000, 1-5 story, Jinshan building, 8 Shanxi Road, Nanjing, Jiangsu. Applicant after: Suning.com group Limited by Share Ltd Address before: 210042 15, 1 Suning Road, Xuanwu District, Nanjing, Jiangsu. Applicant before: Yun Shang Group Plc of Suning |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |