CN105007286B - Coding/decoding method and device and cloud storage method and system - Google Patents
Coding/decoding method and device and cloud storage method and system Download PDFInfo
- Publication number
- CN105007286B CN105007286B CN201410167214.8A CN201410167214A CN105007286B CN 105007286 B CN105007286 B CN 105007286B CN 201410167214 A CN201410167214 A CN 201410167214A CN 105007286 B CN105007286 B CN 105007286B
- Authority
- CN
- China
- Prior art keywords
- data
- decoded
- decoding
- fragmentation
- former
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000006062 fragmentation reaction Methods 0.000 claims abstract description 161
- 238000013467 fragmentation Methods 0.000 claims abstract description 150
- 238000012795 verification Methods 0.000 claims abstract description 50
- 238000012545 processing Methods 0.000 claims abstract description 18
- 239000011159 matrix material Substances 0.000 claims description 104
- 239000012634 fragment Substances 0.000 claims description 76
- 238000004891 communication Methods 0.000 claims description 8
- 238000010586 diagram Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 102100035971 Molybdopterin molybdenumtransferase Human genes 0.000 description 1
- 101710119577 Molybdopterin molybdenumtransferase Proteins 0.000 description 1
- 235000014676 Phragmites communis Nutrition 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/004—Arrangements for detecting or preventing errors in the information received by using forward error control
- H04L1/0056—Systems characterized by the type of code used
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
Abstract
The present invention provides a kind of coding/decoding method and device and cloud storage method and system, belongs to cloud storage technical field.The coding/decoding method includes: the data to be decoded for reading input block, by the moving data to be decoded including former data fragmentation and verification data fragmentation read to output module, and treats decoding data in the output module and is handled；It according to the quantity of data fragmentation in output module Central Plains data field after processing and region to be decoded, chooses corresponding data fragmentation and is decoded operation, decoding operate result is combined into all former data.Decoding operate and the parallel progress of data acquisition work may be implemented in technical solution of the present invention, substantially increases decoding efficiency.The sequence that the present invention treats decoding data does not require, and the data to be decoded comprising former data and verification data can be handled in input block with random order.
Description
Technical field
The present invention relates to cloud storage technical fields, and in particular to a kind of coding/decoding method and device and cloud storage method and is
System.
Background technique
Cloud storage refers to through functions such as cluster application, network technology or distributed file systems, will be a large amount of each in network
The different types of storage equipment of kind gathers collaborative work by application software, and common data storage and the business of externally providing is visited
Ask a system of function.In cloud storage system, file, which is generally fragmented, to be stored in multiple cloud storage service devices.In addition,
In data communication, the data for needing to communicate can also be divided into multiple fragments, send other side to piecewise.
When data store and communicate, in order to solve the problems, such as reliability, correcting and eleting codes (Erasure Codes, letter can be used
Referred to as EC) technology, file is encoded, is divided into several initial data fragments and verification fragment, is stored or led to respectively
News.Reed-Solomon (Reed-Solomon, referred to as RS) correcting and eleting codes are a kind of common EC technologies, it compiles original document
After code, it is divided into the identical m initial data fragment of size and n verification fragment, is stored or communicated respectively.File is deposited
Storage or communication recipient can restore original or data by decoding process as long as obtaining wherein any m fragment,
Therefore it can resist n fragment damage or lose, substantially increase the reliability of system.Computer documents is stored, entangles and deletes
The storage space utilization of code system is m/ (m+n), significantly larger than copy storage mode, and therefore, RS correcting and eleting codes are by using meter
Calculation ability exchanges storage capacity for, significantly reduces carrying cost and O&M cost.
Nineteen sixty, Reed (I.S.Reed) and Saloman (G.Solomon) propose a kind of method for constructing correcting and eleting codes, make
Reed-Solomon code, abbreviation RS code are referred to as with the correcting and eleting codes of this method.Correcting and eleting codes based on RS coding techniques construction then claim
Make RS correcting and eleting codes.One (n, k) correcting and eleting codes is k source data to be encoded to n (n > k) a data, so that being appointed in this n data
K number of anticipating is according to restructural k original source data.Correcting and eleting codes system using m fragment and n verification fragment is exactly (m+
N, m) correcting and eleting codes.
The realization key problem in technology of RS correcting and eleting codes is to find a generator matrix G, and any k arranges equal linear independence in the matrix.
In RS coding techniques, generator matrix can be generalized circular matrix and Cauchy matrix.The coding generated using generalized circular matrix is model
Moral covers code (Vandermond Code), and the coding generated using Cauchy matrix is Cauchy's code (Cauchy Code).It encodes former
Reason is: initial data D [D1, D2 ..Dm] generates m initial data D [D1, D2 ..Dm] and n verification multiplied by generator matrix G
Data P [P1, P2 ..., Pn].Decoding principle is: arbitrarily obtaining m data in m+n initial data and verification data, forms
Data D' to be decoded, such as [D1 ... Di, P1..Pj].Remove n in generator matrix and do not occur that corresponding column of data,
A new generator matrix G is formed, the inverse matrix G', last D'*G'=D of G are then calculated.In order to protect the result of matrix operation
It holds in real number field, operation is changed to be limited to Galois (Galois) domain G (2^8) or G (2^16) progress.Concrete operation process and
Characteristic is studied thoroughly already, can find on related mathematics books and publication, therefore details are not described herein.
But no matter the system of the RS correcting and eleting codes system of generalized circular matrix and Cauchy matrix, have in decoding one altogether
With disadvantage, i.e., D can just be calculated by D'*G'=D, therefore after only collecting arbitrary m fragment, if wherein containing school
Fragment is tested, corresponding decoding generator matrix G' and data D' to be decoded just can determine that system could start to decode.It is received in system
Before m-th of fragment, even if knowing the number of the fragment, corresponding decoding generator matrix G' can be obtained, but due to data to be decoded
D'=[D1 ... Di, P1..Pj], has wherein at least lost a fragment data, therefore not can determine that D'.Only obtain m-th
After fragment, D' could be determined finally, and system could start to decode, therefore system decoding performance is limited by any m fragment most
That slow.In existing system, network transmission module and decoding computing module be it is serial, before collecting m-th of fragment, net
Network transmission module is working, and decodes the computing module free time；After starting decoding, decoding computing module works and network transmission module
Free time reduces system performance.
Another disadvantage is that decoding calculating quantity is O (n), i.e., directly proportional with verification fragment quantity n according to RS decoding algorithm.
Therefore, the original slice contained in data to be decoded is more, and verification fragment is fewer, and decoded performance is better.Consider based on public affairs
The P2P storage model of net configures m+n fragment, and client has received m-k original slice and k verification fragment, starts
It is decoded.Before decoding is done, and t original slice is received.Obviously, indicate that client has received all original point when t=k
Piece can stop to decode.But when t < k, it is desired nonetheless to it is decoded, according to m-k+t original slice and k-t verification fragment solution
Code, the configuration than former m-k original slice+k verification fragment are more preferable.But if terminating coding to restart, then pervious calculating
Amount and time all slattern.Since RS decoding is a kind of relatively time-consuming calculating, less verification fragment is used in decoding
It can be reduced calculation amount, improve performance well.
Therefore, in coding/decoding method of the prior art based on RS correcting and eleting codes, exist in m arbitrary node for be limited by acquisition most
The problem of big delay node, and once after decoding starts, and after obtaining new data fragmentation, system can not be using to reduce
The problem of performance.In addition, existing decoding process, has particular requirement to the sequence of input data, is generally divided into two parts, it is previous
Part is that original data set is closed, and rear portion is to verify the set of data, and coding/decoding method is not flexible.
Summary of the invention
The present invention provides a kind of coding/decoding method and device and cloud storage method and system, can solve and solves in the prior art
Code operation and data acquisition work serially carry out, thus the problem of reducing decoding efficiency.
To achieve the above object, on the one hand, a kind of coding/decoding method is provided in the embodiment of the present invention, comprising steps of
The data to be decoded for reading input block, by read include former data fragmentation and verification data fragmentation to
Decoding data is transferred to output module, and treats decoding data in the output module and handled；
According to the quantity of data fragmentation in output module Central Plains data field after processing and region to be decoded, corresponding number is chosen
It is decoded operation according to fragment, decoding operate result is combined into all former data.
Preferably, in the method, it is described treat in the output module decoding data carry out processing specifically include:
It is if reading verification data fragmentation, the verification data fragmentation is corresponding to the region to be decoded of output module
Data with existing at position carries out xor operation, and using exclusive or result as data with existing new at the position；
If reading former data fragmentation, the former data fragmentation is moved to the corresponding positions of the former data field of output module
Set place, while the former data fragmentation being corresponded to multiplied by generator matrix to whole elements of row respectively, by each result respectively with output
The data with existing of the corresponding position in the region to be decoded of module carries out xor operation, and using exclusive or result as new at the position
Data with existing.
Preferably, described according to data in output module Central Plains data field after processing and region to be decoded in the method
The quantity of fragment, chooses corresponding data fragmentation and is decoded operation and specifically include:
Original when if the sum of quantity of data fragmentation is not less than coding in output module Central Plains data field and region to be decoded
The total quantity m of data fragmentation is then selected by whole m-k former data fragmentations obtainable in former data field and from region to be decoded
K verification data fragmentation is decoded.
Preferably, described be decoded specifically includes:
According to the position of k of loss former data fragmentation and select the k number in region to be decoded according to the position of fragment, from
Corresponding element is chosen in generator matrix and constitutes selection matrix, finds out the inverse matrix of selection matrix as decoding matrix；
The k number selected from region to be decoded is formed into matrix to be decoded according to fragment simultaneously；
The matrix to be decoded is multiplied with the decoding matrix, obtains k former data fragmentation of the loss.
On the other hand, it is gone back in the embodiment of the present invention while a kind of cloud storage method being provided, comprising steps of
Former data are encoded, at least one former data fragmentation and at least one verification data fragmentation are divided into, respectively into
Row storage and/or communication；
When user will obtain the former data, coding/decoding method as described above is executed.
In another aspect, additionally providing a kind of decoding apparatus in the embodiment of the present invention, comprising:
Data acquisition module includes former data fragmentation by what is read for reading the data to be decoded of input block
Moving data to be decoded with verification data fragmentation is treated decoding data and is handled in the output module to output module；
Data decoder module, for according to data fragmentation in output module Central Plains data field after processing and region to be decoded
Quantity chooses corresponding data fragmentation and is decoded operation, and decoding operate result is combined into all former data.
Preferably, the data acquisition module specifically includes:
Input block module, for receiving and storing the number to be decoded including former data fragmentation and verification data fragmentation
According to；
Judgement and read module, for reading the data to be decoded in the module of input block, if reading verification data
The data with existing of the corresponding position in the region to be decoded of the verification data fragmentation and output module is then carried out exclusive or by fragment
Operation, and using exclusive or result as data with existing new at the position；If reading former data fragmentation, by the former data point
Piece is moved to the corresponding position of the former data field of output module, while by the former data fragmentation respectively multiplied by generator matrix pair
The whole elements that should be gone carry out each result different with the data with existing of the corresponding position in the region to be decoded of output module respectively
Or operation, and using exclusive or result as data with existing new at the position.
Preferably, the data decoder module further include:
Fragment chooses module, the sum of the quantity for data fragmentation in former data field in the output module and region to be decoded
When the total quantity m of former data fragmentation when not less than coding, by whole m-k former data fragmentations obtainable in former data field with
The k number selected from region to be decoded is decoded according to fragment.
Preferably, the data decoder module further comprises:
Matrix inversion module, for according to the position of k of loss former data fragmentation and selecting k of region to be decoded
The position of data fragmentation chooses corresponding element from generator matrix and constitutes selection matrix, and the inverse matrix for finding out selection matrix is made
For decoding matrix；
Computing module, the matrix to be decoded that k number for will select from region to be decoded is formed according to fragment with it is described
Decoding matrix is multiplied, and obtains k former data fragmentation of the loss.
On the other hand, a kind of cloud storage system is also provided in the embodiment of the present invention, comprising:
Coding module is divided at least one former data fragmentation and at least one check number for encoding former data
According to fragment, is stored and/or communicated respectively；And decoding apparatus as described above.
The present invention carries out multistage decoding handling one by one by way of data in input block, behind input block
After first data reaches, decoding effort can be started；It, can be complete at once after data all in input block are all disposed
It is operated at data recovery, makes full use of the computing capability of decoding client without leaving unused, thus can be while receiving data
It is decoded, substantially increases decoding efficiency.The sequence that the present invention treats decoding data does not require, and includes former data and verification
The data to be decoded of data, can be handled in input block with random order.
Detailed description of the invention
Fig. 1 is the flow diagram of the coding/decoding method in one embodiment of the present of invention；
Fig. 2 is the structural schematic diagram of the decoding apparatus in one embodiment of the present of invention；
Fig. 3 is the flow diagram of the coding/decoding method provided in a preferred embodiment of the present invention.
Specific embodiment
Technical solution in order to enable those skilled in the art to better understand the present invention, with reference to the accompanying drawing and specific embodiment party
Present invention is further described in detail for formula.
In the prior art, in the coding/decoding method based on RS correcting and eleting codes, data identical with former data bulk and group are being obtained
At data to be decoded, after determining corresponding decoding generator matrix, system just starts to decode, and the communication of data and decoding process are serial
Work, it is restricted to the input sequence of data, it generally requires and first inputs former data, then input validation data, coding/decoding method are ineffective
Living, decoding efficiency is low.
In coding/decoding method of the invention, decoded calculating operation starts from the first any fragment obtained, in acquisition
Achievable whole decoding effort after any m-th of fragment can obtain m fragment in any order, and no matter they are former
Data still verify data, and decoding operate and data acquisition work can carry out parallel, if before decoding is done, the number of input
According to being more than m fragment in set, system can preferentially select wherein all former data, and use as few as possible verifies data.
Fig. 1 is the flow diagram of the coding/decoding method in one embodiment of the present of invention, as shown in Figure 1, the coding/decoding method
Include:
Step S101, the data to be decoded for reading input block include former data fragmentation and check number by what is read
According to the moving data to be decoded of fragment to output module, and decoding data is treated in the output module and is handled.
Specifically, the coding metadata of file, including former data fragmentation number m are obtained first, verify data fragmentation number
N, to construct generator matrix Code [m] [n], monitor input block in data, wherein according to system storage characteristic and
The damaged condition of memory node may both have former data fragmentation in input block, it is also possible to have verification data fragmentation.If reading
To verification data fragmentation, then the corresponding position in the region to be decoded of the verification data fragmentation and output module had into number
It is operated according to exclusive or (XOR) is carried out, and using exclusive or result as data with existing new at the position；If reading former data fragmentation,
The former data fragmentation is then moved to the corresponding position of the former data field of output module, while the former data fragmentation being divided
The whole elements for not corresponding to row multiplied by generator matrix, by each result corresponding position with the region to be decoded of output module respectively
Data with existing carry out xor operation, and using exclusive or result as data with existing new at the position.
For example, if there is the multiple data in input block, it is (such as defeated that verification data fragmentation therein is read in preferential selection
Enter the school inspection data fragmentation Pj occurred at first in buffer area), and by the corresponding position of itself and the region to be decoded of output module
The data with existing of (for j-th of verification data fragmentation Pj, corresponding position is exactly j-th of position in region to be decoded) carries out different
Or operation is (because existing when the primary data 0 using xor operation, the position, which can be considered, carries out xor operation for the first time in the position
Data), exclusive or result becomes data with existing (Pj') new at the position；When not verifying data, former data point therein are selected
Piece (the former data fragmentation Di occurred at first in such as input block), is on the one hand moved into the former data field of output module
Corresponding position (for i-th of former data fragmentation Di, corresponding position is exactly i-th of position of former data field), on the other hand
Correspond to whole elements (the i.e. Code [i] [j], j of row (i.e. the i-th row) multiplied by generator matrix respectively with the original data fragmentation (i.e. Di)
From 1 to n), and by result respectively with the corresponding position in the region to be decoded of output module (j-th of position in region i.e. to be decoded,
J carries out xor operation and storage result from 1 to the data with existing from n)；The number in input block is continued with after being disposed
According to until data are sky in input block.
Step S102, according to the quantity of data fragmentation in output module Central Plains data field after processing and region to be decoded, choosing
It takes corresponding data fragmentation to be decoded operation, decoding operate result is combined into all former data.
Specifically, if the sum of quantity of data fragmentation is not less than coding in output module Central Plains data field and region to be decoded
When former data fragmentation total quantity m when, operation can be decoded.If there be m former data fragmentation in former data field, can not
Subsequent matrix multiple work is carried out again, and m former data fragmentation is directly assembled into former data file in order.If former data
The former data fragmentation (assuming that losing k former data fragmentation, k≤n) in less than m, area, then will last obtainable whole m-k former numbers
According to fragment and from the k number selected in region to be decoded according to fragment (when having in region to be decoded more than k number according to fragment, only with
Machine or sequential selection wherein k) it is decoded.
For example, decoding process are as follows: according to the position of k of loss former data fragmentation and select the k in region to be decoded
The position of a data fragmentation constitutes selection matrix G1 [k] [k] (vacation from corresponding element is chosen in generator matrix Code [m] [n]
It is fixed to lose former data fragmentation Di, while having selected j-th of data fragmentation Pj' in region to be decoded, then choose i-th in generator matrix
The element of row jth column, thus constitutes selection matrix), the inverse matrix of selection matrix is found out as decoding matrix Decode [k] [k]；
The k number selected from region to be decoded is formed into matrix P'[k to be decoded according to fragment simultaneously], by matrix to be decoded and decoding square
Battle array is multiplied and (forms situation according to data, can be multiplied with premultiplication or the right side), k former data fragmentation of loss can be obtained, by its point
Not Cha Ru output module former data field corresponding position, whole m former data fragmentations are then assembled into former number in order
According to file.
Preferably, there is any m data fragment if in input block at the beginning, both comprising former data fragmentation or wrap
The data fragmentation containing verification, and fragment quantity is not further added by, then the above process can do following optimization.Wherein, fragment in buffer area
Sequence can be random ordering, such as use the decoding cases of 4 former data fragmentations, 2 verification data fragmentations, input block data
It may be [D3, P1, D2, P2, D1, D4], it is also possible to [P2, D2, D1, P1, D4, D3] etc..The sequence and system of these data
Reception sequence is related.
Optimization method: scanning buffer data from the beginning to the end, if it is verification data fragmentation Pj, by itself and output module
The corresponding position data with existing exclusive or in region to be decoded.It can be obtained by the following formula the area to be decoded of output module at this time
Data Pj':P at j-th of position in domainj' ⊕=Pj, j ∈ 1...n, wherein the initial value of data Pj' is 0.
If it is former data fragmentation Di, on the one hand Di is moved into the corresponding position (i.e. i-th of the former data field of output module
A position), on the other hand with Di respectively multiplied by generator matrix correspond to row each element Code [i] [k] (k for verification data fragmentation
All numbers, in previous example, as respectively multiplied by Code [i] [1] and Code [i] [2]), and by result respectively with it is defeated
The corresponding position data with existing exclusive or in the region to be decoded of module out.It can be obtained by the following formula output module at this time
Data Pj':P at j-th of position in region to be decodedj'⊕=Di*Code[i][j],i∈1...m,j∈1...n。
In coding/decoding method provided in this embodiment, the decoded calculating operation of decoding system starts from first obtained arbitrarily
Fragment can be obtained in any order after any m-th of fragment of acquisition by the way that whole decoding efforts have been calculated into
M fragment, no matter they are former data or verification data, and decoding operate and data fragmentation, which obtain work, to carry out parallel.
If before decoding is done, in the data acquisition system of input being more than m fragment, system can preferentially select wherein all former data
Fragment, use as few as possible verifies fragment, to improve decoding efficiency.
The present invention also provides a kind of decoding processing devices, the device is for realizing above-described embodiment and preferred implementation side
Formula, the descriptions that have already been made will not be repeated.As used below, term " module " can be the software for realizing predetermined function
And/or the combination of hardware.Although device described in following embodiment is preferably realized with software, hardware or soft
The realization of the combination of part and hardware is also that may and be contemplated and implement.
Fig. 2 is the structural schematic diagram of the decoding apparatus in one embodiment of the present of invention, as shown in Fig. 2, the decoding apparatus
It include: data acquisition module 21 and data decoder module 22.Wherein, data acquisition module 21 be used for read input block to
Decoding data, by the moving data to be decoded including former data fragmentation and verification data fragmentation read to output module, and
Decoding data is treated in the output module to be handled；Data decoder module 22 is used for according to output module Central Plains data after processing
The quantity of data fragmentation in area and region to be decoded, chooses corresponding data fragmentation and is decoded operation, by decoding operate result
It is combined into all former data.
Data acquisition module 21 specifically includes: input block module 211, decoding are established module 212 and judgement and are read
Modulus block 213.
Input block module 211 had both included former data fragmentation, and had also included school for storing data fragmentation to be decoded
Test data fragmentation, minimum capacity m, maximum capacity m+n.Data fragmentation is stored in input block after entering, and works as data
After fragment is processed, input block is removed.Data fragmentation in input block can be handled according to input sequence, can also be pressed
It is handled according to certain rule, such as priority processing verifies data fragmentation, if can preferentially depend on judgement and read module.
Module 212 is established in decoding, for establishing the data needed when decoding, including obtaining member when former encoded data file
Data, including former data fragmentation quantity m, verification data fragmentation quantity n, file size, finally end up whether mend 0 etc coding
Information.By these metadata, the generator matrix Code [m] [n] when module creation original data encoding is established in decoding.Generator matrix
For generating each check code by the matrix in coding.
Whether overtime judgement and read module 213 judge decoding operate, and whether may be used for monitoring input block
To enter next step matrix inversion operation.It is preferential to read verification data fragmentation simultaneously if there is data fragmentation in input block
Processing；Otherwise former data fragmentation is handled.Preferential reading verification data fragmentation herein is not required, can also be according to advanced elder generation
Principle out reads the data fragmentation of input and processing one by one.The principle of processing is: if it is verification data fragmentation, then by its with
The data with existing of output module region corresponding position to be decoded carries out xor operation, and using exclusive or result as new at the position
Data with existing；If it is former data fragmentation, be then moved into the corresponding position of output module original data field, at the same by its
The whole elements for corresponding to row multiplied by generator matrix respectively, by each result respectively with output module region corresponding position to be decoded
Data with existing carries out xor operation, and using exclusive or result as data with existing new at the position.
All data fragmentations in judgement and 213 circular treatment input block of read module, including later in decoding
The data fragmentation being newly added, until input block is empty or time-out (if it times out, decoding failure).Work as input block
Data fragmentation is disposed and the sum of data fragmentation quantity of the former data field of output module and region to be decoded is more than or equal to m
Afterwards, i.e., processing is decoded into next step data decoder module 22.
Data decoder module 22 specifically includes: matrix inversion module 221, computing module 222 and output module 223.
Matrix inversion module 221, principle are as far as possible using more former data fragmentation and less data to be decoded point
Piece.After original data fragmentations whole in output module all uses (total m-k), then select the k in output module region to be decoded
A data fragmentation (composition decoding matrix P'[k]).According to the position and selection region to be decoded of k of loss former data fragmentation
K number according to the position of fragment, choose corresponding element from the generator matrix Code [m] [n] and constitute selection matrix G1 [k] [k],
Its inverse matrix is obtained as decoding matrix Decode [k] [k] by the classical matrix mode of inverting；Computing module 222, will be to be decoded
Matrix is multiplied with decoding matrix, obtains the former data fragmentation of those of loss.Calculated former data fragmentation is inserted into output module
The corresponding position of former data field；Output module 223, for assembling whole m of former data field former data fragmentations in order
For former data file and export.
Preferably, output module 223 further comprises former data field 2231 and region to be decoded 2232.Its Central Plains data field
2231 total m positions respectively correspond original m former data fragmentation.Former data fragmentation Di each of in input block, all
It can be judged and place is set to the corresponding i-th bit in the former data field of output module 2231 by mobile (or duplication) with read module 213.
Region 2232 to be decoded amounts to n position, respectively corresponds original n verification data fragmentation.Each school in input block
Data fragmentation Pj is tested, corresponding j-th of the position in region to be decoded 2232 with read module 213 and output module can be all judged
Place data with existing Pj' carries out xor operation and result is saved as new Pj'.For data fragmentation former each of in input block
Di can be also judged with read module 213 multiplied by Code [i] [j], and carried out xor operation with Pj' and saved as result newly
Pj'。
It solves in existing coding/decoding method and system based on RS correcting and eleting codes, exists in decoding apparatus provided in this embodiment
Communication and decoding process it is completely serial, decoding speed is slow, the problem that system is not flexible and encoding rate is low.Due to system solution
It is no longer limited by m-th of fragment of acquisition at the beginning of code, can further improve decoding speed, especially suitable for network environment
Poor system, such as P2P download environment and slow network based on public network.In field of data communication using the present invention, can do
It is decoded to when listening to data, substantially increases decoding efficiency.
Fig. 3 is the flow chart of the fast decoding method based on RS correcting and eleting codes of a preferred embodiment of the invention, such as Fig. 3 institute
Show, this method comprises the following steps:
The metadata information that module obtains file is established in step S302, decoding.
Specifically, metadata information includes fragment the quantity information m and n of coding, file size, finally ends up and whether mend 0
Etc encoded information.It also include the fragment of Chunk if former data file is stored according to Chunk (big piecemeal)
Information and length.By m and n, the generator matrix Code [m] [n] when module establishes original coding is established in decoding.Pass through when coding
The matrix generates n check code, meanwhile, inverse matrix Decode [n] [n] also is initialized, which at most stores the matrix letter of n*n
Breath.Decode array no data at present.If finally entering in buffer area has m-k former data fragmentation, it is only necessary to use k school
Data fragmentation is tested, then inverse matrix only needs Decode [k] [k] to store data.
Step S304, judgement and read module check whether input block is empty.
Specifically, if the data in input block are sky, S308 is thened follow the steps, if the number in input block
According to being not sky, S306 is thened follow the steps.
Step S306, if there is data in input block, judgement and read module check whether have in input block
Data fragmentation is verified, if so, thening follow the steps S310；If it is not, executing step S312.
As another implementation of the invention, this step can be with affected following amendment: sequence is slow from input first
One data fragmentation of taking-up in area is rushed, then judges whether it is verification data fragmentation again.If it is verification data fragmentation, execute
Step S310, it is no to then follow the steps S312.
Step S308, in the case where input block is empty, judgement and read module check whether output module has m
Data available fragment.
Specifically, data available fragment sum be output module former data field and area data fragment quantity to be decoded
Summation.If there is being more than or equal to m data fragment, S320 is thened follow the steps；If being no more than m, S316 is thened follow the steps.
Step S310, if input block has verification data fragmentation, judgement and read module by itself and output module
The data with existing of the corresponding position in region to be decoded carries out xor operation, and has using exclusive or result as new at the position
Data then go to step 304.
Step S312, if there is former data fragmentation in input block, Di is moved or copied into defeated by judgement and read module
The corresponding position of the former data field of module out.Go to step S314.
Step S314, judgement and read module by Di respectively multiplied by Code [i] [j], and by result and output module to
The data with existing of the corresponding position of decoding region carries out xor operation, and has number using exclusive or result as new at the position
According to i.e. execution Pj' ⊕=Di*Code [i] [j] operation then executes step S304.
Step S316, judgement and read module judge whether decoding operate is overtime.
Specifically, if be not timed-out, S304 is thened follow the steps；If it times out, executing step S318.
Step S318, decoding fail because of time-out, judgement and read module report decoding failed message.
At this point, decoding operate terminates.
Step S320, output module have had at least m data fragment (number including former data field and region to be decoded
According to fragment).
The former data fragmentation for choosing all former data fields with judgment module is read, amounts to m-k；Then in region to be decoded
Middle selection k number forms matrix P'[k to be decoded according to fragment].
Step S322, matrix inversion module, according to the position and selection region to be decoded of k of loss former data fragmentation
K number according to the position of fragment, choose corresponding element from the generator matrix Code [m] [n] and constitute selection matrix G1 [k] [k],
The inverse matrix of selection matrix is found out as decoding matrix Decode [k] [k].
Step S324, according to matrix P'[k to be decoded] and decoding matrix D ecode [k] [k], calculate k original of loss
Data fragmentation.By the corresponding position of the former data field of the former data fragmentation insertion output module of loss.Execute step S326.
All data fragmentations in former data field of output module are reverted to original file or complete by step S326
Long data block (Chunk).In former data field, the existing data fragmentation being put into originally also has the loss data fragmentation calculated.
At this point, terminating entire decoding process.
It should be noted that the quick decoding process shown in Fig. 3 based on RS correcting and eleting codes is simply to illustrate that the present invention is real
Apply the normal realization principle of fast coding process provided by example, when specific implementation may there is also abnormal conditions, therefore also
Need to provide abnormality processing, file is handled by block, handle by file metadata, to it is much more last mend zero or individually to tail block into
The treatment processes such as row assembly.
Technical solution of the present invention may be implemented to be decoded in downloading data fragmentation；It, can be sufficiently sharp when final decoding
With the former data fragmentation of all acquisitions, the quantity that fragment is verified when decoding is reduced.When decoding and the sequence of input data is unrelated, number
According to decoding system can be entered with random order, as long as the data fragmentation quantity entered in input block is more than or equal to m, finally
It can successfully decoded.And when decoding, the matrix for carrying out matrix inversion is needed to become k*k, k≤n, Matrix Calculating inversion from original m*m
It must be more easier, decoding performance can be improved.The process and m of decoding process, n specific value is unrelated, with generator matrix type also without
It closes.Therefore fragment quantity m can be promoted significantly, improve space utilisation.The present invention solves in the prior art, is entangled based on RS
It deletes in the coding/decoding method of code, existing decoding calculating quantity is huge, and matrix inversion is difficult, and decoding speed is slow and system coding rate
Low problem.Meanwhile m-th of fragment of acquisition is no longer limited by the beginning of system decoding, further improve decoding speed
Degree, especially suitable for P2P download environment and slow network based on public network；Or for field of data communication, number can received
According to while be decoded, substantially increase decoding efficiency.
Obviously, those skilled in the art should be understood that each module of the above invention or each step can be with general
Computing device realize that they can be concentrated on a single computing device, or be distributed in multiple computing devices and formed
Network on, optionally, they can be realized with the program code that computing device can perform, so as to be stored in
Executed in hardware, and in some cases, can with the steps shown or described are performed in an order that is different from the one herein, or
They are fabricated to each integrated circuit modules by person, or make multiple modules or steps in them to single integrated
Circuit module is realized.It is combined in this way, the present invention is not only restricted to any specific hardware and software.The present invention and RS correcting and eleting codes make
Generator matrix is unrelated, no matter vandermonde matrix, Cauchy matrix or other any matrixes for meeting RS correcting and eleting codes property
It is applicable in.
It is understood that the principle that embodiment of above is intended to be merely illustrative of the present and the exemplary implementation that uses
Mode, however the present invention is not limited thereto.For those skilled in the art, essence of the invention is not being departed from
In the case where mind and essence, various changes and modifications can be made therein, these variations and modifications are also considered as protection scope of the present invention.
Claims (6)
1. a kind of coding/decoding method, which is characterized in that comprising steps of
The data to be decoded for reading input block will read including former data fragmentation and verify the to be decoded of data fragmentation
Moving data is treated decoding data and is handled in the output module to output module；
According to the quantity of data fragmentation in output module Central Plains data field after processing and region to be decoded, corresponding data point are chosen
Piece is decoded operation, and decoding operate result is combined into all former data；
It is described treat in the output module decoding data carry out processing specifically include:
If reading verification data fragmentation, by the corresponding position in the region to be decoded of the verification data fragmentation and output module
The data with existing at place carries out xor operation, and using exclusive or result as data with existing new at the position；
If reading former data fragmentation, the former data fragmentation is moved to the corresponding position of the former data field of output module
Place, while the former data fragmentation correspond to multiplied by generator matrix to whole elements of row respectively, by each result respectively with export mould
The data with existing of the corresponding position in the region to be decoded of block carries out xor operation, and using exclusive or result as new at the position
Data with existing；
The quantity according to data fragmentation in output module Central Plains data field after processing and region to be decoded chooses corresponding number
Operation is decoded according to fragment to specifically include:
Former data when if the sum of quantity of data fragmentation is not less than coding in output module Central Plains data field and region to be decoded
The total quantity m of fragment is then selected whole m-k original data fragmentations obtainable in former data field and from region to be decoded
K number is decoded according to fragment.
2. coding/decoding method according to claim 1, which is characterized in that described be decoded specifically includes:
According to the position of k of loss former data fragmentation and select the k number in region to be decoded according to the position of fragment, from generation
Corresponding element is chosen in matrix and constitutes selection matrix, finds out the inverse matrix of selection matrix as decoding matrix；
The k number selected from region to be decoded is formed into matrix to be decoded according to fragment simultaneously；
The matrix to be decoded is multiplied with the decoding matrix, obtains k former data fragmentation of the loss.
3. a kind of cloud storage method, which is characterized in that the method includes the steps:
Former data are encoded, is divided at least one former data fragmentation and at least one verification data fragmentation, is deposited respectively
Storage and/or communication；
When user will obtain the former data, such as coding/decoding method of any of claims 1-2 is executed.
4. a kind of decoding apparatus characterized by comprising
Data acquisition module includes former data fragmentation by what is read for reading the data to be decoded of input block module
Moving data to be decoded with verification data fragmentation is treated decoding data and is handled in the output module to output module；
Data decoder module, for the number according to data fragmentation in output module Central Plains data field after processing and region to be decoded
Amount, chooses corresponding data fragmentation and is decoded operation, and decoding operate result is combined into all former data；
The data acquisition module specifically includes:
Input block module, for receiving and storing the data to be decoded including former data fragmentation and verification data fragmentation；
Judgement and read module, for reading the data to be decoded in the module of input block, if reading verification data fragmentation,
The data with existing of the corresponding position in the region to be decoded of the verification data fragmentation and output module is then subjected to xor operation,
And using exclusive or result as data with existing new at the position；If reading former data fragmentation, the former data fragmentation is moved
The corresponding position of the former data field of output module is moved, while the former data fragmentation is corresponded into row multiplied by generator matrix respectively
Whole elements, by each result respectively with the data with existing of the corresponding position in the region to be decoded of output module carry out exclusive or behaviour
Make, and using exclusive or result as data with existing new at the position；
The data decoder module further include:
Fragment chooses module, and the sum of quantity for data fragmentation in former data field in the output module and region to be decoded is not small
When the total quantity m of former data fragmentation when coding, by whole m-k original data fragmentations obtainable in former data field and to
The k number selected in decoding region is decoded according to fragment.
5. decoding apparatus according to claim 4, which is characterized in that the data decoder module further comprises:
Matrix inversion module, for according to the position of k of loss former data fragmentation and selecting the k number evidence in region to be decoded
The position of fragment chooses corresponding element from generator matrix and constitutes selection matrix, finds out the inverse matrix of selection matrix as solution
Code matrix；
Computing module, the matrix to be decoded and the decoding that k number for will select from region to be decoded is formed according to fragment
Matrix multiple obtains k former data fragmentation of the loss.
6. a kind of cloud storage system, which is characterized in that the system comprises:
Coding module is divided at least one former data fragmentation and at least one verification data point for encoding former data
Piece is stored respectively and/or is communicated；And the decoding apparatus as described in any one of claim 4-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410167214.8A CN105007286B (en) | 2014-04-23 | 2014-04-23 | Coding/decoding method and device and cloud storage method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410167214.8A CN105007286B (en) | 2014-04-23 | 2014-04-23 | Coding/decoding method and device and cloud storage method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105007286A CN105007286A (en) | 2015-10-28 |
| CN105007286B true CN105007286B (en) | 2018-12-28 |
Family
ID=54379807
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410167214.8A Active CN105007286B (en) | 2014-04-23 | 2014-04-23 | Coding/decoding method and device and cloud storage method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105007286B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502365B (en) * | 2019-07-11 | 2024-03-01 | 平安科技（深圳）有限公司 | Data storage and recovery method and device and computer equipment |
| CN112052114B (en) * | 2020-08-27 | 2024-05-07 | 江苏超流信息技术有限公司 | Data storage and recovery method, coder and decoder and coding and decoding system |
| CN112286449B (en) * | 2020-10-16 | 2023-04-25 | 山东云海国创云计算装备产业创新中心有限公司 | RS erasure processing equipment and distributed storage system |
| CN116560915B (en) * | 2023-07-11 | 2023-09-19 | 北京谷数科技股份有限公司 | Data recovery method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101034894A (en) * | 2007-02-07 | 2007-09-12 | 华为技术有限公司 | Circuit and method for realizing the decoding |
| CN101488104A (en) * | 2009-02-26 | 2009-07-22 | 北京世纪互联宽带数据中心有限公司 | System and method for implementing high-efficiency security memory |
| CN102681791A (en) * | 2012-04-13 | 2012-09-19 | 华中科技大学 | Energy saving storage method of cluster storage system |
| CN102833040A (en) * | 2012-08-03 | 2012-12-19 | 中兴通讯股份有限公司 | Method and device for decoding, and coding and decoding system |
-
2014
- 2014-04-23 CN CN201410167214.8A patent/CN105007286B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101034894A (en) * | 2007-02-07 | 2007-09-12 | 华为技术有限公司 | Circuit and method for realizing the decoding |
| CN101488104A (en) * | 2009-02-26 | 2009-07-22 | 北京世纪互联宽带数据中心有限公司 | System and method for implementing high-efficiency security memory |
| CN102681791A (en) * | 2012-04-13 | 2012-09-19 | 华中科技大学 | Energy saving storage method of cluster storage system |
| CN102833040A (en) * | 2012-08-03 | 2012-12-19 | 中兴通讯股份有限公司 | Method and device for decoding, and coding and decoding system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105007286A (en) | 2015-10-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11036392B2 (en) | Determining when to use convergent encryption | |
| US11182247B2 (en) | Encoding and storage node repairing method for minimum storage regenerating codes for distributed storage systems | |
| US10372357B2 (en) | Securely recovering stored data in a dispersed storage network | |
| CN104052576B (en) | Data recovery method based on error correcting codes in cloud storage | |
| CN102412847B (en) | Method and apparatus for decoding low density parity check code using united node processing | |
| US10951236B2 (en) | Hierarchical data integrity verification of erasure coded data in a distributed computing system | |
| CN102640422B (en) | Methods and apparatus employing FEC codes with permanent inactivation of symbols for encoding and decoding processes | |
| CN103444087B (en) | Coded and decoded using elastic code and flexible source block mapping | |
| US9647698B2 (en) | Method for encoding MSR (minimum-storage regenerating) codes and repairing storage nodes | |
| US8171102B2 (en) | Smart access to a dispersed data storage network | |
| JP7047133B2 (en) | Indexing and restoration of coded blockchain data | |
| WO2017039795A1 (en) | Systems and methods for verification of code resiliency for data storage | |
| CN105007286B (en) | Coding/decoding method and device and cloud storage method and system | |
| CN111149093A (en) | Data coding, decoding and repairing method of distributed storage system | |
| CN104364765A (en) | Method of data storing and maintenance in a distributed data storage system and corresponding device | |
| US20180089029A1 (en) | Method and apparatus for non-interactive information dispersal | |
| CN104461641B (en) | A kind of data programming method, system, burn writing equipment and target device | |
| US10015152B2 (en) | Securing data in a dispersed storage network | |
| CN113687975B (en) | Data processing method, device, equipment and storage medium | |
| CN110089035A (en) | Storage control, data processing chip and data processing method | |
| CN107003933A (en) | The method that construction method, device and its data of part replica code are repaired | |
| US20170132079A1 (en) | Rebuilding and verifying an encoded data slice utilizing slice verification information | |
| CN105610879A (en) | Data processing method and data processing device | |
| CN107786330B (en) | Method, apparatus, computer storage medium and electronic device for secret sharing | |
| CN113518113B (en) | File storage method based on block chain and Reed-Solomon code, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 210000, 1-5 story, Jinshan building, 8 Shanxi Road, Nanjing, Jiangsu. Applicant after: Suning.com group Limited by Share Ltd Address before: 210042 15, 1 Suning Road, Xuanwu District, Nanjing, Jiangsu. Applicant before: Yun Shang Group Plc of Suning |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |