CN103259856A - Method, device and system for data processing - Google Patents
Method, device and system for data processing Download PDFInfo
- Publication number
- CN103259856A CN103259856A CN201310152513XA CN201310152513A CN103259856A CN 103259856 A CN103259856 A CN 103259856A CN 201310152513X A CN201310152513X A CN 201310152513XA CN 201310152513 A CN201310152513 A CN 201310152513A CN 103259856 A CN103259856 A CN 103259856A
- Authority
- CN
- China
- Prior art keywords
- data
- check code
- data blocks
- stored
- remote storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Storage Device Security (AREA)
Abstract
The invention relates to a method, device and system for data processing. The method comprises the steps of dividing original data into data blocks in a first number in an equal proportion mode, determining check codes in a second number according to the data blocks in the first number, storing the data blocks in the first number and the check codes in the second number into at least one remote storage cluster or storing the data blocks in the first number and the check codes in the second number into at least two different remote storage clusters respectively or storing the check codes in the second number into at least one remote storage cluster, and therefore the original data can be taken from the remote storage clusters when needed.
Description
Technical field
The present invention relates to communication technique field, relate in particular to method, device and system that a kind of data are handled.
Background technology
Along with popularizing of cloud computing technology, the life that the cloud storage also more and more is close to the users.Therefore the user can, can greatly reduce the storage demand to terminal with the deposit data on the terminal on long-range cloud storage system.
Current, user's terminal will copy many parts with a initial data, leave on the different cloud storage systems.When the initial data on the terminal was lost or damaged, terminal can be obtained corresponding initial data from cloud storage system.Terminal is when obtaining initial data, if some cloud storage systems break down, but as long as also have a cloud storage system operate as normal, then terminal just can obtain corresponding initial data.Simultaneously, in order to protect the fail safe of initial data, terminal is stored on the different cloud storage systems after also initial data can being encrypted.
But the scheme that data are handled in the prior art also exposes defective: 1) waste of storage space of cloud storage system is serious.Because current terminal generally copies as initial data three parts, therefore, the demand of memory space reaches 3 times of initial data, and for cloud storage system, carrying cost is too high, and numerous and diverse initial data has also been brought a lot of extra administration overheads.
Summary of the invention
Method, device and system that the embodiment of the invention provides a kind of data to handle in order to improve operating factor of memory space, simultaneously, also save management cost.
In first aspect, the method that the embodiment of the invention provides a kind of data to handle, described method comprises:
The initial data equal proportion is divided into first number of data blocks;
Determine second quantity check code of the described first number of data blocks correspondence;
Described first number of data blocks and described second quantity check code together are stored in 1 remote storage cluster at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different remote storage cluster, perhaps, described second quantity check code is stored in 1 remote storage cluster at least, so that when the described initial data of needs, obtain from described remote storage cluster.
In first kind of possible implementation, second quantity check code of described definite described first number of data blocks correspondence specifically comprises:
Obtain second quantity verification slope value;
According to described verification slope value, form corresponding second quantity checking curve, each described checking curve comprises orthoscopic checking curve and non-rectilinear formula checking curve;
According to each described checking curve, obtain a plurality of data cells in described first number of data blocks that each described checking curve passes;
The a plurality of described data cell of obtaining is carried out XOR, obtain described check code.
In conjunction with first kind of first aspect or first aspect possible implementation, in second kind of possible implementation, the described first number of data blocks linear array is in linear buffering area;
Perhaps the described first number of data blocks annular array is in buffer circle.
In conjunction with second kind of first aspect or first aspect possible implementation, in the third possible implementation, when the described first number of data blocks annular array is in buffer circle, the length value of the described check code that obtains with divide after the length value of described data block identical;
When each described checking curve is non-rectilinear formula checking curve, the length value of the described check code that obtains with divide after the length value of described data block identical.
In conjunction with first aspect or first aspect the third possible implementation, in the 4th kind of possible implementation, described when each described checking curve is non-rectilinear formula checking curve, the length value of the described check code that obtains with divide after identical being specially of length value of described data block:
When each described checking curve is non-rectilinear formula checking curve, only there is 1 intersection point between each described checking curve, and a plurality of data cell quantity in described first number of data blocks that each described checking curve passes are identical, the length value of the described check code that then obtains with divide after the length value of described quantity piece identical.
In the 5th kind of possible implementation, described when the described first number of data blocks annular array is in buffer circle, with described first number of data blocks be written into the address and initial value is stored.
In the 6th kind of possible implementation, described remote storage cluster comprises: memory node or cloud storage;
Described described first number of data blocks and described second quantity check code together are stored in 1 remote storage cluster at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different remote storage cluster, perhaps, described second quantity check code be stored in 1 remote storage cluster at least specifically comprise:
When described remote storage cluster is memory node, described first number of data blocks and described second quantity check code together are stored in 1 memory node by network connection formula storage gateway at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different memory node by network connection formula storage gateway, perhaps, described second quantity check code is stored in 1 remote storage cluster at least;
When described remote storage cluster is the cloud storage, described first number of data blocks and described second quantity check code together are stored in 1 cloud storage at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the storage of different cloud, perhaps, described second quantity check code is stored in 1 remote storage cluster at least.
In second aspect, the device that the embodiment of the invention provides a kind of data to handle, described device comprises:
Division unit is used for the initial data equal proportion is divided into first number of data blocks;
Determining unit is for second quantity check code of determining the described first number of data blocks correspondence;
Memory cell, be used for described first number of data blocks and described second quantity check code together are stored in 1 remote storage cluster at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different remote storage cluster, perhaps, described second quantity check code is stored in 1 remote storage cluster at least, so that when the described initial data of needs, obtain from described remote storage cluster.
In first kind of possible implementation, described determining unit specifically is used for,
Obtain second quantity verification slope value;
According to described verification slope value, form corresponding second quantity checking curve, each described checking curve comprises orthoscopic checking curve and non-rectilinear formula checking curve;
According to each described checking curve, obtain a plurality of data cells in described first number of data blocks that each described checking curve passes;
The a plurality of described data cell of obtaining is carried out XOR, obtain described check code.
In conjunction with first kind of second aspect or second aspect possible implementation, in second kind of possible implementation, the described first number of data blocks linear array after described division unit is divided is in linear buffering area;
Perhaps the described first number of data blocks annular array is in buffer circle.
In conjunction with second kind of second aspect or second aspect possible implementation, in the third possible implementation, described determining unit also specifically is used for,
When the described first number of data blocks annular array was in buffer circle, the length value of the described check code that described determining unit obtains was identical with the length value of data block after described division unit is divided;
When each described checking curve of determining when described determining unit was non-rectilinear formula checking curve, the length value of the data block after the length value of the described check code that described determining unit obtains and the described division unit division was identical.
In conjunction with the third possible implementation of second aspect or second aspect, in the 4th kind of possible implementation, described determining unit also specifically is used for,
When each definite described checking curve of described determining unit is non-rectilinear formula checking curve, only there is 1 intersection point between each described checking curve, and a plurality of data cell quantity in described first number of data blocks that each described checking curve passes are identical, and then the length value of the described check code that obtains of described determining unit is identical with the length value of data block after described division unit is divided.
In the 5th kind of possible implementation, described memory cell also is used for,
When the described first number of data blocks annular array is in buffer circle, with described first number of data blocks be written into the address and initial value is stored.
In the 6th kind of possible implementation, described remote storage cluster comprises: memory node or cloud storage;
Described memory cell specifically is used for,
When described remote storage cluster is memory node, described first number of data blocks and described second quantity check code together are stored in 1 memory node by network connection formula storage gateway at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different memory node by network connection formula storage gateway, perhaps, described second quantity check code is stored in 1 remote storage cluster at least;
When described remote storage cluster is the cloud storage, described first number of data blocks and described second quantity check code together are stored in 1 cloud storage at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the storage of different cloud, perhaps, described second quantity check code is stored in 1 remote storage cluster at least.
In the third aspect, the system that the embodiment of the invention provides a kind of data to handle, described system comprises: the device that the data that provide as second aspect present invention are handled.
In first kind of possible implementation, described system also comprises: network connection formula storage gateway and at least 1 remote storage cluster, described remote storage cluster comprises: memory node.
In second kind of possible implementation, described system also comprises: gateway and at least 1 remote storage cluster, described remote storage cluster comprises: the cloud storage.
Therefore, the method of handling by the data of using the embodiment of the invention and providing, device and system, terminal is divided into first number of data blocks with the initial data equal proportion, determine second quantity check code of the first number of data blocks correspondence, and first number of data blocks and second quantity check code together be stored in 1 remote storage cluster at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different remote storage cluster, perhaps, described second quantity check code is stored in 1 remote storage cluster at least, so that when initial data is lost or damage, obtain from the remote storage cluster, and then avoided the problem that occurs in the prior art, improve operating factor of memory space, also reduced the administration overhead cost, simultaneously, terminal is encrypted initial data by the form of check code, makes data safer.
Description of drawings
The method flow diagram that Fig. 1 handles for data of the present invention;
The data block that Fig. 2 provides for the embodiment of the invention is arranged in linear buffer district sign indicating number schematic diagram;
The data block that Fig. 3 provides for the embodiment of the invention is arranged in buffer circle sign indicating number schematic diagram;
The non-rectilinear formula checking curve that Fig. 4 provides for the embodiment of the invention;
The structure drawing of device that Fig. 5 handles for the data that the embodiment of the invention provides;
The device hardware structure diagram that Fig. 6 handles for the data that the embodiment of the invention provides;
The system that Fig. 7 handles for a kind of data that the embodiment of the invention provides;
The system that Fig. 8 handles for the another kind of data that the embodiment of the invention provides.
Embodiment
For making the purpose, technical solutions and advantages of the present invention clearer, below in conjunction with accompanying drawing the specific embodiment of the invention is described in further detail.
Embodiment one
Be the method that example illustrates the data processing that the embodiment of the invention provides below with Fig. 1, the method flow diagram that Fig. 1 handles for the data that the embodiment of the invention provides, subject of implementation is terminal in embodiments of the present invention.In embodiments of the present invention, described terminal specifically refers to smart mobile phone, PC etc.As shown in Figure 1, this embodiment may further comprise the steps:
Particularly, the user wishes that the initial data of will store in the terminal is stored in the remote storage cluster, and then terminal is divided the initial data equal proportion in local hard drive, obtains first number of data blocks, in embodiments of the present invention, first quantity is specially N (N 〉=1).
Particularly, after terminal is divided into first number of data blocks with initial data, determine second quantity check code of the first number of data blocks correspondence, in embodiments of the present invention, second quantity is specially M (M 〉=1).
Further, terminal determines that second quantity check code of the first number of data blocks correspondence specifically comprises: terminal is obtained second quantity verification slope value, according to the verification slope value, terminal forms corresponding second quantity checking curve, and each described checking curve comprises orthoscopic checking curve and non-rectilinear formula checking curve; Terminal is according to each checking curve, obtains a plurality of data cells in first number of data blocks that each checking curve passes; A plurality of data cells that terminal will be obtained are carried out XOR, obtain check code.In embodiments of the present invention, the algorithm of above-mentioned definite check code is specially advanced storage engines algorithm (abbreviation: the ASE algorithm).
Further, in an example, as shown in Figure 2, terminal is divided into 6 data blocks with the initial data equal proportion, is respectively D1, D2, D3, D4, D5 and D6, and above-mentioned 6 data block linear array are in linear buffering area, terminal is obtained 5 verification slope value, for example, and k1=0; K1=1; K1=-1; K1=2; K1=-2; Need to prove that terminal can form the orthoscopic checking curve of 1 correspondence, perhaps non-rectilinear formula checking curve according to one or more verification slope value of obtaining.In the embodiment of the invention, be the example explanation with k1=0, terminal forms the corresponding check curve according to the verification slope value, and shown in checking curve 21, checking curve 21 is the orthoscopic checking curve; Terminal is according to checking curve 21, obtains 6 data unit in 6 data blocks that checking curve 21 passes; 6 data unit that terminal will be obtained carry out XOR, obtain check code P.
Further, the embodiment of the invention is encrypted initial data by the form of specified data piece corresponding check sign indicating number, makes that data more safely can be reliable.
Need to prove, the length value of the data block after the length value of the check code of determining in the embodiment of the invention and the division both can equate also can be unequal, the length value of the check code of determining in the above-mentioned example with divide after the length value of data block unequal, the situation that the length value of the data block after the length value of check code and the division equates describes in subsequent embodiment.
Be understandable that for the orthoscopic checking curve is that example describes, non-rectilinear formula checking curve illustrates above-mentioned example accordingly with the checking curve in subsequent embodiment.
Particularly, in one implementation, terminal together is stored in first number of data blocks and second quantity check code in 1 remote storage cluster behind second quantity check code of determining the first number of data blocks correspondence; Perhaps, terminal copies many parts with first number of data blocks and second quantity check code, and first number of data blocks after will copying and second quantity check code are stored in a plurality of remote storage clusters, for example, first number of data blocks and second quantity check code are copied 5 parts, be stored in a plurality of remote storage clusters.
In another kind of implementation, terminal is behind second quantity check code of determining the first number of data blocks correspondence, first number of data blocks is stored in 1 remote storage cluster, second quantity check code is stored in 1 remote storage cluster different with the first number of data blocks memory location; Perhaps, terminal is stored in first number of data blocks in 1 remote storage cluster, second quantity check code is copied many parts, and second quantity check code after will copying is stored in a plurality of remote storage clusters different with the first number of data blocks memory location, for example, first number of data blocks is stored in the first remote storage cluster, and second quantity check code is copied 5 parts, is stored in respectively in 5 remote storage clusters different with the first remote storage cluster; Perhaps, terminal is stored in second quantity check code in 1 remote storage cluster, first number of data blocks is copied many parts, and first number of data blocks after will copying is stored in a plurality of remote storage clusters different with the second number of data blocks memory location, for example, second quantity check code is stored in the first remote storage cluster, and first number of data blocks is copied 5 parts, is stored in respectively in 5 remote storage clusters different with the first remote storage cluster; Perhaps, terminal copies many parts respectively with first number of data blocks and second quantity check code, and first number of data blocks after will copying is with in second quantity check code is stored in a plurality of and different remote storage clusters respectively, for example, after first number of data blocks copies 5 parts, be stored in a plurality of remote storage clusters, second quantity check code is copied 6 parts after, be stored in a plurality of remote storage clusters different with the first number of data sign indicating number memory location.
In another kind of implementation, terminal only is stored in second quantity check code in 1 remote storage cluster; Perhaps, terminal copies many parts with second quantity check code, and second quantity check code after will copying is stored in a plurality of remote storage.Terminal can be stored in first number of data blocks in the local data base.
Further, in embodiments of the present invention, described remote storage cluster comprises: memory node or cloud storage.
Terminal together is stored in described first number of data blocks and described second quantity check code in 1 remote storage cluster at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different remote storage cluster, perhaps, described second quantity check code be stored in 1 remote storage cluster at least specifically comprise:
When described remote storage cluster is memory node, described first number of data blocks and described second quantity check code together are stored in 1 memory node by network connection formula storage gateway at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different memory node by network connection formula storage gateway, perhaps, described second quantity check code is stored in 1 remote storage cluster at least;
When described remote storage cluster is the cloud storage, described first number of data blocks and described second quantity check code together are stored in 1 cloud storage at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the storage of different cloud, perhaps, described second quantity check code is stored in 1 remote storage cluster at least.
In embodiments of the present invention, when terminal needed initial data, terminal was obtained from the remote storage cluster.
In one implementation, when terminal needs initial data, that is to say that initial data in the terminal is lost or when damaging, the number that terminal is lost or damaged according to initial data, terminal generates input/output request message, in described input/output request message, comprise lost terminal or damage initial data number information and lose or the positional information of the initial data damaged, terminal sends input/output request message to memory node or cloud storage, the storage of memory node or cloud is according to input/output request message, feeds back to terminal after packing with what be stored in self except the data block corresponding check sign indicating number of losing or damaging.When terminal receive be the check code of memory node or cloud store feedback the time, terminal is reduced check code according to the ASE algorithm again, the data of losing or damage are recovered, and perhaps, the storage of memory node or cloud will be lost or bad data block directly sends to terminal.
In another kind of implementation, when terminal needs initial data, that is to say when not storing initial data in the terminal, terminal generates input/output request message, terminal sends input/output request message to memory node or cloud storage, memory node or cloud are stored according to input/output request message, feed back to terminal after the check code that is stored in self is packed.When terminal receive be the check code of memory node or cloud store feedback the time, terminal is reduced check code according to the ASE algorithm again, and then obtains data block, perhaps, memory node or cloud storage directly send data block to terminal.
Need to prove that it is individual that the number that the initial data in the terminal and check code are lost can not surpass second quantity, otherwise terminal can't utilize the check code of feedback to restore initial data according to the ASE algorithm.
Therefore, the method of handling by the data of using the embodiment of the invention and providing, terminal is divided into first number of data blocks with the initial data equal proportion, determine second quantity check code of the first number of data blocks correspondence, and first number of data blocks and second quantity check code together be stored in 1 remote storage cluster at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different remote storage cluster, perhaps, described second quantity check code is stored in 1 remote storage cluster at least, so that when initial data is lost or damage, obtain from the remote storage cluster, and then avoided the problem that occurs in the prior art, improve operating factor of memory space, also reduced the administration overhead cost, simultaneously, terminal is encrypted initial data by the form of check code, makes data safer.
Alternatively, but the involved first number of data blocks linear array (as shown in Figure 2) in linear buffering area in the embodiment of the invention, the perhaps first number of data blocks annular array (as shown in Figure 3) in buffer circle.In previous embodiment one be that example is when illustrating that the first number of data blocks linear array is in linear buffering area with Fig. 2, terminal is determined the detailed process of check code, in the following embodiments, be that example describes in waking buffering area up with the first number of data blocks annular array.The different arrangement modes of the first quantity number gauge block make that the length value of the quantity piece after length value and the division of the check code that terminal is determined both can be identical, also can be inequality, make things convenient for the deployment of terminal.
Embodiment two
As shown in Figure 3, the data block annular array schematic diagram that provides for the embodiment of the invention of Fig. 3.In Fig. 3, the first number of data blocks annular array is (first number of data blocks connects from beginning to end) in buffer circle, description according to the step 110 in the previous embodiment one, step 120, after terminal is divided into N data block to initial data, according to the ASE algorithm, determine M check code of N data block correspondence.Because first number of data blocks is present in the buffer circle, terminal can be determined a plurality of check codes always circularly, and the length value of any one data block in the length value of the check code that terminal is determined and first number of data blocks is identical; Moreover, because first number of data blocks connects from beginning to end, therefore, when determining check code, terminal also with first number of data blocks be written into the address and initial value is stored, so that when initial data is lost, utilization is written into address and initial value, determine the data block of calculation check sign indicating number at that time, and then obtain corresponding data block or check code from the remote storage cluster, when terminal receive be the check code of memory node or cloud store feedback the time, terminal is reduced check code according to the ASE algorithm again, and then obtains data block.
Be understandable that it also is a kind of common cipher mode that terminal will be written into that address and initial value store.
Embodiment three
In previous embodiment one step step 120, for the orthoscopic checking curve is that example describes, in embodiments of the present invention, be that non-rectilinear formula checking curve is that example describes with the checking curve with the checking curve.
Particularly, as shown in Figure 4, the non-rectilinear formula checking curve that Fig. 4 provides for the embodiment of the invention.In Fig. 4, A-L is first number of data blocks after terminal is divided, and P, Q and R are the check code that terminal is determined.In Fig. 4, be that example describes with checking curve 41.Terminal is obtained second quantity verification slope value, for example, and k1=0; K1=-1, terminal is according to the verification slope value, forms the corresponding check curve, and checking curve 41 is that two straight lines of 0 and-1 constitute the broken line of formation by the verification slope value.Terminal is according to checking curve 41, obtains a plurality of data cells (in Fig. 4, data cell is represented with the circle form) in A-L the data block that checking curve 41 passes.In embodiments of the present invention, checking curve 41 has passed 12 data unit in A-L the data block respectively, and then terminal 12 data unit that will obtain carry out XOR, obtain check code R.
In embodiments of the present invention, because checking curve is non-rectilinear formula checking curve, the length value of the data block after the length value of the check code that then obtains and the division is identical.
Further, when each checking curve is non-rectilinear formula checking curve, only there is 1 intersection point between each checking curve, as the checking curve 41 and checking curve 42 among Fig. 4, checking curve 41 and checking curve 43, all only there is 1 intersection point between checking curve 42 and the checking curve 43, and a plurality of data cell quantity in first number of data blocks that each checking curve passes are identical, as the checking curve 41 among Fig. 4, checking curve 42 and checking curve 43 all pass 12 data unit, the length value of each check code that then obtains with divide after the length value of described quantity piece identical.
Correspondingly, the device that the embodiment of the invention also provides a kind of data to handle, in order to the method that the data that realize in the previous embodiment are handled, as shown in Figure 5, described device comprises: division unit 510, determining unit 520 and memory cell 530.
Determining unit 520 is for second quantity check code of determining the described first number of data blocks correspondence;
Described determining unit 520 specifically is used for, and obtains second quantity verification slope value;
According to described verification slope value, form corresponding second quantity checking curve, each described checking curve comprises orthoscopic checking curve and non-rectilinear formula checking curve;
According to each described checking curve, obtain a plurality of data cells in described first number of data blocks that each described checking curve passes;
The a plurality of described data cell of obtaining is carried out XOR, obtain described check code.
The described first number of data blocks linear array after described division unit 510 is divided is in linear buffering area; Perhaps the described first number of data blocks annular array is in buffer circle.
Described determining unit 520 also specifically is used for, and when the described first number of data blocks annular array was in buffer circle, the length value of the described check code that described determining unit obtains was identical with the length value of data block after described division unit is divided;
When each described checking curve of determining when described determining unit was non-rectilinear formula checking curve, the length value of the data block after the length value of the described check code that described determining unit obtains and the described division unit division was identical.
Described determining unit 520 also specifically is used for, when each definite described checking curve of described determining unit is non-rectilinear formula checking curve, only there is 1 intersection point between each described checking curve, and a plurality of data cell quantity in described first number of data blocks that each described checking curve passes are identical, and then the length value of the described check code that obtains of described determining unit is identical with the length value of data block after described division unit is divided.
Described memory cell 530 also is used for, when the described first number of data blocks annular array is in buffer circle, with described first number of data blocks be written into the address and initial value is stored.
Described remote storage cluster comprises: memory node or cloud storage;
Described memory cell 530 specifically is used for, when described remote storage cluster is memory node, described first number of data blocks and described second quantity check code together are stored in 1 memory node by network connection formula storage gateway at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different memory node by network connection formula storage gateway, perhaps, described second quantity check code is stored in 1 remote storage cluster at least;
When described remote storage cluster is the cloud storage, described first number of data blocks and described second quantity check code together are stored in 1 cloud storage at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the storage of different cloud, perhaps, described second quantity check code is stored in 1 remote storage cluster at least.
Therefore, the device of handling by the data of using the embodiment of the invention and providing, terminal is divided into first number of data blocks with the initial data equal proportion, determine second quantity check code of the first number of data blocks correspondence, and first number of data blocks and second quantity check code together be stored in 1 remote storage cluster at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different remote storage cluster, perhaps, described second quantity check code is stored in 1 remote storage cluster at least, so that when initial data is lost or damage, obtain from the remote storage cluster, and then avoided the problem that occurs in the prior art, improve operating factor of memory space, also reduced the administration overhead cost, simultaneously, terminal is encrypted initial data by the form of check code, makes data safer.
In addition, the device that the data that the embodiment of the invention provides are handled also can be realized by following form, in order to the method that the data that realize in the previous embodiment of the present invention are handled, as shown in Figure 6, described device comprises: network interface 610, processor 620 and memory 630.
When starting, these component softwares are loaded in the memory 630, are visited and carry out as giving an order by processor 620 then:
The initial data equal proportion is divided into first number of data blocks;
Determine second quantity check code of the described first number of data blocks correspondence;
Described first number of data blocks and described second quantity check code together are stored in 1 remote storage cluster at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different remote storage cluster, perhaps, described second quantity check code is stored in 1 remote storage cluster at least, so that when the described initial data of needs, obtain from described remote storage cluster.
Further, behind the component software of described processor access memory 630, carry out the concrete instruction of second quantity check code process of determining the described first number of data blocks correspondence:
Obtain second quantity verification slope value;
According to described verification slope value, form corresponding second quantity checking curve, each described checking curve comprises orthoscopic checking curve and non-rectilinear formula checking curve;
According to each described checking curve, obtain a plurality of data cells in described first number of data blocks that each described checking curve passes;
The a plurality of described data cell of obtaining is carried out XOR, obtain described check code.
Further, behind the component software of described processor access memory 630, carry out the instruction of process:
The described first number of data blocks linear array is in linear buffering area;
Perhaps the described first number of data blocks annular array is in buffer circle.
Further, behind the component software of described processor access memory 630, carry out the instruction of process:
When the described first number of data blocks annular array is in buffer circle, the length value of the described check code that obtains with divide after the length value of described data block identical;
When each described checking curve is non-rectilinear formula checking curve, the length value of the described check code that obtains with divide after the length value of described data block identical.
Further, behind the component software of described processor access memory 630, carry out when each described checking curve is non-rectilinear formula checking curve the concrete instruction of the length value identical process of the described data block after the length value of the described check code that obtains and the division:
When each described checking curve is non-rectilinear formula checking curve, only there is 1 intersection point between each described checking curve, and a plurality of data cell quantity in described first number of data blocks that each described checking curve passes are identical, the length value of the described check code that then obtains with divide after the length value of described quantity piece identical.
Further, behind the component software of described processor access memory 630, carry out the instruction of following process:
When the described first number of data blocks annular array is in buffer circle, with described first number of data blocks be written into the address and initial value is stored.
Further, described remote storage cluster comprises: memory node or cloud storage;
Behind the component software of described processor access memory 630, carry out and described described first number of data blocks and described second quantity check code together are stored in 1 remote storage cluster at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different remote storage cluster, perhaps, described second quantity check code is stored in the concrete instruction of at least 1 remote storage cluster internal procedure:
When described remote storage cluster is memory node, described first number of data blocks and described second quantity check code together are stored in 1 memory node by network connection formula storage gateway at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different memory node by network connection formula storage gateway, perhaps, described second quantity check code is stored in 1 remote storage cluster at least;
When described remote storage cluster is the cloud storage, described first number of data blocks and described second quantity check code together are stored in 1 cloud storage at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the storage of different cloud, perhaps, described second quantity check code is stored in 1 remote storage cluster at least.
Therefore, the device of handling by the data of using the embodiment of the invention and providing, terminal is divided into first number of data blocks with the initial data equal proportion, determine second quantity check code of the first number of data blocks correspondence, and first number of data blocks and second quantity check code together be stored in 1 remote storage cluster at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different remote storage cluster, perhaps, described second quantity check code is stored in 1 remote storage cluster at least, so that when initial data is lost or damage, obtain from the remote storage cluster, and then avoided the problem that occurs in the prior art, improve operating factor of memory space, also reduced the administration overhead cost, simultaneously, terminal is encrypted initial data by the form of check code, makes data safer.
For making the purpose, technical solutions and advantages of the present invention clearer, below in conjunction with accompanying drawing the specific embodiment of the invention is described in further detail.
Be the system that example illustrates the data processing that the embodiment of the invention provides with Fig. 7 and Fig. 8 below, in order to realize the method for the data processing in the previous embodiment of the present invention.
In one implementation, as shown in Figure 7, the system that these data are handled is specially the N+m storage cluster system.Described system comprises: terminal, network connection formula storage (Network-attached storage) gateway and at least 1 memory node.
Terminal is divided into first number of data blocks with the initial data equal proportion, after determining second quantity check code of the first number of data blocks correspondence, by data communications equipment and NAS gateway first number of data blocks and second quantity check code together are stored in 1 memory node at least, perhaps, first number of data blocks and second quantity check code are stored in respectively at least 2 and the different memory node, each memory node is after finishing the data storage of terminal transmission, reply acknowledge message by data communications equipment to terminal, inform the particular location of the data storage of terminal transmission.
In the embodiment of the invention, memory node is specially virtual network interconnection system storage system or virtual network interconnection system storage box.Each memory node all has virtual memory, is used for first number of data blocks and second quantity check code of storage terminal transmission.Each memory node also is connected with network hard disc, is used for when inner virtual memory expires data being stored in the network hard disc.
When terminal needs initial data, terminal generates input, input comprises the positional information of memory node, terminal is sent to NAS gateway place with input by data communications equipment, the positional information of the memory node that the NAS gateway comprises according to input, directly the memory node of input to correspondence sent, memory node is according to input, first number of data blocks of self will be stored in, or second after a quantity check code packs, and is back to terminal by data communications equipment again.
In another kind of implementation, as shown in Figure 8, the system that these data are handled is specially cloud storage system.Described system comprises: terminal, gateway and at least 1 cloud storage.
Terminal is divided into first number of data blocks with the initial data equal proportion, after determining second quantity check code of the first number of data blocks correspondence, by backbone network (Internet Backbone) and gateway first number of data blocks and second quantity check code together are stored in 1 cloud storage at least, perhaps, first number of data blocks and second quantity check code are stored in respectively at least 2 and the storage of different cloud, after each cloud is stored in the storage of the data of terminal transmission is finished, reply acknowledge message to terminal, informed the particular location of the data storage of terminal transmission.
When terminal needs initial data, terminal generates input, input comprises the positional information of cloud storage, terminal is sent to the backbone network place with input, backbone network is according to the positional information of the memory node that comprises according to input, directly the gateway of input to correspondence sent, gateway is forwarded to connected cloud storage with input, the cloud storage is according to input, first number of data blocks of self will be stored in, or second after a quantity check code packs, and is back to terminal by gateway and backbone network again.
Be understandable that the course of work of the above-mentioned system that only handles with the formal specification data of giving an example in actual applications, is not restricted to this.
The professional should further recognize, unit and the algorithm steps of each example of describing in conjunction with embodiment disclosed herein, can realize with electronic hardware, computer software or the combination of the two, for the interchangeability of hardware and software clearly is described, composition and the step of each example described in general manner according to function in the above description.These functions still are that software mode is carried out with hardware actually, depend on application-specific and the design constraint of technical scheme.The professional and technical personnel can specifically should be used for using distinct methods to realize described function to each, but this realization should not thought and exceeds scope of the present invention.
The method of describing in conjunction with embodiment disclosed herein or the step of algorithm can use the software module of hardware, processor execution, and perhaps the combination of the two is implemented.Software module can place the storage medium of any other form known in random asccess memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or the technical field.
Above-described embodiment; purpose of the present invention, technical scheme and beneficial effect are further described; institute is understood that; the above only is the specific embodiment of the present invention; and be not intended to limit the scope of the invention; within the spirit and principles in the present invention all, any modification of making, be equal to replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (17)
1. the method handled of data is characterized in that described method comprises:
The initial data equal proportion is divided into first number of data blocks;
Determine second quantity check code of the described first number of data blocks correspondence;
Described first number of data blocks and described second quantity check code together are stored in 1 remote storage cluster at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different remote storage cluster, perhaps, described second quantity check code is stored in 1 remote storage cluster at least, so that when the described initial data of needs, obtain from described remote storage cluster.
2. the method for data processing according to claim 1 is characterized in that, second quantity check code of described definite described first number of data blocks correspondence specifically comprises:
Obtain second quantity verification slope value;
According to described verification slope value, form corresponding second quantity checking curve, each described checking curve comprises orthoscopic checking curve and non-rectilinear formula checking curve;
According to each described checking curve, obtain a plurality of data cells in described first number of data blocks that each described checking curve passes;
The a plurality of described data cell of obtaining is carried out XOR, obtain described check code.
3. the method for data processing according to claim 2 is characterized in that the described first number of data blocks linear array is in linear buffering area;
Perhaps the described first number of data blocks annular array is in buffer circle.
4. the method handled of data according to claim 3 is characterized in that, when the described first number of data blocks annular array was in buffer circle, the length value of the described data block after the length value of the described check code that obtains and the division was identical;
When each described checking curve is non-rectilinear formula checking curve, the length value of the described check code that obtains with divide after the length value of described data block identical.
5. the method handled of data according to claim 4 is characterized in that, and is described when each described checking curve is non-rectilinear formula checking curve, identical being specially of length value of the described data block after the length value of the described check code that obtains and the division:
When each described checking curve is non-rectilinear formula checking curve, only there is 1 intersection point between each described checking curve, and a plurality of data cell quantity in described first number of data blocks that each described checking curve passes are identical, the length value of the described check code that then obtains with divide after the length value of described quantity piece identical.
6. the method handled of data according to claim 1 is characterized in that, and is described when the described first number of data blocks annular array is in buffer circle, with described first number of data blocks be written into the address and initial value is stored.
7. the method for data processing according to claim 1 is characterized in that described remote storage cluster comprises: memory node or cloud storage;
Described described first number of data blocks and described second quantity check code together are stored in 1 remote storage cluster at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different remote storage cluster, perhaps, described second quantity check code be stored in 1 remote storage cluster at least specifically comprise:
When described remote storage cluster is memory node, described first number of data blocks and described second quantity check code together are stored in 1 memory node by network connection formula storage gateway at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different memory node by network connection formula storage gateway, perhaps, described second quantity check code is stored in 1 remote storage cluster at least;
When described remote storage cluster is the cloud storage, described first number of data blocks and described second quantity check code together are stored in 1 cloud storage at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the storage of different cloud, perhaps, described second quantity check code is stored in 1 remote storage cluster at least.
8. the device handled of data is characterized in that described device comprises:
Division unit is used for the initial data equal proportion is divided into first number of data blocks;
Determining unit is for second quantity check code of determining the described first number of data blocks correspondence;
Memory cell, be used for described first number of data blocks and described second quantity check code together are stored in 1 remote storage cluster at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different remote storage cluster, perhaps, described second quantity check code is stored in 1 remote storage cluster at least, so that when the described initial data of needs, obtain from described remote storage cluster.
9. the device of data processing according to claim 8 is characterized in that described determining unit specifically is used for,
Obtain second quantity verification slope value;
According to described verification slope value, form corresponding second quantity checking curve, each described checking curve comprises orthoscopic checking curve and non-rectilinear formula checking curve;
According to each described checking curve, obtain a plurality of data cells in described first number of data blocks that each described checking curve passes;
The a plurality of described data cell of obtaining is carried out XOR, obtain described check code.
10. the device of data processing according to claim 9 is characterized in that, the described first number of data blocks linear array after described division unit is divided is in linear buffering area;
Perhaps the described first number of data blocks annular array is in buffer circle.
11. the device that data according to claim 10 are handled is characterized in that described determining unit also specifically is used for,
When the described first number of data blocks annular array was in buffer circle, the length value of the described check code that described determining unit obtains was identical with the length value of data block after described division unit is divided;
When each described checking curve of determining when described determining unit was non-rectilinear formula checking curve, the length value of the data block after the length value of the described check code that described determining unit obtains and the described division unit division was identical.
12. the device that data according to claim 11 are handled is characterized in that described determining unit also specifically is used for,
When each definite described checking curve of described determining unit is non-rectilinear formula checking curve, only there is 1 intersection point between each described checking curve, and a plurality of data cell quantity in described first number of data blocks that each described checking curve passes are identical, and then the length value of the described check code that obtains of described determining unit is identical with the length value of data block after described division unit is divided.
13. the device that data according to claim 8 are handled is characterized in that described memory cell also is used for,
When the described first number of data blocks annular array is in buffer circle, with described first number of data blocks be written into the address and initial value is stored.
14. the device that data according to claim 8 are handled is characterized in that described remote storage cluster comprises: memory node or cloud storage;
Described memory cell specifically is used for,
When described remote storage cluster is memory node, described first number of data blocks and described second quantity check code together are stored in 1 memory node by network connection formula storage gateway at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the different memory node by network connection formula storage gateway, perhaps, described second quantity check code is stored in 1 remote storage cluster at least;
When described remote storage cluster is the cloud storage, described first number of data blocks and described second quantity check code together are stored in 1 cloud storage at least, perhaps, described first number of data blocks and described second quantity check code are stored in respectively at least 2 and the storage of different cloud, perhaps, described second quantity check code is stored in 1 remote storage cluster at least.
15. the system that data are handled is characterized in that described system comprises: as the device of claim 8 to each described data processing of claim 14.
16. the system that data according to claim 15 are handled, it is characterized in that described system also comprises: network connection formula storage gateway and at least 1 remote storage cluster, described remote storage cluster comprises: memory node.
17. the system that data according to claim 15 are handled is characterized in that described system also comprises: gateway and at least 1 remote storage cluster, described remote storage cluster comprises: the cloud storage.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310152513XA CN103259856A (en) | 2013-04-27 | 2013-04-27 | Method, device and system for data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310152513XA CN103259856A (en) | 2013-04-27 | 2013-04-27 | Method, device and system for data processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103259856A true CN103259856A (en) | 2013-08-21 |
Family
ID=48963546
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310152513XA Pending CN103259856A (en) | 2013-04-27 | 2013-04-27 | Method, device and system for data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103259856A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070186066A1 (en) * | 2006-02-03 | 2007-08-09 | Emc Corporation | Fast verification of computer backup data |
| CN101251812A (en) * | 2008-02-28 | 2008-08-27 | 浪潮电子信息产业股份有限公司 | Method for cluster system data fault tolerance |
| CN101719086A (en) * | 2009-11-30 | 2010-06-02 | 成都市华为赛门铁克科技有限公司 | Fault-tolerant processing method and device of disk array and fault-tolerant system |
| CN102752402A (en) * | 2012-07-20 | 2012-10-24 | 广东威创视讯科技股份有限公司 | Cloud storage method and cloud storage system |
| CN103064902A (en) * | 2012-12-18 | 2013-04-24 | 厦门市美亚柏科信息股份有限公司 | Method and device for storing and reading data in hadoop distributed file system (HDFS) |
-
2013
- 2013-04-27 CN CN201310152513XA patent/CN103259856A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070186066A1 (en) * | 2006-02-03 | 2007-08-09 | Emc Corporation | Fast verification of computer backup data |
| CN101251812A (en) * | 2008-02-28 | 2008-08-27 | 浪潮电子信息产业股份有限公司 | Method for cluster system data fault tolerance |
| CN101719086A (en) * | 2009-11-30 | 2010-06-02 | 成都市华为赛门铁克科技有限公司 | Fault-tolerant processing method and device of disk array and fault-tolerant system |
| CN102752402A (en) * | 2012-07-20 | 2012-10-24 | 广东威创视讯科技股份有限公司 | Cloud storage method and cloud storage system |
| CN103064902A (en) * | 2012-12-18 | 2013-04-24 | 厦门市美亚柏科信息股份有限公司 | Method and device for storing and reading data in hadoop distributed file system (HDFS) |
Non-Patent Citations (1)
| Title |
|---|
| 宁雄雁等: "应用于海量存储中高度容错的新编码方案", 《计算机应用研究》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109831487B (en) | Fragmented file verification method and terminal equipment | |
| CN103778034B (en) | Data backup disaster recovery method and system based on cloud storage | |
| CN102801784B (en) | A kind of distributed data storage method and equipment | |
| CN104077380B (en) | A kind of data de-duplication method, apparatus and system | |
| CN102591947A (en) | Fast and low-RAM-footprint indexing for data deduplication | |
| EP3474146A1 (en) | Data processing method, storage system and exchange device | |
| CN103677674A (en) | Data processing method and device | |
| CN104866430A (en) | High-availability optimization method of memory computing system in combination with principal-subordinate backup and erasure codes | |
| Hollmann | Storage codes—Coding rate and repair locality | |
| CN104331343A (en) | File backup method and system | |
| CN116578746A (en) | Object de-duplication method and device | |
| CN102780769A (en) | Cloud computing platform-based disaster recovery storage method | |
| CN107608821B (en) | Data reading method, device and equipment based on erasure codes | |
| CN102799659B (en) | Overall repeating data deleting system and method based on non-centre distribution system | |
| CN104268032B (en) | The snap processing method and device of a kind of multi-controller | |
| CN105162869A (en) | Data backup management method and equipment | |
| CN107395772B (en) | Management method and management system for repeated data | |
| CN104270287A (en) | Message disorder detecting method and device | |
| CN110457321A (en) | Date storage method and Related product based on block chain | |
| CN104932986A (en) | Data redistribution method and apparatus | |
| CN104063377A (en) | Information processing method and electronic equipment using same | |
| CN102523205A (en) | Determination method and device for content checksum | |
| CN116932421A (en) | Data storage method, device, equipment and storage medium | |
| CN116303789A (en) | Parallel synchronization method and device for multi-fragment multi-copy database and readable medium | |
| CN115883533A (en) | File synchronization method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20130821 |
|
| RJ01 | Rejection of invention patent application after publication |