CN105677508A - Method and system for modifying erasure code data in cloud storage - Google Patents
Method and system for modifying erasure code data in cloud storage Download PDFInfo
- Publication number
- CN105677508A CN105677508A CN201510945310.5A CN201510945310A CN105677508A CN 105677508 A CN105677508 A CN 105677508A CN 201510945310 A CN201510945310 A CN 201510945310A CN 105677508 A CN105677508 A CN 105677508A
- Authority
- CN
- China
- Prior art keywords
- data
- amendment
- coding
- disk
- decoding
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/085—Error detection or correction by redundancy in data representation, e.g. by using checking codes using codes with inherent redundancy, e.g. n-out-of-m codes
Abstract
The present invention discloses a method and system for modifying erasure code data in cloud storage. The method comprises the steps of performing erasure code coding on original data, and storing the original coded data obtained by coding to a disk; when detecting a modification instruction on the original data, obtaining modification data corresponding to the modification instruction; determining whether the size of the modification data is less than or equal to a preset data size threshold; and when the size of the modification data is less than or equal to the data size threshold, performing erasure code coding on the modification data, and saving the modification coded data obtained by coding as modification coded data corresponding to the original data to a kv database. Write operation performance of the kv database when processing small files and the characteristics that stripes required for erasure codes to calculate verifying blocks are small data are fully utilized and combined to accelerate modification speed of the data, and therefore the purposes of reducing time consumption during the data modification process and improving modification efficiency are effectively achieved.
Description
Technical field
The present invention relates to cloud storage technical field, particularly relate to amending method and the system of correcting and eleting codes data in a kind of cloud storage.
Background technology
At present; cloud storage system is in order to ensure the safety of data; the mode that would generally adopt the many copies of data realizes the redundancy of data; along with data volume continue, constantly increase; the problem that storage efficiency that copy mode is brought is low is more and more obvious, then, occurs in that the data redundancy strategy of correcting and eleting codes mode; when coordinates data redundancy, correcting and eleting codes mode can bring more than at least 50% raising of storage efficiency.
In correcting and eleting codes mode, typically require the checking data block calculating data, for the ease of data management, need first to be divided into by initial data the data block of fixed size when calculating, then carry out the calculating of check block for data block, finally data block and the check block calculated are distributed to different storage disks. In order to ensure transmission and the efficiency of storage, cloud storage system is transmission and storage data in units of 4MB generally, can there are the data of multiple data fragmentation in the data block of a storage. Giving an example, 3+1 correcting and eleting codes data Storage Format as shown in Figure 1, initial data is divided into 3 data blocks, and each data block is deleted data fragmentation formed by multiple being entangled, and the data fragmentation on different pieces of information block forms corresponding check block after calculating. Wherein, the burst in order to ensure computational efficiency data is generally between 4kB～64kB.
When initial data is modified operation by needs, in data fragmentation, the amendment of a byte also implies that recalculating of whole data fragmentation and check block, therefore, need when data modification whole data fragmentation is all read out recovery initial data, again carry out burst for data after amendment after the modification and calculate corresponding check block, then data fragmentation and calculated check block being re-write.
Visible, in existing amending method, need when data modification all to read whole data fragmentation, recover, revise, again burst, recalculate after write again, particularly when revising data and being small block data, it is written as random writing, in SATA hard disc, efficiency is far below being sequentially written in, and length consuming time, efficiency are low, particularly in disk pressure bigger when.
Summary of the invention
In view of this, the invention provides the amending method of correcting and eleting codes data in a kind of cloud storage and system, to realize reducing the purpose of the consuming time and then raising amendment efficiency in data modification process.
For solving above-mentioned technical problem, the present invention provides the amending method of correcting and eleting codes data in a kind of cloud storage, and the method includes:
Initial data is carried out correcting and eleting codes coding, the piginal encoded data that coding obtains is deposited to disk;
When the amendment instruction to described initial data being detected, obtain the amendment data corresponding with described amendment instruction;
Judge that whether the size of described amendment data is less than or equal to preset data amount threshold value;
When the size of the described amendment data of judgement is less than or equal to described data-quantity threshold, described amendment data are carried out correcting and eleting codes coding, and the amendment coding data obtained by coding preserve to kv data base as the amendment coding data corresponding with described initial data.
In said method, it is preferable that also include:
When the size of the described amendment data of judgement is more than described data-quantity threshold, from described disk, obtains described piginal encoded data, and utilize the described piginal encoded data and described amendment data, data after being revised that get;
Data after described amendment are carried out coding data after erasure code is revised, and is stored back to described disk by encoding data after described amendment.
In said method, it is preferable that also include:
When the reading instruction to described initial data being detected, it is judged that whether described kv data base exists the amendment coding data corresponding with described initial data;
When it is determined that the presence of described corresponding amendment and encoding data, from described disk, obtain described piginal encoded data, and encode data from amendment corresponding described in described kv data base;
Respectively described piginal encoded data and described corresponding amendment coding data are carried out correcting and eleting codes decoding, obtain described initial data and described amendment data;
Merge the described initial data that obtains of decoding and described amendment data, and data feedback will be merged after the described amendment obtained to read requests end.
In said method, it is preferable that when judging that being absent from described corresponding amendment in described kv data base encodes data, also include:
From described disk, obtain described piginal encoded data, described piginal encoded data is carried out correcting and eleting codes decoding, and the described initial data that decoding obtains is fed back to described read requests end;
Or,
Encode data after obtaining described amendment from described disk, carry out correcting and eleting codes decoding data feedback extremely described read requests end after described amendment decoding obtained to encoding data after described amendment.
In said method, it is preferable that the described amendment coding data obtained by coding preserve to kv data base as the amendment coding data corresponding with described initial data, including:
The data block identifier of described piginal encoded data is stored in described kv data base as key value;
Described corresponding amendment coding data are stored in described kv data base as the value value corresponding with described key value.
Present invention also offers the amendment system of correcting and eleting codes data in a kind of cloud storage, this system includes:
Data block memory element, for initial data is carried out correcting and eleting codes coding, deposits the piginal encoded data that coding obtains to disk;
Amendment data capture unit, for when the amendment instruction to described initial data being detected, obtaining the amendment data corresponding with described amendment instruction;
Whether threshold decision unit, be used for the size judging described amendment data less than or equal to preset data amount threshold value;
Database storage unit, for when the size of the described amendment data of judgement is less than or equal to described data-quantity threshold, described amendment data are carried out correcting and eleting codes coding, and the amendment coding data obtained by coding preserve to kv data base as the amendment coding data corresponding with described initial data.
In said system, it is preferable that also include:
Data determination unit after amendment, for when the size of the described amendment data of judgement is more than described data-quantity threshold, from described disk, obtain described piginal encoded data, and utilize the described piginal encoded data and described amendment data, data after being revised that get;
Described data block memory element is additionally operable to data after described amendment carry out coding data after erasure code is revised, and is stored back to described disk by encoding data after described amendment.
In said system, it is preferable that also include:
There is judging unit, for when the reading instruction to described initial data being detected, it is judged that whether described kv data base exists the amendment coding data corresponding with described initial data;
Coding data capture unit, for when it is determined that the presence of described corresponding amendment and encoding data, obtaining described piginal encoded data, and encode data from amendment corresponding described in described kv data base from described disk;
Decoding unit, for respectively described piginal encoded data and described corresponding amendment coding data being carried out correcting and eleting codes decoding, obtains described initial data and described amendment data;
Read instruction feedback unit, for merging the described initial data that obtains of decoding and described amendment data, and data feedback will be merged after the described amendment obtained to read requests end.
In said system, preferably, described reading instruction feedback unit is additionally operable to: when judging that being absent from described corresponding amendment in described kv data base encodes data, described piginal encoded data is obtained from described disk, described piginal encoded data is carried out correcting and eleting codes decoding, and the described initial data that decoding obtains is fed back to described read requests end; Or, encode data after obtaining described amendment from described disk, carry out correcting and eleting codes decoding data feedback extremely described read requests end after described amendment decoding obtained to encoding data after described amendment.
In said system, it is preferable that described database storage unit specifically for: the data block identifier of described piginal encoded data is stored in described kv data base as key value; Described corresponding amendment coding data are stored in described kv data base as the value value corresponding with described key value.
Above in a kind of cloud storage provided by the invention in the amending method of correcting and eleting codes data and system, when the first time of initial data writes, deposit initial data being carried out the piginal encoded data that correcting and eleting codes coding obtains to disk with data block form; When the amendment instruction to described initial data being detected, obtain the amendment data corresponding with described amendment instruction, and judge that whether the size of these amendment data is less than or equal to preset data amount threshold value (such as 16k, this threshold value can carry out, according to the bin value of erasure code, the setting that adapts); When judging that the size revising data is less than or equal to described data-quantity threshold, think that these amendment data are small block data, then these amendment data are carried out correcting and eleting codes coding, and the amendment coding data obtained by coding preserve to kv data base as the amendment coding data corresponding with described initial data.
When revising data and being small block data, only need to deposit after it is encoded to kv data base, simultaneously, owing to kv data base has sufficient odds for effectiveness when processing low volume data, visible, compare prior art (to need when data modification all to read whole data fragmentation, recover, amendment, again burst, write again after recalculating, particularly when revising data and being small block data, it is written as random writing, in SATA hard disc, efficiency is far below being sequentially written in, length consuming time, efficiency is low, particularly in disk pressure bigger when), the present invention takes full advantage of the characteristic that the kv data base write operation performance when processing small documents and correcting and eleting codes need band to be small data when calculating check block, by in combination with the amendment speed accelerating data, and then effectively achieve the purpose of the consuming time and then raising amendment efficiency reduced in data modification process.
Accompanying drawing explanation
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, the accompanying drawing used required in embodiment or description of the prior art will be briefly described below, apparently, accompanying drawing in the following describes is only embodiments of the invention, for those of ordinary skill in the art, under the premise not paying creative work, it is also possible to obtain other accompanying drawing according to the accompanying drawing provided.
Fig. 1 is the scene schematic diagram of 3+1 correcting and eleting codes data Storage Format of the prior art;
The flow chart of the amending method of correcting and eleting codes data in a kind of cloud storage that Fig. 2 provides for the embodiment of the present invention;
Another flow chart of the amending method of correcting and eleting codes data in a kind of cloud storage that Fig. 3 provides for the embodiment of the present invention;
The structured flowchart schematic diagram of the amendment system of correcting and eleting codes data in a kind of cloud storage that Fig. 4 provides for the embodiment of the present invention.
Detailed description of the invention
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is only a part of embodiment of the present invention, rather than whole embodiments. Based on the embodiment in the present invention, the every other embodiment that those of ordinary skill in the art obtain under not making creative work premise, broadly fall into the scope of protection of the invention.
The core of the present invention is to provide amending method and the system of correcting and eleting codes data in a kind of cloud storage, to realize reducing the purpose of the consuming time and then raising amendment efficiency in data modification process.
The present invention relates to cloud storage system field, the inefficient problem of data modification is there is in some implementations mainly for cloud storage system uses correcting and eleting codes data redundancy mode, to improve the data write efficiency when correcting and eleting codes redundant fashion, reduce the data modification pressure load to system.
In cloud storage system, the realization of correcting and eleting codes redundant fashion is generally divided into client process and server-side processes two ways, the former carries out the distribution of the calculating of check block and data in client, and the latter then carries out at the server end of cloud storage system, and both mutuals are excellent to be lacked. But, when cloud storage system user bandwidth is limited, the mode of server-side processes can effectively save user bandwidth, by the bandwidth of redundant data in server end consumption, it is possible to increase Consumer's Experience. Therefore, the realization that present invention is generally directed to carry out data process at server end is optimized, and by the feature in conjunction with correcting and eleting codes calculation, makes full use of the kv data base efficiency when processing low volume data, improves the efficiency of correcting and eleting codes data modification.
In order to make those skilled in the art be more fully understood that the present invention program, below in conjunction with the drawings and specific embodiments, the present invention is described in further detail.
First, carry out the several concepts involved by the present invention program individually illustrating:
(11) initial data: the initial data being stored in disk with data block form;
(12) piginal encoded data: initial data is carried out correcting and eleting codes and encodes the data obtained;
(21) amendment data: for the amendment data of initial data, amendment data can include concrete amendment content and this amendment content location in initial data, data after can being revised according to initial data and amendment data;
(22) amendment coding data: amendment data are carried out correcting and eleting codes and encodes the data obtained;
(31) data after amendment: amendment data add data after initial data is revised;
(32) coding data after amendment: data after amendment are carried out correcting and eleting codes and encodes the data obtained.
Illustrating the flow chart of the amending method of correcting and eleting codes data in a kind of cloud storage that the embodiment of the present invention provides with reference to Fig. 2, Fig. 2, the method specifically may include steps of:
Step S100, initial data is carried out correcting and eleting codes coding, the piginal encoded data that coding obtains is deposited to disk.
In the present invention, for the storage of rear end, there are data block and two kinds of data storage format of kv data base, when the first time of initial data writes, initial data is carried out correcting and eleting codes coding and obtains piginal encoded data, and this piginal encoded data is deposited to disk with data block form. Then, select be stored in kv data base or directly modify on former data block basis according to the size of amendment data, thus improving the amendment efficiency of correcting and eleting codes data.
Step S101, when the amendment instruction to initial data being detected, obtain and revise amendment data corresponding to instruction;
In specific implementation process, technical staff can call the amendment corresponding api interface of initial data, then the amendment data for this initial data are inputted, amendment data can include concrete amendment content and this amendment content location in initial data, based on this, if data after initial data and amendment data merging can be revised.
Step S102, judge the size of amendment data whether less than or equal to (not less than) preset data amount threshold value; When judging that the size revising data is less than or equal to data-quantity threshold, enter step S103, otherwise, enter step S104.
For default data-quantity threshold, it can carry out, according to the bin value of erasure code, the setting that adapts, for instance 16k etc., and those skilled in the art can be set according to practical situation. When revising the size of data not less than default data-quantity threshold, it is believed that amendment data are small block data, then the subsequent modification of initial data is operated and need not operate data block, but direct execution step S103 entrance kv database processing pattern; It is understood that when the size revising data exceedes data-quantity threshold, then need to perform step S104 and enter conventional data block tupe.
Step S103, amendment data are carried out correcting and eleting codes coding, and the amendment coding data obtained by coding preserve to kv data base as the amendment coding data corresponding with initial data.
Wherein, kv data base uses key assignments (Key-Value) to store data base, and this is a kind of NoSQL (non-relational database) model, and its data carry out organizing, index and storing according to the form of key-value pair.Kv storage is especially suitable for the business datum being not related to too much data relationship business relations, can effectively reduce the number of times of read-write disk simultaneously, have better readwrite performance than SQL database storage.
Characteristic based on kv data base, in actual applications, first, we can arrange a data block identifier for each piginal encoded data, the piginal encoded data that different pieces of information block identification is corresponding different, it is understood that by a data block identifier, we can find a unique piginal encoded data corresponding thereto; Then, the data block identifier of piginal encoded data is stored in kv data base as key value, the amendment coding data corresponding with piginal encoded data are stored in kv data base as the value value corresponding with above-mentioned key value simultaneously, corresponding value value can be found by the particular content of this key value.
In actual applications, above-mentioned kv data base is specifically as follows Leveldb data base, it is a google very efficient key assignments (key-value) data base realized, it would be preferable to support the data volume of 1,000,000,000 ranks, when small data processes, performance is very high. Visible, the present invention processes advantage in conjunction with characteristic and the leveldb data small documents of cloud storage system data fragmentation, it is achieved the raising of correcting and eleting codes data modification efficiency.
Step S104, from disk, obtain piginal encoded data, and data after utilizing the piginal encoded data that gets and amendment data to be revised.
Concrete, obtain after piginal encoded data from disk, the piginal encoded data got is carried out correcting and eleting codes decoding, obtains initial data; Then data after being revised according to initial data and amendment data.
Step S105, data after amendment are carried out after erasure code revise coding data, and coding data after amendment are stored back to disk.
Above-mentioned steps S104 and step S105 is conventional data block tupe, and the process of implementing is referred to prior art, no longer sets forth herein.
As fully visible, when revising data and being small block data, only need to deposit after it is encoded to kv data base, simultaneously, owing to kv data base has sufficient odds for effectiveness when processing low volume data, visible, compare prior art (to need when data modification all to read whole data fragmentation, recover, amendment, again burst, write again after recalculating, particularly when revising data and being small block data, it is written as random writing, in SATA hard disc, efficiency is far below being sequentially written in, length consuming time, efficiency is low, particularly in disk pressure bigger when), the present invention effectively takes full advantage of the characteristic that the KV data base write operation performance when processing small documents and correcting and eleting codes need band to be small data when calculating check block, by in combination with the amendment speed accelerating data, and then achieve the purpose of the consuming time and then raising amendment efficiency reduced in data modification process.
Based on the technical scheme disclosed in the invention described above embodiment, in an alternative embodiment of the invention, based on practical application, when the reading instruction to initial data being detected, the response contents for this reading instruction includes following three kinds of situations:
(1) initial data changes, and the size of amendment data is not less than preset data amount threshold value, now, not exclusively it is positioned in data block due to data, after needing the amendment merging in kv data base to encode the piginal encoded data in data and data block, data after returning amendment to read requests end;
(2) initial data is from start to finish all without changing, and now, data are fully located at data block, it is to be understood that directly can obtain initial data by taking out with the piginal encoded data of data block form storage in disk, decoding;
(3) initial data changes, and the size of amendment data exceedes preset data amount threshold value, now, data are fully located at data block, directly by disk takes out with coding data after the amendment of data block form storage, decoding can revise after data, after then revising, data feedback is to read requests end.
Visible, except (1st) plants situation, other two kinds only need to can feed back to read requests end after directly data block data corresponding with initial data in disk being read out, decoded. Also, it is to be understood that, no matter which kind of mode, feeding back to the data of read requests end is all up-to-date data.
For above-mentioned three kinds of situations, the flow process that is embodied as in actual applications refer to Fig. 3:
Step S300, judge whether kv data base exists the amendment coding data corresponding with initial data; When it is determined that the presence of corresponding amendment coding data, entering step S301, otherwise, entering step S304.
Specifically, when reading instruction read requests end being detected, first determine whether whether kv data base exists the data being key value with the data block identifier that initial data (piginal encoded data) is corresponding, if existed, it is accomplished by reading value value, then read piginal encoded data, return to read requests end after both being merged, otherwise directly read data block data.
Step S301, from disk, obtain piginal encoded data, and corresponding from kv data base amendment coding data;
Step S302, respectively piginal encoded data and corresponding amendment are encoded data and carry out correcting and eleting codes decoding, obtain initial data and amendment data;
Step S303, merge the initial data that obtains of decoding and amendment data, and data feedback will be merged after the amendment obtained to read requests end.
Step S304, the data block data that acquisition is corresponding with initial data from disk, and after it being read out, decodes, feed back to read requests end.
It is understandable that the above-mentioned data block data corresponding with initial data is coding data after piginal encoded data or amendment, when this data block data is piginal encoded data, piginal encoded data is obtained from disk, piginal encoded data is carried out correcting and eleting codes decoding, and the initial data that decoding obtains is fed back to read requests end; When coding data after this data block data is for amendment, from disk, obtain coding data after amendment, coding data after amendment carried out correcting and eleting codes decoding, and after the amendment that decoding is obtained data feedback to read requests end.
Based on the amending method of correcting and eleting codes data in the cloud storage that the invention described above embodiment provides, the embodiment of the present invention additionally provides the amendment system of correcting and eleting codes data in a kind of cloud storage, and with reference to Fig. 4, this system 400 can include following content:
Data block memory element 401, for initial data is carried out correcting and eleting codes coding, deposits the piginal encoded data that coding obtains to disk;
Amendment data capture unit 402, for when the amendment instruction to initial data being detected, obtaining the amendment data corresponding with amendment instruction;
Threshold decision unit 403, for judging that whether the size revising data is less than or equal to preset data amount threshold value;
Database storage unit 404, for when judging that the size revising data is less than or equal to data-quantity threshold, amendment data are carried out correcting and eleting codes coding, and the amendment coding data obtained by coding preserve to kv data base as the amendment coding data corresponding with initial data.
In the present invention, said system 400 can also include herein below:
Data determination unit after amendment, for when judging that the size revising data is more than data-quantity threshold, obtaining piginal encoded data, and utilize the piginal encoded data got and amendment data, data after being revised from disk;
Data block memory element is additionally operable to data after amendment are carried out coding data after erasure code is revised, and is stored back to disk by encoding data after amendment.
In the present invention, said system 400 can also include herein below:
There is judging unit, for when the reading instruction to initial data being detected, it is judged that whether kv data base exists the amendment coding data corresponding with initial data;
Coding data capture unit, for when it is determined that the presence of corresponding amendment coding data, obtaining piginal encoded data amendment coding data corresponding from kv data base from disk;
Decoding unit, carries out correcting and eleting codes decoding for respectively piginal encoded data and corresponding amendment being encoded data, obtains initial data and amendment data;
Read instruction feedback unit, for merging the initial data that obtains of decoding and amendment data, and data feedback will be merged after the amendment obtained to read requests end.
In the present invention, above-mentioned reading instruction feedback unit is additionally operable to: when judging to be absent from kv data base corresponding amendment coding data, from disk, obtain piginal encoded data, piginal encoded data is carried out correcting and eleting codes decoding, and the initial data that decoding obtains is fed back to read requests end; Or, from disk, obtain coding data after amendment, coding data after amendment carried out correcting and eleting codes decoding, and after the amendment that decoding is obtained data feedback to read requests end.
In the present invention, above-mentioned database storage unit 404 specifically for: the data block identifier of piginal encoded data is stored in kv data base as key value; Corresponding amendment coding data are stored in kv data base as the value value corresponding with key value.
It should be noted that each embodiment in this specification all adopts the mode gone forward one by one to describe, what each embodiment stressed is the difference with other embodiments, between each embodiment identical similar part mutually referring to. For system class embodiment, due to itself and embodiment of the method basic simlarity, so describing fairly simple, relevant part illustrates referring to the part of embodiment of the method.
Above the amending method of correcting and eleting codes data in a kind of cloud storage provided by the present invention and system are described in detail. Principles of the invention and embodiment are set forth by specific case used herein, and the explanation of above example is only intended to help to understand method and the core concept thereof of the present invention. It should be pointed out that, for those skilled in the art, under the premise without departing from the principles of the invention, it is also possible to the present invention carries out some improvement and modification, these improve and modify in the protection domain also falling into the claims in the present invention.
Claims (10)
1. the amending method of correcting and eleting codes data in a cloud storage, it is characterised in that the method includes:
Initial data is carried out correcting and eleting codes coding, the piginal encoded data that coding obtains is deposited to disk;
When the amendment instruction to described initial data being detected, obtain the amendment data corresponding with described amendment instruction;
Judge that whether the size of described amendment data is less than or equal to preset data amount threshold value;
When the size of the described amendment data of judgement is less than or equal to described data-quantity threshold, described amendment data are carried out correcting and eleting codes coding, and the amendment coding data obtained by coding preserve to kv data base as the amendment coding data corresponding with described initial data.
2. the method for claim 1, it is characterised in that also include:
When the size of the described amendment data of judgement is more than described data-quantity threshold, from described disk, obtains described piginal encoded data, and utilize the described piginal encoded data and described amendment data, data after being revised that get;
Data after described amendment are carried out coding data after erasure code is revised, and is stored back to described disk by encoding data after described amendment.
3. method as claimed in claim 2, it is characterised in that also include:
When the reading instruction to described initial data being detected, it is judged that whether described kv data base exists the amendment coding data corresponding with described initial data;
When it is determined that the presence of described corresponding amendment and encoding data, from described disk, obtain described piginal encoded data, and encode data from amendment corresponding described in described kv data base;
Respectively described piginal encoded data and described corresponding amendment coding data are carried out correcting and eleting codes decoding, obtain described initial data and described amendment data;
Merge the described initial data that obtains of decoding and described amendment data, and data feedback will be merged after the described amendment obtained to read requests end.
4. method as claimed in claim 3, it is characterised in that when judging that being absent from described corresponding amendment in described kv data base encodes data, also include:
From described disk, obtain described piginal encoded data, described piginal encoded data is carried out correcting and eleting codes decoding, and the described initial data that decoding obtains is fed back to described read requests end;
Or,
Encode data after obtaining described amendment from described disk, carry out correcting and eleting codes decoding data feedback extremely described read requests end after described amendment decoding obtained to encoding data after described amendment.
5. the method as described in Claims 1-4 any one, it is characterised in that the described amendment coding data obtained by coding preserve to kv data base as the amendment coding data corresponding with described initial data, including:
The data block identifier of described piginal encoded data is stored in described kv data base as key value;
Described corresponding amendment coding data are stored in described kv data base as the value value corresponding with described key value.
6. the amendment system of correcting and eleting codes data in a cloud storage, it is characterised in that this system includes:
Data block memory element, for initial data is carried out correcting and eleting codes coding, deposits the piginal encoded data that coding obtains to disk;
Amendment data capture unit, for when the amendment instruction to described initial data being detected, obtaining the amendment data corresponding with described amendment instruction;
Whether threshold decision unit, be used for the size judging described amendment data less than or equal to preset data amount threshold value;
Database storage unit, for when the size of the described amendment data of judgement is less than or equal to described data-quantity threshold, described amendment data are carried out correcting and eleting codes coding, and the amendment coding data obtained by coding preserve to kv data base as the amendment coding data corresponding with described initial data.
7. system as claimed in claim 6, it is characterised in that also include:
Data determination unit after amendment, for when the size of the described amendment data of judgement is more than described data-quantity threshold, from described disk, obtain described piginal encoded data, and utilize the described piginal encoded data and described amendment data, data after being revised that get;
Described data block memory element is additionally operable to data after described amendment carry out coding data after erasure code is revised, and is stored back to described disk by encoding data after described amendment.
8. system as claimed in claim 7, it is characterised in that also include:
There is judging unit, for when the reading instruction to described initial data being detected, it is judged that whether described kv data base exists the amendment coding data corresponding with described initial data;
Coding data capture unit, for when it is determined that the presence of described corresponding amendment and encoding data, obtaining described piginal encoded data, and encode data from amendment corresponding described in described kv data base from described disk;
Decoding unit, for respectively described piginal encoded data and described corresponding amendment coding data being carried out correcting and eleting codes decoding, obtains described initial data and described amendment data;
Read instruction feedback unit, for merging the described initial data that obtains of decoding and described amendment data, and data feedback will be merged after the described amendment obtained to read requests end.
9. system as claimed in claim 8, it is characterized in that, described reading instruction feedback unit is additionally operable to: when judging that being absent from described corresponding amendment in described kv data base encodes data, described piginal encoded data is obtained from described disk, described piginal encoded data is carried out correcting and eleting codes decoding, and the described initial data that decoding obtains is fed back to described read requests end; Or, encode data after obtaining described amendment from described disk, carry out correcting and eleting codes decoding data feedback extremely described read requests end after described amendment decoding obtained to encoding data after described amendment.
10. the system as described in claim 6 to 9 any one, it is characterised in that described database storage unit specifically for: the data block identifier of described piginal encoded data is stored in described kv data base as key value; Described corresponding amendment coding data are stored in described kv data base as the value value corresponding with described key value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510945310.5A CN105677508B (en) | 2015-12-16 | 2015-12-16 | The amending method and system of correcting and eleting codes data in a kind of cloud storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510945310.5A CN105677508B (en) | 2015-12-16 | 2015-12-16 | The amending method and system of correcting and eleting codes data in a kind of cloud storage |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105677508A true CN105677508A (en) | 2016-06-15 |
| CN105677508B CN105677508B (en) | 2018-12-18 |
Family
ID=56189524
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510945310.5A Active CN105677508B (en) | 2015-12-16 | 2015-12-16 | The amending method and system of correcting and eleting codes data in a kind of cloud storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105677508B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107341070A (en) * | 2017-06-30 | 2017-11-10 | 长江大学 | A kind of random write method and system based on correcting and eleting codes |
| CN107948269A (en) * | 2017-11-17 | 2018-04-20 | 郑州云海信息技术有限公司 | A kind of device and method of quick modification data |
| CN107977285A (en) * | 2017-12-19 | 2018-05-01 | 郑州云海信息技术有限公司 | A kind of data modification method, device and the medium of correcting and eleting codes memory mechanism |
| CN109542939A (en) * | 2018-11-21 | 2019-03-29 | 中国人民解放军国防科技大学 | Method and device for abstracting count-min based on erasure codes |
| CN110874284A (en) * | 2018-09-03 | 2020-03-10 | 阿里巴巴集团控股有限公司 | Data processing method and device |
| CN111106840A (en) * | 2018-10-25 | 2020-05-05 | 贵州白山云科技股份有限公司 | Method, system, medium and computer device for accelerating erasure code decoding |
| CN111273862A (en) * | 2020-01-16 | 2020-06-12 | 上海依图网络科技有限公司 | Data storage method and device, readable medium and system thereof |
| CN112799872A (en) * | 2021-02-19 | 2021-05-14 | 上海交通大学 | Erasure code encoding method and device based on key value pair storage system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101630282A (en) * | 2009-07-29 | 2010-01-20 | 国网电力科学研究院 | Data backup method based on Erasure coding and copying technology |
| US20100211841A1 (en) * | 2007-06-08 | 2010-08-19 | Da Tang Mobile Communications Equipment Co., Ltd. | Method, apparatus and system for coding and decoding of ldpc codes |
| US20140372378A1 (en) * | 2013-06-13 | 2014-12-18 | DataGravity, Inc. | Consolidating analytics metadata |
-
2015
- 2015-12-16 CN CN201510945310.5A patent/CN105677508B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100211841A1 (en) * | 2007-06-08 | 2010-08-19 | Da Tang Mobile Communications Equipment Co., Ltd. | Method, apparatus and system for coding and decoding of ldpc codes |
| CN101630282A (en) * | 2009-07-29 | 2010-01-20 | 国网电力科学研究院 | Data backup method based on Erasure coding and copying technology |
| US20140372378A1 (en) * | 2013-06-13 | 2014-12-18 | DataGravity, Inc. | Consolidating analytics metadata |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107341070A (en) * | 2017-06-30 | 2017-11-10 | 长江大学 | A kind of random write method and system based on correcting and eleting codes |
| CN107341070B (en) * | 2017-06-30 | 2020-07-10 | 长江大学 | Random writing method and system based on erasure codes |
| CN107948269A (en) * | 2017-11-17 | 2018-04-20 | 郑州云海信息技术有限公司 | A kind of device and method of quick modification data |
| CN107977285A (en) * | 2017-12-19 | 2018-05-01 | 郑州云海信息技术有限公司 | A kind of data modification method, device and the medium of correcting and eleting codes memory mechanism |
| CN110874284A (en) * | 2018-09-03 | 2020-03-10 | 阿里巴巴集团控股有限公司 | Data processing method and device |
| CN110874284B (en) * | 2018-09-03 | 2024-03-22 | 阿里巴巴集团控股有限公司 | Data processing method and device |
| CN111106840A (en) * | 2018-10-25 | 2020-05-05 | 贵州白山云科技股份有限公司 | Method, system, medium and computer device for accelerating erasure code decoding |
| CN109542939A (en) * | 2018-11-21 | 2019-03-29 | 中国人民解放军国防科技大学 | Method and device for abstracting count-min based on erasure codes |
| CN111273862A (en) * | 2020-01-16 | 2020-06-12 | 上海依图网络科技有限公司 | Data storage method and device, readable medium and system thereof |
| CN112799872A (en) * | 2021-02-19 | 2021-05-14 | 上海交通大学 | Erasure code encoding method and device based on key value pair storage system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105677508B (en) | 2018-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105677508A (en) | Method and system for modifying erasure code data in cloud storage | |
| CN103136243B (en) | File system duplicate removal method based on cloud storage and device | |
| CN103164490B (en) | A kind of efficient storage implementation method of not fixed-length data and device | |
| CN111400083B (en) | Data storage method and system and storage medium | |
| CN104991738B (en) | A kind of solid-state disk and its read-write operation method | |
| CN101582076A (en) | Data de-duplication method based on data base | |
| CN106293990B (en) | A kind of redundant array of independent disks reading/writing method based on batch write check | |
| CN102945278A (en) | Method and device for redoing logs of database records | |
| CN103034592B (en) | Data processing method and device | |
| CN115576501B (en) | Node updating method, system and related device of RAID card | |
| CN103412929A (en) | Mass data storage method | |
| CN105373487A (en) | Defragmentation method and system for storage operation system | |
| CN101984400A (en) | RAID control method, device and system | |
| CN107205024A (en) | Data duplicate removal method and system in a kind of cloud storage system | |
| CN103049508A (en) | Method and device for processing data | |
| CN107506466B (en) | Small file storage method and system | |
| CN107193693A (en) | A kind of online data storage optimization method based on storage system | |
| CN110427347A (en) | Method, apparatus, memory node and the storage medium of data de-duplication | |
| CN104461384A (en) | Data write-in method and storage device | |
| CN104572492A (en) | Method and apparatus of burning data to FAT (File Allocation Table)32 partition | |
| CN102722450A (en) | Storage method for redundancy deletion block device based on location-sensitive hash | |
| CN102270102A (en) | Method for optimizing writing performance of RAID6 (Redundant Array of Independent Disks) disk array | |
| CN104133970A (en) | Data space management method and device | |
| CN109819013A (en) | A kind of block chain memory capacity optimization method based on cloud storage | |
| CN103294658A (en) | Document storage method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |