CN112463460B - Erasure code-based small block data recovery method and system - Google Patents
Erasure code-based small block data recovery method and system Download PDFInfo
- Publication number
- CN112463460B CN112463460B CN202011456828.XA CN202011456828A CN112463460B CN 112463460 B CN112463460 B CN 112463460B CN 202011456828 A CN202011456828 A CN 202011456828A CN 112463460 B CN112463460 B CN 112463460B
- Authority
- CN
- China
- Prior art keywords
- zero
- data
- unit
- erasure correcting
- erasure
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000011084 recovery Methods 0.000 title claims abstract description 43
- 238000000034 method Methods 0.000 title claims abstract description 26
- 230000004048 modification Effects 0.000 claims description 9
- 238000012986 modification Methods 0.000 claims description 9
- 238000012937 correction Methods 0.000 description 28
- 101000772194 Homo sapiens Transthyretin Proteins 0.000 description 8
- 102100029290 Transthyretin Human genes 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 6
- 230000002349 favourable effect Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 108010001267 Protein Subunits Proteins 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1469—Backup restoration techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
Abstract
The application discloses a method and a system for recovering small block data based on erasure codes, wherein the method comprises the following steps: adding an object attribute in the attribute of the small data; recording all-zero erasure correcting units generated by small data blocks in a creation writing stage by using object attributes, and marking the all-zero erasure correcting units; when the storage system is recovered from the fault, all the all-zero erasure correcting units are obtained from the object attributes; judging whether the current erasure correcting unit to be recovered is an all-zero erasure correcting unit or not; if so, creating an empty file with the same size as zero data in the erasure correcting unit to be recovered to replace the zero data, and executing write-in operation; otherwise, the small data block is directly recovered. The system comprises: the system comprises an object attribute increasing module, a recording and marking module, an all-zero erasure correcting unit obtaining module, a judging module, an empty file creating module and a data recovering module. By the method and the device, the storage space can be saved, and the data recovery efficiency and the data recovery performance are improved.
Description
Technical Field
The present application relates to the field of distributed storage systems, and in particular, to a method and system for recovering small blocks of data based on erasure codes.
Background
With the rapid development of information technology, the data function is more and more prominent, daily life and work can generate a large amount of data, the data are collected, analyzed, refined and summarized, key information can be obtained, and huge economic benefits are created according to the information, so that the rapid development of industries such as big data is promoted, the rapid growth of the data also brings huge pressure to data storage, and more enterprises generally use a copy technology or an erasure code technology to ensure the consistency of the data for ensuring the safe and efficient storage of the rapidly-growing data.
The read-write data in the erasure code technology is divided into K erasure correction stripes by taking the erasure correction stripes as units, the data written into the erasure correction stripes are divided into K parts according to the erasure correction units, the K parts of data are used for obtaining M parts of check data through calculation, the K + M parts of data are stored at different positions, such as different nodes or different hard disks of the same node, when the stored data are read, only the K parts of data in the K + M need to be read, and all the written data can be obtained through calculation. When the erasure code technology is used for data storage, how to recover small data blocks when a storage system is in a fault scene is an important technical problem.
At present, when a storage system is in a failure scene, a method for recovering small blocks of data is generally optimized when the small blocks of data are written. Specifically, if the length of the written data is smaller than the erasure correction stripes, the original erasure correction coding technology needs to complete the part of the erasure correction stripes which are insufficient with zero data, then divide the part into K erasure correction units, judge whether all the erasure correction units are zero data, and if all the erasure correction units are zero data, create an empty file to replace the writing operation during data writing.
However, in the current method for recovering small blocks of data, when a storage system fails, if data to be recovered is an empty file, the empty file is recovered to zero data of a corresponding size, and the zero data causes a waste of space, and also occupies a write bandwidth and increases write time. Therefore, the current method for recovering the small data blocks has the disadvantages of low utilization rate of space resources of the storage system and low data recovery efficiency of the storage system.
Disclosure of Invention
The application provides a method and a system for recovering small block data based on erasure codes, which are used for solving the problems that the utilization rate of space resources of a storage system is low and the data recovery efficiency of the storage system is not high enough by a method for recovering small block data in the prior art.
In order to solve the technical problem, the embodiment of the application discloses the following technical scheme:
an erasure code based method for recovering small blocks of data, the method comprising:
adding an object attribute in the attribute of the small data;
recording an all-zero erasure correcting unit generated by the small data blocks in the creation and writing stage by using the object attributes, and marking the all-zero erasure correcting unit;
when the storage system is recovered from the fault, all the all-zero erasure correcting units are acquired from the object attributes;
judging whether the current erasure correcting unit to be recovered is an all-zero erasure correcting unit or not;
if so, creating an empty file with the same size as zero data in the erasure correcting unit to be recovered to replace the zero data, and executing write-in operation;
if not, the small block data recovery is directly carried out.
Optionally, after recording an all-zero erasure correcting unit generated by the small data in the creation writing stage by using the object attribute, and marking the all-zero erasure correcting unit, the method further includes:
and updating the object attribute in the small block data modification or additional writing stage.
Optionally, the recording, by using the object attribute, an all-zero erasure correcting unit generated by the tile data in a creation writing stage, and marking the all-zero erasure correcting unit includes:
in the small block data creation and writing stage, storing all-zero erasure correcting units in a map container shares;
whether all-zero erasure units exist is marked by all _ zero in the object attribute;
encrypting all _ zero and shards into a first storage space;
and saving the first storage space to a database along with the writing of the object.
Optionally, the updating the object attribute at the phase of modifying, rewriting or appending the small block data includes:
acquiring the object attribute;
according to the object attribute, decrypting to obtain all _ zero and shares;
according to the decrypted all _ zero and shards, taking intersection of the all-zero erasure correcting unit in the current modification or additional writing stage and the all-zero erasure correcting unit in the object attribute;
updating shards of the all-zero erasure unit according to the intersection result, and acquiring updated data;
and rewriting the updated data into the object attribute.
Optionally, the rewriting the updated data into the object attribute includes:
setting the value of all _ zero according to whether the updated shards is empty;
encrypting the updated shards and the set all _ zero into a second storage space;
and storing the second storage space in a database along with the writing of the object.
An erasure code based tile data recovery system, the system comprising:
the object attribute adding module is used for adding an object attribute in the attribute of the small data;
the recording and marking module is used for recording all-zero erasure correcting units generated by the small data blocks in the creation and writing stage by utilizing the object attributes and marking the all-zero erasure correcting units;
an all-zero erasure correcting unit obtaining module, configured to obtain all the all-zero erasure correcting units from the object attribute when performing storage system failure recovery;
the judging module is used for judging whether the erasure correcting unit to be recovered at present is an all-zero erasure correcting unit, if so, starting the empty file creating unit, and if not, directly recovering the small data blocks;
the empty file creating module is used for creating an empty file with the same size as zero data in the erasure correcting unit to be recovered to replace the zero data when the erasure correcting unit to be recovered is an all-zero erasure correcting unit and executing writing operation;
and the data recovery module is used for directly recovering the small data when the current erasure correcting unit to be recovered is not an all-zero erasure correcting unit.
Optionally, the system further includes: and the object attribute updating module is used for updating the object attribute in the stage of modifying or additionally writing the small block data.
Optionally, the recording and marking module comprises:
the first storage unit is used for storing the all-zero erasure correcting unit in a map container shares in a small block data creation writing stage;
a marking unit for marking whether there is an all-zero erasure unit in the object attribute by using all _ zero;
the first encryption unit is used for encrypting the all _ zero and the shares into the first storage space;
and the second storage unit is used for storing the first storage space into a database along with the writing of the object.
Optionally, the object attribute updating module includes:
an object attribute acquisition unit configured to acquire the object attribute;
the decryption unit is used for decrypting to obtain all _ zero and shards according to the object attributes;
an intersection fetching unit, configured to fetch an intersection between an all-zero erasure correcting unit in the current modification or additional writing stage and an all-zero erasure correcting unit in the object attribute according to the decrypted all _ zero and shares;
the updating unit is used for updating shards of the all-zero erasure unit according to the intersection taking result and acquiring updated data;
and the third storage unit is used for rewriting the updated data into the object attribute.
Optionally, the third saving unit includes:
a setting subunit, configured to set a value of all _ zero according to whether the updated shards is empty;
the second encryption unit is used for encrypting the updated shards and the set all _ zero into a second storage space;
and the fourth storage unit is used for storing the second storage space in a database along with the writing of the object.
The technical scheme provided by the embodiment of the application can have the following beneficial effects:
the method comprises the steps of firstly adding an object attribute in the attribute of small block data, recording and marking an all-zero erasure correction unit generated by the small block data in a creation and writing stage by using the object attribute, acquiring all-zero erasure correction units from the object attribute when fault recovery is carried out, then judging whether the erasure correction unit to be recovered currently belongs to the all-zero erasure correction unit, if the erasure correction unit is the all-zero erasure correction unit, creating an empty file to replace zero data to carry out writing operation, and if the erasure correction unit is not the all-zero erasure correction unit, directly carrying out small block data recovery according to the original mode. This embodiment is based on erasure code fault scene, through record and mark all-zero erasure correcting unit, when data recovery, can accurately discern whether present pending erasure correcting unit is all-zero erasure correcting unit, then only carry out the empty file to all-zero erasure correcting unit and establish, non-all-zero erasure correcting unit directly carries out data recovery, and need not restore the empty file for the zero data of corresponding size, thereby save storage space, be favorable to improving storage space resource utilization, avoid occupying more bandwidth of writing in, and save data write-in time, be favorable to improving storage data recovery efficiency.
The embodiment also updates the object attribute at the stages of modification and write addition, so that the newly generated all-zero erasure correction unit is timely recorded and marked by using the object attribute, and the accuracy of data recovery and the data recovery efficiency of the storage system are favorably improved.
The present application further provides a small block data recovery system based on erasure codes, which includes: the system comprises an object attribute increasing module, a recording and marking module, an all-zero erasure correcting unit obtaining module, a judging module, an empty file creating module and a data recovery module. Through object attribute increase module, record and mark module, utilize the object attribute to record and mark the all-zero erasure correcting unit, be favorable to the quick discernment and the differentiation of all-zero erasure correcting unit, need not restore the null file to the zero data of corresponding size, be favorable to saving storage space, improve storage space resource utilization, and owing to need not carry out data recovery to the null file, can avoid occupying more bandwidth of writing in, and save data write-in time, be favorable to improving storage data recovery efficiency. In this embodiment, an all-zero erasure correcting unit for comparison is timely called by an all-zero erasure correcting unit obtaining module when a fault is recovered, a data processing mode is determined by a determining module according to whether the all-zero erasure correcting unit is determined, an empty file creating module is started when an all-zero erasure correcting unit is determined, and a data recovery module is started when a non-all-zero erasure correcting unit is determined. The structure for rapidly processing the all-zero erasure correcting unit can effectively improve the data recovery efficiency.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present application, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a schematic flowchart of a small block data recovery method based on erasure codes according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of a small block data recovery system based on erasure codes according to an embodiment of the present application.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
For a better understanding of the present application, embodiments of the present application are explained in detail below with reference to the accompanying drawings.
Example one
Referring to fig. 1, fig. 1 is a schematic flowchart of a method for recovering small blocks of data based on erasure codes according to an embodiment of the present application. As shown in fig. 1, the method for recovering small block data based on erasure codes in this embodiment mainly includes the following steps:
s1: and adding an object attribute to the attribute of the small data.
The embodiment may add an object attribute EC _ ATTR to the attributes of the small block data.
S2: and recording all-zero erasure correcting units generated by the small data blocks in the creation writing stage by utilizing the object attributes, and marking the all-zero erasure correcting units.
In this embodiment, the all-zero erasure correcting unit is also called an all-zero fragment, that is: and erasure correcting units corresponding to all the zero writing data. The embodiment can store and facilitate the identification of all the all-zero erasure correcting units by recording and marking the all-zero erasure correcting units. In this embodiment, step S2 further includes the following steps:
s21: in the small block data creation writing stage, all-zero erasure correction units are stored in a map container shares.
S22: in the object attribute, whether there is an all-zero erasure unit is marked with all _ zero.
S23: all _ zero and shares are encrypted into the first memory space.
In this embodiment, the first storage space is a buffer.
S24: and saving the first storage space to the database along with the writing of the object.
After all _ zero and shards are encrypted into the first storage space, the first storage space is stored into the object attribute EC _ ATTR, finally, the object attribute EC _ ATTR and other attributes are put into a map container to _ set, and all the attributes are stored into the database together. By the method, time occupation caused by multiple times of writing of the object attribute in the object writing process can be avoided, and the data processing efficiency is improved, so that the data recovery efficiency is improved.
The erasure correcting unit is divided into a data unit and a checking unit, the checking unit is obtained through calculation of the data unit, and all zero writing units only have the data unit. As can be seen from the foregoing steps S21-S24, by adding the object attribute EC _ ATTR, in the small block data creation writing stage, all-zero erasure correction units are stored in one map container shares, and at the same time, a pool type value all _ zero is added to mark whether there is an all-zero erasure correction unit, and if there is an all-zero erasure correction unit, all _ zero is recorded as true, otherwise, all _ zero is recorded as false. Then, all _ zero and shards are encrypted into a buffer list, the buffer list is stored in EC _ ATTR, and the EC _ ATTR is finally stored in a database along with the writing of the object.
Further, after step S2, the present embodiment further includes step S3: and updating the object attribute in the small block data modification or additional writing stage.
S31: and acquiring the object attribute.
S32: decryption results in all _ zero and shards according to the object attributes.
S33: and according to the decrypted all _ zero and shards, taking intersection of the all-zero erasure correcting unit in the current modification or additional writing stage and the all-zero erasure correcting unit in the object attribute.
S34: and updating shards of the all-zero erasure correction unit according to the intersection result, and acquiring updated data.
S35: and rewriting the updated data into the object attribute.
Further, step S35 again includes the following processes:
s351: the value of all _ zero is set according to whether the updated shards is empty.
S352: and encrypting the updated shards and the set all _ zero into the second storage space.
In this embodiment, the second storage space is also buffer.
S353: the second storage space is saved in the database with the writing of the object.
As can be seen from the foregoing steps S31 to S35, in the stage of writing or modifying the small block of data, the EC _ ATTR attribute of the object is first obtained, and is decrypted to obtain all _ zero and shards, the intersection is taken between the all-zero erasure correcting unit and the content recorded in the shards, and the intersection is recorded again in the shards according to the intersection result, so as to update the all-zero erasure correcting unit shards. And then setting the value of all _ zero according to whether the shards are empty, wherein the value of all _ zero is false when the shards are empty, and the value of all _ zero is true when the shards are not empty. Continuing to encrypt the all _ zero and shares to another buffer list, and saving the buffer list in the database along with the object's write.
S4: when the storage system is recovered from the failure, all the all-zero erasure correcting units are acquired from the object attributes.
When the storage system is abnormal and data recovery is performed, all attributes of the object also need to be recovered, in this embodiment, all attributes of the object are obtained through one attribute reading operation, and the object attribute EC _ ATTR is taken out, and the buffer is decrypted to obtain the all _ zero and the shares.
S5: and judging whether the current erasure correcting unit to be recovered is an all-zero erasure correcting unit or not.
S6: if so, an empty file with the same size as zero data in the erasure correcting unit to be recovered currently is created to replace the zero data, and the writing operation is executed.
S7: if not, the small block data recovery is directly carried out.
As can be seen from the above steps S5 to S7, after the buffer obtained by decryption in step S6 obtains all _ zero and shares, it is determined whether the all _ zero erasure correction unit is an all-zero erasure correction unit according to the value of all _ zero, and if the value of all _ zero is false, it indicates that the object does not have all zero units, and the original recovery process is continued. And if the value of the all _ zero is true, entering an erasure correction small block data recovery optimization flow.
The erasure correction small block data recovery optimization process specifically comprises the following steps: firstly, judging whether an erasure correcting unit needing to be recovered at present is a data unit or a check unit, if the erasure correcting unit is the check unit, continuing to perform the original recovery process; if the data unit is the data unit, comparing all erasure correcting units needing to be recovered with records of shards, if the current erasure correcting unit is not recorded in the shards, indicating that the current erasure correcting unit records effective data, and needing to write all the recovered data, otherwise, if the current erasure correcting unit is recorded in the shards, creating an empty file through touch operation.
In summary, in this embodiment, by recording an all-zero erasure correcting unit generated in the writing process of a small block of data, when an abnormality occurs in the storage system, the all-zero erasure correcting unit is obtained, and it is determined whether the current erasure correcting unit to be recovered is the all-zero erasure correcting unit, if so, an empty file is created through touch instead of writing zero data, and the storage system can detect that data has been written. By the method, the storage space can be saved, the zero data recovery writing time can be saved, the bandwidth occupation during data recovery can be reduced, and the data recovery efficiency and the data recovery performance of the storage system can be improved.
Example two
Referring to fig. 2 based on the embodiment shown in fig. 1, fig. 2 is a schematic structural diagram of a small block data recovery system based on erasure codes according to an embodiment of the present application. As can be seen from fig. 2, the small block data recovery system based on erasure codes in the embodiment of the present invention mainly includes: the system comprises an object attribute increasing module, a recording and marking module, an all-zero erasure correcting unit obtaining module, a judging module, an empty file creating module and a data recovery module.
The object attribute adding module is used for adding an object attribute in the attribute of the small data; the recording and marking module is used for recording all-zero erasure correcting units generated by the small data blocks in the creation and writing stage by utilizing the object attributes and marking the all-zero erasure correcting units; the all-zero erasure correcting unit obtaining module is used for obtaining all the all-zero erasure correcting units from the object attributes when the storage system is recovered from the fault; the judging module is used for judging whether the erasure correcting unit to be recovered at present is an all-zero erasure correcting unit, if so, starting the empty file creating unit, and if not, directly recovering the small data blocks; the empty file creating module is used for creating an empty file with the same size as zero data in the erasure correcting unit to be recovered to replace the zero data when the erasure correcting unit to be recovered is an all-zero erasure correcting unit and executing writing operation; and the data recovery module is used for directly recovering the small blocks of data when the current erasure correcting unit to be recovered is not an all-zero erasure correcting unit.
The recording and marking module includes: the device comprises a first storage unit, a marking unit, a first encryption unit and a second storage unit. The first storage unit is used for storing the all-zero erasure correcting unit in a map container shards in a small block data creation writing stage; a marking unit for marking whether there is an all-zero erasure unit in the object attribute with all _ zero; a first encryption unit, configured to encrypt all _ zero and shares into a first storage space; and the second storage unit is used for storing the first storage space into the database along with the writing of the object.
Further, the system for restoring the small block data based on the erasure code in this embodiment further includes an object attribute updating module, configured to update the object attribute at a stage of modifying, rewriting or additionally writing the small block data.
The object attribute updating module comprises: the device comprises an object attribute acquisition unit, a decryption unit, an intersection acquisition unit, an updating unit and a third storage unit. The object attribute acquiring unit is used for acquiring object attributes; the decryption unit is used for decrypting to obtain all _ zero and shares according to the object attribute; the intersection taking unit is used for taking an intersection from the all-zero erasure correcting unit in the current modification or additional writing stage and the all-zero erasure correcting unit in the object attribute according to the decrypted all _ zero and shards; the updating unit is used for updating shards of the all-zero erasure unit according to the intersection taking result and acquiring updated data; and the third storage unit is used for rewriting the updated data into the object attribute.
Further, the third holding unit includes: a sub-unit, a second encryption unit and a fourth saving unit are arranged. The setting subunit is used for setting the value of all _ zero according to whether the updated shards are empty or not; the second encryption unit is used for encrypting the updated shards and the set all _ zero into a second storage space; and the fourth storage unit is used for storing the second storage space in the database along with the writing of the object.
In this embodiment, the working principle and the working method of the erasure code-based small block data recovery system have been elaborated in detail in the embodiment shown in fig. 1, and are not described herein again, and parts not described in detail may be referred to each other between the two embodiments.
The previous description is only an example of the present application, and is provided to enable any person skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (2)
1. A method for recovering small block data based on erasure codes is characterized by comprising the following steps:
adding an object attribute in the attribute of the small data;
recording an all-zero erasure correcting unit generated by the small data blocks in the creation and writing stage by using the object attributes, and marking the all-zero erasure correcting unit;
when the storage system is recovered from the fault, all the all-zero erasure correcting units are acquired from the object attributes;
judging whether the current erasure correcting unit to be recovered is an all-zero erasure correcting unit or not;
if so, creating an empty file with the same size as zero data in the erasure correcting unit to be recovered to replace the zero data, and executing write-in operation;
if not, directly recovering the small data blocks;
the recording of the all-zero erasure correcting unit generated by the small data block in the creation and writing stage by using the object attribute and the marking of the all-zero erasure correcting unit include:
in the small block data creation and writing stage, storing all-zero erasure correcting units in a map container shares;
in the object attribute, whether all-zero erasure units exist is marked by all _ zero;
encrypting all _ zero and shares into a first storage space;
the first storage space is stored in a database along with the writing of the object;
in the stage of modifying or appending writing the small block data, the object attribute is updated, which comprises the following steps:
acquiring the object attribute;
according to the object attribute, decrypting to obtain all _ zero and shares;
according to the decrypted all _ zero and shards, taking intersection of the all-zero erasure correcting unit in the current modification or additional writing stage and the all-zero erasure correcting unit in the object attribute;
updating shards of the all-zero erasure unit according to the intersection result, and acquiring updated data;
rewriting the updated data into the object attribute;
the rewriting of the updated data into the object attribute includes:
setting the value of all _ zero according to whether the updated shards is empty;
encrypting the updated shards and the set all _ zero into a second storage space;
and storing the second storage space in a database along with the writing of the object.
2. A system for recovering data from a small block based on erasure codes, the system comprising:
the object attribute adding module is used for adding an object attribute in the attribute of the small data;
the recording and marking module is used for recording all-zero erasure correcting units generated by the small data blocks in the creation and writing stage by utilizing the object attributes and marking the all-zero erasure correcting units;
an all-zero erasure correcting unit obtaining module, configured to obtain all the all-zero erasure correcting units from the object attribute when performing storage system failure recovery;
the judging module is used for judging whether the erasure correcting unit to be recovered at present is an all-zero erasure correcting unit, if so, starting the empty file creating unit, and if not, directly recovering the small data blocks;
the empty file creating module is used for creating an empty file with the same size as zero data in the erasure correcting unit to be recovered to replace the zero data when the erasure correcting unit to be recovered is an all-zero erasure correcting unit and executing writing operation;
the data recovery module is used for directly recovering the small data when the current erasure correcting unit to be recovered is not an all-zero erasure correcting unit;
the recording and marking module includes:
the first storage unit is used for storing the all-zero erasure correcting unit in a map container shares in a small block data creation writing stage;
a marking unit for marking whether there is an all-zero erasure unit in the object attribute by using all _ zero;
a first encryption unit, configured to encrypt all _ zero and shares into a first storage space;
the second storage unit is used for storing the first storage space into a database along with the writing of the object;
the object attribute updating module comprises:
an object attribute acquisition unit configured to acquire the object attribute;
the decryption unit is used for decrypting to obtain all _ zero and shards according to the object attributes;
an intersection taking unit, configured to take an intersection between an all-zero erasure correcting unit in the current modifying or additionally writing stage and an all-zero erasure correcting unit in the object attribute according to the decrypted all _ zero and shards;
the updating unit is used for updating shards of the all-zero erasure unit according to the intersection taking result and acquiring updated data;
a third saving unit, configured to rewrite the updated data into the object attribute;
the third holding unit includes:
a setting subunit, configured to set a value of all _ zero according to whether the updated shards is empty;
the second encryption unit is used for encrypting the updated shards and the set all _ zero into a second storage space;
and the fourth storage unit is used for storing the second storage space in a database along with the writing of the object.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011456828.XA CN112463460B (en) | 2020-12-11 | 2020-12-11 | Erasure code-based small block data recovery method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011456828.XA CN112463460B (en) | 2020-12-11 | 2020-12-11 | Erasure code-based small block data recovery method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112463460A CN112463460A (en) | 2021-03-09 |
| CN112463460B true CN112463460B (en) | 2022-11-29 |
Family
ID=74801474
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011456828.XA Active CN112463460B (en) | 2020-12-11 | 2020-12-11 | Erasure code-based small block data recovery method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112463460B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105893169A (en) * | 2016-03-31 | 2016-08-24 | 乐视控股（北京）有限公司 | File storage method and system based on erasure codes |
| CN107589916A (en) * | 2017-09-29 | 2018-01-16 | 郑州云海信息技术有限公司 | A kind of entangling based on correcting and eleting codes deletes the creation method and relevant apparatus in pond |
| CN107608820A (en) * | 2017-09-26 | 2018-01-19 | 郑州云海信息技术有限公司 | A kind of file wiring method and relevant apparatus based on correcting and eleting codes |
-
2020
- 2020-12-11 CN CN202011456828.XA patent/CN112463460B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105893169A (en) * | 2016-03-31 | 2016-08-24 | 乐视控股（北京）有限公司 | File storage method and system based on erasure codes |
| CN107608820A (en) * | 2017-09-26 | 2018-01-19 | 郑州云海信息技术有限公司 | A kind of file wiring method and relevant apparatus based on correcting and eleting codes |
| CN107589916A (en) * | 2017-09-29 | 2018-01-16 | 郑州云海信息技术有限公司 | A kind of entangling based on correcting and eleting codes deletes the creation method and relevant apparatus in pond |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112463460A (en) | 2021-03-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20170220256A1 (en) | Retrieve data block from determined devices | |
| CN106708653B (en) | Mixed tax big data security protection method based on erasure code and multiple copies | |
| JP2007012058A (en) | File system for storing transaction records in flash-like media | |
| CN103631721A (en) | Method and system for isolating bad blocks in internal storage | |
| CN104615504A (en) | Method and device for achieving data protection | |
| CN105302665B (en) | A kind of improved Copy on write Snapshot Method and system | |
| CN111400267B (en) | Method and device for recording logs | |
| CN112612576B (en) | Virtual machine backup method and device, electronic equipment and storage medium | |
| CN103049539A (en) | Method and device for storing file data in file system | |
| CN104881242A (en) | Data writing method and data writing device | |
| CN107798063B (en) | Snapshot processing method and snapshot processing device | |
| WO2015116141A1 (en) | Identifying memory regions that contain remapped memory locations | |
| CN112463460B (en) | Erasure code-based small block data recovery method and system | |
| CN103729301B (en) | Data processing method and device | |
| CN112882859A (en) | Virtual machine synthetic backup method and system | |
| CN111858159A (en) | Backup method and system based on chain clone characteristic | |
| CN113641694B (en) | Database massive historical data backup method and database massive historical data recovery method | |
| CN108089942B (en) | Data backup and recovery method and device | |
| CN111736778B (en) | Data updating method, device and system and electronic equipment | |
| CN110209530B (en) | Method and system for recovering IO data of CDP system | |
| CN113553215A (en) | Erasure code data recovery optimization method and device based on environmental information | |
| CN105573862A (en) | Method and equipment for recovering file systems | |
| CN110688071A (en) | Data synchronization method and system for reducing data synchronization quantity | |
| KR101477017B1 (en) | Method and Apparatus for managing index in a shared memory | |
| CN110297728B (en) | Selective data reconstruction method in file reconstruction process based on origin data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |