CN111984443A - Encoding method, decoding method and corresponding devices in distributed system environment - Google Patents
Encoding method, decoding method and corresponding devices in distributed system environment Download PDFInfo
- Publication number
- CN111984443A CN111984443A CN201910439739.5A CN201910439739A CN111984443A CN 111984443 A CN111984443 A CN 111984443A CN 201910439739 A CN201910439739 A CN 201910439739A CN 111984443 A CN111984443 A CN 111984443A
- Authority
- CN
- China
- Prior art keywords
- data
- original data
- original
- distributed system
- partition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0709—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1008—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices
- G06F11/1012—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices using codes or arrangements adapted for a specific type of error
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1471—Saving, restoring, recovering or retrying involving logging of persistent data for recovery
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Error Detection And Correction (AREA)
Abstract
The application provides an encoding method, a decoding method and a corresponding device under a distributed system environment, which relate to the technical field of encoding fault tolerance, and the method comprises the following steps: performing first coding on original data to obtain global check data, and performing second coding on the original data to obtain local check data; storing the raw data in one or more data partitions of a distributed system; and storing the global check data and the local check data corresponding to the original data in the data partition storing the original data. Aiming at the distributed system environment, the recovery cost is optimized on the premise of not influencing the fault tolerance capability and the storage cost of the system.
Description
Technical Field
The invention relates to the technical field of coding fault tolerance, in particular to a coding method, a decoding method and a corresponding device in a distributed system environment.
Background
At present, the storage scale of distributed systems is becoming larger and larger; device errors in distributed systems are also a significant problem. The storage cost and reliability of the data are factors to be considered when designing the distributed system. Erasure codes can minimize the storage overhead of the system while ensuring the same data reliability as it does.
The erasure code can add m parts of encoded data (used for storing the erasure code) to n parts of original data, and can restore the original data through any n parts of data in n + m parts. Such erasure codes are also referred to as maximum distance divisible codes. One of the most widely used conventional codes in the industry is RS coding. It can have the performance of tolerating any number of disc errors by adjusting the parameters k and m, but RS coding has significant drawbacks, mainly in its representation of recovering data. The RS encoding requires the retrieval of data from k data disks for recovery of any erroneous data disk. The amplification of the recovery cost occupies additional network bandwidth, IO resources, and recovery time. In a distributed system environment, cross-region calling data also occupies more network bandwidth, and the response time is further improved.
Disclosure of Invention
The application provides an encoding method, a decoding method and a corresponding device in a distributed system environment, aiming at the distributed system environment, and the recovery cost is optimized on the premise of not influencing the fault tolerance capability and the storage cost of the system.
The technical scheme is as follows:
in a first aspect, the present invention provides a coding method in a distributed system environment, including:
Performing first coding on original data to obtain global check data, and performing second coding on the original data to obtain local check data;
storing the raw data in one or more data partitions of a distributed system;
and storing the global check data and the local check data corresponding to the original data in the data partition storing the original data.
Preferably, according to the data blocks of the original data, the original data is averagely stored in each data partition of the distributed system; the local check data corresponding to the original data stored in each data partition are the same in quantity; the quantity of the global check data corresponding to the original data stored in each data partition is the same.
Preferably, the first code is an RS code; the second code is an MSR code.
In a second aspect, the present invention provides a decoding method in a distributed system environment, including:
reading original data stored in a current partition;
when the damage quantity of the original data is smaller than or equal to a preset first threshold value, performing data recovery on the original data according to a second coding mode by using local check data stored in a current partition to obtain complete original data;
And when the damage quantity of the original data is larger than a preset first threshold value, performing data recovery on the original data according to a first coding mode by using the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system to obtain complete original data.
Preferably, when the damaged amount of the original data is greater than a preset second threshold, performing data recovery on the original data according to a first coding and second coding joint coding mode by using the local check data and the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system to obtain complete original data, wherein the second threshold is greater than the first threshold.
Preferably, the first code is an RS code; the second code is an MSR code.
In a third aspect, the present invention provides an encoding apparatus in a distributed system environment, including:
the encoding module is used for carrying out first encoding on the original data to obtain global check data and carrying out second encoding on the original data to obtain local check data;
a partitioning module configured to store the raw data in one or more data partitions of a distributed system;
And the distribution module is arranged for storing the global check data and the local check data corresponding to the original data in the data partition storing the original data.
Preferably, the partitioning module averagely stores the original data in each data partition of the distributed system according to the data partitioning of the original data; the local check data corresponding to the original data stored in each data partition are the same in quantity; the quantity of the global check data corresponding to the original data stored in each data partition is the same.
In a fourth aspect, the present invention provides a decoding apparatus in a distributed system environment, including:
the communication module is set to read the original data stored in the current partition;
the recovery module is used for performing data recovery on the original data according to a second coding mode by using the local check data stored in the current partition when the damaged quantity of the original data is smaller than or equal to a preset first threshold value to obtain complete original data;
and the recovery module is further configured to perform data recovery on the original data according to a first coding mode by using the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system when the original data damage number is larger than a preset first threshold value, so as to obtain complete original data.
Preferably, the recovery module is further configured to perform data recovery on the original data according to a joint coding mode of a first coding and a second coding by using the local check data and the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system when the original data damage number is greater than a preset second threshold, so as to obtain complete original data, where the second threshold is greater than the first threshold.
Compared with the prior art, the application has the following beneficial effects:
the method combines the RS code and the MSR code into a new code form by using the idea of local check. The new coding mode ensures the data reliability and storage cost of the whole distributed system, and has the excellent performance of low single disk recovery cost of MSR coding. Compared with the traditional RS coding scheme, the performance of the system after using new codes is greatly improved in a distributed system environment; and the method has excellent expandability, and the parameters can be suitable for system environments of various partitions after being adjusted.
Drawings
FIG. 1 is a flowchart of an encoding method in a distributed system environment according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a distributed system storing original data and check data in partitions according to an embodiment of the present invention;
FIG. 3 is a flowchart of a decoding method in a distributed system environment according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of an encoding apparatus in a distributed system environment according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a decoding method in a distributed system environment according to an embodiment of the present invention;
FIG. 6 is a flow chart of a decoding process in a distributed system environment according to an embodiment of the present invention.
Detailed Description
The technical solutions of the present application will be described in more detail below with reference to the accompanying drawings and embodiments.
It should be noted that, if not conflicted, the embodiments and the features of the embodiments can be combined with each other and are within the scope of protection of the present application. Additionally, while a logical order is shown in the flow diagrams, in some cases, the steps shown or described may be performed in an order different than here.
In one configuration, the distributed system includes a plurality of data partitions, each data partition including one or more data disks for storing data, each data disk may include one or more processors (CPUs), input/output interfaces, network interfaces, and memory (memory).
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium. The memory may include one or more modules.
Computer-readable media include both non-transitory and non-transitory, removable and non-removable storage media that can implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
Example one
As shown in fig. 1, an embodiment of the present invention provides an encoding method in a distributed system environment, including:
s101, performing first coding on original data to obtain global check data, and performing second coding on the original data to obtain local check data;
s102, storing the original data in one or more data partitions of a distributed system;
s103, storing the global check data and the local check data corresponding to the original data in the data partition storing the original data.
In the embodiment of the invention, the number of the original data is n as an example, n parts of original data are subjected to first coding to obtain m parts of global check data, and n parts of original data are subjected to second coding to obtain k parts of local check data; and storing the n original data into a data partition of the distributed system, wherein correspondingly, the global check data and the local check data corresponding to the original data also exist in the partition where the original data is located.
Preferably, according to the data blocks of the original data, the original data is averagely stored in each data partition of the distributed system; the local check data corresponding to the original data stored in each data partition are the same in quantity; the quantity of the global check data corresponding to the original data stored in each data partition is the same;
In the embodiment of the invention, the number of the original data is n as an example, and n pieces of original data are averagely stored in each data partition of the distributed system; the local check data stored in each data partition are the same in quantity; the number of the global check data stored in each data partition is the same.
In the embodiment of the present invention, the average storage of n pieces of raw data in each data partition of the distributed system is for load balancing, for example, 21 pieces of raw data are evenly distributed in 3 data partitions, and each partition stores 7 pieces of raw data and its corresponding global check data and local check data. In addition, the average distribution of practical application is to achieve load balancing as much as possible, for example, two data partitions are provided, 21 parts of original data are transmitted, 10 parts of original data and corresponding global check data and local check data thereof can be stored in one data partition, and 11 parts of original data and corresponding global check data and local check data thereof can be stored in the other data partition.
As shown in fig. 2, the embodiment of the present invention is 3AZ (Available Zone in the Available system), where 6 parts of original data are stored in each Zone, 2 parts of local check data are stored in each Zone (denoted as P in the figure), and 12 parts of global check data are stored in each Zone (denoted as G in the figure). The amount of data in the entire distributed system is 18+6+ 12.
Preferably, the first code is an RS code; the second code is an MSR code.
MSR encoding is a kind of regenerative code, which can significantly reduce data recovery cost by changing the conventional encoding method.
In the embodiment of the invention, local check data are generated according to the mode that a data partition uses MSR coding and are stored in the data partition; and generating global check data according to the RS coding scheme and distributing the global check data to each partition. The advantages and the disadvantages of RS coding and MSR coding are integrated, the local check idea is utilized to combine the two codes into a new code, the new code ensures the data reliability and the storage cost of the whole system, and simultaneously has the excellent performance of low single-disc recovery cost of the MSR coding, and compared with an RS coding scheme, the performance of the system after using the new code is greatly improved in a 3AZ environment; in addition, the embodiment of the invention has excellent expandability, and the parameter is applicable to various partitioned system environments after being adjusted.
Example two
As shown in fig. 3, an embodiment of the present invention provides a decoding method in a distributed system environment, including:
s201, reading original data stored in a current partition;
S202, when the damage quantity of the original data is smaller than or equal to a preset first threshold value, performing data recovery on the original data according to a second coding mode by using local check data stored in a current partition to obtain complete original data;
and S203, when the damage quantity of the original data is larger than a preset first threshold value, performing data recovery on the original data according to a first coding mode by using the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system to obtain complete original data.
In the embodiment of the invention, when the damaged quantity of the original data is greater than a preset second threshold, the original data is subjected to data recovery according to a first coding and second coding combined coding mode by using the local check data and the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system, so as to obtain complete original data, wherein the second threshold is greater than the first threshold.
In the embodiment of the invention, the first code is RS code; the second code is an MSR code.
In the embodiment of the invention, when data decoding and recovery are carried out, the discussion is carried out according to the situation, and if the quantity of the lost data is less, for example, only 1 and 2 data are in error. Only the local check data needs to be recovered by adopting the MSR coding recovery scheme, and the MSR coding optimization effect on the recovery cost can be utilized. If a large-scale data error occurs, for example, all data of one data partition fails. The global check data is needed to be recovered, and at this time, the RS coding scheme is used, so the recovery cost is the same as that of the RS coding. In some extreme cases, joint recovery of global parity data and local parity data may be required. Thereby the data reliability of the system is improved.
EXAMPLE III
As shown in fig. 4, an encoding apparatus in a distributed system environment according to an embodiment of the present invention includes:
the encoding module 100 is configured to perform first encoding on original data to obtain global check data, and perform second encoding on the original data to obtain local check data;
a partitioning module 200 configured to store the raw data in one or more data partitions of a distributed system;
the allocation module 300 is configured to store global parity data and local parity data corresponding to the original data in the data partition storing the original data.
In the embodiment of the present invention, an encoding module 100 performs first encoding on n pieces of original data to obtain m pieces of global check data, and performs second encoding on n pieces of original data to obtain k pieces of local check data; the partitioning module 200 stores n pieces of original data into a data partition of the distributed system, and the allocating module 300 stores global check data and local check data corresponding to the original data in a partition where the original data is located.
In the embodiment of the present invention, the partitioning module 200 averagely stores the original data in each data partition of the distributed system according to the data partitioning of the original data; the local check data corresponding to the original data stored in each data partition are the same in quantity; the quantity of the global check data corresponding to the original data stored in each data partition is the same.
Example four
As shown in fig. 5, an embodiment of the present invention provides a decoding apparatus in a distributed system environment, including:
a communication module 400 configured to read original data stored in a current partition;
the recovery module 500 is configured to perform data recovery on the original data according to a second coding mode by using the local check data stored in the current partition when the damaged amount of the original data is less than or equal to a preset first threshold value, so as to obtain complete original data;
the recovery module 500 is further configured to perform data recovery on the original data according to a first coding mode by using the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system when the original data damage amount is greater than a preset first threshold, so as to obtain complete original data.
In this embodiment of the present invention, the recovery module 500 is further configured to, when the damaged amount of the original data is greater than a preset second threshold, perform data recovery on the original data according to a joint coding manner of a first coding and a second coding by using the local verification data and the global verification data stored in the current partition and the global verification data stored in other data partitions of the distributed system, so as to obtain complete original data, where the second threshold is greater than the first threshold.
EXAMPLE five
As shown in fig. 2, the encoding method is as follows: and generating local check data according to the mode that the partitions use MSR codes and storing the local check data in the partitions. And generating global check data according to the RS coding scheme and distributing the global check data to each partition in an average manner. The example of FIG. 2 verifies that the data is 2 copies per partition locally. The total check data is 12 parts, and each partition stores 4 parts. The overall encoding parameter is 18+6+12, with 18 copies of the original data, on average 6 copies stored per partition.
As shown in fig. 6, a data recovery time division situation is discussed, if the amount of lost data is small, for example, only 1 or 2 data errors occur, only local encoding needs to be used to perform recovery by using an MSR encoding recovery scheme, which can use the optimization effect of MSR encoding on recovery cost. If a large-scale data error occurs, for example, a partition fails all over. The global check data is needed to be recovered, and the recovery cost is the same as that of the RS coding due to the used RS coding scheme. In some extreme cases, joint recovery may be required for global and local checks.
The embodiment of the invention integrates the advantages of RS coding and MSR coding, so that the new coding is more suitable for a system in a 3AZ environment in terms of network bandwidth and response time.
In the embodiment of the invention, the local check data is coded by using an MSR coding scheme, and compared with direct XOR operation coding, the method has obvious advantage in the occupation of network bandwidth during recovery. For example, when the partition coding configuration is 4+2, 2 parts of Local check data, 4 parts of global check data, and a scheme of LRC (Local Reconstruction Code, a new type of coding using Local check) coding need to call 4 times of data size, while MSR coding only needs to call 2.5 times of data size.
Although the embodiments of the present invention have been described above, the contents thereof are merely embodiments adopted to facilitate understanding of the technical aspects of the present invention, and are not intended to limit the present invention. It will be apparent to persons skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (10)
1. A method for encoding in a distributed system environment, comprising:
performing first coding on original data to obtain global check data, and performing second coding on the original data to obtain local check data;
Storing the raw data in one or more data partitions of a distributed system;
and storing the global check data and the local check data corresponding to the original data in the data partition storing the original data.
2. The method of claim 1, wherein the raw data is stored in each data partition of the distributed system on average, in terms of data partitions of the raw data; the local check data corresponding to the original data stored in each data partition are the same in quantity; the quantity of the global check data corresponding to the original data stored in each data partition is the same.
3. The method of claim 1 or 2, wherein the first code is an RS code; the second code is an MSR code.
4. A decoding method in a distributed system environment, comprising:
reading original data stored in a current partition;
when the damage quantity of the original data is smaller than or equal to a preset first threshold value, performing data recovery on the original data according to a second coding mode by using local check data stored in a current partition to obtain complete original data;
and when the damage quantity of the original data is larger than a preset first threshold value, performing data recovery on the original data according to a first coding mode by using the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system to obtain complete original data.
5. The method as claimed in claim 4, wherein when the original data damage amount is greater than a preset second threshold, the original data is subjected to data recovery according to a first coding and second coding joint coding mode by using the local check data and the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system, so as to obtain complete original data, wherein the second threshold is greater than the first threshold.
6. The method of claim 4 or 5, wherein the first code is an RS code; the second code is an MSR code.
7. An encoding apparatus in a distributed system environment, comprising:
the encoding module is used for carrying out first encoding on the original data to obtain global check data and carrying out second encoding on the original data to obtain local check data;
a partitioning module configured to store the raw data in one or more data partitions of a distributed system;
and the distribution module is arranged for storing the global check data and the local check data corresponding to the original data in the data partition storing the original data.
8. The apparatus of claim 7, wherein the partitioning module stores raw data in each data partition of a distributed system on average, in terms of data partitions of the raw data; the local check data corresponding to the original data stored in each data partition are the same in quantity; the quantity of the global check data corresponding to the original data stored in each data partition is the same.
9. A decoding apparatus in a distributed system environment, comprising:
the communication module is set to read the original data stored in the current partition;
the recovery module is used for performing data recovery on the original data according to a second coding mode by using the local check data stored in the current partition when the damaged quantity of the original data is smaller than or equal to a preset first threshold value to obtain complete original data;
and the recovery module is further configured to perform data recovery on the original data according to a first coding mode by using the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system when the original data damage number is larger than a preset first threshold value, so as to obtain complete original data.
10. The apparatus of claim 9, wherein the recovery module is further configured to perform data recovery on the original data according to a joint coding manner of a first coding and a second coding by using the local parity data and the global parity data stored in the current partition and the global parity data stored in other data partitions of the distributed system to obtain complete original data when the number of the damaged original data is greater than a preset second threshold, where the second threshold is greater than the first threshold.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910439739.5A CN111984443A (en) | 2019-05-24 | 2019-05-24 | Encoding method, decoding method and corresponding devices in distributed system environment |
| PCT/CN2020/090515 WO2020238653A1 (en) | 2019-05-24 | 2020-05-15 | Encoding method in distributed system environment, decoding method in distributed system environment, and corresponding apparatuses |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910439739.5A CN111984443A (en) | 2019-05-24 | 2019-05-24 | Encoding method, decoding method and corresponding devices in distributed system environment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111984443A true CN111984443A (en) | 2020-11-24 |
Family
ID=73436260
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910439739.5A Pending CN111984443A (en) | 2019-05-24 | 2019-05-24 | Encoding method, decoding method and corresponding devices in distributed system environment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111984443A (en) |
| WO (1) | WO2020238653A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113687975A (en) * | 2021-07-14 | 2021-11-23 | 重庆大学 | Data processing method, device, equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106527993A (en) * | 2016-11-09 | 2017-03-22 | 北京搜狐新媒体信息技术有限公司 | Mass file storage method and device for distributed type system |
| CN107844272A (en) * | 2017-10-31 | 2018-03-27 | 成都信息工程大学 | A kind of cross-packet coding and decoding method for improving error correcting capability |
| US20180365102A1 (en) * | 2017-06-16 | 2018-12-20 | Alibaba Group Holding Limited | Method and system for iterative data recovery and error correction in a distributed system |
| CN109491835A (en) * | 2018-10-25 | 2019-03-19 | 哈尔滨工程大学 | A kind of data fault tolerance method based on Dynamic Packet code |
-
2019
- 2019-05-24 CN CN201910439739.5A patent/CN111984443A/en active Pending
-
2020
- 2020-05-15 WO PCT/CN2020/090515 patent/WO2020238653A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106527993A (en) * | 2016-11-09 | 2017-03-22 | 北京搜狐新媒体信息技术有限公司 | Mass file storage method and device for distributed type system |
| US20180365102A1 (en) * | 2017-06-16 | 2018-12-20 | Alibaba Group Holding Limited | Method and system for iterative data recovery and error correction in a distributed system |
| CN107844272A (en) * | 2017-10-31 | 2018-03-27 | 成都信息工程大学 | A kind of cross-packet coding and decoding method for improving error correcting capability |
| CN109491835A (en) * | 2018-10-25 | 2019-03-19 | 哈尔滨工程大学 | A kind of data fault tolerance method based on Dynamic Packet code |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113687975A (en) * | 2021-07-14 | 2021-11-23 | 重庆大学 | Data processing method, device, equipment and storage medium |
| CN113687975B (en) * | 2021-07-14 | 2023-08-29 | 重庆大学 | Data processing method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2020238653A1 (en) | 2020-12-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11947423B2 (en) | Data reconstruction in distributed storage systems | |
| US9823980B2 (en) | Prioritizing data reconstruction in distributed storage systems | |
| US10353787B2 (en) | Data stripping, allocation and reconstruction | |
| KR101874280B1 (en) | Efficient data reads from distributed storage systems | |
| CN110442535B (en) | Method and system for improving reliability of distributed solid-state disk key value cache system | |
| US8977893B2 (en) | Accelerated rebuild and zero time rebuild in raid systems | |
| CN109491835B (en) | Data fault-tolerant method based on dynamic block code | |
| CN106708653B (en) | Mixed tax big data security protection method based on erasure code and multiple copies | |
| CN103838860A (en) | File storing system based on dynamic transcript strategy and storage method of file storing system | |
| CN102981927A (en) | Distribution type independent redundant disk array storage method and distribution type cluster storage system | |
| CN110427156B (en) | Partition-based MBR (Membrane biological reactor) parallel reading method | |
| US10346066B2 (en) | Efficient erasure coding of large data objects | |
| CN107153661A (en) | A kind of storage, read method and its device of the data based on HDFS systems | |
| CN115098295A (en) | Data local recovery method, equipment and storage medium | |
| CN116501553B (en) | Data recovery method, device, system, electronic equipment and storage medium | |
| EP4170499A1 (en) | Data storage method, storage system, storage device, and storage medium | |
| CN111984443A (en) | Encoding method, decoding method and corresponding devices in distributed system environment | |
| CN107885615B (en) | Distributed storage data recovery method and system | |
| CN112015325B (en) | Method for generating decoding matrix, decoding method and corresponding device | |
| CN112860476A (en) | Approximate erasure code coding method and device based on video layered storage | |
| CN108932176B (en) | Data degradation storage method and device | |
| CN111897676A (en) | File backup method and device based on database index | |
| US11914475B2 (en) | Erasure recovery in a distributed storage system | |
| US20190020359A1 (en) | Systematic coding technique for erasure correction | |
| CN104881252A (en) | Layout structure for disk array based on E code |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |