WO2020238653A1

WO2020238653A1 - Encoding method in distributed system environment, decoding method in distributed system environment, and corresponding apparatuses

Info

Publication number: WO2020238653A1
Application number: PCT/CN2020/090515
Authority: WO
Inventors: 董元元; 赵亚飞
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2019-05-24
Filing date: 2020-05-15
Publication date: 2020-12-03
Also published as: CN111984443A

Abstract

An encoding method in a distributed system environment, a decoding method in a distributed system environment, and corresponding apparatuses, relating to the technical field of encoding fault tolerance. The encoding method comprises: performing first encoding on original data to obtain global verification data, and performing second encoding on the original data to obtain local verification data (S101); storing the original data in one or more data partitions of a distributed system (S102); and storing, in the data partition where the original data is stored, the global verification data and the local verification data corresponding to the original data (S103). The recovery cost of a distributed system environment is optimized, without affecting the system fault tolerance capability and storage cost.

Description

Encoding method, decoding method and corresponding device under distributed system environment

This application claims the priority of a Chinese patent application filed on May 24, 2019 with the application number 201910439739.5 and the invention title "A coding method, decoding method and corresponding device in a distributed system environment", the entire content of which is by reference Incorporated in this application.

Technical field

The present invention relates to the technical field of coding fault tolerance, in particular to a coding method, a decoding method and a corresponding device in a distributed system environment.

Background technique

At present, the storage scale of distributed systems is becoming larger and larger; and equipment errors in distributed systems are also a problem that cannot be ignored. Therefore, the storage cost and reliability of data are factors that need to be considered when designing a distributed system. Erasure codes can minimize the storage overhead of the system on the premise of ensuring the same data reliability.

Erasure codes can add n pieces of original data to m pieces of coded data (used to store correction codes), and can restore to original data through any n pieces of data in n+m pieces. This erasure code is also called the maximum distance separable code. The most widely used traditional encoding in the industry is RS encoding. It can tolerate any number of disk errors by adjusting the parameters k and m, but the RS encoding has obvious defects, which are mainly reflected in its performance of recovering data. RS encoding to restore any data disk with errors requires data of k data disks. The amplification of the recovery cost will take up additional network bandwidth, IO resources and recovery time. In a distributed system environment, calling data across regions will take up more network bandwidth and the response time will be further improved.

Summary of the invention

The present application provides an encoding method, a decoding method, and a corresponding device in a distributed system environment, for the distributed system environment, the restoration cost is optimized without affecting the system's fault tolerance and storage cost.

The technical solutions adopted are as follows:

In the first aspect, the present invention provides an encoding method in a distributed system environment, including:

Perform a first encoding on the original data to obtain global verification data, and perform a second encoding on the original data to obtain local verification data;

Save the original data in one or more data partitions of the distributed system;

The global check data and local check data corresponding to the original data are stored in the data partition where the original data is stored.

Preferably, according to the data block of the original data, the original data is evenly stored in each data partition of the distributed system; each data partition stores the same amount of local verification data corresponding to the original data; The number of global check data corresponding to the original data stored in each data partition is the same.

Preferably, the first code is RS code; the second code is MSR code.

In the second aspect, the present invention provides a decoding method in a distributed system environment, including:

Read the original data stored in the current partition;

When the amount of damage to the original data is less than or equal to the preset first threshold, use the local check data stored in the current partition to perform data recovery on the original data in the second encoding mode to obtain complete original data;

When the amount of damage to the original data is greater than the preset first threshold, use the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system to perform the first encoding method on the original data Perform data recovery and obtain complete original data.

Preferably, when the number of original data damages is greater than the preset second threshold, the local check data and global check data stored in the current partition and the global check data stored in other data partitions of the distributed system are used to compare all the data. The original data is restored in a joint encoding manner of the first encoding and the second encoding to obtain complete original data, wherein the second threshold is greater than the first threshold.

Preferably, the first code is RS code; the second code is MSR code.

In a third aspect, the present invention provides an encoding device in a distributed system environment, including:

The encoding module is configured to perform the first encoding on the original data to obtain the global verification data, and perform the second encoding on the original data to obtain the local verification data;

The partition module is configured to store the original data in one or more data partitions of the distributed system;

The distribution module is configured to store the global check data and the local check data corresponding to the original data in the data partition where the original data is stored.

Preferably, the partition module stores the original data in each data partition of the distributed system on average according to the data partition of the original data; each data partition stores the local verification data corresponding to the original data. The number is the same; the number of global check data corresponding to the original data stored in each data partition is the same.

In a fourth aspect, the present invention provides a decoding device in a distributed system environment, including:

The communication module is set to read the original data stored in the current partition;

The recovery module is configured to use the local verification data stored in the current partition to recover the original data according to the second encoding method when the number of damages to the original data is less than or equal to the preset first threshold to obtain complete original data ；

The recovery module is further configured to use the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system to compare the original data when the amount of damage to the original data is greater than the preset first threshold. The data is restored according to the first encoding method to obtain complete original data.

Preferably, the recovery module is further configured to use the local check data and global check data stored in the current partition and the data stored in other data partitions of the distributed system when the number of original data damages is greater than the preset second threshold. The global check data performs data recovery on the original data in a joint encoding manner of the first encoding and the second encoding to obtain complete original data, wherein the second threshold is greater than the first threshold.

Compared with the prior art, this application has the following beneficial effects:

This application uses the idea of local verification to combine RS coding and MSR coding into a new coding form. The new encoding format guarantees the data reliability and storage cost of the entire distributed system, and at the same time it also has the excellent performance of low-cost single-disk recovery of MSR encoding. In a distributed system environment, compared with the traditional RS coding scheme, the performance of the system will be greatly improved after using the new coding; and it has excellent scalability, and can be applied to various partitioned system environments after adjusting the parameters.

Description of the drawings

Figure 1 is a flowchart of an encoding method in a distributed system environment according to an embodiment of the present invention;

2 is a schematic diagram of the distributed system partitioning storage of original data and verification data according to an embodiment of the present invention;

3 is a flowchart of a decoding method in a distributed system environment according to an embodiment of the present invention;

Figure 4 is a schematic structural diagram of an encoding device in a distributed system environment according to an embodiment of the present invention;

5 is a schematic structural diagram of a decoding method in a distributed system environment according to an embodiment of the present invention;

Fig. 6 is a flowchart of a decoding process in a distributed system environment according to an embodiment of the present invention.

Detailed ways

The technical solution of the present application will be described in more detail below with reference to the drawings and embodiments.

It should be noted that, if there is no conflict, the embodiments of the present application and various features in the embodiments can be combined with each other, and all fall within the protection scope of the present application. In addition, although a logical sequence is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than here.

In one configuration, the distributed system includes multiple data partitions, and each data partition includes one or more data disks for storing data. Each data disk may include one or more processors (CPU), input/output Output interface, network interface and memory (memory).

The memory may include non-permanent memory in a computer readable medium, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer readable media. The memory may include one or more modules.

Computer-readable media include permanent and non-permanent, removable and non-removable storage media, and information storage can be realized by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, Magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media can be used to store information that can be accessed by computing devices.

Example one

As shown in Fig. 1, an embodiment of the present invention provides an encoding method in a distributed system environment, including:

S101. Perform a first encoding on the original data to obtain global verification data, and perform a second encoding on the original data to obtain local verification data;

S102. Save the original data in one or more data partitions of the distributed system.

S103: Store the global check data and the local check data corresponding to the original data in the data partition where the original data is stored.

In the embodiment of the present invention, the number of original data is n as an example, the first encoding is performed on n pieces of original data to obtain m pieces of global check data, and the n pieces of original data are subjected to second coding to obtain k pieces of local check. Verify data; save n copies of original data in the data partition of the distributed system. Correspondingly, the global verification data and local verification data corresponding to the original data also exist in the partition where the original data is located.

Preferably, according to the data block of the original data, the original data is evenly stored in each data partition of the distributed system; each data partition stores the same amount of local verification data corresponding to the original data; The number of global check data corresponding to the original data stored in each data partition is the same;

In the embodiment of the present invention, the number of original data is n as an example, n pieces of original data are averagely stored in each data partition of the distributed system; the number of local verification data stored in each data partition is the same; The number of global check data stored in each data partition is the same.

In the embodiment of the present invention, n pieces of original data are evenly stored in each data partition of the distributed system for load balancing. For example, 21 pieces of original data are equally distributed among 3 data partitions, and each partition stores 7 pieces of original data. And its corresponding global check data and local check data. In addition, the actual application is evenly distributed, try to achieve load balancing, for example, there are two data partitions, 21 copies of original data can be transmitted, 10 copies of original data and its corresponding global check data and local check data can be stored in one data partition. Another data partition stores 11 pieces of original data and its corresponding global check data and local check data.

As shown in Figure 2, the embodiment of the present invention is 3AZ (Available Zone refers to the available partitions in the system), where the original data has 6 copies for each partition, and the local check data is 2 copies for each partition (denoted as P in the figure) There are 12 copies of global check data, and each partition stores 4 copies (denoted as G in the figure). The number of data in the entire distributed system is 18+6+12.

Preferably, the first code is RS code; the second code is MSR code.

MSR coding is a kind of regeneration code, which can significantly reduce the cost of data recovery by changing the traditional coding method.

In the embodiment of the present invention, the local verification data is generated according to the data partition using MSR encoding and stored in the data partition; the global verification data is generated according to the RS encoding scheme and distributed to each partition. Integrating the advantages and disadvantages of RS coding and MSR coding, using the idea of local verification to combine the two codes into a new code. The new code guarantees the data reliability and storage cost of the entire system and also has MSR. The coded single-disk recovery has excellent performance with low cost. Compared with the RS coding scheme under the 3AZ environment, the system performance will be greatly improved after the new code is used; in addition, the embodiment of the present invention has excellent scalability. It can be applied to various partitioned system environments.

Example two

As shown in Figure 3, an embodiment of the present invention provides a decoding method in a distributed system environment, including:

S201: Read the original data stored in the current partition;

S202: When the amount of damage to the original data is less than or equal to the preset first threshold, use the local verification data stored in the current partition to perform data recovery on the original data in a second encoding mode to obtain complete original data;

S203. When the amount of damage to the original data is greater than the preset first threshold, use the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system to compare the original data according to the first Encoding method for data recovery to obtain complete original data.

In the embodiment of the present invention, when the original data damage quantity is greater than the preset second threshold, the local check data and global check data stored in the current partition and the global check data stored in other data partitions of the distributed system are used. The data performs data recovery on the original data according to the first coding and the second coding joint coding mode to obtain complete original data, wherein the second threshold is greater than the first threshold.

In the embodiment of the present invention, the first encoding is RS encoding; the second encoding is MSR encoding.

In the embodiment of the present invention, when data decoding and recovery are performed, the situation is discussed. If the amount of lost data is small, for example, only one or two pieces of data are wrong. It is only necessary to use the local verification data to use the MSR code recovery scheme to recover, and then the optimization effect of the MSR code on the recovery cost can be used. If a large-scale data error occurs, for example, all data in a data partition fails. The global check data needs to be used for recovery. In this case, the RS encoding scheme is used, so the recovery cost is the same as the RS encoding. In some extreme cases, it may be necessary to perform joint recovery of the global check data and the local check data. Based on the data reliability of the system.

Example three

As shown in Figure 4, an embodiment of the present invention provides an encoding device in a distributed system environment, including:

The encoding module 100 is configured to perform the first encoding on the original data to obtain the global verification data, and to perform the second encoding on the original data to obtain the local verification data;

The partition module 200 is configured to store the original data in one or more data partitions of the distributed system;

The distribution module 300 is configured to store the global check data and the local check data corresponding to the original data in the data partition where the original data is stored.

In the embodiment of the present invention, the encoding module 100 performs first encoding on n pieces of original data to obtain m pieces of global verification data, and performs second encoding on n pieces of original data to obtain k pieces of local verification data; The n pieces of original data are stored in the data partition of the distributed system, and the distribution module 300 stores the global check data and the local check data corresponding to the original data in the partition where the original data is located.

In the embodiment of the present invention, the partition module 200 stores the original data in each data partition of the distributed system on average according to the data partition of the original data; each data partition stores the local data corresponding to the original data. The number of check data is the same; the number of global check data corresponding to the original data stored in each data partition is the same.

Example four

As shown in FIG. 5, an embodiment of the present invention provides a decoding device in a distributed system environment, including:

The communication module 400 is configured to read the original data stored in the current partition;

The restoration module 500 is configured to use the local check data stored in the current partition to restore the original data according to the second encoding method when the amount of damage to the original data is less than or equal to the preset first threshold to obtain a complete original data;

The recovery module 500 is further configured to use the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system to compare the damage to the original data when the number of original data damages is greater than the preset first threshold The original data is restored according to the first encoding method to obtain complete original data.

In the embodiment of the present invention, the recovery module 500 is further configured to use the local check data and global check data stored in the current partition and other data of the distributed system when the original data damage quantity is greater than the preset second threshold. The global check data stored in the partition performs data recovery on the original data according to a joint encoding method of the first encoding and the second encoding to obtain complete original data, wherein the second threshold is greater than the first threshold.

Example five

As shown in Figure 2, the encoding method is as follows: Local verification data is generated and stored in the corresponding partition according to the partition using MSR encoding. According to the RS coding scheme, global check data is generated and distributed evenly among the partitions. In the example shown in Figure 2, the local verification data has 2 copies for each partition. There are 12 copies of global verification data, and 4 copies are stored in each partition. The entire coding parameter is 18+6+12, of which 18 original data are stored, and each partition stores 6 copies on average.

As shown in Figure 6, the time division of data recovery is discussed. If the amount of lost data is small, for example, only one or two pieces of data are wrong, you only need to use the local encoding and use the MSR encoding recovery scheme to recover, which can be used The optimization effect of MSR coding on recovery cost. If there is a large-scale data error, such as a partition failure. It is necessary to use the global check data for recovery. Because of the RS encoding scheme used, the recovery cost is the same as the RS encoding. In some extreme cases, global check and local check may be required for joint recovery.

The embodiments of the present invention integrate the advantages of RS coding and MSR coding, so that the new coding is more suitable for systems in a 3AZ environment in terms of network bandwidth and response time.

In the embodiment of the present invention, the local check data is encoded using the MSR encoding scheme, which has obvious advantages in terms of network bandwidth occupation during restoration, compared with the direct exclusive OR operation encoding. For example, when the partition code is configured as 4+2, 2 copies of local verification data and 4 copies of global verification data. The LRC (Local Reconstruction Code, a new type of code that uses local verification) coding scheme requires 4 times the amount of data to be called. The MSR encoding only needs to call 2.5 times the amount of data.

Although the embodiments disclosed in the present invention are as described above, the contents are only used to facilitate the understanding of the technical solutions of the present invention and are not used to limit the present invention. Any person skilled in the technical field of the present invention can make any modifications and changes in the implementation form and details without departing from the core technical solution disclosed in the present invention. However, the protection scope defined by the present invention remains The scope defined by the appended claims shall prevail.

Claims

A coding method in a distributed system environment, which is characterized in that it includes:

Perform a first encoding on the original data to obtain global verification data, and perform a second encoding on the original data to obtain local verification data;

Save the original data in one or more data partitions of the distributed system;

The global check data and local check data corresponding to the original data are stored in the data partition where the original data is stored.
The method according to claim 1, wherein the original data is stored in each data partition of the distributed system on average according to the data partition of the original data; each data partition stores the corresponding original data The number of local verification data is the same; the number of global verification data corresponding to the original data stored in each data partition is the same.
The method according to claim 1 or 2, wherein the first encoding is RS encoding; and the second encoding is MSR encoding.
A decoding method in a distributed system environment, characterized in that it includes:

Read the original data stored in the current partition;

When the amount of damage to the original data is less than or equal to the preset first threshold, use the local check data stored in the current partition to perform data recovery on the original data in the second encoding mode to obtain complete original data;

When the amount of damage to the original data is greater than the preset first threshold, use the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system to perform the first encoding method on the original data Perform data recovery and obtain complete original data.
The method of claim 4, wherein when the number of original data damages is greater than the preset second threshold, the local check data and global check data stored in the current partition and other data of the distributed system are used The global check data stored in the partition performs data recovery on the original data in a joint encoding manner of the first encoding and the second encoding to obtain complete original data, wherein the second threshold is greater than the first threshold.
The method according to claim 4 or 5, wherein the first encoding is RS encoding; and the second encoding is MSR encoding.
An encoding device in a distributed system environment, characterized in that it comprises:

The encoding module is configured to perform the first encoding on the original data to obtain the global verification data, and perform the second encoding on the original data to obtain the local verification data;

The partition module is configured to store the original data in one or more data partitions of the distributed system;

The distribution module is configured to store the global check data and the local check data corresponding to the original data in the data partition where the original data is stored.
The device according to claim 7, wherein the partition module stores the original data in each data partition of the distributed system on average according to the data partition of the original data; each data partition stores the The quantity of local check data corresponding to the original data is the same; the quantity of global check data corresponding to the original data stored in each data partition is the same.
A decoding device in a distributed system environment, characterized in that it comprises:

The communication module is set to read the original data stored in the current partition;

The recovery module is configured to use the local check data stored in the current partition to recover the original data according to the second encoding method when the number of damages to the original data is less than or equal to the preset first threshold to obtain complete original data ；

The recovery module is further configured to use the global check data stored in the current partition and the global check data stored in other data partitions of the distributed system to compare the original data when the amount of damage to the original data is greater than the preset first threshold. The data is restored according to the first encoding method to obtain complete original data.
The device according to claim 9, wherein the recovery module is further configured to use the local check data and the global check data stored in the current partition when the number of original data damages is greater than a preset second threshold. The global check data stored in other data partitions of the distributed system performs data recovery on the original data according to the first coding and the second coding joint coding mode to obtain complete original data, wherein the second threshold is greater than the first threshold.