CN112799872B - A kind of erasure code encoding method and device based on key-value pair storage system - Google Patents

Abstract

The application discloses an erasure code coding method and device based on a key value pair storage system, wherein the method comprises the following steps: acquiring an incidence relation between keywords in a key value pair storage system; acquiring at least two corresponding target data with strong relevance or strong time limitation from the key value pair storage system according to the incidence relation between the keywords; dividing at least two target data into the same coding group for coding to obtain corresponding data blocks and check blocks; and writing the obtained data block and the check block into corresponding storage nodes by adopting load balancing. Through implementing this application, can combine the design with traditional erasure coding technique and novel key value to storage system, combine simultaneously to consider the relation between the data in the key value to storage system to reduce data access's number of times and time, promote data recovery efficiency.

Description

A kind of erasure code encoding method and device based on key-value pair storage system

technical field

The present application relates to the technical field of cloud storage, and in particular, to a method and device for erasure code encoding based on a key-value pair storage system.

Background technique

With the emergence of emerging storage technologies and hardware, and the performance improvement of applications by the underlying design of key-value storage systems, more and more server clusters begin to use key-value storage systems to store data. To ensure the reliability and availability of data, erasure coding technology is usually used to store data.

However, the traditional erasure coding technology is mainly aimed at the traditional fixed-length block storage devices. When faced with the non-fixed-length block storage background of the new key-value storage system, it cannot be well adapted to the key-value storage system. and provide technical support. Moreover, there is always an association between different blocks in the key-value pair storage system, which is difficult to reflect and utilize in traditional erasure coding technology.

Therefore, there is an urgent need to propose an erasure coding scheme suitable for key-value pair storage systems.

SUMMARY OF THE INVENTION

In order to overcome the above-mentioned deficiencies in the prior art, the purpose of this application is to provide an erasure coding method and device based on a key-value pair storage system, combining traditional erasure coding technology with a novel key-value pair storage system to design, Considering the relationship between data in the key-value pair storage system, it can reduce the number and time of data access and improve the efficiency of data recovery.

In order to achieve the above-mentioned and other purposes, the present application proposes a method for encoding erasure codes based on a key-value pair storage system, comprising the following steps:

Obtain the association relationship between keywords in the key-value pair storage system, the association relationship is used to indicate that the data corresponding to the keywords have strong association or strong time locality, and the key-value pair storage system uses the key to store data in the form of value pairs;

Acquire corresponding at least two target data with strong correlation or strong time limitation from the key-value pair storage system according to the association relationship between the keywords;

Divide at least two of the target data into the same encoding group for encoding to obtain corresponding data blocks and check blocks;

The obtained data block and the check block are written into the corresponding storage node by using load balancing.

Optionally, the association relationship includes at least one of the following: a parent-child inclusion relationship, a relationship with strong access correlation, or a relationship with strong access sequence.

Optionally, the strong time limitation between the data is obtained by analyzing data access characteristics, and the strong correlation between the data is obtained by analyzing the attribute information of the data.

Optionally, the key-value pair storage system stores keywords, values corresponding to the keywords, correspondence between keywords and values, and associations between keywords.

Optionally, the data block stores at least two associated data of the target data, and the method further includes:

Receive a data recovery request, where the data recovery request is used to request to read the data block or check block in the target node, so as to realize the recovery of lost data based on the data block or check block, and the lost data is any Part of the data in the target data;

In response to the data recovery request, perform recovery calculation on the read data block or check block in the target node to obtain the lost data, and at the same time according to the associated data as having a strong association with any of the target data Prepare for access to the next target data with sexual or strong time constraints.

In order to achieve the above and other purposes, the present application also provides an erasure code encoding device based on a key-value pair storage system, including:

The obtaining unit is used to obtain the association relationship between keywords in the key-value pair storage system, and the association relationship is used to indicate that the data corresponding to the keywords have strong association or strong temporal locality, and the key value The storage system uses the form of key-value pairs to store data;

The obtaining unit is further configured to obtain at least two corresponding target data with strong correlation or strong time limitation from the key-value pair storage system according to the association relationship between the keywords;

an encoding unit, configured to divide at least two of the target data into the same encoding group for encoding to obtain corresponding data blocks and check blocks;

The writing unit is configured to use load balancing to write the obtained data block and the check block into the corresponding storage node.

Optionally, the association relationship includes at least one of the following: a parent-child inclusion relationship, a relationship with strong access correlation, or a relationship with strong access sequence.

Optionally, the strong time limitation between the data is obtained by analyzing data access characteristics, and the strong correlation between the data is obtained by analyzing the attribute information of the data.

Optionally, the key-value pair storage system stores keywords, values corresponding to the keywords, correspondence between keywords and values, and associations between keywords.

Optionally, at least two associated data of the target data are stored in the data block, and the device further includes a receiving unit and a restoring unit,

The receiving unit is further configured to receive a data recovery request, where the data recovery request is used to request to read a data block or a check block in the target node, so as to realize the recovery of lost data based on the data block or the check block, The missing data is any partial data in the target data;

The recovery unit is further configured to, in response to the data recovery request, perform recovery calculation on the read data block or check block in the target node, to obtain the lost data, and at the same time according to the associated data to be related to any data block. One of the target data has a strong correlation or strong time constraints to prepare for the access of the next target data.

It can be seen from the above that the present application provides an erasure code encoding method and device based on a key-value pair storage system, which can achieve the following beneficial effects: the present application combines the traditional erasure code technology with a novel key-value pair storage system to design, At the same time, the relationship between data in the key-value pair storage system is introduced into the design consideration, so that the key-value pair storage system can also use the erasure coding technology for data recovery, so that the strongly correlated data can be recovered faster during data recovery. Compared with the existing technology, the efficiency of data recovery is improved to ensure better service quality for users. In addition, since the present application puts data with strong correlation or strong time limitation into the same coding group for coding, the access times of nodes and the access time of the entire data access can also be reduced during data access.

Description of drawings

FIG. 1 is a schematic flowchart of a method for encoding erasure codes based on a key-value pair storage system provided by an embodiment of the present application.

FIG. 2 is a schematic structural diagram of an erasure correction code encoding apparatus based on a key-value pair storage system provided by an embodiment of the present application.

FIG. 3 is a schematic structural diagram of another erasure code encoding apparatus based on a key-value pair storage system provided by an embodiment of the present application.

Detailed ways

The embodiments of the present application will be described below with reference to specific examples and accompanying drawings, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific examples, and various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the spirit of the present application.

Please refer to FIG. 1 , which is a schematic flowchart of an erasure code encoding method based on a key-value pair storage system provided by an embodiment of the present application. The method shown in FIG. 1 includes the following implementation steps.

S101. Obtain an association relationship between keywords in a key-value pair storage system, where the association relationship is used to indicate that the data corresponding to the keywords have strong association or strong temporal locality, and the key-value pair storage system Data is stored in the form of key-value pairs.

The key-value pair (key-value) storage system of the present application adopts the form of key-value pair (key-value) to store data. A lot of data content is stored in the key-value pair storage system. For example, it may include but not limited to a keyword (key), a value (value) corresponding to a keyword (key), and the existence of a keyword (key) and value (value). Correspondence, the association between keywords (keys) and keywords (keys), etc.

In an optional embodiment, when the data is stored, the system can determine or obtain at least one associated data that has strong temporal locality with the data by analyzing the access characteristics of the data; At least one associated data with strong correlation, and then store the data in the form of key-value pairs, and create corresponding associations between the keywords corresponding to the data and the keywords corresponding to each associated data, and store them in the key-value on the storage system. It is convenient to obtain corresponding data from the key-value pair storage system for subsequent processing.

The association relationship between keywords involved in this application includes, but is not limited to, any one or a combination of the following relationships: parent-child inclusion relationship, relationship with strong access correlation, or relationship with strong access sequence, etc. .

S102. Acquire corresponding at least two target data with strong correlation or strong time limitation from the key-value pair storage system according to the association relationship between the keywords.

The erasure coding system will segment the data when storing the data. Specifically, this application will base on the association relationship between keywords (such as parent-child inclusion relationship, or the strong association between the corresponding data of the two keys when they are accessed). or strong sequential access relationship, etc.), obtain at least two target data with strong correlation or strong time limitation corresponding to the correlation relationship.

S103: Divide at least two of the target data into the same encoding group for encoding to obtain corresponding data blocks and check blocks.

In the present application, at least two target data with strong correlation or strong time limitation can be divided into the same coding group, and then the coding operation can be performed to obtain the corresponding data block and check block by coding. In other words, when the present application performs erasure code encoding, at least two target data with a relatively close relationship can be put into the same encoding group for encoding according to the relationship between the data.

S04, using load balancing to write the obtained data block and the check block into a corresponding storage node.

Finally, the present application can use the load balancing technology to write the data blocks and check blocks obtained by encoding into corresponding different storage nodes (eg, disks). Optionally, any data block obtained after encoding includes associated data for the at least two target data, and the associated data is used to indicate that there is a strong correlation or strong time limitation between the two target data. .

It should be noted that, since the data access in the erasure coding system requires data to be accessed from different storage receptions, in this application, after the related data are divided into the same coding group for coding and storage, when the data is accessed The number of accesses to storage nodes can be reduced/reduced, and the data access time can be reduced. For example, taking data A and data B that are related as an example, when a certain data A is accessed, there is a high probability that data B will be accessed immediately. The traditional erasure coding scheme does not consider this point, and the scheme of the present application considers that the data A and the data B are encoded and stored in the same erasure coding group. Therefore, when accessing data A, data blocks related to accessing data B can be obtained at the same time, and preparations for accessing data B can be made in advance, so that the data B that may need to be accessed immediately in a short period of time can be better obtained, and the data access efficiency can be improved.

In an optional embodiment, since the erasure coding scheme designed in the present application takes into account the relationship between data, the strongly correlated data can be recovered at the same time faster during data recovery. Compared with the traditional random grouping technology, the It has higher data recovery efficiency and provides users with better service quality. During specific implementation, the present application can receive a data recovery request, the data recovery request is used to request the recovery of the lost book data in the damaged node, which can specifically be used to request to read the corresponding data block or checksum from the corresponding target node block, and then use these read data blocks and check blocks to perform calculations to recover the corresponding lost data. Further, in response to the data recovery request, the corresponding data blocks and check blocks can be read from the corresponding target nodes, and then the data blocks and check blocks can be recovered and calculated to obtain the lost data. At the same time, according to the associated data in the read data block, preparations are made for the access of the next target data that is strongly associated with the current data or has a strong time limitation.

By implementing this application, the application combines the traditional erasure coding technology with the new key-value pair storage system, and at the same time introduces the relationship between data in the key-value pair storage system into design considerations, so that the key-value pair storage system can also be used Erasure code technology is used for data recovery, so that strongly related data can be recovered more quickly during data recovery. Compared with the existing technology, the efficiency of data recovery is improved to ensure better service for users. quality. In addition, since the present application puts data with strong correlation or strong time limitation into the same coding group for coding, the access times of nodes and the access time of the entire data access can also be reduced during data access.

Please refer to FIG. 2 , which is a schematic structural diagram of an erasure correction code encoding apparatus based on a key-value pair storage system provided by an embodiment of the present application. The apparatus shown in FIG. 2 includes an acquisition unit 201, an encoding unit 202 and a writing unit, wherein:

The obtaining unit 201 is configured to obtain the association relationship between keywords in the key-value pair storage system, where the association relationship is used to indicate that the data corresponding to the keywords have strong association or strong temporal locality, so The key-value pair storage system uses the form of key-value pairs to store data;

The obtaining unit 201 is further configured to obtain at least two corresponding target data with strong correlation or strong time limitation from the key-value pair storage system according to the association relationship between the keywords;

The encoding unit 202 is configured to divide at least two of the target data into the same encoding group for encoding to obtain corresponding data blocks and check blocks;

The writing unit 203 is configured to use load balancing to write the obtained data block and the check block into the corresponding storage node.

Optionally, the association relationship includes at least one of the following: a parent-child inclusion relationship, a relationship with strong access correlation, or a relationship with strong access sequence.

Optionally, the strong time limitation between the data is obtained by analyzing data access characteristics, and the strong correlation between the data is obtained by analyzing the attribute information of the data.

Optionally, the key-value pair storage system stores keywords, values corresponding to the keywords, correspondence between keywords and values, and associations between keywords.

Optionally, at least two associated data of the target data are stored in the data block, and the apparatus further includes a receiving unit 204 and a restoring unit 205,

The receiving unit 204 is further configured to receive a data recovery request, where the data recovery request is used to request to read a data block or a check block in the target node, so as to realize the recovery of lost data based on the data block or check block , the missing data is any partial data in the target data;

The recovery unit 205 is further configured to, in response to the data recovery request, perform recovery calculation on the read data block or check block in the target node, to obtain the lost data, and at the same time according to the associated data as the Any one of the target data has a strong correlation or a strong time limitation to prepare for the access of the next target data.

By implementing this application, the application combines the traditional erasure coding technology with the new key-value pair storage system, and at the same time introduces the relationship between data in the key-value pair storage system into design considerations, so that the key-value pair storage system can also be used Erasure code technology is used for data recovery, so that strongly related data can be recovered faster during data recovery. Compared with the existing technology, the efficiency of data recovery is improved to ensure better service for users. quality. In addition, since the present application puts data with strong correlation or strong time limitation into the same coding group for coding, the access times of nodes and the access time of the entire data access can also be reduced during data access.

The above-mentioned embodiments merely illustrate the principles and effects of the present application, but are not intended to limit the present application. Any person skilled in the art can modify and change the above embodiments without departing from the spirit and scope of the present application. Therefore, the scope of protection of the right of this application should be as listed in the claims.

Claims (8)

1. a kind of erasure code coding method based on key-value pair storage system, is characterized in that, comprises: Obtain the association relationship between keywords in the key-value pair storage system, the association relationship is used to indicate that the data corresponding to the keywords have strong association or strong time locality, and the key-value pair storage system uses the key to store data in the form of value pairs; Acquire corresponding at least two target data with strong correlation or strong time limitation from the key-value pair storage system according to the association relationship between the keywords; Divide at least two of the target data into the same encoding group for encoding to obtain corresponding data blocks and check blocks; Using load balancing to write the obtained data block and the verification block into the corresponding storage node; The association relationship includes at least one of the following: a parent-child inclusion relationship, a relationship with strong access correlation, or a relationship with strong access sequence. 2. the erasure coding method based on key-value pair storage system according to claim 1, is characterized in that, having strong time limitation between described data is obtained by analyzing data access characteristic, between described data. Strong correlation is obtained by analyzing the attribute information of the data. 3. the erasure code coding method based on key-value pair storage system according to claim 1, is characterized in that, in described key-value pair storage system, there is a keyword, the value corresponding to the keyword, the difference between the keyword and the value. Correspondence between and the relationship between keywords. 4. the erasure coding method based on key-value pair storage system according to any one of claims 1-3, is characterized in that, in described data block, the associated data of at least two described target data is stored, The method also includes: Receive a data recovery request, where the data recovery request is used to request to read the data block or check block in the target node, so as to realize the recovery of lost data based on the data block or check block, and the lost data is any Part of the data in the target data; In response to the data recovery request, perform recovery calculation on the read data block or check block in the target node to obtain the lost data, and at the same time according to the associated data as having a strong association with any of the target data Prepare for access to the next target data with sexual or strong time constraints. 5. a kind of erasure code coding device based on key-value pair storage system, is characterized in that, comprises: The obtaining unit is used to obtain the association relationship between keywords in the key-value pair storage system, and the association relationship is used to indicate that the data corresponding to the keywords have strong association or strong temporal locality, and the key value The storage system uses the form of key-value pairs to store data; The obtaining unit is further configured to obtain at least two corresponding target data with strong correlation or strong time limitation from the key-value pair storage system according to the association relationship between the keywords; an encoding unit, configured to divide at least two of the target data into the same encoding group for encoding to obtain corresponding data blocks and check blocks; a writing unit, configured to use load balancing to write the obtained data block and the check block into the corresponding storage node; The association relationship includes at least one of the following: a parent-child inclusion relationship, a relationship with strong access correlation, or a relationship with strong access sequence. 6. The erasure code encoding device based on a key-value pair storage system according to claim 5, wherein the data with strong time limitation is obtained by analyzing data access characteristics, and between the data Strong correlation is obtained by analyzing the attribute information of the data. 7. The erasure code encoding device based on a key-value pair storage system according to claim 5, wherein, in the key-value pair storage system, a keyword, a value corresponding to the keyword, a keyword and a value are stored in the key-value pair storage system. Correspondence between and the relationship between keywords. 8. The erasure code encoding device based on a key-value pair storage system according to any one of claims 5-7, wherein the data block stores at least two associated data of the target data, The device also includes a receiving unit and a recovery unit, The receiving unit is further configured to receive a data recovery request, where the data recovery request is used to request to read a data block or a check block in the target node, so as to realize the recovery of lost data based on the data block or the check block, The missing data is any partial data in the target data; The recovery unit is further configured to, in response to the data recovery request, perform recovery calculation on the read data block or check block in the target node, to obtain the lost data, and at the same time according to the associated data to be related to any data block. One of the target data has a strong correlation or strong time constraints to prepare for the access of the next target data.

Images