CN116107797A

CN116107797A - Data storage method and device, electronic equipment and storage medium

Info

Publication number: CN116107797A
Application number: CN202111320489.7A
Authority: CN
Inventors: 梁松涛; 何钧; 王一; 周全
Original assignee: Shanghai Bilibili Technology Co Ltd
Current assignee: Shanghai Bilibili Technology Co Ltd
Priority date: 2021-11-09
Filing date: 2021-11-09
Publication date: 2023-05-12
Also published as: WO2023082629A1

Abstract

The disclosure provides a data storage method, a data storage device, data storage equipment and a data storage medium, and relates to the technical field of distributed storage. The implementation scheme is as follows: acquiring a plurality of data blocks to be stored and storage requirement information of the plurality of data blocks; determining a target encoding strategy of the plurality of data blocks based on the storage requirement information; encoding the plurality of data blocks based on a target encoding strategy to generate a plurality of check blocks corresponding to the plurality of data blocks; and storing the plurality of data blocks and the plurality of check blocks in association.

Description

Data storage method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of distributed storage technologies, and in particular, to a data storage method and apparatus, an electronic device, a computer readable storage medium, and a computer program product.

Background

Distributed storage is a storage technique that disperses data across multiple independent devices. Accordingly, a distributed storage system is a storage system employing distributed storage technology that includes multiple devices for storing data. In a distributed storage system, data is stored in a redundant manner, so that reliability and fault tolerance of the distributed storage system are guaranteed, and data loss caused by errors of partial equipment is avoided.

The approaches described in this section are not necessarily approaches that have been previously conceived or pursued. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, the problems mentioned in this section should not be considered as having been recognized in any prior art unless otherwise indicated.

Disclosure of Invention

The present disclosure provides a data storage method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

According to an aspect of the present disclosure, there is provided a data storage method including: acquiring a plurality of data blocks to be stored and storage requirement information of the plurality of data blocks; determining a target encoding strategy of the plurality of data blocks based on the storage requirement information; encoding the plurality of data blocks based on the target encoding strategy to generate a plurality of check blocks corresponding to the plurality of data blocks; and storing the plurality of data blocks and the plurality of check blocks in association.

According to another aspect of the present disclosure, there is also provided a data storage device including: an acquisition unit configured to acquire a plurality of data blocks to be stored and storage demand information of the plurality of data blocks; a determining unit configured to determine a target encoding policy of the plurality of data blocks based on the storage demand information; an encoding unit configured to encode the plurality of data blocks based on the target encoding policy to generate a plurality of check blocks corresponding to the plurality of data blocks; and a storage unit configured to store the plurality of data blocks and the plurality of check blocks in association.

According to another aspect of the present disclosure, there is also provided an electronic apparatus including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores a computer program which, when executed by the at least one processor, implements a method according to the above.

According to another aspect of the present disclosure, there is also provided a non-transitory computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements a method according to the above.

According to another aspect of the present disclosure, there is also provided a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements a method according to the above.

According to one or more embodiments of the present disclosure, an appropriate encoding policy can be dynamically selected based on storage demand information of a user, so that flexible customization of the encoding policy is realized, and different demands of different users on data storage can be satisfied, thereby avoiding unnecessary storage space waste, improving storage efficiency, and reducing storage cost.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The accompanying drawings illustrate exemplary embodiments and, together with the description, serve to explain exemplary implementations of the embodiments. The illustrated embodiments are for exemplary purposes only and do not limit the scope of the claims. Throughout the drawings, identical reference numerals designate similar, but not necessarily identical, elements.

FIG. 1 illustrates a flow chart of a data storage method according to an embodiment of the present disclosure;

FIG. 2 illustrates a schematic diagram of stripe storage according to an embodiment of the present disclosure;

FIG. 3 illustrates a block diagram of a storage system according to some embodiments of the present disclosure;

FIG. 4 illustrates a block diagram of a storage system according to further embodiments of the present disclosure;

FIG. 5 illustrates a block diagram of a storage system according to further embodiments of the present disclosure;

FIG. 6 illustrates a block diagram of a data storage device according to an embodiment of the present disclosure; and

fig. 7 illustrates a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

In the present disclosure, the use of the terms "first," "second," and the like to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of the elements, unless otherwise indicated, and such terms are merely used to distinguish one element from another. In some examples, a first element and a second element may refer to the same instance of the element, and in some cases, they may also refer to different instances based on the description of the context.

The terminology used in the description of the various illustrated examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly indicates otherwise, the elements may be one or more if the number of the elements is not specifically limited. Furthermore, the term "and/or" as used in this disclosure encompasses any and all possible combinations of the listed items.

In a distributed storage system (such as a distributed object storage system, a distributed file storage system, etc.), data is generally stored in a redundant manner, so as to ensure the reliability and fault tolerance of the distributed storage system and avoid data loss caused by errors of part of devices. When the erasure coding technology is adopted to code the original data line, the ratio of the original data to the data amount of the check data needs to be specified.

Erasure Coding (EC) is a common redundancy scheme in distributed storage systems. The basic idea of erasure codes is to encode original data to obtain check data, and when part of the original data is lost, the check data can be used for decoding to obtain the original data so as to achieve the purpose of fault tolerance. When the erasure coding technology is adopted to code the original data line, the ratio of the original data to the data amount of the check data needs to be specified.

The coding and decoding modes of erasure codes generally comprise two modes, a generation matrix-based mode and a generation polynomial-based mode. The inventor finds that the encoding and decoding modes based on the generation matrix are simple to calculate, so that the distributed storage system can be guaranteed to have higher throughput (namely the data quantity processed in unit time), and the read-write performance is improved. However, the method has poor perception capability on the silent errors, and when the silent errors are encountered, if the integrity check mechanism of the data at the system level is not available, the decoding error of the data can be caused, and the problem of inconsistent data is caused. The encoding and decoding modes based on the generator polynomials have higher computational complexity, so that the throughput of the system is lower, the read-write performance is limited, but the perception capability of silent errors is stronger (the silent errors can be detected), and the fault tolerance is better.

The inventor notes that the distributed storage system generally adopts a fixed coding scheme (including the matching of the data amounts of the original data and the check data and the coding and decoding modes) to realize erasure code storage, and the coding schemes of all users are the same, so that the coding schemes cannot be flexibly customized for different users, and the differentiated requirements of the users cannot be met. If the coding scheme provided by the system exceeds the requirement of the user, unnecessary storage space waste, storage efficiency reduction and service cost increase may be caused; if the coding scheme provided by the system does not meet the requirements of the user, the loss of the user may be caused.

In view of this, embodiments of the present disclosure provide a data storage method, which can dynamically select an appropriate coding policy based on storage requirement information of a user, so that the coding policy is matched with a requirement of the user, thereby implementing an elastic customization of the coding policy, being capable of meeting differentiated requirements of different users on data storage, avoiding unnecessary storage space waste, improving storage efficiency, and reducing storage cost.

Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

Fig. 1 illustrates a flow chart of a data storage method 100 according to an embodiment of the present disclosure. The method 100 may be applied, for example, in a distributed storage system (hereinafter referred to as a "storage system"), and in particular, in any device of the storage system (i.e., a server for storing data).

As shown in fig. 1, the method 100 may include:

step S110, obtaining a plurality of data blocks to be stored and storage requirement information of the plurality of data blocks;

step S120, determining target coding strategies of a plurality of data blocks based on storage requirement information;

step S130, encoding the plurality of data blocks based on a target encoding strategy to generate a plurality of check blocks corresponding to the plurality of data blocks; and

step S140, a plurality of data blocks and a plurality of check blocks are stored in an associated mode.

According to the embodiment of the disclosure, a proper coding strategy can be dynamically selected based on the storage demand information of the user, so that the flexible customization of the coding strategy is realized, and the differentiated demands of different users on data storage can be met, thereby avoiding unnecessary storage space waste, improving storage efficiency and reducing storage cost.

The various steps of method 100 are described in detail below.

In step S110, a plurality of data blocks to be stored and storage requirement information of the plurality of data blocks are acquired.

In embodiments of the present disclosure, a data block refers to a set of data that contains a certain amount of data. A check block refers to a set of data obtained by encoding a data block. According to some embodiments, the amount of data included in each data block may be preset. Typically, each data block may include an amount of data on the order of KB-MB, e.g., 16MB. According to some embodiments, the amount of data included in each check block may be the same as the amount of data included in each data block.

According to some embodiments, the plurality of data blocks to be stored may be obtained by dividing a data stream uploaded by the client device.

The client device may be, for example, but not limited to, an internet of things device (e.g., an industrial sensor, a road camera, an intelligent vehicle device, an intelligent home device, etc.), a mobile computing device (e.g., a cell phone, a tablet computer, an intelligent wearable device, a notebook computer, etc.), a stationary computing device (e.g., a desktop computer, an industrial personal computer, a local area network gateway, a local area network server, etc.), etc.

The client device may perform data collection or data processing, and upload collected or processed data to the storage system, specifically to one or some servers in the storage system, in a data stream. After receiving the data stream uploaded by the client device, the server divides the data stream according to the data volume included in a preset single data block to obtain a plurality of data blocks. For example, the client device may be a road camera. The road camera collects image data in real time, and uploads the collected image data to the server for storage. Each time 16MB of image data is received by the server, it is divided into one data block. As the road camera continuously collects and uploads image data, the server will divide a plurality of data blocks of 16MB in size.

According to other embodiments, the plurality of data blocks to be stored may also be obtained by dividing data uploaded by the client device in batches. For example, the client device may be a desktop computer that uploads 128MB of data to the server for storage at a time based on user interaction. After receiving 128MB of data uploaded by the desktop computer, the server divides each data block into 128/18=8 data blocks according to 16MB of data.

The storage requirement information is used to represent the user's requirement for data storage. According to some embodiments, the storage requirement information includes at least one of storage cost requirement information, storage reliability requirement information, read-write performance requirement information.

The storage cost requirement information is used to represent the requirement of the user for the storage cost. According to some embodiments, the user may be provided with a list of candidate values for the storage cost requirement information, such that the user selects one candidate value from the list as the storage cost requirement information. Candidate values for the storage cost requirement information may include, for example, standard, low cost, extremely low cost, etc. It should be understood that the present disclosure is not limited to the number and content of candidate values storing cost demand information.

The storage reliability requirement information is used to represent the requirement of the user for storage reliability. According to some embodiments, the user may be provided with a list of candidate values for storing the reliability requirement information, such that the user selects one candidate value from the list as storing the reliability requirement information. Candidates for storing reliability requirement information may include, for example, 99.999% (i.e., 5 9), 99.99999% (i.e., 7 9), 99.9999999% (i.e., 9), 99.999999999% (i.e., 11 9), etc. It should be understood that the present disclosure is not limited to the number and content of candidate values storing reliability requirement information.

The read-write performance requirement information is used to represent the user's requirement for read-write performance of the storage system. In some embodiments, the read-write performance of the storage system may also be referred to as quality of service (Quality of Service, qoS). According to some embodiments, a list of candidate values for the read-write performance requirement information may be provided to the user, such that the user selects one candidate value from the list as the read-write performance requirement information. The candidate values of the read-write performance requirement information may include, for example, read priority, write priority, read-write balancing, and the like. It should be understood that the present disclosure is not limited to the number and content of candidates for read-write performance requirement information.

According to some embodiments, the storage requirement information may be preset. For example, in a storage system, each user may correspond to one or more storage addresses, and storage requirement information corresponding to each storage address may be preset. When a user stores a plurality of data blocks, a storage address of the plurality of data blocks needs to be specified, and storage requirement information corresponding to the storage address is the storage requirement information of the plurality of data blocks.

For another example, in an object storage system, a user may create one or more storage spaces (also called "buckets") for storing data, and the storage requirement information may be set as an attribute of the storage spaces, when the storage spaces are created, or after the storage spaces have been created. When a user stores a plurality of data blocks, the storage space to which the plurality of data blocks correspond (i.e., to which storage space the plurality of data blocks are to be stored) may be specified. After the server obtains the plurality of data blocks to be stored, the storage requirement information of the plurality of data blocks can be obtained by reading the attribute values of the storage spaces corresponding to the plurality of data blocks.

In step S120, a target encoding policy for a plurality of data blocks may be determined based on the acquired storage requirement information. The target encoding strategy is used to encode a plurality of data blocks.

According to some embodiments, the correspondence between the storage requirement information and the encoding policy may be preset. Accordingly, in step S120, a target encoding policy of the plurality of data blocks may be determined based on a preset correspondence between the storage requirement information and the encoding policy. Specifically, the correspondence of the storage requirement information and the encoding policy may be represented by a correspondence table. That is, a correspondence table of storage requirement information and encoding strategies may be preset, and the target encoding strategy may be determined by looking up a table.

For example, the correspondence between exemplary storage requirement information and encoding strategy may be as shown in table 1 below:

TABLE 1

Based on table 1, for example, when the storage cost requirement information is "standard", the storage reliability requirement information is "9 pieces of 9", and the read/write performance requirement information is "read priority", the corresponding target encoding policy is "policy 1".

The target encoding strategy is an erasure code encoding strategy for encoding a plurality of data blocks to be encoded. According to some embodiments, the target encoding policy includes a first number of corresponding data blocks encoded at a time and a number of check blocks generated by each encoding. Accordingly, in step S130, the plurality of data blocks to be stored may be divided into at least one data group, each data group including the first number of data blocks, based on the first number and the second number included in the target encoding policy; and encoding each data set respectively to generate a second number of check blocks corresponding to the data set. In step S140, each data set and the check blocks corresponding to the data set may be stored as one stripe, i.e., each stripe includes one data set (including the first number of data blocks) and the second number of check blocks corresponding to the data set. The first number and the second number corresponding to the stripe are referred to as the "parameter ratio" of the stripe. For example, if a stripe includes 8 data blocks and 4 check blocks, the parameter ratio of the stripe is 8+4.

For the above embodiment, the correspondence between the exemplary storage requirement information and the encoding policy may be as follows in table 2:

TABLE 2

Based on table 2, for example, in the case where the storage cost requirement information is "standard", the storage reliability requirement information is "9" and the read-write performance requirement information is "read-priority", the corresponding target encoding policy includes the first number y of data blocks encoded each time ₁ And a second number z of check blocks generated by each encoding ₁ . The first number may be, for example, 8, 12, 14, 28, etc., and the second number may be, for example, 2, 4, 6, etc.

Generally, the lower the storage cost, the higher the write performance, the larger the corresponding first number, i.e. the more data blocks are encoded at a time. In some cases, very low cost may be implemented in connection with hardware facilities of a storage system, such as employing lower cost memory (e.g., disk) to store data.

It will be appreciated that in some embodiments, only the first number and the second number may be included in the set of target encoding strategies corresponding to the storage requirement information (as shown in table 2).

In other embodiments, the target encoding policy corresponding to the set of requirement information may include, in addition to the first number and the second number, a first range of values to which the first number of values belongs and a second range of values to which the second number of values belongs. In this case, the first number is an initial value in the first range of values and the second number is an initial value in the second range of values.

The correspondence between the exemplary storage requirement information and the encoding strategy of the above embodiment may be as follows in table 3:

TABLE 3 Table 3

In table 3, the coding strategy corresponding to the i-th group (i=1, 2,3, …) of storage requirement information includes a first number y _i A second number z _i 、y _i A first value range [ p ] to which the value of (a) belongs _i ,q _i ]Z _i A second value range [ s ] to which the value of (a) belongs _i ,t _i ]. It will be appreciated that the first number y _i Is a first numerical range [ p ] _i ,q _i ]An initial value, a second quantity z _i Is a second range of values s _i ,t _i ]An initial value in the memory.

Based on the above embodiment, in step S120, the first number of data blocks encoded each time, the second number of check blocks generated by each encoding, the first numerical value range to which the numerical value of the first number belongs, and the second numerical value range to which the numerical value of the second number belongs may be determined based on the storage requirement information. Accordingly, in step S130, the plurality of data blocks may be divided into at least one data group, each data group including the first number of data blocks, based on the determined first number and second number; and encoding each data set respectively to generate a second number of check blocks corresponding to the data set. After the storage of the plurality of data blocks is completed, the access frequency of the plurality of data blocks can be counted, and the first number and the second number of values are respectively adjusted in the determined first number range and the determined second number range based on the access frequency, so that the data access efficiency is further improved on the premise of meeting the user requirements. The specific steps of adjusting the first number and the second number based on the access frequency will be described in detail below.

According to some embodiments, the target encoding strategy may include only the first number and the second number described above, or only the first number, the second number, the first range of values, and the second range of values described above, without including the target encoding mode. In this case, the storage system will employ a default encoding mode (e.g., a first encoding mode based on a generator polynomial or a second encoding mode based on a generator matrix) to encode each data set.

According to further embodiments, the target encoding strategy may further comprise a target encoding mode, which may comprise, for example, a first encoding mode based on a generator polynomial and a second encoding mode based on a generator matrix.

In the case where the target encoding strategy includes the target encoding mode, the correspondence between exemplary storage requirement information and the encoding strategy may be as shown in table 4 below:

TABLE 4 Table 4

It should be noted that, the correspondence between the storage requirement information and the encoding mode in table 4 is merely taken as an example, so as to facilitate understanding of the embodiments of the present disclosure. The correspondence relation between the storage demand information and the coding mode is not limited to that shown in table 4.

Based on the corresponding relation between the preset storage requirement information and the coding strategy, a target coding mode can be determined. Accordingly, in step S130, each data set may be encoded based on the determined target encoding mode (e.g., the first encoding mode based on the generator polynomial or the second encoding mode based on the generator matrix) to generate a second number of check blocks corresponding to the data set.

According to some embodiments, in case the target encoding mode is a first encoding mode based on a generator polynomial, the data set may be encoded with the generator polynomial to generate a second number of check blocks, wherein the second number of check blocks comprises a target check block, which is an exclusive or of the first number of data blocks comprised by the data set, for fault repair.

There are a variety of expressions for the generator polynomial. According to some embodiments, the generator polynomial may be, for example, a generator polynomial in an RS (Reed-Solomon) encoding algorithm, expressed as:

g(x)＝(x-α ⁰ )(x-α ¹ )…(x-α ^d-1 ) (1)

where g (x) is a generator polynomial of variable x, d is a second number, α is a generator of Galois Field (GF), and for Galois Field GF (2) ⁸ ) For α=2.

The process of encoding the first number of data blocks within the data set using the generator polynomial of equation (1) may be as follows:

the data set is noted as m= [ M ₀ m ₁ …m _k-1 ] ^T Wherein m is _i (i=0, 2, …, k-1) is the i-th data block in the data group, k is the first number, and T represents the transpose.

The data polynomial M (x) corresponding to the data set M is:

encoding the data set M by using a generator polynomial g (x) in the formula (1), and obtaining a check polynomial b (x) corresponding to the second number of check blocks, wherein the check polynomial b (x) is as follows:

b(x)＝x ^d m(x)mod g(x) (3)

In the formula (3), mod represents the remainder.

The check polynomial b (x) can be expressed in the form of the following formula (4):

b in formula (4) _i I.e. a second number of check blocks generated based on the RS coding algorithm. Based on RS coding, the following properties exist: using a generator polynomial g (x) = (x- α) ⁰ )(x-α ¹ )…(x-α ^d-1 ) Encoding data set M results in an exclusive OR of all check blocks equal to the exclusive OR of all data blocks (in the Galois field, the exclusive OR operation is identical to the addition operation), i.e.

In an embodiment of the present disclosure, a second number of check blocks b generated based on the RS encoding algorithm may be used _i Any of the check blocks is replaced by an exclusive or of all the data blocks, e.g. the first check block b ₀ Exclusive or, i.e. command, of all check blocks

Therefore, when only one block in the stripe (comprising the data group and the second number of check blocks corresponding to the data group) fails (the failed block is marked as a failure block), the failure block can be quickly repaired, and the repair efficiency of single failure is improved. The fault repair process will be described in detail below.

According to further embodiments, in case the target encoding mode is a second encoding mode based on a generator matrix, the generator matrix may be used to encode the data set to generate a second number of check blocks corresponding to the data set. Specifically, the generator matrix includes a plurality of code vectors, and a second number of code vectors in the generator matrix may be acquired; and encoding the data set with a second number of encoding vectors, respectively (e.g., multiplying the encoding vectors with the data set), to generate a second number of check blocks corresponding to the second number of encoding vectors, respectively.

The generation matrix may be various. According to some embodiments, the generator matrix may be, for example, but not limited to, a vandermonde matrix, a cauchy matrix, and the like. The size of the generator matrix of the embodiments of the present disclosure may be, for example, e×k, that is, the generator matrix includes e rows and k columns, where e is greater than or equal to the second number and k is the first number. Preferably, the value of e is set to be greater than the second number, thereby facilitating clipping or expanding the check block in the second encoding mode. The specific steps of check block clipping and expansion will be described in detail below.

As described above, in step S130, the plurality of data blocks may be divided into at least one data group based on the target encoding policy (including at least one of a first number of data blocks each encoded, a second number of check blocks generated each encoding, and a first numerical range to which a numerical value of the first number belongs, a second numerical range to which a numerical value of the second number belongs, and the target encoding policy), each data group including the first number of data blocks; and respectively encoding each data group to generate a second number of check blocks corresponding to the data group, thereby obtaining a plurality of check blocks corresponding to a plurality of data blocks to be stored.

After obtaining the plurality of check blocks corresponding to the plurality of data blocks in step S130, step S140 may be performed to store the plurality of data blocks in association with the plurality of check blocks. In particular, according to some embodiments, a plurality of data sets and a plurality of parity chunks may be stored as at least one stripe, each stripe including one data set and a parity chunk corresponding to the data set.

In an embodiment of the present disclosure, one data set (including a first number of data blocks) and its corresponding second number of check blocks are denoted as one stripe. According to some embodiments, each block in the stripe may be stored separately in different servers to improve reliability and fault tolerance of data storage. For example, a stripe comprises 8 data blocks and 4 check blocks, i.e. the stripe comprises 12 blocks in total, which 12 blocks can be stored in different servers, respectively. Fig. 2 shows a schematic diagram of the block storage of this embodiment. As shown in FIG. 2, storage system 200 includes, for example, 12 servers 201-212. 8 data blocks 251-258 included in stripe 220 (shown in phantom in FIG. 2) are stored in servers 201-208, respectively, and 4 parity blocks 261-264 are stored in servers 209-212, respectively. It will be appreciated that the servers 201-212 may store data blocks and/or parity blocks for other stripes in addition to the respective blocks of stripe 220. For example, the server 205 further stores a check block 281, the server 210 further stores a data block 271, and the server 211 further stores a check block 282.

According to some embodiments, in the case where the target encoding strategy includes a first range of values and a second range of values, after storing the plurality of data blocks in association with the plurality of check blocks, the method 100 may further include: counting the access frequency of a plurality of data blocks in a preset time period; adjusting the value of at least one of the first number and the second number based on the access frequency, wherein the adjusted value of the first number belongs to a first value range and the adjusted value of the second number belongs to a second value range; and encoding the plurality of data blocks based on the adjusted first number and second number.

Based on the above embodiment, the values of the first number and the second number can be adjusted in the first numerical range and the second numerical range respectively based on the access frequency, and the plurality of data blocks are recoded based on the adjusted first number and second number, so that the data access efficiency is further improved on the premise of meeting the user requirements.

According to some embodiments, in an object storage system, a user's storage space (socket) may have an attribute that dynamically adjusts a flag bit that may be used to turn on or off a function that dynamically adjusts a first number and/or a second number based on an access frequency. For example, if the dynamic adjustment flag bit is 1, the function of dynamically adjusting the first number and/or the second number based on the access frequency is turned on, if the dynamic adjustment flag bit is 0, the function of dynamically adjusting the first number and/or the second number based on the access frequency is turned off.

According to some embodiments, the ratio between the adjusted first number of values and the adjusted second number of values is inversely related to the access frequency. That is, the higher the access frequency, the smaller the value of the first number of data blocks encoded at a time relative to the second number of check blocks, so that the data recovery efficiency and the data access efficiency can be improved.

In particular, only the first number, or only the second number, or both the first number and the second number may be adjusted. For example, if the access frequency of a socket is high (e.g., may be above a predetermined threshold) in the last 7 days, the first value and/or the second value may be adjusted to decrease the ratio of the first number of values to the second number of values. For example, the current first number is 8, the second number is 4, and the ratio of the first number to the second number is 8/4=2. If only the first number is adjusted, the value of the first number may be reduced, for example, the first number is adjusted to 6, and the ratio of the adjusted first number to the second number is 6/4=1.5. If only the second number is adjusted, the value of the second number may be increased, for example, the second number is adjusted to 5, and the ratio of the adjusted first number to the second number is 8/5=1.6. If the first number and the second number are adjusted at the same time, the first number and the second number can be increased or decreased, so long as the ratio of the adjusted first number to the adjusted second number is ensured to be smaller than the original ratio (namely 2).

As described above, in the case where each data group is encoded using the first encoding mode based on the generator polynomial, the target parity chunk is included in the second number of parity chunks corresponding to each data group. Based on the target check block, rapid fault repair can be achieved.

Accordingly, in accordance with some embodiments, where each data set is encoded using a first encoding mode based on a generator polynomial, the method 100 may further comprise: performing fault detection on at least one strip, wherein the strip comprises any data group and a check block corresponding to the data group; and in response to determining that at least one of the at least one stripe includes at least one failed block, repairing the at least one failed block based on the number and type of the at least one failed block and the target parity block.

According to some embodiments, the failure detection of the stripe may be automated by the storage system. For example, the storage system may periodically employ an algorithm such as CRC (Cyclic Redundancy Check ) to detect the correctness of the data for each block in the stripe, and if the data for a block is incorrect, the block is marked as a failed block. The fault block may be a data block or a check block. That is, the fail block includes two types of data blocks and check blocks.

Repairing the at least one failed block based on the number and type of the at least one failed block and the target parity block may specifically include at least one of four cases.

According to some embodiments, in a first case, in response to determining that the number of at least one failed block is one and that the failed block is a data block, replacing the failed block with an exclusive or of other data blocks in the stripe other than the failed block with the target parity block.

According to the above embodiment, if only one data block in the stripe fails, the exclusive or of the target check block and other data blocks that do not fail may be calculated, and then the failed data block is replaced with the calculated exclusive or result, so that the repair of the failed data block may be completed. For example, the data block in a stripe is m ₀ ，m ₁ ，…，m ₇ The check block is b ₀ ，b ₁ ，b ₂ ，b ₃ Wherein b ₀ For the target check block, there are

If the fault block is data block m ₂ I.e. data block m ₂ If the fault occurs, the repaired data block can be calculated to be m ₂ ’＝b ₀ +m ₀ +m ₁ +m ₃ +m ₄ +m ₅ +m ₆ +m ₇ 。

According to some embodiments, in the second case, in response to determining that at least one of the failed blocks is a check block, marking the target check block as a failed block, and recoding the data set in the stripe to obtain a repaired check block corresponding to each failed block.

According to the above embodiment, if only the check block in the stripe fails, the target check block is marked as a failed block regardless of whether the target check block fails. The data sets in the stripe may then be re-encoded to obtain a repaired parity block. For example, the data block in a stripe is m ₀ ，m ₁ ，…，m ₇ The check block is b ₀ ，b ₁ ，b ₂ ，b ₃ Wherein b ₀ For the target check block, there are

If the fault block is the check block b ₁ Then the target check block b ₀ Also marked as a faulty block, and then employing a first coding mode based on a generator polynomial for data block m ₀ ，m ₁ ，…，m ₇ Recoding to obtain a repaired target check block b ₀ ' and repaired check block b ₁ ’。

According to some embodiments, in a third case, in response to determining that the number of at least one failed block is greater than one and that the at least one failed block is a data block, determining an exclusive or of the target parity block and other parity blocks as a repaired target parity block, and repairing the at least one failed block with a preset decoding algorithm based at least on the repaired target parity block.

The preset decoding algorithm may be, for example, a decoding algorithm corresponding to the RS encoding algorithm, i.e., an RS decoding algorithm. The RS decoding algorithm may include, for example, the following steps 1-4:

And step 1, performing a standard RS decoding flow based on a generator polynomial, and calculating an accompanying polynomial.

And 2, creating an error positioning polynomial.

And 3, marking in the error positioning polynomial according to the position of the fault block.

Step 4, error correction is performed based on a PGZ (Peterson-Gorenstein-Ziegler) decoding algorithm (for specific steps reference may be made to Gorenstein, daniel; peterson, W.Wesley; ziegler, neal (1960), "Two-Error Correcting Bose-Chaudhuri Codes are Quasi-Perfect", information and Control,3 (3): 291-294, doi:10.1016/s0019-9958 (60) 90877-9.).

According to the above embodiment, by determining the exclusive or of the target parity block and the other parity blocks as the repaired target parity block, the second number of parity blocks can be restored to the original state of the parity block based on the RS encoding. And then, repairing the data block with the fault by adopting an RS decoding algorithm, so as to recover the correct data block.

For example, the data block in a stripe is m ₀ ，m ₁ ，…，m ₇ The check block is b ₀ ，b ₁ ，b ₂ ，b ₃ Wherein b ₀ For the target check block, there are

If the fault block is data block m ₂ ，m ₃ Then the repaired target check block can be calculated

b ₀ ’、b ₁ ，b ₂ ，b ₃ Namely, the original check block generated based on the RS coding algorithm. Subsequently, based on b ₀ ’、b ₁ ，b ₂ ，b ₃ And decoding the stripe by adopting an RS decoding algorithm to obtain the repaired correct data block.

According to some embodiments, in a fourth case, in response to determining that the number of at least one failed block is greater than one and that the at least one failed block includes a data block and a check block, marking the target check block as a failed block, and repairing the at least one failed block using the decoding algorithm described above.

According to the above embodiment, if there are a plurality of failed blocks in the stripe and the failed block includes both the data block and the check block, the target check block is marked as the failed block regardless of whether the target check block fails. And then, repairing the fault block by adopting an RS decoding algorithm, so as to recover the correct data block and the correct check block.

If the fault block is data block m ₂ And check block b ₁ Then the target check block b ₀ Also labeled as a fault block. And then, decoding the stripe by adopting an RS decoding algorithm to obtain the repaired correct data block and the repaired correct check block.

As described above, in the case where each data group is encoded using the second encoding mode based on the generation matrix, quick clipping and expansion of the check block can be achieved.

According to some embodiments, in case of encoding the respective data sets using a second encoding mode based on the generation matrix, the generated second number of check blocks is arranged in the order of the corresponding encoding vectors in the generation matrix. The method 100 may further include the following check block clipping step: responding to the check block cutting request, and determining a third number of check blocks to be deleted; and deleting a third number of check blocks in the second number of check blocks that follow in order.

According to the embodiment, quick cutting of the check blocks can be realized, and the number of the check blocks is reduced on the premise that the number of the data blocks (namely the first number) is unchanged, so that the storage efficiency is improved, and the storage cost is saved.

According to some embodiments, the check block clipping request may be issued, for example, by a user, or may be issued automatically by the storage system (e.g., as described above, the storage system may automatically reduce the number of values of the second number, i.e., clip the check block, based on the access frequency).

For example, the second number is 4, the data group is encoded by using a second encoding mode based on the generating matrix, and the second number of check blocks are b respectively ₀ ，b ₁ ，b ₂ ，b ₃ . Based on the check block clipping request, it is determined that 2 check blocks need to be deleted, i.e. the third number is 2, then from check block b ₀ ，b ₁ ，b ₂ ，b ₃ In which 2 check blocks are deleted in order from the back to the front, i.e. check block b ₂ And b ₃ 。

In accordance with some embodiments, where the data set is encoded using a second encoding mode based on a generator matrix, the method 100 may further comprise the following check block expansion step: determining a fourth number of parity chunks to be added in response to the parity chunk expansion request; obtaining a fourth number of encoded vectors from the generator matrix, the fourth number of encoded vectors being located after the second number of encoded vectors; and respectively encoding the data group by adopting a fourth number of encoding vectors to generate a fourth number of check blocks to be increased, which correspond to the fourth number of encoding vectors respectively.

According to the embodiment, the quick expansion of the check blocks can be realized, and the number of the check blocks is increased on the premise that the number of the data blocks (namely the first number) is unchanged, so that the fault tolerance of the storage system is improved.

According to some embodiments, the check block expansion request may be issued, for example, by a user, or may be issued automatically by the storage system (e.g., as described above, the storage system may automatically increment a second number of values, i.e., expand the check block, based on the access frequency).

For example, the second number is 4, the data group is encoded by using a second encoding mode based on the generating matrix, and the second number of check blocks are b respectively ₀ ，b ₁ ，b ₂ ，b ₃ Check block b ₀ ，b ₁ ，b ₂ ，b ₃ Based on the coded vector v in the generator matrix, respectively ₀ ，v ₁ ，v ₂ ，v ₃ And obtaining the product. Based on the check block expansion request, it is determined that 2 check blocks need to be added, i.e. the fourth number is2, obtaining the position code vector v from the generation matrix ₃ The following code vector v ₄ ，v ₅ Using the code vector v ₄ ，v ₅ Encoding the data sets respectively to obtain added check blocks b ₄ ，b ₅ 。

The above embodiments describe the clipping and expansion process of the check block with the number of data blocks (i.e., the first number) remaining unchanged.

In other embodiments, the trimming and expanding of the check blocks may also be performed with a change in the first number of data blocks. In this case, clipping of the check blocks means that the ratio of the second number of check blocks to the first number of data blocks decreases, and expanding of the check blocks means that the ratio of the second number of check blocks to the first number of data blocks increases. For example, a stripe currently includes 8 data blocks, 4 check blocks, and the ratio of the second number to the first number is 4/8=0.5. If the stripe is adjusted to include 16 data blocks and 4 check blocks, the ratio of the second number to the second number is adjusted to be 4/16=0.25, and the ratio of the second number to the first number is reduced, so the adjustment is the clipping of the check blocks.

According to some embodiments, the method 100 may further comprise: and responding to the coding mode switching instruction, and adopting a preset conversion matrix to realize conversion among the corresponding check blocks of different target coding modes.

According to some embodiments, the coding mode switch instruction may be issued by a user, for example, or may be issued automatically by the storage system. Through the conversion among the corresponding check blocks of different target coding modes, the user requirements can be further met, the storage efficiency is improved, and the storage cost is saved.

According to some embodiments, the generator polynomial has equivalence with the generator matrix, and any one of the generator polynomial-based coding schemes (e.g., the generator polynomial-based RS coding scheme) can be converted into the generator matrix-based coding scheme under the same parameters under the condition that d.ltoreq.k-1 is satisfied (k is a first number and d is a second number). The proving process is as follows:

in the galois field GF (2^8), the generator polynomial is g (x) = (x- α) ⁰ )(x-α ¹ )…(x-α ^d-1 ) Namely formula (1) above. α=2. From the nature of the generator polynomial, it is possible to:

m(1)＝b(1) (5)

α ^d m(α)＝b(α) (6)

…α ^d(d-1) m(α ^d-1 )＝b(α ^d-1 ) (7)

in the above formulas (5) - (7), m () is a data polynomial, b () is a check polynomial, and d is a second number of check blocks.

And (3) making:

the above formulas (5) - (7) can be expressed as:

Λ[A _l |A _r ]M＝A _l [b ₀ …b _d-1 ] ^T (11)

In the above formula (11), "|" represents a matrix concatenation operation, i.e., [ a ] _l |A _r ]Representing matrix A _l And matrix A _r Spliced in the row direction (i.e., spliced side-to-side in the horizontal direction). M is the data set to be encoded. The above formula (11) can be converted into:

[b ₀ …b _d-1 ] ^T ＝A _l ^-1 Λ[A _l |A _r ]M (12)

that is, based on the generator polynomial g (x) = (x- α) ⁰ )(x-α ¹ )…(x-α ^d-1 ) Is a coding scheme of (a)Equivalent to based on the generator matrix A _l ^-1 Λ[A _l |A _r ]Is a coding scheme of (a).

According to some embodiments, if the check block of the current data set is generated based on the first coding mode of the generator polynomial, the corresponding coding vector v in the generator matrix (e.g., vandermonde matrix) may be generated by ₀ ，…，v _d-1 Multiplying the data group M to obtain a check block f in a second coding mode based on a generation matrix ₀ ，…，f _d-1 Thereby enabling a conversion of the check block of the first encoding mode to the second encoding mode. In this embodiment, the preset conversion matrix is the generation matrix.

Based on equation (12), if the check block of the current data set is generated based on the second encoding mode of the generation matrix, it is possible by combining matrix A _l ^-1 Λ[A _l |A _r ]Multiplying the data set M to obtain a check block b in a first coding mode based on a generator polynomial ₀ ，…，b _d-1 Thereby enabling a conversion of the second encoding mode into a check block of the first encoding mode. In this embodiment, the predetermined conversion matrix is matrix A _l ^-1 Λ[A _l |A _r ]。

According to the data storage method 100 of the embodiment of the present disclosure, the data block can be elastically encoded based on the requirement of the user, and expansion and clipping of the check block and conversion between the check blocks of different encoding modes can be realized.

Fig. 3 illustrates a block diagram of a storage system 300, according to an embodiment of the present disclosure. As shown in fig. 3, the storage system 300 includes an encoding module 310, a decoding module 320, an expanding and cropping module 330, and a conversion module 340. It is understood that modules 310-340 are all software modules.

The encoding module 310 may be configured to perform steps S110-S140 of the data storage method 100 implemented by the present disclosure, dynamically select an appropriate target encoding policy based on storage demand information of a user, encode a plurality of data blocks to be stored with the target encoding policy to generate a plurality of check blocks, and store the plurality of data blocks in association with the plurality of check blocks.

The decoding module 320 may be configured to perform the step of data reading, i.e. decoding the stored data and returning to the user. It is understood that the decoding strategy used by the decoding module 320 corresponds to the encoding strategy used by the encoding module 310.

The expand and crop module 330 may be configured to perform the steps of expanding and cropping check blocks in the data storage method 100 of an embodiment of the present disclosure.

The conversion module 340 may be configured to perform conversion steps between check blocks of different encoding modes in the data storage method 100 of an embodiment of the present disclosure.

Fig. 4 illustrates a block diagram of a storage system 400 according to further embodiments of the present disclosure. As shown in FIG. 4, the storage system 400 includes a multi-user requirements module 410, an access control module 420, a system control module 430, a decision control module 440, flexible codecs 450-1-450-3, and storage modules 460-1-460-3. Modules 410-440 may be implemented, for example, as software modules, and modules 450 (including 450-1-450-3), 460 (including 460-1-460-3) may be implemented, for example, as hardware modules.

The multi-user requirements module 410 may be configured to set and store the correspondence of storage requirements information and encoding strategies.

The access control module 420 may be configured to perform the steps of the data storage method 100 of adjusting the first number and the second number based on the access frequency.

The system control module 430 may be configured to provide commands and interfaces for adjusting system parameters in order to adjust system parameters involved in the data storage method 100. The system parameters include, for example, a system default coding mode (generator polynomial based or generator matrix based), an expression of the generator polynomial, an expression of the generator matrix, an initial value of the first number or the second number, a first range of values, a second range of values, etc.

The decision control module 440 may be configured to perform step S120 of the data storage method 100, determining a target encoding strategy based on the storage demand information.

The flexible codec modules 450-1 to 450-3 may be implemented as CPUs of respective servers in the storage system 400 for performing the encoding and decoding calculation tasks.

The storage modules 460-1-460-3 may be implemented as memory in each server in the storage system 400 for persisting each data block and each parity block. The storage modules 460-1-460-3 may be, for example, HDD (Hard Disk Drive), SSD (Solid State Disk), SMR (Shingled Magnetic Recording), magnetic tape, etc.

The encoding process and the check block clipping process according to the embodiment of the present disclosure are described below with reference to fig. 5.

An exemplary encoding process for embodiments of the present disclosure is as follows:

step 1, the storage system receives the data stream requested to be written, and divides the data stream into a plurality of data blocks with the same size (for example, 16 MB).

Step 2, the disk allocation module 510 initiates a request to the decision control module 520 to obtain the target encoding strategy from the decision control module 520. The target encoding strategy includes a first number of data blocks encoded at a time (e.g., 8), a second number of parity blocks generated at a time (e.g., 4), and a target encoding pattern (e.g., an RS encoding pattern based on a generator polynomial).

And 3, based on a target coding strategy, each stripe comprises 8 data blocks and 4 check blocks, and the total number of the data blocks and the check blocks is 12. The disk allocation module 510 allocates 12 disk blocks, 16MB each, to the 12 blocks, located at different storage nodes (servers) and creates one encoded computing task.

And 4, binding the coding calculation task with a calculation node (server) with idle calculation resources in a pull or push mode. The flexible codec module 530 in the compute node loads this task, determining that the task is an 8+4 generator polynomial based RS encoding task.

And 5, the elastic encoding and decoding module 530 in the computing node encodes 8 data blocks to obtain 4 check blocks.

Step 6, the elastic codec module 530 in the computing node writes the 12 blocks to the disk blocks of the pre-allocated storage modules 540-1 to 540-3 through the interfaces provided by the storage modules concurrently.

And 7, recording the metadata related to the stripes, and finishing the encoding task of the single stripe.

An exemplary check block clipping process of an embodiment of the present disclosure is as follows:

let the current stripe parameter ratio of a certain user be 8+4, i.e. a stripe comprises 8 data blocks and 4 check blocks.

Step 1, the storage system 500 obtains the current operation as check block clipping, and the new stripe parameter ratio is 16 (first number) +4 (second number).

Step 2, the storage system 500 selects two 8+4 stripes 550 under the same bucket through the data stream, and marks the blocks in the two stripes as n1-n12, m1-m12, respectively. And acquiring storage position information of each block through a metadata management module.

Step 3, the disk allocation module 510 of the storage system takes n1-n8, m1-m8 as data blocks, and allocates disk blocks for the newly generated check blocks. n1-n8, m1-m8 and the newly allocated disk blocks are on different storage nodes. And creates a clipping task.

And 4, enabling a computing node to acquire the current clipping task in a pull or push mode.

Step 5, the elastic encoding and decoding module 530 in the computing node encodes n1-n8, m1-m8 according to the new encoding algorithm assigned by the clipping task, generates 4 new check blocks, writes the new 4 check blocks into the disk blocks of the corresponding storage modules 540-1 to 540-3, and updates the metadata information.

And 6, marking n9-n12, m9-m12 and 8 original check blocks as a recovery task through a GC (Garbage Collection) module of the storage system, marking metadata of corresponding disk blocks as available in an asynchronous mode, and recovering disk space.

And 7, marking the access control of the new stripe as 0 (if the stripe id is multiplexed) through the decision control module 520, ending the cutting task and releasing the occupied memory resource.

According to another aspect of the present disclosure, there is also provided a data storage device. As shown in fig. 6, the apparatus 600 may include: an acquisition unit 610 configured to acquire a plurality of data blocks to be stored and storage demand information of the plurality of data blocks; a determining unit 620 configured to determine a target encoding policy of the plurality of data blocks based on the storage requirement information; an encoding unit 630 configured to encode the plurality of data blocks based on the target encoding policy, so as to generate a plurality of check blocks corresponding to the plurality of data blocks; and a storage unit 640 configured to store the plurality of data blocks and the plurality of check blocks in association.

It should be appreciated that the various units 610-640 of the apparatus 600 shown in fig. 6 may correspond to the various steps S110-S140 in the method 100 described with reference to fig. 1. Thus, the operations, features and advantages described above with respect to method 100 apply equally to apparatus 600 and the units comprised thereby. For brevity, certain operations, features and advantages are not described in detail herein.

It should also be appreciated that various techniques may be described herein in the general context of software hardware elements or program modules. The various units described above with respect to fig. 6 may be implemented in hardware or in hardware in combination with software and/or firmware. For example, the units may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units may be implemented as hardware logic/circuitry. For example, in some embodiments, one or more of the acquisition unit 610, the determination unit 620, the encoding unit 630, the storage unit 640 may be implemented together in a System on Chip (SoC). The SoC may include an integrated circuit chip including one or more components of a processor (e.g., a central processing unit (Central Processing Unit, CPU), microcontroller, microprocessor, digital signal processor (Digital Signal Processor, DSP), etc.), memory, one or more communication interfaces, and/or other circuitry, and may optionally execute received program code and/or include embedded firmware to perform functions.

Referring to fig. 7, a block diagram of an electronic device 700 that may be a server or a client of the present disclosure, which is an example of a hardware device that may be applied to aspects of the present disclosure, will now be described. The electronic devices may be different types of computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 7, the electronic device 700 may include at least one processor 701, a working memory 702, an input unit 704, a display unit 705, a speaker 706, a storage unit 707, a communication unit 708, and other output units 709 that can communicate with each other through a system bus 703.

The processor 701 may be a single processing unit or multiple processing units, all of which may include a single or multiple computing units or multiple cores. The processor 701 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. The processor 701 may be configured to obtain and execute computer readable instructions stored in the working memory 702, the storage unit 707, or other computer readable media, such as program code of the operating system 702a, program code of the application programs 702b, and the like.

The working memory 702 and the storage unit 707 are examples of computer-readable storage media for storing instructions that are executed by the processor 701 to implement the various functions described previously. The working memory 702 may include both volatile memory and nonvolatile memory (e.g., RAM, ROM, etc.). In addition, the storage unit 707 may include hard drives, solid state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CD, DVD), storage arrays, network attached storage, storage area networks, and the like. The working memory 702 and the storage unit 707 may both be referred to herein collectively as memory or computer-readable storage medium, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code that may be executed by the processor 701 as a particular machine configured to implement the operations and functions described in the examples herein.

The input unit 706 may be any type of device capable of inputting information to the electronic device 700, the input unit 706 may receive input numeric or character information and generate key signal inputs related to user settings and/or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone, and/or a remote control. The output unit may be any type of device capable of presenting information and may include, but is not limited to, a display unit 705, a speaker 706, and other output units 709, which other output units 709 may include, but are not limited to, a video/audio output terminal, a vibrator, and/or a printer. The communication unit 708 allows the electronic device 700 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunications networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and/or a chipset, such as bluetooth ^TM Devices, 802.11 devices, wi-Fi devices, wiMAX devices, cellular communication devices, and/or the like.

The application 702b in the working register 702 may be loaded to perform the various methods and processes described above, such as step S110-step S140 in fig. 1. For example, in some embodiments, the method 100 described above may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 707. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 700 via the storage unit 707 and/or the communication unit 708. One or more of the steps of the method 100 described above may be performed when a computer program is loaded and executed by the processor 701. Alternatively, in other embodiments, processor 701 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.

Although embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it is to be understood that the foregoing methods, systems, and apparatus are merely exemplary embodiments or examples, and that the scope of the present invention is not limited by these embodiments or examples but only by the claims following the grant and their equivalents. Various elements of the embodiments or examples may be omitted or replaced with equivalent elements thereof. Furthermore, the steps may be performed in a different order than described in the present disclosure. Further, various elements of the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after the disclosure.

Claims

1. A data storage method, comprising:

acquiring a plurality of data blocks to be stored and storage requirement information of the plurality of data blocks;

determining a target encoding strategy of the plurality of data blocks based on the storage requirement information;

encoding the plurality of data blocks based on the target encoding strategy to generate a plurality of check blocks corresponding to the plurality of data blocks; and

and storing the plurality of data blocks and the plurality of check blocks in an associated manner.

2. The method of claim 1, wherein the determining a target encoding policy for the plurality of data blocks based on the storage requirement information comprises:

and determining target coding strategies of the plurality of data blocks based on the corresponding relation between the preset storage demand information and the coding strategies.

3. The method of claim 1 or 2, wherein the storage requirement information comprises at least one of:

cost requirement information, reliability requirement information and read-write performance requirement information are stored.

4. The method of any of claims 1-3, wherein the target encoding strategy comprises encoding a first number of corresponding data blocks at a time and encoding a second number of generated parity blocks at a time, and wherein,

the encoding the plurality of data blocks based on the target encoding policy to generate a plurality of check blocks corresponding to the plurality of data blocks includes:

dividing the plurality of data blocks into at least one data group, each data group comprising the first number of data blocks; and

each data set is encoded separately to generate the second number of check blocks corresponding to the data set.

5. The method of claim 4, wherein storing the plurality of data blocks and the plurality of parity blocks in association comprises:

Storing the plurality of data blocks and the plurality of check blocks as at least one stripe, wherein each stripe comprises a data group and a check block corresponding to the data group.

6. The method of claim 4 or 5, wherein the target encoding strategy comprises a first range of values to which the first number of values belongs and a second range of values to which the second number of values belongs,

the method further comprises the steps of:

counting the access frequency of the plurality of data blocks in a preset time period;

adjusting the value of at least one of the first number and the second number based on the access frequency, wherein the adjusted value of the first number belongs to the first value range and the adjusted value of the second number belongs to the second value range; and

the plurality of data blocks is encoded based on the adjusted first number and second number.

7. The method of claim 6, wherein a ratio between the adjusted first number of values and the adjusted second number of values is inversely related to the access frequency.

8. The method of any of claims 4-7, wherein the target coding strategy comprises a target coding mode comprising a first coding mode based on a generator polynomial, and wherein,

The encoding each data set separately to generate the second number of check blocks corresponding to the data set includes:

and encoding the data set by adopting the generating polynomial to generate a second number of check blocks, wherein the second number of check blocks comprises a target check block which is the exclusive OR of the first number of data blocks included in the data set and is used for fault repair.

9. The method of claim 8, further comprising:

performing fault detection on at least one strip; and

in response to determining that at least one of the at least one stripe includes at least one failed block, the at least one failed block is repaired based on the number and type of the at least one failed block and the target parity block.

10. The method of claim 9, wherein the repairing the at least one failed block based on the number and type of the at least one failed block and the target parity block comprises at least one of:

in response to determining that the number of the at least one failed block is one and the failed block is a data block, replacing the failed block with an exclusive or of other data blocks in the stripe, except the failed block, with the target parity block;

In response to determining that the at least one failed block is a check block, marking the target check block as a failed block, and recoding a data set in the stripe to obtain repaired check blocks corresponding to the failed blocks;

in response to determining that the number of the at least one fault block is greater than one and that the at least one fault block is a data block, determining an exclusive or of the target check block and other check blocks as a repaired target check block, and repairing the at least one fault block with a preset decoding algorithm based at least on the repaired target check block; and

in response to determining that the number of the at least one failed block is greater than one and that the at least one failed block includes a data block and a check block, marking the target check block as a failed block, and employing the decoding algorithm to repair the at least one failed block.

11. The method of any of claims 4-7, wherein the target coding strategy comprises a target coding mode comprising a second coding mode based on a generator matrix comprising a plurality of coding vectors, and wherein,

acquiring a second number of coded vectors in the generator matrix; and

and respectively encoding the data group by adopting the second number of encoding vectors to generate a second number of check blocks respectively corresponding to the second number of encoding vectors.

12. The method of claim 11, wherein the second number of check blocks is arranged in the order of corresponding encoding vectors in the generator matrix, the method further comprising:

responding to the check block cutting request, and determining a third number of check blocks to be deleted; and

and deleting a third number of check blocks in the second number of check blocks in sequence.

13. The method of claim 11, further comprising:

determining a fourth number of parity chunks to be added in response to the parity chunk expansion request;

acquiring a fourth number of encoded vectors located after the second number of encoded vectors; and

and respectively encoding the data group by adopting the fourth number of encoding vectors to generate a fourth number of check blocks to be increased, which correspond to the fourth number of encoding vectors respectively.

14. The method of any of claims 8-13, further comprising:

and responding to the coding mode switching instruction, and adopting a preset conversion matrix to realize conversion among the corresponding check blocks of different target coding modes.

15. A data storage device, comprising:

an acquisition unit configured to acquire a plurality of data blocks to be stored and storage demand information of the plurality of data blocks;

a determining unit configured to determine a target encoding policy of the plurality of data blocks based on the storage demand information;

an encoding unit configured to encode the plurality of data blocks based on the target encoding policy to generate a plurality of check blocks corresponding to the plurality of data blocks; and

and a storage unit configured to store the plurality of data blocks and the plurality of check blocks in association.

16. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the method comprises the steps of

The memory stores a computer program which, when executed by the at least one processor, implements the method of any of claims 1-14.

17. A non-transitory computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the method of any of claims 1-14.

18. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method of any of claims 1-14.