CN116185698A

CN116185698A - Data storage method and device and computer equipment

Info

Publication number: CN116185698A
Application number: CN202111434140.6A
Authority: CN
Inventors: 李�杰; 唐小虎; 侯韩旭; 韩永祥; 韩伟; 白铂; 张弓
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-11-29
Filing date: 2021-11-29
Publication date: 2023-05-30

Abstract

The application discloses a data storage method and device and computer equipment, wherein the method comprises the following steps: dividing a source file into a plurality of pieces of data to be stored; encoding target data by adopting a local check matrix in a local code of a PMDS array code to obtain first local check data, wherein the local code adopts an epsilon-MSR code, and the target data is data to be stored in a first local group; adopting a global check matrix and a local check matrix of a PMDS array code to encode a plurality of data to be stored and the first local check data to obtain second local check data and global check data; storing a plurality of data to be stored in system nodes of a plurality of local groups, storing first local check data in local check nodes of the first local group, storing second local check data in local check nodes of the second local group, and storing global check data in global check nodes of the second local group.

Description

Data storage method and device and computer equipment

Technical Field

The present invention relates to the field of storage technologies, and in particular, to a data storage method and apparatus, and a computer device.

Background

With the rapid increase in data magnitude, traditional storage technologies (such as copy and backup) become economically infeasible, which drives the development of distributed storage systems based on erasure coding, which require processing of data during data writing: the source file is generally divided into a plurality of pieces of data to be stored, then the plurality of pieces of data to be stored are encoded by erasure coding, local check data and global check data are generated, and the plurality of pieces of data to be stored, the local check data and the global check data are stored in a plurality of distributed nodes.

Currently, erasure coding commonly used in distributed storage systems includes very large distance separable (maximum distance separable, MDS) codes, partial very large distance separable (partial maximum distance separable, PMDS) codes, PMDS array codes, and the like. The PMDS array code adopted in the related art combines a PMDS code and a minimum storage reproduction (minimum storage regenerating, MSR) code. When the MSR code is used as the local code of the PMDS array code to process the data to be stored, the number of the packets of the PMDS array code is large, and further the calculation order is large in the encoding process, so that the resource cost is high and the requirements on software and hardware are high.

Disclosure of Invention

The application provides a data storage processing method and device and computer equipment, which are particularly suitable for large data distributed storage and can reduce the packet number when PMDS array codes are adopted for coding.

In a first aspect, the present application provides a data storage method, the method comprising:

dividing a source file into a plurality of pieces of data to be stored, wherein the plurality of pieces of data to be stored are used for storing a plurality of local groups of storage equipment, each local group in the plurality of local groups comprises a plurality of system nodes and a first number of local check nodes, and part of local groups in the plurality of local groups also comprise a second number of global check nodes;

encoding target data by adopting a local check matrix in local codes of partial maximum distance separable PMDS array codes to obtain first local check data, wherein the local codes adopt epsilon-minimum storage regeneration epsilon-MSR codes of which matrix blocks are diagonal matrixes, second-order upper triangular matrixes or second-order lower triangular matrixes, and the target data are to-be-stored data to be stored in a first local group;

adopting a global check matrix and a local check matrix of the PMDS array code to encode the data to be stored and the first local check data to obtain second local check data and global check data;

Storing the data to be stored in the plurality of local groups to the system nodes of the plurality of local groups, storing the first local check data to the local check nodes of the first local group, storing the second local check data to the local check nodes of the second local group, and storing the global check data to the global check nodes of the second local group, wherein the first local group is a local group not comprising the global check nodes, and the second local group is a local group comprising the global check nodes.

In the embodiment of the present application, in the PMDS array code, an epsilon-MSR code whose matrix block is a diagonal matrix, a second-order upper triangular matrix, or a second-order lower triangular matrix is used as a local code, so as to implement a function (for example, a repair function) of the PMDS array code.

Optionally, the encoding the target data by using the local check matrix in the local code of the partial maximum distance separable PMDS array code to obtain first local check data includes:

acquiring coding parameters, and determining a coding algorithm corresponding to the coding parameters, wherein the coding algorithm comprises a local check matrix of the local code;

Calculating the first local check data based on the local check matrix and the target data;

the encoding algorithm further includes the global check matrix, and the encoding the multiple data to be stored and the first local check data by using the global check matrix and the local check matrix of the PMDS array code to obtain second local check data and global check data, including:

and calculating the second local check data and the global check data based on the global check matrix, the local check matrix, the plurality of data to be stored and the first local check data.

Wherein the global check matrix and the local check matrix may also be referred to as a local check equation and a global check equation.

Optionally, the encoding algorithm is determined based on a relationship between the encoding parameter and the encoding algorithm, the encoding parameter includes the first number, the second number, and a third number, the third number is a number of packets, and the relationship between the encoding parameter and the encoding algorithm includes:

when the first number r is greater than or equal to 2, the second number s is 2, and the third number l is r ^n’ Or 2, the matrix block of the global check matrix is a diagonal matrix, wherein n' is more than or equal to 2 and less than or equal to n, and n is the total node number in the local group;

When the first number r is greater than or equal to 2, the second number s is 3, and the third number l is r ^n’ In this case, the global check matrix includes a plurality of third-order cauchy-vandermonde matrices.

Optionally, when the first number r is greater than or equal to 2, the second number s is 2, and the third number l is r ^n’ Wherein when n' is more than or equal to 2 and less than or equal to n,

the local check matrix is as follows:

the global check matrix is as follows:

wherein mu is the number of local groups, i is the number of local groups, H _i Is a local check matrix, P _i Is a global check matrix, I is a unit matrix, A _j In the form of a matrix block,

is A _j To the power of r, θ ₀ ,θ ₁ ,…,θ _μ-1 To be from finite field F _q The sequence of one element is selected from the mu cosets contained in the sequence.

Alternatively, when the first number r is 2, the second number s is 2, the third number l is 2,

the local check matrix is as follows:

/>

the global check matrix is as follows:

is A _j To the power of r, θ ₀ ,θ ₁ ,…,θ _μ-1 To be from finite field F _q The mu cosets contained in the sequence are respectively selected to be a sequence composed of one element, lambda ₀ ,λ ₁ ,…,λ _n-1 To be from finite field F _q Two-by-two unequal n elements.

Optionally, when the first number r is greater than or equal to 2, the second number s is 3, and the third number l is r ^n’ Wherein when n' is more than or equal to 2 and less than or equal to n,

the local check matrix is as follows:

the global check matrix is as follows:

wherein mu is the number of local groups, i is the number of local groups, H _i Is a local check matrix, P _i Is a global check matrix, I is a unit matrix, A _j 、A′ _j 、A′ _j ' is a block of a matrix and,

is A _j To the power of r, θ ₀ ,θ ₁ ,…,θ _μ-1 To be from finite field F _q The mu cosets contained in the sequence are respectively selected to be a sequence composed of one element, delta ₀ ,δ ₁ ,…,δ _μ-1 To be from finite field F _q Wherein any 3 elements are selected from mu elements in the finite field +.>

The upper linearity is irrelevant.

Optionally, the method further comprises:

in response to a reconfiguration instruction, determining the number of nodes of unconnected nodes in each local group and the total number of unconnected nodes in the storage device when unconnected nodes exist in the storage device, wherein the unconnected nodes comprise at least one of offline nodes and failure nodes;

determining a corresponding reconstruction algorithm based on the number of nodes of the unconnected nodes in each local group and the total number of unconnected nodes in the storage device;

And adopting the determined reconstruction algorithm to solve the data in the unconnected nodes and reconstructing the source file.

Optionally, the determining a corresponding reconstruction algorithm based on the number of nodes of the unconnected nodes in each local group and the total number of unconnected nodes in the storage device includes:

when the number of the nodes which are not connected in each local group does not exceed r, the reconstruction algorithm is as follows:

wherein H is _i Is a local check matrix, f _i,0 Representing data in node 0 in local group i, i representing the local group number.

when the number of nodes with 1 partial group of unconnected nodes in the plurality of partial groups exceeds r, and the total number of unconnected nodes in the storage device does not exceed r+s, the reconstruction algorithm is as follows:

wherein H is _i Is a local check matrix, P _i R is the first number, 0, for the global check matrix _rl×1 For the rl row 1 column matrix, the elements in the matrix are 0,l, μ is the number of partial groups, and f _i,0 The data in the 0 th node in the local group i is represented, i and j represent local group serial numbers, and the local group with the number of unconnected nodes exceeding r is the local group i.

when the number of nodes with 2 local group unconnected nodes in the plurality of local groups exceeds r, and the total number of unconnected nodes in the storage device does not exceed 2r+s, the reconstruction algorithm is as follows:

wherein H is _i Is a local check matrix, P _i R is the first number, 0, for the global check matrix _2rl×1 A matrix with 2rl rows and 1 column, wherein the elements in the matrix are 0,l as packet numbers, mu as the number of partial groups and f _i,0 The data in the 0 th node in the local group i is represented, i, j and k represent local group numbers, and the local groups with the number of nodes of unconnected nodes exceeding r are the local groups i and j.

when the number of nodes with 3 local group unconnected nodes in the plurality of local groups exceeds r, and the total number of unconnected nodes in the storage device does not exceed 3r+s, the reconstruction algorithm is as follows:

Wherein H is _i Is a local check matrix, P _i R is the first number, 0, for the global check matrix _3rl×1 A matrix with 3rl rows and 1 column, wherein the elements in the matrix are 0,l as packet numbers, mu as the number of partial groups and f _i,0 The data in the 0 th node in the local group i is represented, i, j, k and m represent local group numbers, and the local groups with the number of nodes of unconnected nodes exceeding r are the local group i, the local group j and the local group k.

Optionally, the method further comprises:

determining a repair algorithm corresponding to the coding algorithm in response to a storage node failure in the storage device;

and repairing the data in the failure node by adopting the determined repairing algorithm.

Optionally, the determining a repair algorithm corresponding to the coding algorithm includes:

when the coding algorithm is a coding algorithm corresponding to the first coding parameter or the third coding parameter, the repair algorithm is as follows:

wherein f _k,i Representing data in the failed node, wherein the failed node is an ith node in the local group k, S _i ＝V _i,0 +V _i,1 +…+V _i,r-1 J represents a node sequence number;

the first encoding parameters include: the first number r is greater than or equal to 2, the second number s is 2, and the third number l is r ^n’ Wherein n' is more than or equal to 2 and less than or equal to n; the third encoding parameters include: when the first number r is greater than or equal to 2, the second number s is 3, and the third number l is r ^n’ 。

when the coding algorithm is the coding algorithm corresponding to the second coding parameter, the repairing algorithm is as follows:

if the number i of the failed node is even, then:

wherein s= (1 0);

if the number i of the failure node is an odd number, then:

wherein f _k,i Representing data in the failed node in question,

the second encoding parameters include: the first number is 2, the second number is 2, and the third number is 2.

Optionally, the repairing the data in the failed node by using the determined repairing algorithm includes:

if the local group where the failure node is located has no other failure node or busy node, determining the relation between the first node in the local group and the serial number of the failure node;

when the relation between the serial numbers of the first node and the failure node in the local group is a first relation, downloading all data from the first node; downloading the product of all data and coefficients from the first node when the relation between the serial numbers of the first node and the failure node in the local group is a second relation;

and repairing the data in the failure node by adopting the repairing algorithm based on the downloaded data.

Optionally, the first relationship is: j≡i mod n';

the second relation is:

wherein i is the serial number of the failure node, and j is the serial number of the first node. The application provides three PMDS array codes, which correspond to the three coding algorithms in sequence. Wherein the first and second PMDS array codes are substantially reduced in finite field size; the second PMDS array code can greatly reduce the input/output I/O consumed during node repair; the third PMDS array code may support more global checks.

In a second aspect, the present application provides a data storage device, the device comprising:

the system comprises a segmentation module, a storage module and a storage module, wherein the segmentation module is used for segmenting a source file into a plurality of pieces of data to be stored, the plurality of pieces of data to be stored are used for storing a plurality of local groups of storage equipment, each local group in the plurality of local groups comprises a plurality of system nodes and a first number of local check nodes, and part of local groups in the plurality of local groups also comprise a second number of global check nodes;

the processing module is used for encoding target data by adopting a local check matrix in local codes of partial maximum distance separable PMDS array codes to obtain first local check data, wherein the local codes adopt epsilon-minimum storage regeneration epsilon-MSR codes of which matrix blocks are diagonal matrixes, second-order upper triangular matrixes or second-order lower triangular matrixes, and the target data are to-be-stored data to be stored in a first local group; adopting a global check matrix and a local check moment of the PMDS array code to encode the data to be stored and the first local check data to obtain second local check data and global check data;

The output module is used for storing the data to be stored into the system nodes of the local groups, storing the first local check data into the local check nodes of the first local group, storing the second local check data into the local check nodes of the second local group, storing the global check data into the global check nodes of the second local group, wherein the first local group is a local group which does not comprise the global check nodes, and the second local group is a local group which comprises the global check nodes.

In a third aspect, the present application provides a computer device comprising a processor and a memory; the memory is used for storing software programs and modules, and the processor implements the method in any one of the possible implementation manners of the first aspect by running or executing the software programs and/or modules stored in the memory.

Optionally, the processor is one or more, and the memory is one or more.

Alternatively, the memory may be integrated with the processor or the memory may be separate from the processor.

In a specific implementation process, the memory may be a non-transient (non-transitory) memory, for example, a Read Only Memory (ROM), which may be integrated on the same chip as the processor, or may be separately disposed on different chips, where the type of the memory and the manner of disposing the memory and the processor are not limited in this application.

In a fourth aspect, the present application provides a computer program (product), the computer program (product) comprising: computer program code which, when run by a computer, causes the computer to perform the method of any one of the possible implementations of the first aspect described above.

In a fifth aspect, the present application provides a computer readable storage medium for storing program code for execution by a processor, the program code comprising instructions for implementing the method of any one of the possible implementations of the first aspect.

In a sixth aspect, a chip is provided, comprising a processor for calling from a memory and executing instructions stored in said memory, such that a communication device on which said chip is mounted performs the method of any of the possible implementations of the first aspect described above.

In a seventh aspect, there is provided another chip comprising: the device comprises an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected through an internal connection path, the processor is used for executing codes in the memory, and when the codes are executed, the processor is used for executing the method in any possible implementation manner of the first aspect.

Drawings

FIG. 1 is a schematic diagram of a distributed storage system according to an embodiment of the present application;

fig. 2 is a schematic structural diagram of a controller according to an embodiment of the present application;

FIG. 3 is a schematic diagram illustrating a distribution of storage nodes according to an embodiment of the present application;

FIG. 4 is a flow chart of a data storage method provided in an embodiment of the present application;

FIG. 5 is a flow chart of a data storage method provided in an embodiment of the present application;

FIG. 6 is a schematic diagram of a data storage device according to an embodiment of the present disclosure;

fig. 7 is a schematic structural diagram of a computer device according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

In order to facilitate understanding of the technical solution provided in the embodiments of the present application, an application scenario of the present application is first introduced. The application scenario may be a mass distributed storage system or a data center. The application scenario of the present application will be described below by taking a distributed storage system as an example.

Fig. 1 is a schematic structural diagram of a distributed storage system according to an embodiment of the present application. Referring to fig. 1, the distributed storage system includes a host 10, a controller 20, and storage devices 30, and the host 10 is connected to the storage devices 30 through the controller 20, respectively.

In some possible implementations of the present application, the host 10 is configured to receive external instructions and requests, and convert them into instructions and requests sent to the controller 20, so as to implement uploading and downloading of data. When uploading data, the host 10 selects encoding parameters of the PMDS array code according to the size and storage requirement of the source file, and sends the source file and the encoding parameters to the controller 20, so that the controller realizes encoding and storage of the source file. When downloading data, the host 10 requests the controller 20 to download data in the storage node and reconstruct the source file.

In some possible implementations of the present application, the controller 20 is configured to receive instructions and requests from the host 10, and implement data encoding, reconstruction, and repair.

Fig. 2 is a schematic structural diagram of a controller according to an embodiment of the present application. Referring to fig. 2, the controller includes an encoder 21, a reconstructor (also called a decoder) 22, and a healer 23. The encoder 21 is configured to implement an encoding function of the controller 20, for example, dividing a source file into multiple parts, then encoding the divided data, and writing the encoded data into the storage device 30. The reconstructor 22 is configured to implement a reconstruction function of the controller 20, such as completing the reconstruction of the source file by downloading data from the storage device 30 upon receipt of a reconstruction instruction. The healer 23 is configured to implement a repair function of the controller 20, for example, when a storage node failure occurs in the storage device 30, download data from some surviving storage nodes, repair the data in the failed storage node by the downloaded data, and rewrite a new storage node to replace the failed storage node after the repair is completed.

In some possible implementations of the present application, the storage device 30 stores data written by the controller. The storage device 30 includes a plurality of storage nodes 31, and when a storage node 31 fails, a new storage node 31 is inserted for replacement, and the repaired data by the repair unit 23 is stored.

Fig. 3 is a schematic distribution diagram of a storage node according to an embodiment of the present application. Referring to fig. 3, the storage devices 30 have a total of μn storage nodes (abbreviated as nodes in the figure) 31, the storage nodes 31 are divided into μlocal groups 32, each local group 32 includes n storage nodes 31, r of the n storage nodes 31 in each local group 32 are local check nodes for storing local check data, s of the μn storage nodes 31 in the storage devices 30 are global check nodes for storing global check data, the global check nodes may be distributed in any location, for example, in the last or several local groups 32, and other storage nodes except the local check nodes and the global check nodes are all used for storing data of a source file, which may be called a system node. Each storage node 31 is capable of storing data comprising l elements, i.e. a packet number of l.

Fig. 4 is a flowchart of a data storage method according to an embodiment of the present application. The method may be performed by a controller in the application scenario shown in fig. 1, as shown in fig. 4, the method comprising the following steps.

S41: the source file is divided into a plurality of pieces of data to be stored.

The data storage system comprises a plurality of local groups, a plurality of local check nodes and a plurality of global check nodes, wherein the plurality of data to be stored are used for storing the plurality of local groups to the storage device, each local group in the plurality of local groups comprises a plurality of system nodes and a first number of local check nodes, and part of local groups in the plurality of local groups also comprise a second number of global check nodes.

S42: and encoding target data by adopting a local check matrix in local codes of partial maximum distance separable PMDS array codes to obtain first local check data, wherein the local codes adopt epsilon-minimum storage regenerated epsilon-MSR codes of which matrix blocks are diagonal matrixes, second-order upper triangular matrixes or second-order lower triangular matrixes, and the target data are to-be-stored data to be stored in a first local group.

Wherein the epsilon-MSR code is a code with a repair bandwidth larger than that of the MSR code and a packet number smaller than that of the MSR code, wherein the repair bandwidth of the epsilon-MSR code is 1+epsilon times that of the MSR code, and the epsilon-MSR code can be also called as an MDS array code with a smaller packet number and almost optimal repair bandwidth.

It should be noted that the repair bandwidth of the epsilon-MSR code, while larger than the MSR code, is much smaller than the conventional PMDS code.

The matrix blocks are one of the matrices, and one matrix can be divided into a plurality of matrix blocks arranged in an array.

The diagonal matrix refers to a matrix with elements other than the main diagonal being 0. The upper triangular matrix refers to a square matrix with zero below the main diagonal. The lower triangular matrix refers to a square matrix with zeros above the main diagonal. The second order matrix refers to a matrix of two rows and two columns.

In the method, the epsilon-MSR code is realized by using the matrix blocks as the local check matrix of the diagonal matrix, the second-order upper triangular matrix or the second-order lower triangular matrix, so that the proper global check matrix can be determined, and the construction of the PMDS array code is completed.

S43: and coding the multiple data to be stored and the first local check data by adopting the global check matrix and the local check matrix of the PMDS array code to obtain second local check data and global check data.

S44: storing a plurality of data to be stored in system nodes of a plurality of local groups, storing first local check data in local check nodes of the first local group, storing second local check data in local check nodes of the second local group, and storing global check data in global check nodes of the second local group, wherein the first local group is a local group which does not comprise the global check nodes, and the second local group is a local group which comprises the global check nodes.

It should be noted that, the storage of the data to be stored and the storage of the check codes (the first local check data, the second local check data and the global check data) may be performed simultaneously or separately, for example, the storage of the data to be stored may also be performed before steps S42 and S43, and the storage of the check codes may be performed after steps S42 and S43. Alternatively, both the storing of the data to be stored and the storing of the check code are performed after steps S42 and S43.

In the embodiment of the application, in the PMDS array code, the epsilon-MSR code with the matrix block being a diagonal matrix, a second-order upper triangular matrix or a second-order lower triangular matrix is adopted as the local code, so that the function (such as a repairing function) of the PMDS array code is realized, and the defect of the MSR code is overcome due to the very small packet number of the epsilon-MSR code, so that the calculation magnitude is reduced, the resource cost is reduced, and the software and hardware implementation of the PMDS array code are facilitated.

Fig. 5 is a flowchart of a data storage method according to an embodiment of the present application. The method may be performed by a controller in the application scenario shown in fig. 1, as shown in fig. 5, the method comprising the following steps.

S51: coding parameters and source files are acquired.

Wherein the encoding parameters comprise a first number, a second number and a third number, the third number being a fractional number, i.e. the number of elements of the data stored in each storage node.

The encoding parameters may be sent to the controller by the host, and the host may select the encoding parameters of the PMDS array code according to the size of the source file and the storage requirement.

S52: the source file is divided into a plurality of pieces of data to be stored.

In some possible implementations of the present application, the partitioning of the source file is performed according to a third number of the aforementioned encoding parameters.

Illustratively, the source file is composed of l [ mu (n-r) -s]F (F) _q Is composed of the elements in (a). Wherein r is a first number, s is a second number, l is a third number, μ is a local group number, n is a total node number in the local group, F _q Refers to a finite field.

The source file may be partitioned into μ (n-r) -s shares, each containing l elements, which may be laid out in a column, called a column vector.

Step S52 may include: dividing the source file into mu (n-r) -s portions, each portion containing l F _q Elements of (a) are respectively denoted as F with length l _q Column vector f above _i,0 ,f _i,1 ,…,f _i,n-r-1 I=0, …, μ -2, and f _μ-1,0 ,f _μ-1,1 ,…,f _μ-1,n-r-s-1 。

The column vector f _i,0 ,f _i,1 ,…,f _i,n-r-1 And corresponding to a local group, wherein the local group also comprises r local check nodes. Column vector f _μ-1,0 ,f _μ-1,1 ,…,f _μ-1,n-r-s-1 And the local group also comprises r local check nodes and s global check nodes. In this implementation, s global check nodes are located in the same local group, e.g., the last local group. In other implementations, s global check nodes may be located in multiple local groups.

S53: and determining a coding algorithm corresponding to the coding parameters, wherein the coding algorithm comprises a local check matrix and a global check matrix of the local code.

The local check matrix and the global check matrix are determined in advance, and a corresponding relation is established between the determined local check matrix and the global check matrix and the coding parameters. Therefore, after the controller receives the coding parameters sent by the host, the controller can select the corresponding local check matrix and the global check matrix to carry out data coding work.

In some possible implementations of the present application, the coding algorithm is determined based on a relation between the coding parameter and the coding algorithm, where the relation between the coding parameter and the coding algorithm includes:

when the first number r is greater than or equal to 2, the second number s is 3, the third numberl is r ^n’ In this case, the global check matrix includes a plurality of third-order cauchy-vandermonde matrices.

Illustratively, when the first number r is greater than or equal to 2, the second number s is 2, and the third number l is r ^n’ When n' is more than or equal to 2 and less than or equal to n, the corresponding coding algorithm I is as follows:

the local check matrix is:

the global check matrix is:

When the first number r is 2, the second number s is 2, and the third number l is 2, the corresponding coding algorithm II is as follows:

the local check matrix is:

the global check matrix is:

When the first number is equal to 3, when the first number r is greater than or equal to 2, the second number s is 3, and the third number l is r ^n’ When n' is more than or equal to 2 and less than or equal to n, the corresponding coding algorithm III is as follows:

the local check matrix is:

/>

the global check matrix is:

The upper linearity is irrelevant.

In this embodiment of the present application, the first number of rows is related to the number of rows of matrix blocks in the local check matrix, for example, if the first number is 2, the number of rows of matrix blocks in the local check matrix is 2, and if the first number is greater than 2, the number of rows of matrix blocks in the local check matrix is also greater than 2.

H in the three coding algorithms _i And P _i The way of obtaining (c) is respectively described:

coding algorithm one:

1) Let r be greater than or equal to 2, s=2, l=r ^n’ Wherein n' is less than or equal to 2 and less than or equal to n.

2) Let phi be:

3) Selecting a prime power q to make finite field F _q Comprising a multiplication subgroup G of at least phi elements and mu cosets, from each of which one element is selected, in turn denoted as theta ₀ ,θ ₁ ,…,θ _μ-1 。

4) Selecting rn elements lambda from G _j,0 ,λ _j,1 ,…,λ _j,r-1 ，0≤j<n is such that:

a、λ _j,u ≠λ _i,v for all 0.ltoreq.u, v<r，0≤j,i<n and j≡i mod n' are all true;

b、λ _j,u ≠λ _j+gn’,u for all 0.ltoreq.u<r，1≤g<n/n’，0≤j<n 'and j+gn'<n is all true;

c. for any 0.ltoreq.j<n，λ _j,0 ,λ _j,1 ,…,λ _j,r-1 Two are mutually unequal;

the parameter lambda meeting the above requirements _j,0 ,λ _j,1 ,…,λ _j,r-1 ，0≤j<n is easier to select and will not be described in detail here.

5) Order e ₀ ,e ₁ ,...,e _l-1 Is that

Standard orthonormal basis on, i.e. e _i = (0, …,0,1,0, …, 0), only the i-th component is 1, 0.ltoreq.i<l. Definition { e ₀ ,e ₁ ,...,e _l-1 Subset V of } _i,t The method comprises the following steps:

V _i,t ＝{e _a |a _i ＝t,0≤a<l},0≤i<n’,0≤t<r,

wherein a is _i Is the ith component in the r-ary expansion of a. For simplicity, V is also used _i,t Representing a matrix of l/rxl whose row vectors are defined by V _i,t Vector e of (a) _j The components are arranged in ascending order.

6) Design matrix A ₀ ,A ₁ ,...,A _n-1 The following are provided:

wherein the method comprises the steps of

Represents the remainder after j modulo n', where A _i Is a matrix.

7) Design matrix H _i ,0≤i<Mu is as follows:

wherein I is an identity matrix, namely the elements on the main diagonal are all 1, and the rest are all 0.

8) Design matrix P _i ,0≤i<Mu is as follows:

coding algorithm II:

1) Let r=s=l=2. Selecting a prime power q to make the finite field F _q Comprising a multiplicative subgroup G of at least n elements and mu cosets, from each of which one element is selected, in turn denoted as θ ₀ ,θ ₁ ,…,θ _μ-1 。

2) From finite field F _q Optionally n elements lambda which are mutually different from each other ₀ ,λ ₁ ,…,λ _n-1 。

3) Design matrixA ₀ ,A ₁ ,...,A _n-1 The following are provided:

4) Design matrix H _i ,0≤i<Mu is as follows:

here I denotes a 2-order identity matrix.

5) Design matrix P _i ,0≤i<Mu is as follows:

and a coding method III:

1) Let r be equal to or greater than 2, s= 3,l =r ^n’ Wherein n' is less than or equal to 2 and less than or equal to n.

2) Let phi be

3) Selecting a prime power q ₀ So that the finite field

Comprising a multiplication subgroup G' of at least phi+1 elements and mu cosets, from each of which one element is selected, in turn denoted as theta ₀ ,θ ₁ ,…,θ _μ-1 。

4) From finite fields

Optionally 2 different elements d ₀ ,d ₁ Let->

5) From Ω\ { d ₀ The rn elements lambda are selected from } _i,0 ,λ _i,1 ,…,λ _i,r-1 ，0≤i<n, such that:

a、λ _i,u ≠λ _j,v for all 0.ltoreq.u, v<r，0≤i,j<n and j≡i mod n' are all true;

b、λ _i,u ≠λ _i+gn’,u for all 0.ltoreq.u<r，1≤g<n/n’，0≤i<n 'and i+gn'<n is all true;

c. for any 0.ltoreq.i<n，λ _i,0 ,λ _i,1 ,…,λ _i,r-1 The two are mutually unequal.

The parameter lambda meeting the above requirements _i,0 ,λ _i,1 ,…,λ _i,r-1 ，0≤i<n is easier to select and will not be described in detail here.

6) Order

From F _q Selecting mu elements delta ₀ ,δ ₁ ,…,δ _μ-1 So that any 3 elements are in the finite field

The upper linearity is irrelevant, and elements meeting the constraint condition are easy to select, and are not described in detail herein.

7) Order e ₀ ,e ₁ ,...,e _l-1 Is that

V _i,t ＝{e _a |a _i ＝t,0≤a<l},0≤i<n’,0≤t<r,

wherein a is _i Is the ith component in the r-ary expansion of a. For simplicity, V is also used _i,t Representing a l/rxlMatrix with row vector of V _i,t Vector e of (a) _j The components are arranged in ascending order.

8) Design matrix A ₀ ,A ₁ ,...,A _n-1 The following are provided:

wherein, the liquid crystal display device comprises a liquid crystal display device,

the remainder after j modulo n' is represented.

9) Design matrix A' ₀ ,A′ ₁ ,...,A′ _n-1 The following are provided:

10 Design matrix A ₀ ,A″ ₁ ,...,A″ _n-1 The following are provided:

11 A) and a design matrix H _i ,0≤i<Mu is as follows:

12 A) and a design matrix P _i ,0≤i<Mu is as follows:

s54: first local check data is calculated based on the local check matrix and the target data.

Illustratively, step S54 includes: the local verification data of the first local group is calculated according to the following formula:

wherein H is _i Is a local check matrix, f _i,0 ,f _i,1 ,…,f _i,n-r-1 For the data to be stored in the first partial group, f _i,n-r …,f _i,n-1 For locally verifying the data.

S55: and calculating second local check data and global check data based on the global check matrix, the local check matrix, the plurality of data to be stored and the first local check data.

Illustratively, step S55 includes: calculating the local check data and the global check data of the second local group according to the following formula:

wherein H is _μ-1 Is a local check matrix, P ₀ …,P _μ-1 Is a global check matrix, f _μ-1,0 ,f _μ-1,1 ,…,f _μ-1,n-r-s-1 For the data to be stored in the second partial group, f _μ-1,n-r-s …,f _μ-1,n-s-1 For checking data locally, f _μ-1,n--s …,f _μ-1,n-1 For global check data, 0 _rl×1 For a rl row 1 column matrix, the elements in the matrix are all 0.

S56: storing a plurality of data to be stored in system nodes of a plurality of local groups, storing first local check data in local check nodes of the first local group, storing second local check data in local check nodes of the second local group, and storing global check data in global check nodes of the second local group, wherein the first local group is a local group which does not comprise the global check nodes, and the second local group is a local group which comprises the global check nodes.

Optionally, the method further comprises:

s57: and in response to the reconfiguration instruction, determining the number of nodes of the unconnected nodes in each local group and the total number of unconnected nodes in the storage device when unconnected nodes exist in the storage device.

Wherein the unconnected nodes include at least one of offline nodes and failed nodes.

When the host needs to read the source file from the storage device, a reconstruction instruction is sent to the controller, and after the controller receives the reconstruction instruction, the reconstruction (or decoding) of the source file is performed based on the data in the storage device. In the reconstruction, there may be several storage nodes that are not connected, and the number of nodes that are not connected is different, and the reconstruction algorithm used is different.

It should be noted that, when all the nodes are connected, the data of each node is directly read and combined into the source file, and steps S57 to S59 are not required to be executed.

S58: the corresponding reconstruction algorithm is determined based on the number of nodes of the unconnected nodes in each local group and the total number of unconnected nodes in the storage device.

Illustratively, step S58 includes:

when the number of nodes of the unconnected nodes in each local group does not exceed r, a reconstruction algorithm I is as follows:

When the number of nodes with 1 partial group of unconnected nodes in the plurality of partial groups exceeds r and the total number of unconnected nodes in the storage device does not exceed r+s, a reconstruction algorithm II is as follows:

wherein H is _i Is a local check matrix, P _i R is the first number, 0, for the global check matrix _rl×1 For a matrix of rl row 1 columns,the elements in the matrix are 0,l as the packet number, mu as the local group number, f _i,0 The data in the 0 th node in the local group i is represented, i and j represent local group serial numbers, and the local group with the number of unconnected nodes exceeding r is the local group i.

When the number of nodes with 2 local group unconnected nodes in the plurality of local groups exceeds r and the total number of unconnected nodes in the storage device does not exceed 2r+s, a reconstruction algorithm III is as follows:

When the number of nodes with 3 local group unconnected nodes in the plurality of local groups exceeds r and the total number of unconnected nodes in the storage device does not exceed 3r+s, a reconstruction algorithm is as follows:

wherein H is _i Is a local check matrix, P _i R is the first number, 0, for the global check matrix _3rl×1 A matrix with 3rl rows and 1 column, wherein the elements in the matrix are 0,l as packet numbers, mu as the number of partial groups and f _i,0 The data in the 0 th node in the local group i is represented, i, j, k and m represent local group numbers, and the local groups with the number of nodes of unconnected nodes exceeding r are the local group i, the local group j and the local group k. It should be noted that, in the matrix provided in the present application, a blank portion (e.g., H in the first matrix on the left side in the above formula _i And P _i The part in between) is filled with 0.

S59: and adopting the determined reconstruction algorithm to solve the data in the unconnected nodes and reconstructing the source file.

That is, the data in each partial group is calculated according to the formula determined in step S58, and then synthesized as the source file.

Optionally, the method further comprises:

s510: and determining a repair algorithm corresponding to the coding algorithm in response to a failure of a storage node in the storage device.

When a storage node in the storage device fails, a new storage node is inserted for replacement, but no data in the failed storage node exists in the newly inserted storage node, at this time, the data in the failed storage node is determined by repairing, and then the data is written into the newly inserted storage node.

Illustratively, step S510 includes:

when the coding algorithm is the coding algorithm corresponding to the first coding parameter or the third coding parameter, the repairing algorithm is as follows:

wherein f _k,i Representing data in a failed node, wherein the failed node is an ith node in a local group k, S _i ＝V _i,0 +V _i,1 +…+V _i,r-1 J represents a node sequence number;

is easy to calculate from the downloaded data.

When the coding algorithm is the coding algorithm corresponding to the second coding parameter, the repairing algorithm II is as follows:

if the number i of the failed node is even, then:

wherein s= (1 0);

if the number i of the failed node is an odd number, then:

wherein f _k,i Representing the data in the failed node and,

wherein the following equation is easily calculated from the downloaded data:

S511: and repairing the data in the failure node by adopting the determined repairing algorithm.

Illustratively, step S511 includes:

when the relation between the serial numbers of the first node and the failure node in the local group is the first relation, downloading all data from the first node; when the relation between the serial numbers of the first node and the failure node in the local group is the second relation, downloading the product of all data and coefficients from the first node;

and repairing the data in the failure node by adopting a repairing algorithm based on the downloaded data.

Wherein, the first relation is: j≡i mod n';

the second relationship is:

wherein i is the serial number of the failure node, and j is the serial number of the first node.

For example, assume that node i in local group k fails, i.e., f _k,i Is corrupted.

If there are no other failed nodes or busy nodes in the local group k where the failed node is located, for the remaining nodes j in the local group k, if j≡i mod n', all the data f is downloaded from the nodes j in the local group k _k,j If (if)

Downloading data R from node j in local group k _i,j f _k,j Wherein R is _i,j ＝V _i,0 +V _i,1 +…+V _i,r-1 ；

The following will briefly describe the differences in downloaded data in the following two cases. All data f _k,j Refers to all data in the node, e.g. 4 elements a, b, c, d, and data R _i,j f _k,j Refers to the portion of data in the node, e.g., a, b. From the downloaded data, the aforementioned S can be obtained _i A _j For example S _i A _j A+b, etc., and will not be described in detail herein.

Optionally, if there are other dead nodes or busy nodes in the local group k, a reconstructor in the controller is called to reconstruct, that is, steps S57 to S59 are performed.

The data storage method provided in fig. 5 is exemplarily described below:

example one

In this example, the parameter settings are as follows: r=2, s=2, μ=2N' =2, n=8, the storage device contains n=16 storage nodes, k=10 system nodes, the number of packets l=4, the finite field used is F ₁₇ A source file size of 40 finite fields F ₁₇ The element in (a) is denoted as f _0,0 ,f _0,1 ,…,f _0,5 ,f _1,0 ,f _1,1 ,…,f _1,3 Either (e.g. f _0,0 ) Is F ₁₇ The last length 4 column vector.

The data f to be stored in the check node after the encoding is calculated by adopting an encoding algorithm _0,6 ,f _0,7 ,f _1,4 ,f _1,5 ,f _1,6 ,f _1,7 。

For example, the multiplication subgroup g= {2,2 is selected ² ,…,2 ⁸ And the corresponding 2 cosets are G and 3G. The following elements may be selected from G:

λ _0,0 ＝2,λ _1,0 ＝2 ² ,λ _2,0 ＝2 ³ ,λ _3,0 ＝2 ⁴ ,λ _4,0 ＝2 ⁵ ,λ _5,0 ＝2 ⁶ ,λ _6,0 ＝2 ⁷ ,λ _7,0 ＝2 ⁸ ，

λ _0,1 ＝2 ⁵ ,λ _1,1 ＝2 ⁶ ,λ _2,1 ＝2 ⁷ ,λ _3,1 ＝2 ⁸ ,λ _4,1 ＝2,λ _5,1 ＝2 ² ,λ _6,1 ＝2 ³ ,λ _7,1 ＝2 ⁴ 。

Selecting θ from G ₀ =2, select θ from 3G ₁ ＝6。

The subset of the upper orthonormal basis set is: v (V) _0,0 ＝{e ₀ ,e ₁ },V _0,1 ＝{e ₂ ,e ₃ },V _1,0 ＝{e ₀ ,e ₂ },V _1,1 ＝{e ₁ ,e ₃ Matrix A ₀ ,…,A ₇ ,H _i ,P ₀ ,P ₁ The method comprises the following steps of:

from the MDS encoding algorithm, it can be calculated:

/>

i.e.

Further, it can be calculated that:

from this, the content that each storage node needs to store can be obtained.

In reconstructing the source file, assuming that each partial group has 3 nodes (the first 3 nodes can be set) unconnected, f can be recovered by the following equation according to the reconstruction algorithm III _0,0 ,f _0,1 ,f _0,2 ,f _1,0 ,f _1,1 ,f _1,2 ：

In node repair, node 0 is assumed to fail without loss of generality, i.e. f _0,0 Data loss in (1), according to repair algorithm one, downloading f _0,2 ,f _0,4 ,f _0,6 ,f _0,i (0)+f _0,i (2),f _0,i (1)+f _0,i (3) I=1, 3,5,7, where f _0,i (j) Represents f _0,i The j-th component of (b). Node repair may be performed by the following formula:

in this example, by using an [8,6] epsilon-MSR code (or MDS array code with smaller packet number) as the local code and matching 2 brand new global checks to construct the PMDS array code, the technical problem that the packet number of the PMDS array code in the related art is too large can be solved, and meanwhile, the finite field size can be greatly reduced, and the complexity of system implementation can be greatly reduced.

Based on the distributed storage system of the embodiment of the application, the packet number is only 4, the minimum packet number can be implemented on GF (17), but in the related art, the packet number under the same parameters in the scheme needs to be 256, and the implementation on the operating field GF (q) with q being more than or equal to 80 is needed. Because the adopted local code is an MDS array code with smaller packet number, the packet number can be greatly reduced, and meanwhile, the global verification is optimized, so that the finite field size is further reduced.

Example two

In this example, the parameter settings are as follows: r=2, s=2, μ=2, N' =2, n=8, the storage device contains n=16 storage nodes, k=10 system nodes, the number of packets l=4, the finite field used is F ₁₇ Source file size is 20 finite fields F ₁₇ The element in (a) is denoted as f _0,0 ,f _0,1 ,…,f _0,5 ,f _1,0 ,f _1,1 ,…,f _1,3 Either (e.g. f _0,0 ) Is F ₁₇ Up a column vector of length 2.

The data f to be stored in the check node after the encoding is calculated by adopting the encoding algorithm II _0,6 ,f _0,7 ,f _1,4 ,f _1,5 ,f _1,6 ,f _1,7 。

λ ₀ ＝2,λ ₁ ＝2 ² ,λ ₂ ＝2 ³ ,λ ₃ ＝2 ⁴ ,λ ₄ ＝2 ⁵ ,λ ₅ ＝2 ⁶ ,λ ₆ ＝2 ⁷ ,λ ₇ ＝2 ⁸ ，

selecting θ from G ₀ =2, select θ from 3G ₁ ＝6。

Finite field F ₁₇ Upper matrix A ₀ ,…,A ₇ ,H _i ,P ₀ ,P ₁ The method comprises the following steps of:

the process of computing the check code, reconstructing the source file is similar to that of example one and is omitted here.

In node repair, it is assumed that node 0 fails, i.e. f _0,0 Data loss in the file, and downloading f according to a second repair algorithm _0,2 ,f _0,4 ,f _0,6 ,f _0,i (0) I=1, 3,5,7, where f _0,i (j) Representation off _0,i The j-th component of (b). Node repair may be performed by the following formula:

assuming node 1 fails, i.e. f _0,1 Data loss in the file, and downloading f according to a second repair algorithm _0,3 ,f _0,5 ,f _0,7 ,f _0,i (0) I=0, 2,4,6. Node repair may be performed by the following formula:

in this example, by adopting an MDS array code with a packet number of 2 of [8,6] as a local code, and having small input/output (I/O) consumption during node repair, and configuring a PMDS array code with 2 brand-new global checks, the technical problems of too large packet number of the PMDS array code and excessive I/O during node repair in the related art can be solved. Compared with the related art, the ratio of the I/O consumed in node repair to the file size can be reduced from 0.7 to 0.5, and the I/O consumed in node repair is smaller.

Example three

In this example, the parameter settings are as follows: r=2, s=3, μ=3, N' =2, n=8, the storage device contains n=24 storage nodes, k=15 system nodes, the number of packets l=4, the finite field used is

Source file size 60 finite fields

The element in (a) is denoted as f _0,0 ,f _0,1 ,…,f _0,5 ,f _0,0 ,f _1,1 ,…,f _1,5 ,f _2,0 ,f _2,1 ,f _2,2 Either (e.g. f _0,0 ) Is->

Up a column vector of length 4.

The data f to be stored in the check node after the encoding is calculated by adopting the encoding algorithm three _0,6 ,f _0,7 ,f _1,6 ,f _1,7 ,f _2,3 ,f _2,4 ,f _2,5 ,f _2,6 ,f _2,7 。

For example, select finite field F ₃₁ The multiplicative subgroup g= {27,27 in (a) ² ,…,27 ⁹ And the corresponding 3 cosets are G,3G and 9G. Selecting delta from G ₀ =27, select δ from 3G ₁ =19, select δ from 9G ₂ =26. From finite field F ₃₁ D is arbitrarily selected from ₀ ＝1,d ₁ =3, Ω= {19,26,30,4,5,11,2,18,7,1}, from Ω the following elements can be selected:

λ _0,0 ＝19,λ _1,0 ＝26,λ _2,0 ＝30,λ _3,0 ＝4,λ _4,0 ＝5,λ _5,0 ＝11,λ _6,0 ＝2,λ _7,0 ＝18，

λ _0,1 ＝5,λ _1,1 ＝11,λ _2,1 ＝2,λ _3,1 ＝18,λ _4,1 ＝19,λ _5,1 ＝26,λ _6,1 ＝30,λ _7,1 ＝4。

order the

Epsilon is +.>

Primitive element of (a), let θ ₀ ＝1+ε+ε ² ,θ ₁ ＝1+2ε+4ε ² ,θ ₂ ＝1+3ε+9ε ² Theta is then ₀ ,θ ₁ ,θ ₂ In finite field F ₃₁ The upper linearity is irrelevant.

The subset of the upper orthonormal basis set is: v (V) _0,0 ＝{e ₀ ,e ₁ },V _0,1 ＝{e ₂ ,e ₃ },V _1,0 ＝{e ₀ ,e ₂ },V _1,1 ＝{e ₁ ,e ₃ }. Matrix A ₀ ,…,A ₇ ,A′ ₀ ,…,A′ ₇ ,A″ ₀ ,…,A″ ₇ ,H _i ,P ₀ ,P ₁ The method comprises the following steps of:

/>

the calculation of the check code, reconstruction of the source file, and node repair process are similar to example one and are omitted here.

In this example, by adopting an MDS array code with a packet number of 4 of [8,6] as a local code and collocating 3 brand new global checks to construct a PMDS array code, the technical problem that the PMDS array code constructed on a smaller finite field in the related art only supports 2 global checks can be solved, and the supportable global checks are promoted from 2 to 3.

In summary, the application constructs the PMDS array code with smaller packet number, smaller repair bandwidth and I/O by adopting the MDS array code with smaller packet number as the local code and skillfully matching the corresponding global check equation, provides 3 solutions for contradiction between the packet number, the repair bandwidth and the I/O in the PMDS code, and provides 3 implementation methods of the distributed storage system/device. The coding algorithm I and the coding algorithm III greatly reduce the sub-packet number of the PMDS array code, provide a flexible compromise scheme about the sub-packet number and the repair bandwidth, and have small finite field. The packet number of the coding algorithm II is only 2, the I/O overhead during repair can be greatly reduced, the finite field is small, and the implementation is easy. The third coding algorithm is PMDS array codes of the first type supporting certainty of 3 global checks on a smaller finite field. And, the coding complexity of the three coding algorithms is low. Table 1 below compares the specific parameters of the PMDS array code in the related art and the three PMDS array codes proposed in the present application, wherein for more intuitiveness, the coding scheme one of the present application is split into two cases where t=1 and t is any positive integer (table 3 rd and 4 th rows below).

TABLE 1

Wherein r is ^n/t I.e. the aforementioned r ^n’ 。

As can be seen from the above results, the PMDS array code provided by the present application can greatly reduce the packet count compared with the related art. Compared with the related technology, the PMDS array code I (corresponding to the first coding algorithm) II (corresponding to the second coding algorithm) provided by the application can also greatly reduce the size of the finite field; compared with the related technology, the PMDS array code II provided by the application can also greatly reduce the I/O consumed during node repair; compared with the related technology, the PMDS array code III (corresponding to the coding algorithm III) can support more global verification.

Fig. 6 is a block diagram of a data storage device according to an embodiment of the present application. The data storage device may be implemented in software, hardware, or a combination of both as all or part of the controller. The data storage device may include: a segmentation module 701, a processing module 702, and an output module 703.

The partitioning module 701 is configured to partition a source file into a plurality of to-be-stored data, where the plurality of to-be-stored data is used for a plurality of local groups stored in a storage device, each of the plurality of local groups includes a plurality of system nodes and a first number of local check nodes, and a part of the plurality of local groups further includes a second number of global check nodes;

The processing module 702 is configured to encode target data by using a local check matrix in a local code of a partial maximum distance separable PMDS array code to obtain first local check data, where the local code uses an epsilon-minimum storage regeneration epsilon-MSR code in which a matrix block is a diagonal matrix, a second-order upper triangular matrix or a second-order lower triangular matrix, and the target data is data to be stored in a first local group; adopting a global check matrix and a local check matrix of a PMDS array code to encode a plurality of data to be stored and the first local check data to obtain second local check data and global check data;

the output module 703 is configured to store a plurality of data to be stored in system nodes of a plurality of local groups, store first local check data in local check nodes of the first local group, store second local check data in local check nodes of the second local group, store global check data in global check nodes of the second local group, where the first local group is a local group that does not include the global check nodes, and the second local group is a local group that includes the global check nodes.

Optionally, the processing module 702 is configured to obtain the encoding parameter, determine an encoding algorithm corresponding to the encoding parameter, where the encoding algorithm includes a local check matrix of the local code; calculating first local check data based on the local check matrix and the target data; the encoding algorithm further includes a global check matrix, and a processing module 702, which is further configured to calculate second local check data and global check data based on the global check matrix, the local check matrix, the plurality of data to be stored, and the first local check data.

Optionally, the encoding algorithm is determined based on a relationship between the encoding parameter and the encoding algorithm, the encoding parameter including a first number, a second number, and a third number, the third number being a number of packets, the relationship between the encoding parameter and the encoding algorithm including:

when the first number r is greater than or equal to 2, the second number s is 2, and the third number l is r ^n’ When the matrix block of the global check matrix is a diagonal matrix, wherein n' is more than or equal to 2 and less than or equal to n, and n is the total node number in the local group;

when the first number r is 2, the second number s is 2, and the third number l is 2, the matrix block of the global check matrix is a diagonal matrix;

Alternatively, whenThe first number r is greater than or equal to 2, the second number s is 2, and the third number l is r ^n’ Wherein when n' is more than or equal to 2 and less than or equal to n,

the local check matrix is:

the global check matrix is:

the local check matrix is:

the global check matrix is:

Alternatively, when the first number r is greater than or equal to 2, the second number s is 3, and the third number l is r ^n’ Wherein when n' is more than or equal to 2 and less than or equal to n,

the local check matrix is:

/>

the global check matrix is:

wherein mu is the number of local groups, i is the number of local groups, H _i Is a local check matrix, P _i Is a global check matrix, I is a unit matrix, A _j 、A′ _j 、A″ _j In the form of a matrix block,

The upper linearity is irrelevant.

Optionally, the processing module 702 is further configured to determine, in response to the reconfiguration instruction, when there are unconnected nodes in the storage device, a number of nodes of the unconnected nodes in each local group and a total number of unconnected nodes in the storage device, where the unconnected nodes include at least one of an offline node and a failure node; determining a corresponding reconstruction algorithm based on the number of nodes of the unconnected nodes in each local group and the total number of unconnected nodes in the storage device; and adopting the determined reconstruction algorithm to solve the data in the unconnected nodes and reconstructing the source file.

Optionally, the processing module 702 is configured to, when the number of nodes of the unconnected nodes in each local group does not exceed r, reconstruct as follows:

Optionally, the processing module 702 is configured to, when the number of nodes in which 1 local group of unconnected nodes exists in the plurality of local groups exceeds r, and the total number of unconnected nodes in the storage device does not exceed r+s, reconstruct as follows:

Optionally, the processing module 702 is configured to, when the number of nodes in which 2 local groups of unconnected nodes exist in the plurality of local groups exceeds r, and the total number of unconnected nodes in the storage device does not exceed 2r+s, reconstruct as follows:

wherein H is _i Is a local check matrix, P _i R is the first number, 0, for the global check matrix _2rl×1 A matrix with 2rl rows and 1 column, wherein the elements in the matrix are 0,l as packet numbers, mu as the number of partial groups and f _i,0 Representing localThe data in the 0 th node in the group i, j and k represent local group numbers, and the local groups with the number of nodes of unconnected nodes exceeding r are the local group i and the local group j.

Optionally, the processing module 702 is configured to, when the number of nodes in which 3 local groups of unconnected nodes exist in the plurality of local groups exceeds r, and the total number of unconnected nodes in the storage device does not exceed 3r+s, reconstruct as follows:

Optionally, the processing module 702 is further configured to determine a repair algorithm corresponding to the encoding algorithm in response to a failure of a storage node in the storage device; and repairing the data in the failure node by adopting the determined repairing algorithm.

Optionally, the processing module 702 is configured to, when the coding algorithm is a coding algorithm corresponding to the first coding parameter or the third coding parameter, repair the algorithm as follows:

the first encoding parameters include: the first number r is greater than or equal to 2, the second number s is 2, and the third number l is r ^n’ Wherein n' is more than or equal to 2 and less than or equal to n; the third encoding parameters include: when the first numberThe quantity r is greater than or equal to 2, the second quantity s is 3, and the third quantity l is r ^n’ 。

Optionally, a processing module 702 is configured to

if the number i of the failed node is even, then:

wherein s= (1 0);

if the number i of the failed node is an odd number, then:

wherein f _k,i Representing the data in the failed node and,

Optionally, the processing module 702 is configured to determine a relationship between the sequence numbers of the first node and the failed node in the local group if there are no other failed nodes or busy nodes in the local group where the failed node is located; when the relation between the serial numbers of the first node and the failure node in the local group is the first relation, downloading all data from the first node; when the relation between the serial numbers of the first node and the failure node in the local group is the second relation, downloading the product of all data and coefficients from the first node; and repairing the data in the failure node by adopting a repairing algorithm based on the downloaded data.

Optionally, the first relationship is: j≡i mod n';

the second relationship is:

wherein i is the serial number of the failure node, and j is the serial number of the first node. The application provides three PMDS array codes, which correspond to the three coding algorithms in sequence. Wherein the first and second PMDS array codes are substantially reduced in finite field size; the second PMDS array code can greatly reduce I/O consumed during node repair; the third PMDS array code may support more global checks.

It should be noted that: in the data storage device provided in the above embodiment, only the division of the above functional units is used for illustration, and in practical application, the above functional allocation may be performed by different functional units according to needs, that is, the internal structure of the device is divided into different functional units, so as to complete all or part of the functions described above. In addition, the data storage device and the data storage method provided in the foregoing embodiments belong to the same concept, and specific implementation processes of the data storage device and the data storage method are detailed in the method embodiments and are not repeated herein.

The descriptions of the processes corresponding to the drawings have emphasis, and the descriptions of other processes may be referred to for the parts of a certain process that are not described in detail.

Fig. 7 shows a schematic structural diagram of a computer device 150 according to an exemplary embodiment of the present application. The computer device 150 shown in fig. 7 is used to perform the operations involved in the data storage method shown in fig. 4 or 5 described above. The computer device 150 may be implemented by a general bus architecture.

As shown in fig. 7, computer device 150 includes at least one processor 151, memory 153, and at least one communication interface 154.

Processor 151 is, for example, a general-purpose central processing unit (central processing unit, CPU), digital signal processor (digital signal processor, DSP), network processor (network processer, NP), graphics processor (Graphics Processing Unit, GPU), neural-network processor (neural-network processing units, NPU), data processing unit (Data Processing Unit, DPU), microprocessor, or one or more integrated circuits for implementing aspects of the present application. For example, processor 151 includes an application-specific integrated circuit (ASIC), a programmable logic device (programmable logic device, PLD) or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. PLDs are, for example, complex programmable logic devices (complex programmable logic device, CPLD), field-programmable gate arrays (field-programmable gate array, FPGA), general-purpose array logic (generic array logic, GAL), or any combination thereof. Which may implement or execute the various logical blocks, modules, and circuits described in connection with the present disclosure. The processor may also be a combination that performs the function of a computation, e.g., including one or more microprocessors, a combination of a DSP and a microprocessor, and so forth.

Optionally, the computer device 150 further comprises a bus. The bus is used to transfer information between the components of the computer device 150. The bus may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, only one thick line is shown in fig. 7, but not only one bus or one type of bus.

The Memory 153 is, for example, but not limited to, a read-only Memory (ROM) or other type of static storage device that can store static information and instructions, as well as a random access Memory (random access Memory, RAM) or other type of dynamic storage device that can store information and instructions, as well as an electrically erasable programmable read-only Memory (electrically erasable programmable read-only Memory, EEPROM), compact disc read-only Memory (compact disc read-only Memory) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, blu-ray disc, etc.), magnetic disk storage media, or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. The memory 153 is, for example, independent and is connected to the processor 151 via a bus. Memory 153 may also be integrated with processor 151.

The communication interface 154 uses any transceiver-like device for communicating with other devices or communication networks, which may be ethernet, radio Access Network (RAN) or wireless local area network (wireless local area networks, WLAN), etc. Communication interface 154 may include a wired communication interface and may also include a wireless communication interface. Specifically, the communication interface 154 may be an Ethernet (Fast Ethernet) interface, a Fast Ethernet (FE) interface, a Gigabit Ethernet (GE) interface, an asynchronous transfer mode (Asynchronous Transfer Mode, ATM) interface, a wireless local area network (wireless local area networks, WLAN) interface, a cellular network communication interface, or a combination thereof. The ethernet interface may be an optical interface, an electrical interface, or a combination thereof. In embodiments of the present application, the communication interface 154 may be used to communicate with the computer device 150 with other devices.

In a specific implementation, processor 151 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG. 7, as an embodiment. Each of these processors may be a single-core (single-CPU) processor or may be a multi-core (multi-CPU) processor. A processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).

In a particular implementation, as one embodiment, computer device 150 may include multiple processors, such as processor 151 and processor 155 shown in FIG. 7. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).

In a specific implementation, computer device 150 may also include an output device and an input device, as one embodiment. The output device communicates with the processor 151 and information may be displayed in a variety of ways. For example, the output device may be a liquid crystal display (liquid crystal display, LCD), a light emitting diode (light emitting diode, LED) display device, a Cathode Ray Tube (CRT) display device, or a projector (projector), or the like. The input device(s) are in communication with the processor 151 and may receive user input in a variety of ways. For example, the input device may be a mouse, a keyboard, a touch screen device, a sensing device, or the like.

In some embodiments, memory 153 is used to store program code 1510 that performs aspects of the present application, and processor 151 may execute program code 1510 stored in memory 153. That is, the computer device 150 may implement the data storage method provided by the method embodiment through the processor 151 and the program code 1510 in the memory 153. One or more software modules may be included in the program code 1510. Alternatively, processor 151 itself may store program code or instructions for performing the aspects of the present application.

In a specific embodiment, the computer device 150 of the embodiment of the present application may correspond to the controller in the above-described method embodiments, and the processor 151 in the computer device 150 reads the instructions in the memory 153, so that the computer device 150 shown in fig. 7 can perform all or part of the operations performed by the controller.

Specifically, the processor 151 is configured to divide the source file into a plurality of to-be-stored data, where the plurality of to-be-stored data is used for storing a plurality of local groups in the storage device, each local group in the plurality of local groups includes a plurality of system nodes and a first number of local check nodes, and a part of local groups in the plurality of local groups further includes a second number of global check nodes; encoding target data by adopting a local check matrix in local codes of partial maximum distance separable PMDS array codes to obtain first local check data, wherein the local codes adopt epsilon-minimum storage regeneration epsilon-MSR codes of which matrix blocks are diagonal matrixes, second-order upper triangular matrixes or second-order lower triangular matrixes, and the target data are to-be-stored data to be stored in a first local group; adopting a global check matrix and a local check matrix of the PMDS array code to encode the data to be stored and the first local check data to obtain second local check data and global check data; storing the data to be stored in the plurality of local groups to the system nodes of the plurality of local groups, storing the first local check data to the local check nodes of the first local group, storing the second local check data to the local check nodes of the second local group, and storing the global check data to the global check nodes of the second local group, wherein the first local group is a local group not comprising the global check nodes, and the second local group is a local group comprising the global check nodes.

Other optional embodiments are not described here again for brevity.

Wherein the steps of the data storage method shown in fig. 4 or 5 are accomplished by instructions in the form of integrated logic circuits or software of hardware in the processor of the computer device 150. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in a hardware processor for execution, or in a combination of hardware and software modules in the processor for execution. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads information in the memory, and in combination with its hardware, performs the steps of the above method, which will not be described in detail here to avoid repetition.

The embodiment of the application also provides a chip, which comprises: the input interface, the output interface, the processor and the memory are connected through an internal connection path, the processor is used for executing codes in the memory, and when the codes are executed, the processor is used for executing any data storage method.

It is to be appreciated that the processor described above may be a CPU, but may also be other general purpose processors, DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or any conventional processor or the like. It should be noted that the processor may be a processor supporting the ARM architecture.

Further, in an alternative embodiment, the processor is one or more, and the memory is one or more. Alternatively, the memory may be integrated with the processor or the memory may be separate from the processor. The memory may include read only memory and random access memory and provide instructions and data to the processor. The memory may also include non-volatile random access memory. For example, the memory may also store a reference block and a target block.

The memory may be volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The non-volatile memory may be ROM, PROM, EPROM, EEPROM or flash memory, among others. The volatile memory may be RAM, which acts as external cache. By way of example, and not limitation, many forms of RAM are available. For example SRAM, DRAM, SDRAM, DDR SDRAM, ESDRAM, SLDRAM and DR RAM.

In an embodiment of the present application, there is also provided a computer-readable storage medium storing computer instructions that, when executed by a computer device, cause the computer device to perform the above-provided data storage method.

In an embodiment of the present application, there is also provided a computer program product containing instructions which, when run on a computer device, cause the computer device to perform the data storage method provided above.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the present application are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk), etc.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The foregoing is merely an optional embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that may be easily contemplated by those skilled in the art within the technical scope of the present application should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Unless defined otherwise, technical or scientific terms used herein should be given the ordinary meaning as understood by one of ordinary skill in the art to which this application belongs. The terms "first," "second," "third," and the like in the description and in the claims, are not used for any order, quantity, or importance, but are used for distinguishing between different elements. Likewise, the terms "a" or "an" and the like do not denote a limitation of quantity, but rather denote the presence of at least one. The word "comprising" or "comprises", and the like, is intended to mean that elements or items that are present in front of "comprising" or "comprising" are included in the word "comprising" or "comprising", and equivalents thereof, without excluding other elements or items.

The foregoing is merely an embodiment of the present application and is not intended to limit the present application, and any modifications, equivalent substitutions, improvements, etc. that fall within the spirit and principles of the present application are intended to be included in the scope of the present application.

Claims

1. A method of data processing, the method comprising:

adopting the global check matrix and the local check matrix of the PMDS array code to encode the data to be stored and the first local check data to obtain second local check data and global check data;

2. The method of claim 1, wherein the encoding the target data using the local check matrix in the local code of the PMDS array code to obtain the first local check data includes:

the encoding algorithm further includes the global check matrix, the global check matrix and the local check matrix using the PMDS array code encode the plurality of data to be stored and the first local check data to obtain second local check data and global check data, including:

3. The method of claim 2, wherein the encoding algorithm is determined based on a relationship of the encoding parameters and the encoding algorithm, the encoding parameters including the first number, the second number, and a third number, the third number being a number of packets, the relationship of the encoding parameters and the encoding algorithm including:

4. A method according to claim 3, wherein, when said first phase isThe number r is greater than or equal to 2, the second number s is 2, and the third number l is r ^n’ Wherein when n' is more than or equal to 2 and less than or equal to n,

The local check matrix is as follows:

0≤i<μ；

the global check matrix is as follows:

0≤i<μ；

5. A method according to claim 3, wherein when the first number r is 2, the second number s is 2, and the third number l is 2,

the local check matrix is as follows:

0≤i<μ；

the global check matrix is as follows:

0≤i<μ；

6. A method according to claim 3, wherein when the first number r is greater than or equal to 2, the second number s is 3, and the third number l is r ^n’ Wherein when n' is more than or equal to 2 and less than or equal to n,

the local check matrix is as follows:

0≤i<μ；

The global check matrix is as follows:

0≤i<μ；

is A _j To the power of r, θ ₀ ,θ ₁ ,…,θ _μ-1 To be from finite field F _q The mu cosets contained in the sequence are respectively selected to be a sequence composed of one element, delta ₀ ,δ ₁ ,…,δ _μ-1 To be from finite field F _q Selected from mu elements, and any 3 of themThe element is in finite field F _q0 The upper linearity is irrelevant.

7. A data storage device, the device comprising:

the processing module is used for encoding target data by adopting a local check matrix in local codes of partial maximum distance separable PMDS array codes to obtain first local check data, wherein the local codes adopt epsilon-minimum storage regeneration epsilon-MSR codes of which matrix blocks are diagonal matrixes, second-order upper triangular matrixes or second-order lower triangular matrixes, and the target data are to-be-stored data to be stored in a first local group; adopting the global check matrix and the local check matrix of the PMDS array code to encode the data to be stored and the first local check data to obtain second local check data and global check data;

8. The apparatus of claim 7, wherein the processing module is configured to obtain a coding parameter, determine a coding algorithm corresponding to the coding parameter, the coding algorithm comprising a local check matrix of the local code;

the encoding algorithm further comprises the global check matrix, and the processing module is further configured to calculate the second local check data and the global check data based on the global check matrix, the local check matrix, the plurality of pieces of data to be stored and the first local check data.

9. The apparatus of claim 8, wherein the encoding algorithm is determined based on a relationship of the encoding parameter and the encoding algorithm, the encoding parameter comprising the first number, the second number, and a third number, the third number being a number of packets, the relationship of the encoding parameter and the encoding algorithm comprising:

10. The apparatus of claim 9, wherein when the first number r is greater than or equal to 2, the second number s is 2, and the third number l is r ^n’ Wherein when n' is more than or equal to 2 and less than or equal to n,

the local check matrix is as follows:

0≤i<μ；

the global check matrix is as follows:

0≤i<μ；

11. The apparatus of claim 9, wherein when the first number r is 2, the second number s is 2, and the third number l is 2,

the local check matrix is as follows:

0≤i<μ；

the global check matrix is as follows:

0≤i<μ；

12The apparatus of claim 9, wherein when the first number r is greater than or equal to 2, the second number s is 3, and the third number l is r ^n’ Wherein when n' is more than or equal to 2 and less than or equal to n,

the local check matrix is as follows:

0≤i<μ；

the global check matrix is as follows:

0≤i<μ；

The upper linearity is irrelevant.

13. A computer device, the computer device comprising a processor and a memory; the memory is used for storing a software program, and the processor is used for enabling the computer device to realize the method according to any one of claims 1 to 6 by running or executing the software program stored in the memory.

14. A computer readable storage medium for storing program code for execution by a processor, the program code comprising instructions for implementing the method of any one of claims 1 to 6.