WO2023098048A1

WO2023098048A1 - Method and apparatus for expanding erasure code storage system

Info

Publication number: WO2023098048A1
Application number: PCT/CN2022/101302
Authority: WO
Inventors: 沈志荣; 杜知城; 范瑞彬; 张开翔; 李辉忠; 李成博
Original assignee: 深圳前海微众银行股份有限公司; 厦门大学
Priority date: 2021-12-02
Filing date: 2022-06-24
Publication date: 2023-06-08
Also published as: CN114237970A

Abstract

Disclosed in the present invention are a method and apparatus for expanding an erasure code storage system. The method comprises: determining data in a storage system, encoding the data, dispersedly storing the data in nodes, and obtaining spatial position distribution information of the nodes; determining the number of newly added nodes on each stripe on the basis of expansion demand information, and determining expansion node information on each stripe on the basis of the number of newly added nodes and the spatial position distribution information, wherein each stripe comprises a data block and a check block which have an encoding relationship; determining an expansion group on the basis of the expansion node information and the rule of the least common multiple, and splitting the expansion group, so as to obtain a target group comprising a plurality of selected stripes; and executing an expansion algorithm on the target group, so as to obtain a corresponding target expanded group, wherein the target expanded group comprises an expanded data block and an expanded check block. On the basis of the method, the expansion efficiency of an erasure code storage system can be improved.

Description

A method and device for extending an erasure code storage system

Cross References to Related Applications

This application claims the priority of the Chinese patent application with the application number 202111459202.9 and the application name "A Method and Device for Extended Erasure Code Storage System" submitted to the China Patent Office on December 02, 2021, the entire contents of which are incorporated by reference incorporated in this application.

technical field

Embodiments of the present invention relate to the field of financial technology (Fintech), and in particular to a method and device for expanding an erasure code storage system.

Background technique

With the development of computer technology, more and more technologies are applied in the financial field, and the traditional financial industry is gradually transforming into financial technology. However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on technology.

At present, the storage system is deployed on a large number of storage nodes, and it is the main backbone supporting various upper-level applications such as information retrieval, machine learning, and video streaming. In order to ensure the reliability of data in the storage system, the storage system often relies on replication and erasure code technology, both of which need to store additional data redundancy in advance, so that the system can use redundancy to recover lost data, which is similar to replication Compared with erasure code technology, it can achieve higher data reliability with the same storage overhead.

Moreover, with the continuous growth of data, higher requirements are placed on the scalability of the storage system. Specifically, the implementation of the storage expansion function requires the storage system to perform two operations: data relocation and check block update. However, in the solutions provided in the prior art, in data relocation and check block update, the expansion of the storage system inevitably causes a large amount of data transmission, resulting in poor transmission parallelism and a long expansion process, that is, the expansion efficiency and The effect is poor.

Contents of the invention

The invention provides a method and device for expanding an erasure code storage system, which is used to solve the problem of low expansion efficiency of an erasure code storage system in the prior art.

In a first aspect, the present invention provides a method for expanding an erasure code storage system, the method comprising: determining data in the storage system, encoding the data, and dispersively storing the data in each node, Obtain the spatial location distribution information of each node; determine the number of newly added nodes on each strip based on the extended demand information, and determine each Extended node information on a stripe; the stripe includes a data block and a check block having an encoding relationship; based on the extended node information and the least common multiple rule, an extended group is determined, and the extended group is split, Obtaining a target group including a plurality of selected strips; the expansion group is composed of a plurality of strips that satisfy the expansion requirement and the condition that the spatial position distribution law remains unchanged; perform an expansion algorithm on the target group, A corresponding target extension group is obtained, where the target extension group includes an extended data block and an extended parity block.

In the above method, a new scaling mechanism is proposed with the aim of reducing traffic and exploring the scaling mechanism of transmission parallelism in continuous scaling. In this extension mechanism, a new stripe layout is designed, which utilizes locally stored data blocks for parity block update, thereby reducing the data transmission for parity block update. Therefore, the data transmission for parity block update can be reduced, thereby improving the expansion efficiency.

Optionally, the data is encoded, and the data is scattered and stored in each node, and the spatial position distribution information of each node is obtained, including: dividing the data into K data blocks of the same size; K is A positive integer greater than 1; performing an intra-domain matrix operation on the K data blocks and a preset encoding matrix to obtain M check blocks; M is a positive integer greater than 1 and less than K; the K data blocks and the M check blocks form multiple strips; the data blocks and check blocks on the same strip are scattered on different K+M nodes, and it is determined that the K data blocks and the M check blocks are in distribution information of each node, and obtain the spatial location distribution information based on the distribution information.

In the above method, specific processing of data and a manner of distributed storage of data blocks and check blocks are provided. Based on this method, a good implementation basis can be provided for subsequent expansion and updating of check blocks and data blocks based on the new stripe layout, thereby improving expansion efficiency.

Optionally, determining the extended node information on each stripe based on the newly added number of nodes and the spatial location distribution information includes: determining the data stored on each stripe based on the spatial location distribution information The first number of nodes of the block, and the second number of nodes storing the check block on each stripe; adding the first number of nodes and the number of newly added nodes to obtain the third number of nodes number, use the number of the third node as the number of expanded storage data blocks on each stripe; and use the number of the second node as the expanded storage checksum on each stripe The number of blocks to determine the extended node information on each stripe.

Based on the above method, the expanded node information on each stripe, the number of expanded storage data blocks and storage check blocks on each stripe can be accurately and quickly determined. In this way, a basis is provided for filling data of subsequent data blocks and check blocks, so as to quickly realize the migration of data blocks and update of check blocks, and improve the expansion efficiency.

Optionally, based on the extended node information and the least common multiple rule, determine an extended group, and split the extended group to obtain a target group including stripes with corresponding relationships, including: based on the extended node information and the least common multiple rule to determine an extended group; the extended group includes V extended strips; the V extended strips are split to determine P basic groups and R adjustment groups; each of the basic groups includes Vp basic strips, each of the adjustment groups includes Vr adjustment strips; P and R are positive integers greater than 1; K basic strips are selected from the basic group, and from the adjustment group Select D adjustment strips, and determine a target group based on the K basic strips and the D adjustment strips; the target group includes K+D strips.

Based on the above method, the expansion group can be split to determine the basic group including the stripes that need to be updated according to the data blocks on the newly added nodes, and the adjustment group including the stripes that send data blocks to the basic group, so that Based on the adjustment group and the basic group, the fast migration of the data block and the fast update of the check block can be realized.

Optionally, the least common multiple rule is determined by the following formula:

V=LCM(K, K+D+1)(K+D)(K+1)/K

Wherein, the LCM() is used to represent a function for obtaining the least common multiple, k is used to represent the number of nodes storing data blocks before extension on each stripe; d is used to represent the number of newly added nodes.

Based on the above method, the number of stripes included in the extended group can be accurately and quickly determined.

Optionally, an extension algorithm is executed on the target group to obtain a corresponding target extension group, the target extension group includes extended data blocks and extended check blocks, including: K+D blocks in any of the target groups Numbering the stripe, and numbering the K+M+D nodes after the storage system is expanded; calculating the difference check block for adjusting the data blocks on the stripe in the first K+1 nodes, and based on the difference check block Updating the first check block of the basic stripe on the same node; transferring the data block on the adjusted stripe to the basic stripe on the same node according to the round robin mode, and obtaining the expanded initial extended group; A preset operation is performed on the initial extended group to obtain a corresponding target extended group.

Based on the above method, the data block migration and verification block update of the erasure code storage system expansion are performed in parallel, that is, during the expansion process, some nodes are scheduled to perform the data block migration operation, and at the same time, the transmission task is assigned to another part of the nodes to perform the verification Block update operations, in this way, can improve scaling efficiency.

Optionally, after obtaining the target extended group, the method further includes: determining the logical relationship of the stripes corresponding to the target extended group, and the first spatial distribution information corresponding to each extended data block and extended parity block; The spatial distribution information adjusts the order of the logical relationship so that the logical layout of the first spatial distribution information is the same as that of the spatial distribution information.

Based on the above method, it can be supported that the space distribution does not need to be adjusted when the erasure code storage system performs the next expansion, thereby reducing unnecessary overhead. Additionally, functionality is provided to support continuous scaling of erasure coded storage systems.

In a second aspect, the present invention provides a device for expanding an erasure code storage system, the device comprising:

The first processing unit is configured to determine the data in the storage system, encode the data, and dispersely store the data in each node, and obtain the spatial location distribution information of each node; the second processing unit, It is used to determine the number of newly added nodes on each strip based on the extended demand information, and determine the expanded node information on each strip based on the number of newly added nodes and the spatial location distribution information; The strip includes a data block and a check block with an encoding relationship; the third processing unit is configured to determine an extended group based on the extended node information and the least common multiple rule, and split the extended group to obtain the A target group of a plurality of selected strips; the expansion group is composed of a plurality of strips satisfying the condition that the expansion requirement can be completed and the spatial position distribution law is unchanged; the obtaining unit is used to execute the target group An extension algorithm is used to obtain a corresponding target extension group, where the target extension group includes an extension data block and an extension check block.

Optionally, the first processing unit is configured to: divide the data into K data blocks of the same size; K is a positive integer greater than 1; make the K data blocks and the preset encoding matrix into domain Matrix operation to obtain M check blocks; M is a positive integer greater than 1 and less than K; the K data blocks and the M check blocks form multiple strips; the data blocks on the same strip and the The check blocks are scattered on different K+M nodes, the distribution information of the K data blocks and the M check blocks on each node is determined, and the spatial position distribution information is obtained based on the distribution information.

Optionally, the second processing unit is configured to: determine the first number of nodes storing data blocks on each stripe and the number of first nodes storing check blocks on each stripe based on the spatial position distribution information. Two number of nodes; add the first number of nodes and the number of newly added nodes to obtain the third number of nodes, and use the third number of nodes as the expanded number of nodes on each strip The number of stored data blocks; and, using the second number of nodes as the number of expanded storage check blocks on each stripe to determine the expanded node information on each stripe.

Optionally, the third processing unit is configured to: determine an extension group based on the extension node information and the least common multiple rule; the extension group includes V extension strips; and disassemble the V extension strips points, determine P basic groups and R adjustment groups; each of the basic groups includes Vp basic strips, and each of the adjustment groups includes Vr adjustment strips; P and R are positive integers greater than 1; Select K basic strips from the basic group, and select D adjustment strips from the adjustment group, and determine a target group based on the K basic strips and the D adjustment strips; the The target group includes K+D stripes.

V=LCM(K, K+D+1)(K+D)(K+1)/K

Wherein, the LCM() is used to characterize the function of obtaining the least common multiple, k is used to represent the number of nodes storing data blocks before expansion on each stripe; d is used to represent the number of newly added nodes.

The optional obtaining unit is specifically configured to: number K+D stripes in any one of the target groups, and number K+M+D nodes after the storage system is expanded; calculate the first K Adjust the difference check block of the data block on the stripe in +1 node, and update the first check block of the basic stripe on the same node based on the difference check block; The upper data block is transmitted to the basic stripe on the same node to obtain the expanded initial extended group; the preset operation is performed on the initial extended group to obtain the corresponding target extended group.

Optionally, the device further includes an adjustment unit, configured to: determine the logical relationship of the stripes corresponding to the target extended group, and the first spatial distribution information corresponding to each extended data block and extended parity block; according to the spatial distribution information, adjusting the order of the logical relationship so that the logical layout of the first spatial distribution information is the same as that of the spatial distribution information.

For the beneficial effects of the above-mentioned second aspect and each optional device of the second aspect, reference may be made to the beneficial effects of the above-mentioned first aspect and each optional method of the first aspect, which will not be repeated here.

In a third aspect, the present invention provides a computer device, including a program or an instruction, and when the program or instruction is executed, is used to execute the above-mentioned first aspect and each optional method of the first aspect.

In a fourth aspect, the present invention provides a storage medium, including a program or an instruction, and when the program or instruction is executed, is used to execute the above-mentioned first aspect and each optional method of the first aspect.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings that need to be used in the description of the embodiments.

Fig. 1 is a schematic diagram of the data block migration stage of the traditional storage system extended erasure code RS (2,1,4) process;

Fig. 2 is a schematic diagram of the check block update stage of the extended erasure code RS(2,1,4) process of the traditional storage system;

FIG. 3 is a schematic diagram of an optional application scenario provided by an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of an optional erasure code storage system provided by an embodiment of the present invention;

FIG. 5 is a schematic flowchart of steps of a method for extending an erasure code storage system provided by an embodiment of the present invention;

FIG. 6 is a schematic diagram of an encoding process of an erasure correction code RS(k,m) in a stripe according to an embodiment of the present invention;

Fig. 7 is a schematic diagram of erasure correction code storage distribution of erasure correction code RS(2,2) and erasure correction code RS(3,2) in the erasure correction code storage system provided by the embodiment of the present invention;

FIG. 8 is a schematic diagram of the parallel algorithm for updating check blocks and relocating data blocks provided by an embodiment of the present invention for erasure code RS(2,1,4);

FIG. 9 is a schematic diagram of a work flow diagram of an expansion process of an erasure code (2, 2, 3) provided by an embodiment of the present invention;

Fig. 10 is a schematic diagram of the result graph of the test in different bandwidth impact experiments provided by the embodiment of the present invention;

Fig. 11 is a schematic diagram of the result graph of testing the impact experiment on data blocks of different sizes provided by the embodiment of the present invention;

FIG. 12 is a schematic diagram of a test result diagram of an experiment on the influence of different numbers of newly added nodes provided by an embodiment of the present invention;

Fig. 13 is a schematic diagram of the result graph of the extended process flow numerical analysis test under the general configuration of the erasure code storage system provided by the embodiment of the present invention;

Fig. 14 is a schematic diagram of a numerical analysis experiment result diagram of the impact of the expansion process of the erasure code storage system provided by the embodiment of the present invention on the flow bandwidth under different numbers of newly added nodes;

FIG. 15 is a schematic diagram of a result graph of a numerical analysis of bandwidth utilization in different expansion processes of a code erasure storage system provided by an embodiment of the present invention;

FIG. 16 is a schematic structural diagram of an extended erasure code storage system device provided by an embodiment of the present invention.

Detailed ways

In order to better understand the above-mentioned technical solution, the above-mentioned technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention. To illustrate, rather than limit, the technical solutions of the present invention, the embodiments of the present invention and the technical features in the embodiments may be combined without conflict.

It should be noted that the terms "first" and "second" in the description and claims of the present invention are used to distinguish similar objects, and not necessarily used to describe a specific order or sequence. It is to be understood that the images so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. The implementations described in the following exemplary examples do not represent all implementations consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with aspects of the invention as recited in the appended claims.

In order to facilitate the understanding of the technical solutions provided by the embodiments of the present invention, some key terms or processes used in the embodiments of the present invention are first explained here:

1. Erasure Code (EC): It is a forward error correction technology (Forward Error Correction, FEC), which can add n copies of original data to m copies of data, and can pass any of n+m copies n copies of data, restored to the original data. That is, if any data less than or equal to m is invalid, it can still be restored through the remaining data. It is mainly used to avoid packet loss during network transmission, and is used by storage systems to improve storage reliability.

2. There are three main types of erasure code technology applications in distributed storage systems: array erasure codes, RS (Reed-Solomon) Reed-Solomon class erasure codes, and LDPC (LowDensity Parity Check Code) low-density parity checks Erasure coding. In the embodiment of the present invention, the extension of the RS type erasure code corresponding to the distributed storage system is mainly described.

The design idea of the embodiment of the present invention is briefly introduced below:

Please refer to FIG. 1 , which is a schematic diagram of a data block migration stage with extended erasure code parameters (2, 1, 4) in a traditional storage system in the prior art. And, please refer to FIG. 2 , which is a schematic diagram of an update phase of a parity block with extended erasure code parameters (2, 1, 4) in a traditional storage system in the prior art. Among them, S in Figure 1 and Figure 2 is used to represent a stripe, N is used to represent a node, D is used to represent a data block, and P is used to represent a check block.

Apparently, the migration of data blocks and the update of check blocks in the prior art will inevitably lead to a large amount of data transmission, resulting in poor transmission parallelism and a long expansion process, that is, poor expansion efficiency and effect.

In view of this, the present invention provides a method for extending an erasure code storage system, which proposes a brand-new extension mechanism, with the purpose of reducing traffic and exploring the extension mechanism of transmission parallelism in continuous scaling. In this extension mechanism, a new stripe layout is designed, which utilizes locally stored data blocks for parity block update, thereby reducing the data transmission for parity block update. It can be seen that the method for extending the erasure code storage system provided by the present invention can reduce the data transmission for updating the check block, thereby improving the expansion efficiency.

After introducing the design ideas of the embodiments of the present invention, the following briefly introduces the application scenarios applicable to the technical solution of the extended erasure code storage system in the embodiments of the present invention. It should be noted that the application scenarios described in the embodiments of the present invention are In order to illustrate the technical solutions of the embodiments of the present invention more clearly, it does not constitute a limitation to the technical solutions provided by the embodiments of the present invention. Those of ordinary skill in the art know that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention The scheme is also applicable to similar technical problems.

The information provided in the embodiments of the present invention can be applied to most storage systems that need to perform storage expansion functions. Wherein, the storage system is, for example, a business order storage system, or a transaction data storage system, and so on. Referring to FIG. 3 , it is a schematic diagram of a scene provided by an embodiment of the present invention. In the schematic diagram of the scene, it includes a plurality of electronic devices 301 deployed with proxy nodes and a metadata server 302 deployed with a global coordinator, and the electronic devices 301 can communicate with the metadata server 302 of the global coordinator through a network 303, for example The direct or indirect connection is performed through wired or wireless communication, which is not limited in the present invention. Wherein, the electronic device 301-1, the electronic device 301-2, ..., the electronic device 301-n may be deployed by different proxy nodes.

In this embodiment of the present invention, the electronic device 301 may be, for example, a server, but it is not limited thereto. Wherein, each electronic device 301 may include one or more processors 3011, memory 3012, and an I/O interface 3013 for interacting with other servers, and the like.

In the embodiment of the present invention, the metadata server 302 deployed with the global coordinator can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, and can also provide cloud services, cloud databases, Cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN (Content Delivery Network, content distribution network), and cloud of basic cloud computing services such as big data and artificial intelligence platforms server.

In this scenario, the metadata server 302 deployed with the global coordinator is responsible for managing the metadata of the stripes. In addition, each round of transmission tasks can be issued to each electronic device 301 to perform data block migration or check block update. operate. When each electronic device 301 transmits the confirmation signal to the metadata server 302 equipped with the global coordinator, the metadata server 302 equipped with the global coordinator can execute and send the next round of transmission commands to each electronic device 301 .

In this scenario, each electronic device 301 needs to receive the transmission command sent by the coordinator, analyze the transmission command and execute the task content of the transmission command. Specifically, after each electronic device 301 sends the data block or check block that needs to be sent to the corresponding electronic device 301, the electronic device 301 will send a confirmation signal to the metadata server 302 deployed with the global coordinator to inform the deployment of the global The metadata server 302 of the coordinator has finished sending, so that the next round of transmission commands can be prepared for execution.

Referring to FIG. 4 , it is a schematic structural diagram of an erasure code storage system provided by an embodiment of the present invention. Among them, the metadata server can issue a command to update the check block in the proxy node in the existing node, and issue a command to migrate the data block of the proxy node in the existing node to the proxy node in the new node .

Certainly, the method provided by the embodiment of the present invention is not limited to the application scenario shown in FIG. 1 , and may also be used in other possible application scenarios, which is not limited by the embodiment of the present invention.

In order to further illustrate the scheme of the method for extending the erasure correction code storage system provided by the embodiment of the present invention, it will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of the present invention provide method operation steps as shown in the following embodiments or drawings, more or less operation steps may be included in the method based on routine or no creative effort. In the steps that logically do not have a necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiment of the present invention. The method can be executed sequentially or in parallel according to the methods shown in the embodiments or drawings during the actual processing process or when the device is executed (for example, a parallel processor or an application environment for multi-thread processing).

The method for extending the erasure code storage system in the embodiment of the present invention will be described below with reference to the method flowchart shown in FIG. 5 , and the method flow in the embodiment of the present invention will be introduced below.

Referring to FIG. 5 , it is an implementation flowchart of a method for extending an erasure code storage system provided by an embodiment of the present invention. The method can be executed by a metadata server, and the specific implementation process is as follows:

Step 501: Determine the data in the storage system, encode the data, and store the data scattered in each node, and obtain the spatial location distribution information of each node.

In the embodiment of the present invention, the metadata server can select an RS type erasure code that meets the storage system's fault tolerance requirements and storage efficiency according to the storage system's reliability requirements and storage overhead limitations, and use the RS type erasure code as the storage system data in .

In the embodiment of the present invention, the metadata server may divide the data into K data blocks of the same size; wherein, K is a positive integer greater than 1. Then, an intra-domain matrix operation can be performed on the K data blocks and the preset encoding matrix to obtain M check blocks. Wherein, M is a positive integer greater than 1 and less than K. In addition, K data blocks and M parity blocks may form multiple stripes.

Furthermore, the data blocks and check blocks on the same strip can be dispersed on different K+M nodes, the distribution information of K data blocks and M check blocks on each node can be determined, and based on the distribution information, the Spatial location distribution information.

In the embodiment of the present invention, the parameters of the RS-type erasure code include three parameters, and the three parameters are represented by K, M, and W, for example, where K means that the RS-type erasure code has K data blocks, and M means that the RS The type erasure code has M check blocks, and W is used to represent the number of bits corresponding to the RS type erasure code; wherein, W can generally take values: 4, 8, 16, 32. In the embodiment of the present invention, w=8 is taken as an example for description in the following.

In the embodiment of the present invention, the metadata server can obtain the check block according to the parameters of the RS type erasure code and the preset coding matrix. Specifically, the metadata server may perform an intra-domain matrix operation on the above K data blocks and the generated preset encoding matrix limited in the Galois field, so as to obtain M check blocks. Exemplarily, the check block can be obtained by performing a bitwise operation on the data block and the number of the preset coding matrix. Wherein, the foregoing preset encoding matrix may be a Vandermonde matrix or a Cauchy matrix, which is not limited in this embodiment of the present invention.

For example, please refer to FIG. 6 , which is a schematic diagram of a process of encoding an RS-type erasure code provided in an embodiment of the present invention. Wherein, the metadata server may determine the encoding matrix based on the identity matrix and the generating matrix, and determine the encoding matrix as a preset encoding matrix, and then multiply the preset encoding matrix and k data blocks to obtain m check blocks, thereby It is possible to store k data blocks and m data blocks on k+m nodes.

In the embodiment of the present invention, according to the parameters of the erasure correction code and the preset encoding matrix, K data blocks are encoded to generate M corresponding check blocks, represented by a tuple (K, M).

In the embodiment of the present invention, after the data block and the check block having the encoding relationship with the data block are determined, the stripe may be determined based on the given data block and the check block having the encoding relationship with the data block. Furthermore, the data blocks and check blocks of the same stripe can be distributed and stored in different nodes.

Specifically, in the embodiment of the present invention, the spatial distribution scheme for distributing data blocks and check blocks on the same strip on different K+M nodes is set as follows: store one checksum on the first K+1 nodes check blocks and K data blocks; among them, the positions of the check blocks are arranged diagonally on K+1 strips; the check blocks other than 1 check block are stored on the last M-1 nodes piece.

In the embodiment of the present invention, the metadata server may determine spatial location distribution information according to the distribution of data blocks and check blocks on different nodes. Specifically, the spatial location distribution information can be understood as location information of data blocks located in stripes and nodes, and location information of check blocks located in stripes and nodes.

For example, please refer to FIG. 7 . FIG. 7 is a schematic diagram of parameters of RS-type erasure codes (2, 2) and parameters of RS-type erasure codes (3, 2) provided in the embodiment of the present invention. Among them, S is used to represent a stripe, N is used to represent a node, D is used to represent a data block, and P is used to represent a check block. Based on FIG. 7 , the storage distribution of the data blocks and the check blocks corresponding to the data blocks in the embodiment of the present invention can be clearly known.

Step 502: Determine the number of newly added nodes on each stripe based on the extended demand information, and determine the extended node information on each stripe based on the number of newly added nodes and spatial location distribution information; the stripe includes The data block and check block of the encoding relationship.

In the embodiment of the present invention, based on the spatial location distribution information, determine the first number of nodes storing data blocks on each stripe, and the second number of nodes storing check blocks on each stripe; Add the number and the number of newly added nodes to obtain the number of the third node, and use the number of the third node as the number of expanded storage data blocks on each stripe; and, use the number of the second node , as the number of extended storage check blocks on each stripe, so as to determine the extended node information on each stripe.

Exemplarily, according to system adjustment reliability and stripe length requirements, it is determined that the number of newly added nodes is D, the number of first nodes storing data blocks on each stripe is K, and the number of nodes storing data blocks on each stripe is The number of second nodes of the verification block is M, so that the number of expanded storage data blocks on each stripe can be determined as K+D, and the number of expanded storage verification blocks on each stripe can be determined for M.

Step 503: Determine the expansion group based on the expansion node information and the least common multiple rule, and split the expansion group to obtain a target group including multiple selected stripes; It is composed of multiple strips under the condition that the distribution law does not change.

In the embodiment of the present invention, the metadata server can determine the extension group based on the extension node information and the least common multiple rule, where the extension group is formed by a plurality of strips that meet the expansion requirements and the spatial position distribution law is unchanged. formed, and the extension group includes V extension stripes. It can be seen that V expansion strips can fulfill the condition of expansion requirements and the spatial location distribution law remains unchanged.

Specifically, the least common multiple rule is determined by the following formula:

V=LCM(K, K+D+1)(K+D)(K+1)/K

In the embodiment of the present invention, after the metadata server determines the V extended strips, it can split the V extended strips to determine P basic groups and R adjustment groups; each basic group includes Vp basic strips , each adjustment group includes Vr adjustment strips; P and R are positive integers greater than 1.

In the embodiment of the present invention, the metadata server can represent the V stripes as two types of groups according to the different functions of the data blocks and check blocks in the V extended stripes, namely the aforementioned basic group and adjustment group . Among them, the check blocks of the stripes in the basic group need to be updated according to the data blocks on the newly added nodes, and the data blocks of the stripes in the adjustment group are sent to the stripes of the basic group.

Specifically, Vp basic strips and Vr adjustment strips need to satisfy the equation: Vp:Vr=K:D, so that Vp and Vr can be determined based on the following formula:

Vp=LCM(K, K+D+1)(K+1); Vr=LCM(K, K+D+1)D(K+1)/K.

In the embodiment of the present invention, during the expansion process, the data blocks of the adjustment group may be correspondingly transmitted to the metadata server in the stripe of the stripe basic group having a corresponding relationship. Specifically, according to the order of the stripes, it may be determined that the K (K+1) basic stripes in the basic group correspond to the D(K+1) stripes in the adjustment group. Therefore, the corresponding relationship can be expressed as: in the basic group: {(i-1)K(K+1)+1, (i-1)K(K+1)+2, ..., iK(K+1)}, These K(K+1) strips, in the adjustment group: {(i-1)D(K+1)+1, (i-1)D(K+1)+2, ..., iD(K +1)} These D(K+1) strips have a corresponding relationship. Wherein, 0<i<LCM(K, K+D+1)/K. It can be seen that there are LCM(K, K+D+1)/K pairs of strips with a corresponding relationship among each V strips.

In the embodiment of the present invention, after determining the P basic groups and R adjustment groups and their respective corresponding strips, from the K (K+1) strips in the basic group, according to the strip sequence, each group Select K basic strips. And, from the D(K+1) strips in the adjustment group corresponding to the K(K+1) strips, D adjustment strips are continuously selected at intervals of K adjustment strips according to the order of the strips. Therefore, the above (K+D)(K+1) strips can be combined into (K+1) groups, so as to determine the target group based on K basic strips and D adjustment strips; wherein, each target A group includes K+D strips.

It can be seen that any target group includes: independently select {(i-1)K+1, (i-1)K+2,...,iK} from the K(K+1) stripes in the basic group K basic strips, and {D(K+1)-i, (D-1)(K+1)-i, ..., (K+1)-i} in the adjustment group corresponding to them D adjustment strips selected from strips.

In the embodiment of the present invention, after the target group is determined, perform step 504 for each target group: execute the expansion algorithm on the target group to obtain the corresponding target expansion group, and the target expansion group includes the expansion data block and the expansion check block.

Specifically, determining the corresponding target extension group may adopt but not limited to the following steps:

Step a: Number K+D stripes in any target group, and number K+M+D nodes after the storage system is expanded.

In this embodiment of the present invention, the K+D slices in any target group may be numbered as: {1, 2, ..., K+D}. And, the K+M+D nodes can be numbered as: {1, 2, . . . , K+M+D}.

Step b: Calculate the difference check blocks of the data blocks on the adjustment stripe in the first K+1 nodes, and update the first check block of the basic stripe on the same node based on the difference check blocks.

In the embodiment of the present invention, first, the first check blocks of the first K stripes may be updated. Specifically, the check blocks may be updated in a manner of calculating difference check blocks. Specifically, among the first K+1 nodes, {K+1, K+2,...,K+D} can be calculated to adjust the difference check blocks of the data blocks on the stripe, and update the difference check blocks on the same node based on the difference check blocks The first checksum of the basic stripe.

Step c: Transfer the data blocks on the adjusted stripe to the basic stripe on the same node according to the round robin mode, and obtain the expanded initial extended group.

In the embodiment of the present invention, for the data block migration part, it is possible to transmit {K+1, K+2, ..., K+D} in a round-robin manner to adjust the data blocks on the stripe to {1, 2, ..., K} on the basic strip. Specifically, each round selects D data blocks on D nodes from K nodes that store data blocks in sequence, and selects D data blocks from {i,i+1,...,i+D-1} in the i-th round Nodes, correspondingly transmit {K+1, K+2, ..., K+D} D data blocks on D adjustment stripes to D new nodes. When i+D-1>K, from Start with the first storage data block node and continue to select nodes.

Step d: Execute preset operations on the initial expansion group to obtain the corresponding target expansion group.

In the embodiment of the present invention, the number of data blocks transmitted to the new node is counted by means of a global counter, and after sending (K+1)D data blocks to the newly added node, the subsequent D basic stripes The logical position of the first check block and the check block of the corresponding node adjustment stripe is replaced.

Specifically, the foregoing process can be expressed as: in the i-th basic stripe for logical location replacement, perform location replacement on the first check block of the stripe and the i-th data block of the corresponding node adjustment stripe, After the replacement, the above data block migration algorithm is also executed. Wherein, the value range of i is greater than 0 and less than D.

It can be seen that after K rounds, when executing part of the data block migration algorithm, only D nodes are occupied in each round of K nodes that originally stored data blocks, and the remaining (K-D) nodes are idle.

In the embodiment of the present invention, the following operations can be performed for updating (M-1) check blocks except the first check block in the baseband strip:

Step1: There are M nodes storing check blocks in the adjustment stripe, among which (M-1) nodes overlap with nodes that only store check blocks in the basic stripe. Each round transmits a linear combination of (M-1) check blocks from these (M-1) nodes to other (M-1) nodes.

Specifically, the process can be expressed as: In the i-th round, the (M-1) nodes transmit the linear combination of the check blocks to the nodes that are i positions away from each other. When the position of the node exceeds (M-1) When , starting from the first node, continue to select nodes for transmission, it can be seen that it will end after (M-2) rounds, where the value range of i is greater than 0 and less than M-2.

Step2: One storage check block node and (K-M) storage data block nodes in the adjustment stripe do not overlap with the remaining (M-1) storage check block nodes in the basic stripe. Each round selects (M-1) nodes from these (K-M+1) nodes, and transmits a linear combination of data blocks or a linear combination of check blocks to the remaining (M-1) storage in the basic stripe Check block node.

Specifically, the process can be expressed as: in the i-th round, select {i,i+1,...,i+M-1} nodes from the (K-M+1) nodes to transmit a linear combination or collation of data blocks The linear combination of check blocks is stored in nodes corresponding to {1,2,...,M-1} basic strips, and it can be seen that (K-M+1) rounds will end. Wherein, the value range of i is greater than 0 and less than K-M+1.

In summary, after (K-1) rounds, the linear combination of data blocks and check blocks required for the operation of updating check blocks is completed, and the operation of updating check blocks can be completed through the update algorithm designed in the present invention.

It should be noted that, in the embodiment of the present invention, the aforementioned restriction on the check block update part is: each round of data block migration needs to occupy D nodes for storing data blocks and the maximum requirement in Step 2 of updating the check block ( M-1) storage data block nodes cannot overlap, that is, it is required to satisfy the inequality: K is greater than or equal to D+M-1.

In the embodiment of the present invention, it is necessary to limit the sending and receiving of each node to full-duplex work, that is, in each round, each node can only receive and send one block at the same time, which needs to be achieved based on a preset algorithm in each block. Maximize the utilization of nodes within a round. In the actual implementation process, when the data center management personnel determine the number of new nodes, they need to determine that the parameters K, M, and D satisfy this restriction.

In the embodiment of the present invention, the last (M-1) nodes in the basic stripe that only store check blocks, after receiving the linear combination of (K-M) data blocks and (M-1) check blocks from other nodes, The linear combination of blocks and the linear combination of the check blocks on the adjustment strip calculated on the own node can calculate the difference correction corresponding to the last (M-1) check blocks on the basic strip through the algorithm of erasure code decoding. check block, and then XOR the check block of the basic stripe and the calculated difference check block, and the updated check block, that is, the extended check block, can be calculated.

In the embodiment of the present invention, after all the stripes of an extended group are updated, the corresponding target extended group can be obtained. In addition, according to the spatial distribution information, the logical relationship sequence of the stripes can be adjusted to meet the overall spatial distribution scheme before expansion. In this way, when the next expansion is performed on the storage system, there is no need to adjust the space distribution to bring unnecessary overhead.

In the embodiment of the present invention, please refer to FIG. 8 . FIG. 8 is a schematic diagram of a parallel algorithm for parity block update and data block relocation of RS(2,1,4) provided by the present invention. And please refer to FIG. 9 , which is a schematic diagram of a process of extending RS (2, 1, 4) provided by an embodiment of the present invention.

In the embodiment of the present invention, the logical relationship of the stripes corresponding to the target extended group, and the first spatial distribution information corresponding to each extended data block and extended parity block may also be determined. Then, according to the spatial distribution information, the order of the logical relationship is adjusted so that the logical layout of the first spatial distribution information is the same as that of the spatial distribution information. In this way, when the next expansion is performed on the storage system, there is no need to adjust the space distribution to bring unnecessary overhead.

In the embodiment of the present invention, when all stripes of the basic group have completed the expansion operation, all data blocks and parity blocks of the adjustment group in the storage system may also be deleted. In this way, the waste and consumption of resources can be minimized.

It can be seen that in the method for expanding the erasure code storage system provided by the embodiment of the present invention, the input/output of the expansion process, that is, the I/O overhead is small, that is, the amount of data that needs to be read and written in the expansion process and the amount of data transmitted in the network are reduced. amount of data. And, the time delay of the expansion process is short. On the basis of small I/O overhead and full-duplex communication, the new update check block algorithm increases the available bandwidth resources in the storage system, and reduces the expansion by executing the scheduling expansion algorithm in parallel. The time delay of the process. In addition, it can also support continuous expansion, that is, after a single expansion process, the overall space distribution of the present invention is consistent with that before the expansion. Therefore, when the storage system performs the next expansion, there is no need to adjust the space distribution to bring unnecessary overhead.

In the specific implementation process, the solutions proposed in the embodiments of the present invention were tested. Specifically, in the embodiment of the present invention, the solution proposed in the embodiment of the present invention is tested from two test modes of real platform research and simulation experiment.

Mode 1: Test the solution proposed in the embodiment of the present invention based on a real platform.

In the embodiment of the present invention, the specific experimental environment includes 19 virtual servers of type ecs.g6.large, and each virtual server is configured with 2vCPU (2.5GHz Intel Xeon Platinum 8269CY) and 8GB memory. And 40GB storage, the running operating system is Ubuntu18.04. The maximum network bandwidth between any two servers is about 3Gb/s. One of the 19 servers is used as the global coordinator, and the remaining 18 servers are agents running the server program of the present invention. Among them, the default setting of the experiment is that the block size is 64MB, the erasure code scheme is RS(6,3) and RS(10,4), and the number of new nodes varies according to different experiments.

Specifically, each experiment is repeated multiple times, and the measured parameter is the time consumption of the expansion process, that is, the time for all blocks to be transmitted to the corresponding node. Specifically, the expansion time is defined as the expansion time consumption of the average stripe, and the average The shorter the expansion time consumption, the higher the efficiency of the expansion process.

In addition, the test uses a comparative experiment to compare two advanced erasure code storage system expansion mechanisms, Scale-RS and NCScale. In actual implementation, it may also be tested or used in other experimental testing environments, which is not limited in this embodiment of the present invention.

In a specific implementation process, the expansion time when the network bandwidth changes from 1 Gb/s to 2 Gb/s can be measured. Specifically, the test results are shown in FIG. 10 . Referring to FIG. 10 , among the three extension mechanisms, the method provided by the embodiment of the present invention requires the least extension traffic, and improves the parallelism of transmission compared with the other two mechanisms. Overall, when the network bandwidth is 1Gb/s, compared with Scale-RS and NCScale, the present invention reduces the average by 49.8% and 58.9%. And, when the network bandwidth increases to 2Gb/s, the average reduction is 50.8% and 58.8% respectively.

Apparently, when the bandwidth increases, the average expansion time of the present invention is shorter than Scale-RS and NCScale. It can be seen that the expansion performance of the providing method of the present invention is better than Scale-RS and NCScale.

In the specific implementation process, you can also test and study the expansion time under different block sizes, for example, from 32MB to 64MB. During this test, the network bandwidth can be set to 3Gb/s. Please refer to Fig. 11, the expansion time increases with the increase of the block size, and the method provided by the present invention shortens the scaling time by 49.1-53.0% and 24.1-76.9% respectively compared with Scale-RS and NCScale. Moreover, it can be seen that the method provided by the present invention and Scale-RS have achieved quite stable performance in the continuous expansion process, while the scaling time of NCScale is in the second expansion operation, that is, the (8,3,10) expansion process significantly increased in.

In the specific implementation process, it is also possible to test and study the influence of the number of newly added nodes (that is, the number of newly added nodes, that is, the parameter D) on the scaling time. Specifically, the network bandwidth can be fixed at 3Gb/s, and the case where the parameter D is from 2 to 3 will be studied. Please refer to Figure 12, under different numbers of new nodes, the average expansion time of the three mechanisms has not been significantly affected. The most fundamental reason is that we enable all methods to have transmission parallelism to achieve fairness In comparison, the newly added D nodes can receive the migrated data in parallel, and the method provided by the present invention reduces the expansion time of the Scale-RS and NCScale mechanisms by 49.8-51.4% and 23.6-76.3% respectively, significantly improving Scale efficiency.

Mode 2: Test the solution proposed in the embodiment of the present invention based on a simulation test.

In a specific implementation process, a traffic simulation test under a general configuration may be performed. As an example, please refer to Figure 13. This test is to evaluate the flow of successive expansion processes of different expansion mechanisms, and consider the two situations of RS(6,3) and RS(10,4), and set the parameters The value of d is set to 2.

Please continue to refer to Figure 13. It can be seen that under different expansion process parameters, the scheme provided by the present invention performs well in the continuous expansion process. Compared with Scale-RS, it starts from RS (6,3) and RS (10,4) When expanding, the solution provided by the present invention reduces the expansion flow by 22.9-26.7% and 19.4-21.7% respectively, and compared with NCScale, the expansion flow reduces by 8.3% to 62.8%, that is, the solution provided by the present invention reduces resource consumption. consume.

In a specific implementation process, a simulation experiment can be carried out in which the number of expanded nodes affects the expanded bandwidth. The experiment measures the influence of the present invention on the efficiency of the expansion process brought about by adding different numbers of nodes. Exemplarily, please refer to Figure 14, use the two parameters RS(6,3) and RS(10,4) before the expansion, and then change the number of newly added nodes (ie, parameter D) from 2 Change to 10. It can be seen that the expansion traffic increases with the number of new nodes added. The reason for this is that adding more nodes requires more blocks to be transmitted for relocation and check block updates. However, the scheme provided by the present invention still maintains the advantages of compressing and expanding traffic. Compared with Scale-RS and NCScale, the expansion process of the present invention can reduce the expansion traffic by 35.2% and 38.1% respectively on average, that is, the scheme provided by the present invention reduces resources. consumption.

In the specific implementation process, different simulation experiments of the average bandwidth utilization can be performed. The final evaluation of the experiment is the average bandwidth utilization, and the average bandwidth utilization is defined as the average amount of data transmitted per time unit and the average amount of data per time unit The ratio of the theoretical maximum amount of data that can be transmitted for data block relocation and check block update.

Referring to FIG. 15 , it can be seen that, compared with Scale-RS and NCScale, the solution provided by the present invention achieves near-optimal bandwidth utilization. In particular, the scheme provided by the present invention achieves a bandwidth utilization rate of 96.7% in the extended process RS(18,4,20). On average, the bandwidth utilization of the present invention is 41.7-46.7% and 61.9-78.3% higher than Scale-RS and NCScale, respectively.

In summary, the present invention proposes a rapid continuous expansion mechanism for the phenomenon that the expansion process of the erasure code storage system consumes a lot of I/O, the bandwidth utilization rate is low, and the continuous expansion consumption increases. Moreover, the present invention analyzes the expansion process of the erasure code storage system from a continuous perspective, and designs a new space distribution scheme and a check block update algorithm to increase node bandwidth utilization and block transmission execution. The invention reduces the expansion process time and bandwidth flow consumption on the basis of ensuring system reliability.

As shown in Figure 16, the present invention provides an apparatus for extending an erasure code storage system, the apparatus including: a first processing unit 1601, configured to determine data in the storage system, encode the data, and The data is distributedly stored in each node, and the spatial location distribution information of each node is obtained; the second processing unit 1602 is configured to determine the number of newly added nodes on each stripe based on the extended demand information, and based on the The number of newly added nodes and the spatial position distribution information are used to determine the extended node information on each stripe; the stripe includes data blocks and check blocks with encoding relationships; the third processing unit 1603 is used to Based on the extended node information and the least common multiple rule, an extended group is determined, and the extended group is split to obtain a target group including a plurality of selected strips; Composed of multiple strips under the condition that the distribution of spatial positions remains unchanged; the obtaining unit 1604 is configured to perform an expansion algorithm on the target group to obtain a corresponding target expansion group, and the target expansion group includes expansion data blocks and expansion parity block.

Optionally, the first processing unit 1601 is configured to: divide the data into K data blocks of the same size; K is a positive integer greater than 1; In-domain matrix operation to obtain M check blocks; M is a positive integer greater than 1 and less than K; the K data blocks and the M check blocks form multiple stripes; the data blocks on the same stripe The check blocks are distributed on different K+M nodes, the distribution information of the K data blocks and the M check blocks on each node is determined, and the spatial position distribution information is obtained based on the distribution information .

Optionally, the second processing unit 1602 is configured to: based on the spatial location distribution information, determine the first number of nodes storing data blocks on each stripe, and the number of first nodes storing check blocks on each stripe The second number of nodes; add the first number of nodes and the number of newly added nodes to obtain the third number of nodes, and use the third number of nodes as the expanded number of each strip The number of stored data blocks; and, using the second number of nodes as the number of expanded storage check blocks on each stripe to determine the expanded node information on each stripe.

Optionally, the third processing unit 1603 is configured to: determine an extension group based on the extension node information and the least common multiple rule; the extension group includes V extension strips; Splitting and determining P basic groups and R adjustment groups; each of the basic groups includes Vp basic strips, and each of the adjustment groups includes Vr adjustment strips; P and R are positive integers greater than 1 ; Select K basic strips from the basic group, and select D adjustment strips from the adjustment group, and determine a target group based on the K basic strips and the D adjustment strips; The target group includes K+D strips.

V=LCM(K, K+D+1)(K+D)(K+1)/K

Optionally, the obtaining unit 1604 is specifically configured to: number K+D stripes in any one of the target groups, and number K+M+D nodes after the storage system is expanded; calculate Adjust the difference check block of the data block on the stripe in the first K+1 nodes, and update the first check block of the basic stripe on the same node based on the difference check block; The data blocks on the stripe are transmitted to the basic stripe on the same node to obtain an expanded initial extended group; a preset operation is performed on the initial extended group to obtain a corresponding target extended group.

An embodiment of the present invention provides a computer device, including a program or an instruction, and when the program or instruction is executed, it is used to execute a method for extending an erasure code storage system and any optional method provided in an embodiment of the present invention .

An embodiment of the present invention provides a storage medium, including a program or an instruction, and when the program or instruction is executed, it is used to execute a method for extending an erasure code storage system and any optional method provided in an embodiment of the present invention .

Finally, it should be noted that: those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the invention. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these modifications and variations.

Claims

A method for expanding an erasure code storage system, characterized in that the method comprises:

Determining the data in the storage system, encoding the data, and dispersely storing the data in each node, and obtaining the spatial location distribution information of each node;

Determine the number of newly added nodes on each strip based on the extended demand information, and determine the expanded node information on each strip based on the number of newly added nodes and the spatial location distribution information; The band includes a data block and a check block with an encoding relationship;

Based on the extended node information and the least common multiple rule, an extended group is determined, and the extended group is split to obtain a target group including a plurality of selected strips; Composed of multiple strips under the condition that the distribution of spatial positions remains unchanged;

Executing an expansion algorithm on the target group to obtain a corresponding target expansion group, where the target expansion group includes an extended data block and an extended check block.
The method according to claim 1, wherein said data is encoded, and said data is distributedly stored in each node, and said each node spatial position distribution information is obtained, comprising:

Divide the data into K data blocks of the same size; K is a positive integer greater than 1;

performing an intra-domain matrix operation on the K data blocks and the preset encoding matrix to obtain M check blocks; M is a positive integer greater than 1 and less than K; the K data blocks and the M check blocks constitute multiple strips;

Distribute the data blocks and check blocks on the same strip on different K+M nodes, determine the distribution information of the K data blocks and the M check blocks on each node, and based on the distribution The information obtains the spatial location distribution information.
The method according to claim 1 or 2, wherein, based on the newly added number of nodes and the spatial location distribution information, determining the extended node information on each stripe includes:

Based on the spatial position distribution information, determine the first number of nodes storing data blocks on each stripe, and the second number of nodes storing check blocks on each stripe;

Adding the first number of nodes and the number of newly added nodes to obtain a third number of nodes, and using the third number of nodes as the number of expanded storage data blocks on each stripe number; and, using the second number of nodes as the number of expanded storage check blocks on each stripe, so as to determine the expanded node information on each stripe.
The method according to claim 1, wherein an extension group is determined based on the extension node information and the least common multiple rule, and the extension group is split to obtain a target group including strips with corresponding relationships ,include:

Determine an extension group based on the extension node information and the least common multiple rule; the extension group includes V extension strips;

Splitting the V extended strips to determine P basic groups and R adjustment groups; each of the basic groups includes Vp basic strips, and each of the adjustment groups includes Vr adjustment strips; P and R are positive integers greater than 1;

Select K basic strips from the basic group, and select D adjustment strips from the adjustment group, and determine a target group based on the K basic strips and the D adjustment strips; the The target group includes K+D stripes.
The method according to claim 4, wherein the least common multiple rule is determined by the following formula:

V=LCM(K, K+D+1)(K+D)(K+1)/K

Wherein, the LCM() is used to represent a function for obtaining the least common multiple, k is used to represent the number of nodes storing data blocks before extension on each stripe; d is used to represent the number of newly added nodes.
The method according to claim 4, wherein an expansion algorithm is performed on the target group to obtain a corresponding target expansion group, and the target expansion group includes an expansion data block and an expansion check block, including:

Numbering K+D stripes in any one of the target groups, and numbering K+M+D nodes after the storage system is expanded;

Calculating the difference check blocks of the data blocks on the adjusted stripes in the first K+1 nodes, and updating the first check block of the basic stripe on the same node based on the difference check blocks;

Transmitting the data blocks on the adjusted stripe to the basic stripe on the same node according to the round robin mode to obtain an expanded initial extended group;

A preset operation is performed on the initial extended group to obtain a corresponding target extended group.
The method according to claim 1, characterized in that, after obtaining the target extension group, the method further comprises:

Determine the logical relationship of the stripes corresponding to the target extended group, and the first spatial distribution information corresponding to each extended data block and extended parity block;

According to the spatial distribution information, the order of the logical relationship is adjusted so that the logical layout of the first spatial distribution information is the same as that of the spatial distribution information.
A device for expanding an erasure code storage system, characterized in that the device includes:

The first processing unit is configured to determine the data in the storage system, encode the data, and store the data dispersedly in each node, and obtain the spatial location distribution information of each node;

The second processing unit is configured to determine the number of newly added nodes on each stripe based on the extended demand information, and determine the number of nodes on each stripe based on the number of newly added nodes and the spatial location distribution information. Extended node information; the strip includes a data block and a check block with an encoding relationship;

The third processing unit is configured to determine an extension group based on the extension node information and the least common multiple rule, and split the extension group to obtain a target group including multiple selected strips; the extension group It is composed of multiple strips that meet the conditions that the expansion requirements can be completed and the spatial distribution law remains unchanged;

The obtaining unit is configured to execute an expansion algorithm on the target group to obtain a corresponding target expansion group, where the target expansion group includes an extended data block and an extended check block.
A computer device, characterized by including programs or instructions, when the programs or instructions are executed, the method according to any one of claims 1 to 7 is executed.
A storage medium is characterized by including programs or instructions, and when the programs or instructions are executed, the method according to any one of claims 1 to 7 is executed.