WO2018040583A1

WO2018040583A1 - Data storage method in data storage system and coordinating storage node

Info

Publication number: WO2018040583A1
Application number: PCT/CN2017/081073
Authority: WO
Inventors: 刘强; 蒋中斌
Original assignee: 华为技术有限公司
Priority date: 2016-09-05
Filing date: 2017-04-19
Publication date: 2018-03-08
Also published as: CN106383665A; CN106383665B

Abstract

A data storage method in a data storage system and a coordinating storage node (103), relating to the technical field of information. The method comprises: the coordinating storage node (103) equally divides data having a length of M into X data fragments and obtains Y check fragments; the coordinating storage node (103) sends an initial address and a length of the i^th data fragment to the i^th storage node (103) in N storage nodes (103); when X is greater than N, the coordinating storage node (103) allocates a first storage node (103) to each of (X-N) data fragments and the Y check fragments; when X is not greater than N, the coordinating storage node (103) allocates a second storage node (103) to each of the Y check fragments. By means of the method, when data is switched from a multi-copy storage mode to an EC storage mode for storage, it is unnecessary to send the data to a special coordinating storage node; instead, the data is distributed by means of a storage node (103) capable of serving as a data coordination node, so that the data transmission amount is greatly decreased and network resource consumption in the system is reduced.

Description

Data storage method and coordinated storage node in data storage system

Technical field

The present invention relates to the field of information technology, and in particular, to a data storage method and a coordinated storage node in a data storage system.

Background technique

In order to improve the security of data and avoid data loss during transmission, redundant data is often used to store data in distributed storage. At present, the redundancy modes mainly include copy redundancy and EC (Erasure code) redundancy. Among them, multi-copy redundancy is to copy the original data into multiple copies of data, and each copy data is stored on one storage node. EC redundancy divides the original data into X pieces of data, and uses the EC algorithm to verify the X pieces of data, and obtains Y pieces of check data, and then stores X+Y pieces into different pieces. On the storage node.

The most widely used data storage method is a combination of multiple copy redundancy and EC redundancy. The specific storage process is as follows: the client copies the original data, obtains multiple copies of the data, stores the plurality of copies of the data on different copy storage nodes, and sends any copy data to the EC coordination node, and the EC coordination node copies the copy. The data is divided into X pieces of data, and the X pieces of data are verified by the EC algorithm to obtain Y pieces of check data, and the EC coordination node stores one piece of data, and then X+Y-1 The fragments are sent to different EC storage nodes for storage.

When the data storage mode is converted from replica redundancy to EC redundancy, any replica data needs to be sent to the EC coordination node first, and then the EC coordination node will include multiple segments including fragment data and fragmentation verification data. The slices are sent to different EC storage nodes for storage. This process transmits a large amount of data, resulting in a large consumption of network resources in the storage system.

Summary of the invention

In order to reduce the network resource consumption in the storage system when the data is transferred from the multiple copy storage mode to the EC storage mode, the embodiment of the present invention provides a data storage method and a coordinated storage node in the storage system. The technical solution is as follows:

A first aspect provides a data storage method in a storage system, where the storage system includes a plurality of storage nodes, and N storage nodes of the plurality of storage nodes store data of length M, and the N storage nodes include A coordinated storage node, where N is a natural number greater than 1, the method comprising:

In the storage system, when the data is converted from the multiple copy storage mode to the EC storage mode for storage according to the business requirement, the coordinated storage node divides the length of the M data into X data fragments, and each data slice has one. The starting address and length, the starting address of each data fragment is the position of the data fragment in the data of length M, and the length of each data fragment is M/X. Based on the obtained X data fragments, the coordinated storage node obtains Y verification fragments by verifying X data fragments, and the length of each verification fragment is also M/X.

For the i-th data fragment in the X data fragments, the coordination storage node sends the start address and length of the i-th data fragment to the i-th storage node of the N storage nodes, where i is greater than 1 and less than or equal to the natural number of D, D is the minimum of X and N.

When X is greater than N, the coordinated storage node allocates each of the XN data fragments and the Y parity fragments to a first storage node, where the first storage node is a plurality of storage nodes other than N. Any storage node.

When X is less than or equal to N, the coordination storage node allocates a second storage node for each of the Y verification fragments, the second storage node being any one of the plurality of storage nodes except X storage nodes. Storage node.

Since the data storage node as a coordinated storage node has a dual function, not only can the complete data be stored, but also The shards including data sharding and parity sharding can also be distributed, so that it is not necessary to send the complete data to a dedicated coordinated storage node, which greatly reduces the amount of data transmission in the storage system and saves the storage system. The consumption of network resources within.

With reference to the first aspect, in a first possible implementation manner of the first aspect, the storage system where the coordination node is located further includes a view management node, and the coordinated storage node is configured to slice the i-th data based on the view management node. When the start address and the length are sent to the i th storage node of the N storage nodes, the storage node information sent by the view management node may be received first, where the storage node information includes information of a storage node storing data of length M, the information Includes storage node IDs and more. The coordination storage node sends the starting address and length of the i-th data fragment to the i-th storage node of the N storage nodes in a targeted manner according to the received storage node information.

In conjunction with the first aspect, in a second possible implementation of the first aspect, the coordination storage node further receives a storage success message, which is stored by the storage node storing the X data fragments and the Y parity fragments. The other storage nodes except the coordination storage node are sent. By receiving the storage success message, the coordination storage node can learn the storage condition of each fragment, so that when any fragment is not successfully stored, the fragment is sent to the corresponding storage node for storage in time to ensure data integrity.

In conjunction with the first possible implementation of the first aspect, in a third possible implementation of the first aspect, the coordinated storage node is capable of targeting the starting address and length of the ith data fragment. The view management node in the storage system may perform hash calculation on the data of length M to obtain a hash value, and determine the data of length M according to the hash value and the previously constructed hash space. The position in the Greek space, and starting from the position, the N storage nodes in the specified direction are used as the storage node storing the data of length M and storing the N points in the X data fragments and the Y verification fragments. Slice storage node.

A second aspect provides a coordinated storage node, where the coordinated storage node is located in a storage system, where the storage system includes a plurality of storage nodes and a view management node, and the storage nodes of the plurality of storage nodes have a length of The data of the M, the coordinated storage node is included in the N storage nodes, where N is a natural number greater than 1, the coordinated storage node is configured to execute the method performed by the coordinated storage node in the first aspect, and the view management node uses The method performed by the view management node in performing the above first aspect.

A third aspect provides a coordinated storage node, where the coordinated storage node is located in a storage system, where the storage system includes a plurality of storage nodes, and N of the plurality of storage nodes stores data of length M, And the coordinated storage node is included in N storage nodes, wherein N is natural data greater than 1, the coordinated storage node includes a memory, a processor, and a bus, and the memory and the processor are directly connected through the bus;

The memory is for storing computer instructions for executing computer instructions for performing memory storage for performing the data storage method in the storage system of the first aspect described above.

Specifically, the coordinated storage node of the foregoing second aspect and the third aspect may perform the following steps through a specific structural unit or a processor:

Dividing the data of length M into X data fragments, and verifying the X data fragments to obtain Y verification fragments, wherein X and Y are natural numbers, and each data The length of the slice and each check slice is M/X;

Sending a starting address and a length of the i th data fragment to an i th storage node of the N storage nodes, Where i is a natural number greater than 1 and less than or equal to D, and D is a minimum of X and N;

When X is greater than N, a first storage node is allocated for each of the XN data fragments and the Y verification fragments, and the first storage node is divided into the multiple storage nodes. Any storage node other than the N storage nodes.

Transmitting a starting address and a length of the i th data fragment to an i th storage node of the N storage nodes, where i is a natural number greater than 1 and less than or equal to D, and D is a minimum of X and N value;

When X is not greater than N, a second storage node is allocated for each of the Y verification fragments, and the second storage node is the X storage nodes among the plurality of storage nodes. Any storage node other than .

The beneficial effects brought by the technical solutions provided by the embodiments of the present invention are:

When data is transferred from the multi-copy storage mode to the EC storage mode for storage, there is no need to send data to a dedicated coordinated storage node, and data distribution can be performed by using a storage node that can serve as a data coordination node, thereby greatly reducing data transmission amount and saving. The network resource consumption in the system.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present invention. Other drawings may also be obtained from those of ordinary skill in the art in light of the inventive work.

1 is a schematic diagram of a structure of a storage system according to an embodiment of the present invention;

2 is a flowchart of a data storage method in a storage system according to another embodiment of the present invention;

3 is a schematic diagram of a consistent hash algorithm according to another embodiment of the present invention;

FIG. 4 is a schematic diagram of a distribution of storage nodes in a storage system according to another embodiment of the present invention; FIG.

FIG. 5 is a schematic diagram of a data storage process according to another embodiment of the present invention; FIG.

6 is a schematic diagram of a data storage process according to another embodiment of the present invention;

FIG. 7 is a schematic structural diagram of a coordinated storage node according to another embodiment of the present invention; FIG.

FIG. 8 is a schematic structural diagram of a coordinated storage node according to another embodiment of the present invention; FIG.

9 is an illustrative computer architecture of a coordinated storage node for use in another embodiment of the present invention.

detailed description

The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

Referring to FIG. 1, a storage system is illustrated that includes a view management node 101, a client 102, and a storage node 103.

The view management node 101 can be a single computing device or a computing cluster composed of multiple computing devices. The view management node is configured to receive the heartbeat information reported by the storage node 103, and according to the storage node 103. The reported heartbeat information updates the node status of the storage node 103. The view management node 101 is also used to allocate storage nodes for data in the client 102. The view management node 101 also maintains a view of the correspondence between the hash value and the storage node for the client 102 to query data from the storage node.

The client 102 can be a smart phone, a tablet, a laptop, etc., and the present embodiment does not specifically limit the product type of the client 102. The client 102 is configured to receive a view pushed by the view management node 101 and query data according to the view.

The storage node 103 is a physical medium for storing data in the storage system. The storage node reports heartbeat information to the view management node for a preset duration, so that the view management node maintains the state of the storage node.

The view management node 101 and the client 102 and the data node 103 can communicate through a wired network or a wireless network, and the client 102 and the data node 103 can communicate through a wired network or a wireless network.

Based on the database system shown in FIG. 1 , an embodiment of the present invention provides a data storage method, where the storage system includes a plurality of storage nodes and a view management node, wherein N storage nodes of the plurality of storage nodes store lengths For the data of M, and the N storage nodes include a coordinated storage node, referring to FIG. 2, the method flow provided by the embodiment of the present invention includes:

201. The view management node allocates storage node information for data of length M.

In this embodiment, the number of storage nodes included in the storage system is large, and the data of the length M is stored in a redundant manner for the purpose of minimizing the network resource consumption in the system during the data transmission. The view management node will also use the storage node selection algorithm to allocate storage nodes for data of length M.

The storage node selection algorithm may include a consistent hash algorithm and the like. Consistent hashing algorithm principle is as follows: a configuration 0 to the Hash space ^232-1 of the Hash space 0 ~ ^232-1 ring composed of a hash, the hash ring into a plurality of equal intervals, each The hash values within the intervals have the same range of differences. FIG. 3 shows a hash ring composed of a hash space. Referring to FIG. 3, the hash ring is equally divided into 20 intervals, which are respectively P0 to P19 and the hash values of P0 to P19 have the same difference range.

Limited by the performance of the storage system itself, when the storage system stores the data of length M in a redundant manner, the number of duplicate data (the data of length M obtained by copying), the number of data fragments, and the fragmentation of data The number of verification slices obtained by the verification is determined. When the storage system system in this embodiment is stored in a redundant manner, the number of duplicate data is N, the number of data fragments is X, and the number of verification fragments is Y, where N is greater than The natural number of 1, X and Y are natural numbers.

Specifically, the view management node may include the following steps 2011 to 2013 when the storage node selection algorithm is used to allocate the storage node to the data of length M:

In 2011, the view management node hashes the data of length M to obtain a hash value.

The view management node may perform hash calculation on the data of length M by using a specified hash algorithm to obtain a hash value. The specified hash algorithm includes an addition hash algorithm, a multiplication hash algorithm, a division hash algorithm, a bit operation hash algorithm, a table look hash algorithm, and the like.

2012, the view management node determines the position of the data of length M in the hash space according to the hash value.

In this embodiment, each interval in the hash space corresponds to a hash value range. When the hash value is in any interval of the hash space, the position where the interval is located may be the data of length M. The position in the hash space. For example, the data KEY of length M is hash-calculated, and the obtained hash value is k1, and the k1 is located in the area where P0 is located in FIG. In the meantime, therefore, the position where P0 is located can be used as the position of the data KEY of length M in the hash space.

In 2013, the view management node takes the location as a starting point, and uses N storage nodes in the specified direction as storage nodes for storing data of length M.

The specified direction may be a clockwise direction starting from a position where the hash value is located, or a counterclockwise direction starting from a position where the hash value is located. Since the data of length M is redundantly stored by using N copies in this embodiment, the view management node needs to allocate N storage nodes in the copy storage mode for the data of length M. Based on the hash value obtained by hashing the data of the length M, the view management node takes the location of the hash value as a starting point, and uses N storage nodes in the specified direction as the storage node storing the data of length M. The N storage nodes may be selected continuously, that is, the N consecutive storage nodes in the specified direction starting from the location of the hash value are used as storage nodes for storing data of length M, and may also be discontinuous. The N non-contiguous storage nodes in the specified direction starting from the location of the hash value are used as storage nodes for storing data of length M.

In 2014, the view management node takes the location as a starting point, and uses N storage nodes in the specified direction as storage nodes that store N data slices and N slices in the Y parity slices.

Based on the hash value obtained by hashing the data of length M, the view management node takes the location of the hash value as a starting point, and stores N storage nodes in the specified direction as storing X data fragments and Y schools. The storage node of the N fragments in the fragmentation slice, the selection manner of the N storage nodes in the X data fragment and the Y parity slices and the selection of the N storage nodes storing the data of the length M In the same manner, if N storage nodes storing data of length M are selected in a continuous manner, N consecutive storage nodes starting from the location of the hash value are used as storage X data fragments and Y schools. The N storage nodes in the fragmentation slice, if the N storage nodes storing the data of length M are selected in a discontinuous manner, the N discontinuous storage nodes starting from the location of the hash value As a storage node storing X pieces of data and N pieces of Y check pieces. Regardless of how the method is selected, it is ensured that the N storage nodes storing the data of length M and the storage nodes storing the N data fragments and the N fragments of the Y verification fragments are maximally overlapped. .

Taking the redundant storage mode of 3 copy data, 4 data slice, and 2 check slice as an example, the view management node allocates a storage node for data of length M, and can refer to the following example.

(1) For the data KEY of length M, the view management node performs hash calculation on the data KEY of length M to obtain a hash value k1.

(2) The view management node determines the location of the hash value k1 from the hash space according to the hash value k1, and uses the position as the starting point, and stores the three storage nodes in the clockwise direction as the storage length M. For the storage node of the data, see FIG. 3, P0, P1, and P2 are storage nodes that store data of length M.

(3) The view management node determines the location of the hash value k1 from the hash space according to the hash value k1, and uses the location as the starting point, and stores three data nodes in the clockwise direction as four data. For the storage node of the slice and the three slices of the two check slices, refer to FIG. 3, P0, P1, and P2 are three slices of the data slice and two of the two check slices. Storage node.

The storage node allocated by the view management node for the length M data is actually a virtual storage node where each interval in the hash space is located, and the virtual storage node cannot truly store data, and the virtual storage node and the storage system need to be The physical storage node establishes a mapping relationship, and persists data on the corresponding virtual storage node to the physical storage node. To facilitate management of virtual storage nodes and physical storage nodes, the view management node also maintains a single a mapping table in which a mapping relationship between each interval of the hash space and the physical storage node is stored. Usually, the number of physical storage nodes is smaller than the number of intervals in the hash space, so that each physical storage node is actually Data corresponding to multiple intervals, that is, multiple virtual storage nodes, is stored on the same physical storage node. Referring to FIG. 3, the hash space is equally divided into 20 intervals of P0 to P19, and the storage system includes four physical storage nodes, namely, a physical storage node 1, a physical storage node 2, a physical storage node 3, and a physical storage node 4. Among them, A(P0), E(P4), I(P8), M(P12), Q(P16) are mapped to physical node 1, B(P1), F(P5), J(P9), N (P13), R(P17) are mapped to physical node 2, C(P3), G(P6), K(P10), 0(P14), S(P18) are mapped to physical node 3, D(( P4), H(P7), L(P11), P(P15), T(P19) are mapped onto the physical node 4.

Referring to FIG. 4, based on the mapping table maintained by the view management node, when the client needs to query the data KEY1 from the storage system, the client may first calculate the hash value k1 of the KEY1, and determine that the hash value k1 is in the hash space. The location, and then according to the mapping table, find the physical storage node where the hash value is located, thereby querying the data KEY1 from the physical storage node.

202. The view management node sends the storage node information to the coordination storage node and the client.

The storage node storing data of length M and the storage node storing N pieces of data and N pieces of Y check pieces are the key for the client to perform multiple copy storage and coordinate storage nodes for EC storage. In order to enable the client and the coordinated storage node to perform targeted storage, the view management node may store the storage node information of the N storage nodes allocated for the data of length M, and store X data fragments and Y schools. The storage node information of the storage nodes of the N fragments in the fragmentation slice is sent to the client and the coordination storage node. The storage node information includes a storage node identifier and the like.

203. The client copies the data of length M according to the storage node information, and sends the copied data to the N storage nodes for storage.

When receiving the storage node information sent by the view management node, the client copies N shares of data of length M to obtain N data of length M, and then sends N data of length M to the allocated N storages. Nodes are stored by each storage node.

In order to improve the data access rate, the storage system can convert the storage mode of the data according to the amount of access by the client. When the client accesses any data to a certain value, the storage system can trigger the coordinated storage node to convert the data from the copy storage mode. Store for EC storage mode.

204. The coordination storage node divides the data of length M into X data fragments, and performs verification on X data fragments to obtain Y verification fragments.

The storage system in this embodiment uses X data fragments and Y parity fragments to store in a redundant manner. Therefore, in the EC storage mode, the coordinated storage node needs to store the length of the node as M. The data is equally divided into X data fragments. Based on the divided X data fragments, the coordinated storage node uses a specified check algorithm to verify X data fragments to obtain Y parity fragments. The specified check algorithm includes a parity check algorithm, a CRC cyclic redundancy check algorithm, and the like.

205. The coordinating storage node sends the starting address and length of the i th data fragment to the i th storage node of the N storage nodes.

Where i is a natural number greater than 1 and less than or equal to D, and D is a minimum of X and N. Since the data of length M has been stored in the N storage nodes, in order to save data, the network resources consumed in the system are transmitted. The storage node may send the fragmentation information to the N storage nodes, and each storage node cuts the stored data of length M according to the fragmentation information to obtain a data fragment and store it. The fragmentation information includes a starting address and a length of the data fragment. For the i-th data fragment in the X data fragments, the coordination storage node sends the start address and length of the i-th data fragment to the i-th storage node of the N storage nodes, by the ith storage The node cuts the stored data of length M according to the starting address and length of the i-th data fragment, obtains data fragments corresponding to the starting address and length, and stores the data fragment.

206. When X is greater than N, the coordinated storage section allocates a first storage node for each of the X-N data fragments and the Y verification fragments.

The first storage node is any storage node other than the N storage nodes among the plurality of storage nodes. When X is greater than N, it indicates that the number of data fragments is greater than the number of copies of data of length M. At this time, the coordinated storage node needs to allocate one for each of XN data fragments and Y parity fragments. The first storage node sends XN data fragments and Y parity fragments to the allocated first storage node, and is stored by the first storage node.

207. When X is not greater than N, the coordinated storage section allocates a second storage node for each of the Y check slices.

The second storage node is any storage node other than the X storage nodes among the plurality of storage nodes. When X is not greater than N, it indicates that the number of data fragments is less than or equal to the number of copies of data of length M. At this time, the coordinated storage section needs to allocate a second storage node for each of the Y verification fragments. And sending the Y check fragments to the second storage node for storage by the second storage node.

For the above process, for ease of understanding, a specific example will be described below.

Referring to FIG. 5, the set data size is 4M, and the storage node allocated by the view management node for the data with the storage length of 4M is N1, N2, and N3, and the storage node allocated for the four data fragments and the two parity fragments is N1, N2, N3, N4, N5, N6, the coordination storage node N1 divides the data of length 4M into 4 1M data fragments, and verifies 4 data fragments to obtain 2 check scores. sheet. When it is necessary to convert the data of length M from the multiple copy storage mode to the EC storage mode for storage, the coordinated storage node N1 stores the length of the node to be 4M according to the fragmentation information of the start address of 0 and the length of 1M. The data is cut, the data is fragmented and stored, and the fragment information with the start address of 1M and the length of 1M is sent to the storage node N2, and the fragment information with the start address of 2M and the length of 1M is sent to the storage node N3. The data of the length 4M stored by the node is cut by the storage nodes N2 and N3 to obtain data fragments and stored. For the storage nodes N4, N5, and N6, the coordinated storage node N1 can store the data with the start address of 3M and the length of 1M to the storage node N4, and store the check fragment with the start address of 0 and the length of 1M. It is sent to the storage node N5 for storage, and the verification slice with the start address of 1M and the length of 1M is sent to the storage node N6 for storage.

In order to enable the client to know the storage request of the data in the EC storage mode, the designated storage node will also send a storage success message to the coordinated storage node when each data fragment or check fragment is successfully stored. The designated storage node is a storage node other than the coordinated storage node among the storage nodes storing the X data fragments and the Y parity fragments.

6 is a process diagram for performing data storage when data is transferred from a multiple copy storage mode to an EC storage mode. Referring to FIG. 6, the view management node allocates storage nodes N1, N2, and N3 for data of length M, which are four. The storage nodes allocated by the data fragment and the two parity fragments are N1, N2, N3, N4, N5, and N6. When the multi-copy storage mode is switched to the EC storage mode, the coordinated storage node N1 divides the stored data of length M into 4 data fragments, and 4 data pieces. The data fragment is verified to obtain two verification fragments. The coordination storage node N1N1 cuts the data of length M stored by the node according to the fragmentation information <offset1, len1>, obtains data fragmentation and stores it, and sends the fragmentation information <offset2, len2> to the storage node N2. The storage node N2 cuts the stored data of length M according to <offset2, len2> to obtain a data fragment and stores it; sends fragmentation information <offset3, len3> to N3, according to <offset3 by storage node N3. , len3> cuts the stored data of length M to obtain a data fragment and stores it; sends the fragment of the fragment information to <offset4, len4> to the storage node N4 for storage, and two schools The test pieces are sent to storage nodes N5 and N6 for storage.

The method provided by the embodiment of the present invention, when the data is transferred from the multiple copy storage mode to the EC storage mode for storage, does not need to send data to a dedicated coordinated storage node, but performs data distribution by using a storage node that can serve as a data coordination node. The data transmission amount is greatly reduced, and the network resource consumption in the system is saved.

Referring to FIG. 7, an embodiment of the present invention provides a coordinated storage node, where the coordinated storage node is located in a storage system, where the storage system includes multiple storage nodes, and N storage nodes of the plurality of storage nodes are stored with a length of M. Data, the coordination storage node is included in N storage nodes, where N is a natural number greater than 1, and the coordination storage node includes:

The dividing unit 701 is configured to divide the data of length M into X data fragments, and verify the X data fragments to obtain Y verification fragments, wherein X and Y are natural numbers, and each The length of one data slice and each check slice is M/X;

The sending unit 702 is configured to send the start address and the length of the ith data fragment to the ith storage node of the N storage nodes, where i is a natural number greater than 1 and small equal to D, and D is X. And the minimum value in N;

The allocating unit 703 is configured to allocate, when X is greater than N, a first storage node for each of the XN data fragments and the Y verification fragments, where the first storage node is a plurality of storage nodes. Any storage node other than N storage nodes.

In another embodiment of the present invention, the allocating unit 703 is configured to allocate, when X is not greater than N, a second storage node for each of the Y verification fragments, and the second storage node is multiple Any storage node other than X storage nodes in the storage node.

Optionally, in another embodiment of the present invention, the allocating unit 703 is configured to allocate a first storage node for each of the XN data fragments and the Y parity fragments when X is greater than N. The first storage node is any one of the plurality of storage nodes except the N storage nodes; when X is not greater than N, a second storage node is allocated for each of the Y verification fragments. The second storage node is any storage node other than the X storage nodes among the plurality of storage nodes.

Referring to FIG. 8, in another embodiment of the present invention, the storage system further includes a view management node, and the coordination storage node further includes a receiving unit 704;

The receiving unit 704 is configured to receive storage node information sent by the view management node, where the storage node information includes information about a storage node that stores data of length M;

The sending unit 702 is specifically configured to send, according to the storage node information, the start address and the length of the i th data fragment to the i th storage node of the N storage nodes.

In another embodiment of the present invention, the receiving unit 704 is further configured to receive a storage success message, where the storage success message is other than the coordinated storage node in the storage node that stores the X data fragments and the Y parity fragments. The storage node sends.

In another embodiment of the present invention, the view management node is configured to perform hash calculation on the data of length M to obtain a hash value; and determine the position of the data of length M in the hash space according to the hash value. Starting from the position, the N storage nodes in the specified direction are used as storage nodes for storing data of length M; starting from the position, the N storage nodes in the specified direction are stored as X data fragments and Y. Verify the storage nodes of the N slices in the slice.

In summary, the system provided by the embodiment of the present invention does not need to send data to a dedicated coordinated storage node when the data is transferred from the multiple copy storage mode to the EC storage mode, and the storage can be used as a data coordination node. The node performs data distribution, which greatly reduces the amount of data transmission and saves network resource consumption in the system.

Referring to Figure 9, an illustrative computer architecture of a coordinated storage node 900 for use in one embodiment of the present invention is shown. The coordinated storage node 900 is a server or a storage device. The coordinated storage node 900 includes a processor 901, a memory 902, and a bus 903, wherein the processor 901 and the memory 902 are connected by a bus 903. Optionally, the coordinated storage node 900 further includes a basic input/output system (I/O system) 904 that facilitates transferring information between various devices within the computing device and for storing operating systems, applications, and other program modules. Mass storage device 905.

Alternatively, the mass storage device 905 is connected to the processor 901 by a mass storage controller (not shown) connected to the bus 903. The mass storage device 905 and its associated computer readable medium provide non-volatile storage for the coordinating storage node 900. That is, the mass storage device 905 can include a computer readable medium (not shown) such as a hard disk or a CD-ROM drive.

Without loss of generality, the computer readable medium can include computer storage media and communication media.

Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid state storage technologies, CD-ROM, DVD or other optical storage, tape cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage medium is not limited to the above.

According to various embodiments of the present invention, the coordinated storage node 900 may also be operated by a remote computer connected to the network through a network such as the Internet. That is, the coordination storage node 900 can be connected to the network 907 through a network interface unit 906 connected to the bus 903, or can also be connected to other types of networks or remote computer systems using the network interface unit 906 (not shown) ). The network interface unit 906 can be a network card (also referred to as a network interface card) or a host bus adapter.

The computer device provided by the embodiment of the present invention does not need to send data to a dedicated coordinated storage node when data is transferred from a multiple copy storage mode to an EC storage mode, but is distributed by using a storage node that can serve as a data coordination node. , greatly reducing the amount of data transmission, saving network resource consumption within the system.

It should be noted that, when the data is stored in the foregoing embodiment, the coordinated storage node only exemplifies the division of each functional module. In actual applications, the function distribution may be completed by different functional modules as needed. The internal structure of the coordination storage node is divided into different functional modules to perform all or part of the functions described above. In addition, the embodiment of the data storage method in the foregoing embodiment is the same as the embodiment of the data storage method in the data storage system. For details, refer to the method embodiment, and details are not described herein.

A person skilled in the art may understand that all or part of the steps of implementing the above embodiments may be completed by hardware, or may be instructed by a program to execute related hardware, and the program may be stored in a computer readable storage medium. The storage medium mentioned may be a read only memory, a magnetic disk or an optical disk or the like.

The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalents, improvements, etc., which are within the spirit and scope of the present invention, should be included in the protection of the present invention. Within the scope.

Claims

A data storage method in a storage system, wherein the storage system includes a plurality of storage nodes, and N storage nodes of the plurality of storage nodes store data of length M, the N storages The node includes a coordinated storage node, where N is a natural number greater than 1, the method comprising:

The coordinated storage node divides the data of length M into X data fragments, and performs verification on the X data fragments to obtain Y verification fragments, where X and Y are natural numbers. , and the length of each data slice and each check slice is M/X;

The coordination storage node sends the start address and length of the i-th data fragment to the i-th storage node of the N storage nodes, where i is a natural number greater than 1 and less than or equal to D, and D is X And the minimum value in N;

When X is greater than N, the coordinated storage section allocates a first storage node for each of the XN data fragments and the Y verification fragments, and the first storage node is the plurality of Any storage node other than the N storage nodes;

When X is not greater than N, the coordinated storage section allocates a second storage node for each of the Y verification fragments, and the second storage node is a middle of the plurality of storage nodes. Any storage node other than X storage nodes.
The method according to claim 1, wherein the storage system further comprises a view management node, wherein the coordinated storage node sends a start address and a length of the i-th data slice to the N storage nodes. The i-th storage node, including:

The coordination storage node receives the storage node information sent by the view management node, where the storage node information includes information of a storage node that stores the data of length M;

And the coordinated storage node sends the start address and length of the ith data fragment to the i-th storage node of the N storage nodes according to the storage node information.
The method of claim 1 further comprising:

The coordinated storage node receives a storage success message, and the storage success message is sent by a storage node other than the coordinated storage node among the storage nodes storing the X data fragments and the Y parity fragments.
The method of claim 2, wherein the method further comprises:

The view management node performs hash calculation on the data of length M to obtain a hash value;

Determining, by the view management node, a location of the data of length M in a hash space according to the hash value;

The view management node takes the location as a starting point, and uses N storage nodes in a specified direction as a storage node that stores the data of length M;

The view management node takes the location as a starting point and uses N storage nodes in a specified direction as storage nodes that store the X data fragments and the N slices of the Y parity fragments.
A coordinated storage node, wherein the coordinated storage node is located in a storage system, the storage system includes a plurality of storage nodes, and N of the plurality of storage nodes stores data of length M The coordinated storage node is included in the N storage nodes, where N is a natural number greater than 1, the coordinated storage Nodes include:

a dividing unit, configured to divide the data of length M into X data fragments, and perform verification on the X data fragments to obtain Y verification fragments, wherein X and Y are natural numbers , and the length of each data slice and each check slice is M/X;

a sending unit, configured to send a starting address and a length of the i th data fragment to an i th storage node of the N storage nodes, where i is a natural number greater than 1 and small equal to D, where D is The minimum of X and N;

An allocating unit, configured to allocate, when X is greater than N, a first storage node for each of the XN data fragments and the Y verification fragments, where the first storage node is the Any storage node other than the N storage nodes.
The coordination storage node according to claim 5, wherein the allocation unit is further configured to allocate a second storage for each of the Y verification fragments when X is not greater than N a node, the second storage node being any one of the plurality of storage nodes except the X storage nodes.
The coordinated storage node according to claim 5, wherein the storage system further comprises a view management node, the coordinated storage node further comprising a receiving unit;

The receiving unit is configured to receive storage node information sent by the view management node, where the storage node information includes information of a storage node that stores the data of length M;

The sending unit is specifically configured to send, according to the storage node information, a starting address and a length of the i th data fragment to the i th storage node of the N storage nodes.
The coordination storage node according to claim 5, wherein the receiving unit is further configured to receive a storage success message, where the storage success message is a storage node that stores X data fragments and Y parity fragments. The other storage nodes other than the coordinated storage node are sent.
The coordination storage node according to claim 7, wherein the view management node is configured to perform hash calculation on the data of length M to obtain a hash value, and determine the location according to the hash value. The position of the data of length M in the hash space; starting from the position, the N storage nodes in the specified direction are used as storage nodes storing the data of length M; starting from the position, The N storage nodes in the specified direction are used as storage nodes that store the X data fragments and the N slices of the Y parity fragments.
A coordinated storage node, wherein the coordinated storage node is located in a storage system, the storage system includes a plurality of storage nodes, and N of the plurality of storage nodes stores data of length M The coordinated storage node is included in the N storage nodes, wherein N is a natural number greater than 1, the coordinated storage node includes a memory, a processor, and a bus, and the memory and the processor pass the Direct bus connection;

The memory is for storing computer instructions for executing the memory stored computer instructions for performing the data storage method in the storage system of any one of claims 1 to 4.