WO2018040583A1 - Procédé de mémorisation de données dans un système de mémorisation de données, et nœud de mémorisation coordinateur - Google Patents

Procédé de mémorisation de données dans un système de mémorisation de données, et nœud de mémorisation coordinateur Download PDF

Info

Publication number
WO2018040583A1
WO2018040583A1 PCT/CN2017/081073 CN2017081073W WO2018040583A1 WO 2018040583 A1 WO2018040583 A1 WO 2018040583A1 CN 2017081073 W CN2017081073 W CN 2017081073W WO 2018040583 A1 WO2018040583 A1 WO 2018040583A1
Authority
WO
WIPO (PCT)
Prior art keywords
storage
data
node
storage node
length
Prior art date
Application number
PCT/CN2017/081073
Other languages
English (en)
Chinese (zh)
Inventor
刘强
蒋中斌
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2018040583A1 publication Critical patent/WO2018040583A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0626—Reducing size or complexity of storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Definitions

  • the present invention relates to the field of information technology, and in particular, to a data storage method and a coordinated storage node in a data storage system.
  • the redundancy modes mainly include copy redundancy and EC (Erasure code) redundancy.
  • multi-copy redundancy is to copy the original data into multiple copies of data, and each copy data is stored on one storage node.
  • EC redundancy divides the original data into X pieces of data, and uses the EC algorithm to verify the X pieces of data, and obtains Y pieces of check data, and then stores X+Y pieces into different pieces. On the storage node.
  • the most widely used data storage method is a combination of multiple copy redundancy and EC redundancy.
  • the specific storage process is as follows: the client copies the original data, obtains multiple copies of the data, stores the plurality of copies of the data on different copy storage nodes, and sends any copy data to the EC coordination node, and the EC coordination node copies the copy.
  • the data is divided into X pieces of data, and the X pieces of data are verified by the EC algorithm to obtain Y pieces of check data, and the EC coordination node stores one piece of data, and then X+Y-1
  • the fragments are sent to different EC storage nodes for storage.
  • any replica data needs to be sent to the EC coordination node first, and then the EC coordination node will include multiple segments including fragment data and fragmentation verification data.
  • the slices are sent to different EC storage nodes for storage. This process transmits a large amount of data, resulting in a large consumption of network resources in the storage system.
  • the embodiment of the present invention provides a data storage method and a coordinated storage node in the storage system.
  • the technical solution is as follows:
  • a first aspect provides a data storage method in a storage system, where the storage system includes a plurality of storage nodes, and N storage nodes of the plurality of storage nodes store data of length M, and the N storage nodes include A coordinated storage node, where N is a natural number greater than 1, the method comprising:
  • the coordinated storage node divides the length of the M data into X data fragments, and each data slice has one.
  • the starting address and length, the starting address of each data fragment is the position of the data fragment in the data of length M, and the length of each data fragment is M/X.
  • the coordinated storage node obtains Y verification fragments by verifying X data fragments, and the length of each verification fragment is also M/X.
  • the coordination storage node For the i-th data fragment in the X data fragments, the coordination storage node sends the start address and length of the i-th data fragment to the i-th storage node of the N storage nodes, where i is greater than 1 and less than or equal to the natural number of D, D is the minimum of X and N.
  • the coordinated storage node allocates each of the XN data fragments and the Y parity fragments to a first storage node, where the first storage node is a plurality of storage nodes other than N. Any storage node.
  • the coordination storage node allocates a second storage node for each of the Y verification fragments, the second storage node being any one of the plurality of storage nodes except X storage nodes. Storage node.
  • the data storage node as a coordinated storage node has a dual function, not only can the complete data be stored, but also The shards including data sharding and parity sharding can also be distributed, so that it is not necessary to send the complete data to a dedicated coordinated storage node, which greatly reduces the amount of data transmission in the storage system and saves the storage system. The consumption of network resources within.
  • the storage system where the coordination node is located further includes a view management node, and the coordinated storage node is configured to slice the i-th data based on the view management node.
  • the storage node information sent by the view management node may be received first, where the storage node information includes information of a storage node storing data of length M, the information Includes storage node IDs and more.
  • the coordination storage node sends the starting address and length of the i-th data fragment to the i-th storage node of the N storage nodes in a targeted manner according to the received storage node information.
  • the coordination storage node further receives a storage success message, which is stored by the storage node storing the X data fragments and the Y parity fragments.
  • the other storage nodes except the coordination storage node are sent.
  • the coordination storage node can learn the storage condition of each fragment, so that when any fragment is not successfully stored, the fragment is sent to the corresponding storage node for storage in time to ensure data integrity.
  • the coordinated storage node is capable of targeting the starting address and length of the ith data fragment.
  • the view management node in the storage system may perform hash calculation on the data of length M to obtain a hash value, and determine the data of length M according to the hash value and the previously constructed hash space.
  • the position in the Greek space, and starting from the position, the N storage nodes in the specified direction are used as the storage node storing the data of length M and storing the N points in the X data fragments and the Y verification fragments.
  • Slice storage node may perform hash calculation on the data of length M to obtain a hash value, and determine the data of length M according to the hash value and the previously constructed hash space.
  • the position in the Greek space, and starting from the position, the N storage nodes in the specified direction are used as the storage node storing the data of length M and storing the N points in the X data fragments and the Y verification fragments.
  • a second aspect provides a coordinated storage node, where the coordinated storage node is located in a storage system, where the storage system includes a plurality of storage nodes and a view management node, and the storage nodes of the plurality of storage nodes have a length of The data of the M, the coordinated storage node is included in the N storage nodes, where N is a natural number greater than 1, the coordinated storage node is configured to execute the method performed by the coordinated storage node in the first aspect, and the view management node uses The method performed by the view management node in performing the above first aspect.
  • a third aspect provides a coordinated storage node, where the coordinated storage node is located in a storage system, where the storage system includes a plurality of storage nodes, and N of the plurality of storage nodes stores data of length M, And the coordinated storage node is included in N storage nodes, wherein N is natural data greater than 1, the coordinated storage node includes a memory, a processor, and a bus, and the memory and the processor are directly connected through the bus;
  • the memory is for storing computer instructions for executing computer instructions for performing memory storage for performing the data storage method in the storage system of the first aspect described above.
  • the coordinated storage node of the foregoing second aspect and the third aspect may perform the following steps through a specific structural unit or a processor:
  • a first storage node is allocated for each of the XN data fragments and the Y verification fragments, and the first storage node is divided into the multiple storage nodes. Any storage node other than the N storage nodes.
  • the coordinated storage node of the foregoing second aspect and the third aspect may perform the following steps through a specific structural unit or a processor:
  • a second storage node is allocated for each of the Y verification fragments, and the second storage node is the X storage nodes among the plurality of storage nodes. Any storage node other than .
  • FIG. 1 is a schematic diagram of a structure of a storage system according to an embodiment of the present invention.
  • FIG. 2 is a flowchart of a data storage method in a storage system according to another embodiment of the present invention.
  • FIG. 3 is a schematic diagram of a consistent hash algorithm according to another embodiment of the present invention.
  • FIG. 4 is a schematic diagram of a distribution of storage nodes in a storage system according to another embodiment of the present invention.
  • FIG. 5 is a schematic diagram of a data storage process according to another embodiment of the present invention.
  • FIG. 6 is a schematic diagram of a data storage process according to another embodiment of the present invention.
  • FIG. 7 is a schematic structural diagram of a coordinated storage node according to another embodiment of the present invention.
  • FIG. 8 is a schematic structural diagram of a coordinated storage node according to another embodiment of the present invention.
  • 9 is an illustrative computer architecture of a coordinated storage node for use in another embodiment of the present invention.
  • a storage system is illustrated that includes a view management node 101, a client 102, and a storage node 103.
  • the view management node 101 can be a single computing device or a computing cluster composed of multiple computing devices.
  • the view management node is configured to receive the heartbeat information reported by the storage node 103, and according to the storage node 103.
  • the reported heartbeat information updates the node status of the storage node 103.
  • the view management node 101 is also used to allocate storage nodes for data in the client 102.
  • the view management node 101 also maintains a view of the correspondence between the hash value and the storage node for the client 102 to query data from the storage node.
  • the client 102 can be a smart phone, a tablet, a laptop, etc., and the present embodiment does not specifically limit the product type of the client 102.
  • the client 102 is configured to receive a view pushed by the view management node 101 and query data according to the view.
  • the storage node 103 is a physical medium for storing data in the storage system.
  • the storage node reports heartbeat information to the view management node for a preset duration, so that the view management node maintains the state of the storage node.
  • the view management node 101 and the client 102 and the data node 103 can communicate through a wired network or a wireless network, and the client 102 and the data node 103 can communicate through a wired network or a wireless network.
  • an embodiment of the present invention provides a data storage method, where the storage system includes a plurality of storage nodes and a view management node, wherein N storage nodes of the plurality of storage nodes store lengths For the data of M, and the N storage nodes include a coordinated storage node, referring to FIG. 2, the method flow provided by the embodiment of the present invention includes:
  • the view management node allocates storage node information for data of length M.
  • the number of storage nodes included in the storage system is large, and the data of the length M is stored in a redundant manner for the purpose of minimizing the network resource consumption in the system during the data transmission.
  • the view management node will also use the storage node selection algorithm to allocate storage nodes for data of length M.
  • the storage node selection algorithm may include a consistent hash algorithm and the like. Consistent hashing algorithm principle is as follows: a configuration 0 to the Hash space 232-1 of the Hash space 0 ⁇ 232-1 ring composed of a hash, the hash ring into a plurality of equal intervals, each The hash values within the intervals have the same range of differences.
  • FIG. 3 shows a hash ring composed of a hash space. Referring to FIG. 3, the hash ring is equally divided into 20 intervals, which are respectively P0 to P19 and the hash values of P0 to P19 have the same difference range.
  • the number of duplicate data (the data of length M obtained by copying), the number of data fragments, and the fragmentation of data
  • the number of verification slices obtained by the verification is determined.
  • the number of duplicate data is N
  • the number of data fragments is X
  • the number of verification fragments is Y, where N is greater than
  • the natural number of 1, X and Y are natural numbers.
  • the view management node may include the following steps 2011 to 2013 when the storage node selection algorithm is used to allocate the storage node to the data of length M:
  • the view management node hashes the data of length M to obtain a hash value.
  • the view management node may perform hash calculation on the data of length M by using a specified hash algorithm to obtain a hash value.
  • the specified hash algorithm includes an addition hash algorithm, a multiplication hash algorithm, a division hash algorithm, a bit operation hash algorithm, a table look hash algorithm, and the like.
  • the view management node determines the position of the data of length M in the hash space according to the hash value.
  • each interval in the hash space corresponds to a hash value range.
  • the position where the interval is located may be the data of length M.
  • the position in the hash space For example, the data KEY of length M is hash-calculated, and the obtained hash value is k1, and the k1 is located in the area where P0 is located in FIG. In the meantime, therefore, the position where P0 is located can be used as the position of the data KEY of length M in the hash space.
  • the view management node takes the location as a starting point, and uses N storage nodes in the specified direction as storage nodes for storing data of length M.
  • the specified direction may be a clockwise direction starting from a position where the hash value is located, or a counterclockwise direction starting from a position where the hash value is located. Since the data of length M is redundantly stored by using N copies in this embodiment, the view management node needs to allocate N storage nodes in the copy storage mode for the data of length M. Based on the hash value obtained by hashing the data of the length M, the view management node takes the location of the hash value as a starting point, and uses N storage nodes in the specified direction as the storage node storing the data of length M.
  • the N storage nodes may be selected continuously, that is, the N consecutive storage nodes in the specified direction starting from the location of the hash value are used as storage nodes for storing data of length M, and may also be discontinuous.
  • the N non-contiguous storage nodes in the specified direction starting from the location of the hash value are used as storage nodes for storing data of length M.
  • the view management node takes the location as a starting point, and uses N storage nodes in the specified direction as storage nodes that store N data slices and N slices in the Y parity slices.
  • the view management node Based on the hash value obtained by hashing the data of length M, the view management node takes the location of the hash value as a starting point, and stores N storage nodes in the specified direction as storing X data fragments and Y schools.
  • the storage node of the N fragments in the fragmentation slice, the selection manner of the N storage nodes in the X data fragment and the Y parity slices and the selection of the N storage nodes storing the data of the length M In the same manner, if N storage nodes storing data of length M are selected in a continuous manner, N consecutive storage nodes starting from the location of the hash value are used as storage X data fragments and Y schools.
  • the N storage nodes in the fragmentation slice if the N storage nodes storing the data of length M are selected in a discontinuous manner, the N discontinuous storage nodes starting from the location of the hash value As a storage node storing X pieces of data and N pieces of Y check pieces. Regardless of how the method is selected, it is ensured that the N storage nodes storing the data of length M and the storage nodes storing the N data fragments and the N fragments of the Y verification fragments are maximally overlapped. .
  • the view management node allocates a storage node for data of length M, and can refer to the following example.
  • the view management node For the data KEY of length M, the view management node performs hash calculation on the data KEY of length M to obtain a hash value k1.
  • the view management node determines the location of the hash value k1 from the hash space according to the hash value k1, and uses the position as the starting point, and stores the three storage nodes in the clockwise direction as the storage length M.
  • P0, P1, and P2 are storage nodes that store data of length M.
  • the view management node determines the location of the hash value k1 from the hash space according to the hash value k1, and uses the location as the starting point, and stores three data nodes in the clockwise direction as four data.
  • P0, P1, and P2 are three slices of the data slice and two of the two check slices. Storage node.
  • the storage node allocated by the view management node for the length M data is actually a virtual storage node where each interval in the hash space is located, and the virtual storage node cannot truly store data, and the virtual storage node and the storage system need to be
  • the physical storage node establishes a mapping relationship, and persists data on the corresponding virtual storage node to the physical storage node.
  • the view management node also maintains a single a mapping table in which a mapping relationship between each interval of the hash space and the physical storage node is stored.
  • the number of physical storage nodes is smaller than the number of intervals in the hash space, so that each physical storage node is actually Data corresponding to multiple intervals, that is, multiple virtual storage nodes, is stored on the same physical storage node.
  • the hash space is equally divided into 20 intervals of P0 to P19, and the storage system includes four physical storage nodes, namely, a physical storage node 1, a physical storage node 2, a physical storage node 3, and a physical storage node 4.
  • A(P0), E(P4), I(P8), M(P12), Q(P16) are mapped to physical node 1
  • B(P1), F(P5), J(P9), N (P13), R(P17) are mapped to physical node 2
  • C(P3), G(P6), K(P10), 0(P14), S(P18) are mapped to physical node 3
  • D(( P4), H(P7), L(P11), P(P15), T(P19) are mapped onto the physical node 4.
  • the client when the client needs to query the data KEY1 from the storage system, the client may first calculate the hash value k1 of the KEY1, and determine that the hash value k1 is in the hash space. The location, and then according to the mapping table, find the physical storage node where the hash value is located, thereby querying the data KEY1 from the physical storage node.
  • the view management node sends the storage node information to the coordination storage node and the client.
  • the storage node storing data of length M and the storage node storing N pieces of data and N pieces of Y check pieces are the key for the client to perform multiple copy storage and coordinate storage nodes for EC storage.
  • the view management node may store the storage node information of the N storage nodes allocated for the data of length M, and store X data fragments and Y schools.
  • the storage node information of the storage nodes of the N fragments in the fragmentation slice is sent to the client and the coordination storage node.
  • the storage node information includes a storage node identifier and the like.
  • the client copies the data of length M according to the storage node information, and sends the copied data to the N storage nodes for storage.
  • the client When receiving the storage node information sent by the view management node, the client copies N shares of data of length M to obtain N data of length M, and then sends N data of length M to the allocated N storages. Nodes are stored by each storage node.
  • the storage system can convert the storage mode of the data according to the amount of access by the client.
  • the storage system can trigger the coordinated storage node to convert the data from the copy storage mode. Store for EC storage mode.
  • the coordination storage node divides the data of length M into X data fragments, and performs verification on X data fragments to obtain Y verification fragments.
  • the storage system in this embodiment uses X data fragments and Y parity fragments to store in a redundant manner. Therefore, in the EC storage mode, the coordinated storage node needs to store the length of the node as M.
  • the data is equally divided into X data fragments. Based on the divided X data fragments, the coordinated storage node uses a specified check algorithm to verify X data fragments to obtain Y parity fragments.
  • the specified check algorithm includes a parity check algorithm, a CRC cyclic redundancy check algorithm, and the like.
  • the coordinating storage node sends the starting address and length of the i th data fragment to the i th storage node of the N storage nodes.
  • the storage node may send the fragmentation information to the N storage nodes, and each storage node cuts the stored data of length M according to the fragmentation information to obtain a data fragment and store it.
  • the fragmentation information includes a starting address and a length of the data fragment.
  • the coordination storage node For the i-th data fragment in the X data fragments, the coordination storage node sends the start address and length of the i-th data fragment to the i-th storage node of the N storage nodes, by the ith storage The node cuts the stored data of length M according to the starting address and length of the i-th data fragment, obtains data fragments corresponding to the starting address and length, and stores the data fragment.
  • the coordinated storage section allocates a first storage node for each of the X-N data fragments and the Y verification fragments.
  • the first storage node is any storage node other than the N storage nodes among the plurality of storage nodes.
  • X is greater than N, it indicates that the number of data fragments is greater than the number of copies of data of length M.
  • the coordinated storage node needs to allocate one for each of XN data fragments and Y parity fragments.
  • the first storage node sends XN data fragments and Y parity fragments to the allocated first storage node, and is stored by the first storage node.
  • the coordinated storage section allocates a second storage node for each of the Y check slices.
  • the second storage node is any storage node other than the X storage nodes among the plurality of storage nodes.
  • X is not greater than N, it indicates that the number of data fragments is less than or equal to the number of copies of data of length M.
  • the coordinated storage section needs to allocate a second storage node for each of the Y verification fragments. And sending the Y check fragments to the second storage node for storage by the second storage node.
  • the set data size is 4M
  • the storage node allocated by the view management node for the data with the storage length of 4M is N1, N2, and N3, and the storage node allocated for the four data fragments and the two parity fragments is N1, N2, N3, N4, N5, N6, the coordination storage node N1 divides the data of length 4M into 4 1M data fragments, and verifies 4 data fragments to obtain 2 check scores. sheet.
  • the coordinated storage node N1 stores the length of the node to be 4M according to the fragmentation information of the start address of 0 and the length of 1M.
  • the data is cut, the data is fragmented and stored, and the fragment information with the start address of 1M and the length of 1M is sent to the storage node N2, and the fragment information with the start address of 2M and the length of 1M is sent to the storage node N3.
  • the data of the length 4M stored by the node is cut by the storage nodes N2 and N3 to obtain data fragments and stored.
  • the coordinated storage node N1 can store the data with the start address of 3M and the length of 1M to the storage node N4, and store the check fragment with the start address of 0 and the length of 1M. It is sent to the storage node N5 for storage, and the verification slice with the start address of 1M and the length of 1M is sent to the storage node N6 for storage.
  • the designated storage node will also send a storage success message to the coordinated storage node when each data fragment or check fragment is successfully stored.
  • the designated storage node is a storage node other than the coordinated storage node among the storage nodes storing the X data fragments and the Y parity fragments.
  • FIG. 6 is a process diagram for performing data storage when data is transferred from a multiple copy storage mode to an EC storage mode.
  • the view management node allocates storage nodes N1, N2, and N3 for data of length M, which are four.
  • the storage nodes allocated by the data fragment and the two parity fragments are N1, N2, N3, N4, N5, and N6.
  • the coordinated storage node N1 divides the stored data of length M into 4 data fragments, and 4 data pieces. The data fragment is verified to obtain two verification fragments.
  • the coordination storage node N1N1 cuts the data of length M stored by the node according to the fragmentation information ⁇ offset1, len1>, obtains data fragmentation and stores it, and sends the fragmentation information ⁇ offset2, len2> to the storage node N2.
  • the storage node N2 cuts the stored data of length M according to ⁇ offset2, len2> to obtain a data fragment and stores it; sends fragmentation information ⁇ offset3, len3> to N3, according to ⁇ offset3 by storage node N3.
  • len3> cuts the stored data of length M to obtain a data fragment and stores it; sends the fragment of the fragment information to ⁇ offset4, len4> to the storage node N4 for storage, and two schools The test pieces are sent to storage nodes N5 and N6 for storage.
  • the method provided by the embodiment of the present invention when the data is transferred from the multiple copy storage mode to the EC storage mode for storage, does not need to send data to a dedicated coordinated storage node, but performs data distribution by using a storage node that can serve as a data coordination node.
  • the data transmission amount is greatly reduced, and the network resource consumption in the system is saved.
  • an embodiment of the present invention provides a coordinated storage node, where the coordinated storage node is located in a storage system, where the storage system includes multiple storage nodes, and N storage nodes of the plurality of storage nodes are stored with a length of M.
  • the coordination storage node is included in N storage nodes, where N is a natural number greater than 1, and the coordination storage node includes:
  • the dividing unit 701 is configured to divide the data of length M into X data fragments, and verify the X data fragments to obtain Y verification fragments, wherein X and Y are natural numbers, and each The length of one data slice and each check slice is M/X;
  • the sending unit 702 is configured to send the start address and the length of the ith data fragment to the ith storage node of the N storage nodes, where i is a natural number greater than 1 and small equal to D, and D is X. And the minimum value in N;
  • the allocating unit 703 is configured to allocate, when X is greater than N, a first storage node for each of the XN data fragments and the Y verification fragments, where the first storage node is a plurality of storage nodes. Any storage node other than N storage nodes.
  • the allocating unit 703 is configured to allocate, when X is not greater than N, a second storage node for each of the Y verification fragments, and the second storage node is multiple Any storage node other than X storage nodes in the storage node.
  • the allocating unit 703 is configured to allocate a first storage node for each of the XN data fragments and the Y parity fragments when X is greater than N.
  • the first storage node is any one of the plurality of storage nodes except the N storage nodes; when X is not greater than N, a second storage node is allocated for each of the Y verification fragments.
  • the second storage node is any storage node other than the X storage nodes among the plurality of storage nodes.
  • the storage system further includes a view management node, and the coordination storage node further includes a receiving unit 704;
  • the receiving unit 704 is configured to receive storage node information sent by the view management node, where the storage node information includes information about a storage node that stores data of length M;
  • the sending unit 702 is specifically configured to send, according to the storage node information, the start address and the length of the i th data fragment to the i th storage node of the N storage nodes.
  • the receiving unit 704 is further configured to receive a storage success message, where the storage success message is other than the coordinated storage node in the storage node that stores the X data fragments and the Y parity fragments.
  • the storage node sends.
  • the view management node is configured to perform hash calculation on the data of length M to obtain a hash value; and determine the position of the data of length M in the hash space according to the hash value.
  • the N storage nodes in the specified direction are used as storage nodes for storing data of length M; starting from the position, the N storage nodes in the specified direction are stored as X data fragments and Y. Verify the storage nodes of the N slices in the slice.
  • the system provided by the embodiment of the present invention does not need to send data to a dedicated coordinated storage node when the data is transferred from the multiple copy storage mode to the EC storage mode, and the storage can be used as a data coordination node.
  • the node performs data distribution, which greatly reduces the amount of data transmission and saves network resource consumption in the system.
  • the coordinated storage node 900 is a server or a storage device.
  • the coordinated storage node 900 includes a processor 901, a memory 902, and a bus 903, wherein the processor 901 and the memory 902 are connected by a bus 903.
  • the coordinated storage node 900 further includes a basic input/output system (I/O system) 904 that facilitates transferring information between various devices within the computing device and for storing operating systems, applications, and other program modules.
  • the mass storage device 905 is connected to the processor 901 by a mass storage controller (not shown) connected to the bus 903.
  • the mass storage device 905 and its associated computer readable medium provide non-volatile storage for the coordinating storage node 900. That is, the mass storage device 905 can include a computer readable medium (not shown) such as a hard disk or a CD-ROM drive.
  • the computer readable medium can include computer storage media and communication media.
  • Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
  • Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid state storage technologies, CD-ROM, DVD or other optical storage, tape cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices.
  • RAM random access memory
  • ROM read-only memory
  • EPROM Erasable programmable read-only memory
  • EEPROM electrically erasable programmable read-only memory
  • flash memory or other solid state storage technologies
  • CD-ROM compact discs
  • DVD or other optical storage CD-ROM, DVD or other optical storage
  • tape cartridges magnetic tape
  • magnetic disk storage magnetic disk storage devices
  • the coordinated storage node 900 may also be operated by a remote computer connected to the network through a network such as the Internet. That is, the coordination storage node 900 can be connected to the network 907 through a network interface unit 906 connected to the bus 903, or can also be connected to other types of networks or remote computer systems using the network interface unit 906 (not shown) ).
  • the network interface unit 906 can be a network card (also referred to as a network interface card) or a host bus adapter.
  • the computer device provided by the embodiment of the present invention does not need to send data to a dedicated coordinated storage node when data is transferred from a multiple copy storage mode to an EC storage mode, but is distributed by using a storage node that can serve as a data coordination node. , greatly reducing the amount of data transmission, saving network resource consumption within the system.
  • the coordinated storage node when the data is stored in the foregoing embodiment, the coordinated storage node only exemplifies the division of each functional module. In actual applications, the function distribution may be completed by different functional modules as needed. The internal structure of the coordination storage node is divided into different functional modules to perform all or part of the functions described above.
  • the embodiment of the data storage method in the foregoing embodiment is the same as the embodiment of the data storage method in the data storage system. For details, refer to the method embodiment, and details are not described herein.
  • a person skilled in the art may understand that all or part of the steps of implementing the above embodiments may be completed by hardware, or may be instructed by a program to execute related hardware, and the program may be stored in a computer readable storage medium.
  • the storage medium mentioned may be a read only memory, a magnetic disk or an optical disk or the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

L'invention concerne un procédé de mémorisation de données dans un système de mémorisation de données et un nœud de mémorisation coordinateur (103), se rapportant au domaine technique des informations. Selon le procédé : le nœud de mémorisation coordinateur (103) divise de manière égale des données ayant une longueur de M en X fragments de données, et obtient Y fragments de contrôle ; le nœud de mémorisation coordinateur (103) envoie une adresse initiale et la longueur de l'ième fragment de données à l'ième nœud de mémorisation (103) dans N nœuds de mémorisation (103) ; lorsque X est supérieur à N, le nœud de mémorisation coordinateur (103) attribue un premier nœud de mémorisation (103) à chacun des (X - N) fragments de données et aux Y fragments de contrôle ; lorsque X n'est pas supérieur à N, le nœud de mémorisation coordinateur (103) attribue un second nœud de mémorisation (103) à chacun des Y fragments de contrôle. Au moyen du procédé, lorsque des données passent d'un mode de mémorisation multicopie à un mode de mémorisation EC pour la mémorisation, il n'est pas nécessaire d'envoyer les données à un nœud de mémorisation coordinateur spécial. À la place, les données sont distribuées grâce à un nœud de mémorisation (103) pouvant servir de nœud de coordination de données, de sorte que la quantité de transmission de données soit nettement diminuée et que la consommation de ressources réseau dans le système soit réduite.
PCT/CN2017/081073 2016-09-05 2017-04-19 Procédé de mémorisation de données dans un système de mémorisation de données, et nœud de mémorisation coordinateur WO2018040583A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610803072.9 2016-09-05
CN201610803072.9A CN106383665B (zh) 2016-09-05 2016-09-05 数据存储系统中的数据存储方法及协调存储节点

Publications (1)

Publication Number Publication Date
WO2018040583A1 true WO2018040583A1 (fr) 2018-03-08

Family

ID=57938089

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/081073 WO2018040583A1 (fr) 2016-09-05 2017-04-19 Procédé de mémorisation de données dans un système de mémorisation de données, et nœud de mémorisation coordinateur

Country Status (2)

Country Link
CN (1) CN106383665B (fr)
WO (1) WO2018040583A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110149373A (zh) * 2019-04-28 2019-08-20 平安科技（深圳）有限公司 基于点对点网络的数据存储方法、装置、介质及服务器
CN111309801A (zh) * 2018-12-11 2020-06-19 航天信息股份有限公司 一种联盟链分片的方法及装置
CN113691581A (zh) * 2021-07-08 2021-11-23 杭州又拍云科技有限公司 一种高效的cdn刷新分片的方法

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106383665B (zh) * 2016-09-05 2018-05-11 华为技术有限公司 数据存储系统中的数据存储方法及协调存储节点
CN107436733B (zh) 2017-06-29 2020-11-06 华为技术有限公司 分片管理方法和分片管理装置
CN107491361B (zh) * 2017-07-21 2019-04-02 山东九州信泰信息科技股份有限公司 对数据表中列进行分级别冗余存储的方法
CN108780386B (zh) 2017-12-20 2020-09-04 华为技术有限公司 一种数据存储的方法、装置和系统
CN108647523B (zh) * 2018-04-28 2020-01-17 华南理工大学 一种基于区块链的电子证明系统及存证、文件恢复方法
CN109062512B (zh) * 2018-07-26 2022-02-18 郑州云海信息技术有限公司 一种分布式存储集群、数据读写方法、系统及相关装置
CN113986944B (zh) * 2021-12-29 2022-03-25 天地伟业技术有限公司 分片数据的写入方法、系统及电子设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102110154A (zh) * 2011-02-21 2011-06-29 中国科学院计算技术研究所 一种集群文件系统中文件冗余存储方法
US8458287B2 (en) * 2009-07-31 2013-06-04 Microsoft Corporation Erasure coded storage aggregation in data centers
CN103631539A (zh) * 2013-12-13 2014-03-12 百度在线网络技术（北京）有限公司 基于擦除编码机制的分布式存储系统及其存储方法
CN103984607A (zh) * 2013-02-08 2014-08-13 华为技术有限公司 分布式存储的方法、装置和系统
CN105573680A (zh) * 2015-12-25 2016-05-11 北京奇虎科技有限公司 副本数据的存储方法及装置
CN106383665A (zh) * 2016-09-05 2017-02-08 华为技术有限公司 数据存储系统中的数据存储方法及协调存储节点

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8458287B2 (en) * 2009-07-31 2013-06-04 Microsoft Corporation Erasure coded storage aggregation in data centers
CN102110154A (zh) * 2011-02-21 2011-06-29 中国科学院计算技术研究所 一种集群文件系统中文件冗余存储方法
CN103984607A (zh) * 2013-02-08 2014-08-13 华为技术有限公司 分布式存储的方法、装置和系统
CN103631539A (zh) * 2013-12-13 2014-03-12 百度在线网络技术（北京）有限公司 基于擦除编码机制的分布式存储系统及其存储方法
CN105573680A (zh) * 2015-12-25 2016-05-11 北京奇虎科技有限公司 副本数据的存储方法及装置
CN106383665A (zh) * 2016-09-05 2017-02-08 华为技术有限公司 数据存储系统中的数据存储方法及协调存储节点

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111309801A (zh) * 2018-12-11 2020-06-19 航天信息股份有限公司 一种联盟链分片的方法及装置
CN110149373A (zh) * 2019-04-28 2019-08-20 平安科技（深圳）有限公司 基于点对点网络的数据存储方法、装置、介质及服务器
CN113691581A (zh) * 2021-07-08 2021-11-23 杭州又拍云科技有限公司 一种高效的cdn刷新分片的方法

Also Published As

Publication number Publication date
CN106383665A (zh) 2017-02-08
CN106383665B (zh) 2018-05-11

Similar Documents

Publication Publication Date Title
WO2018040583A1 (fr) Procédé de mémorisation de données dans un système de mémorisation de données, et nœud de mémorisation coordinateur
US10664492B2 (en) Replication of data objects from a source server to a target server
US9606869B2 (en) Retrieving data segments from a dispersed storage network
US9749419B2 (en) Check operation dispersed storage network frame
US9501341B2 (en) Directory synchronization of a dispersed storage network
US9098441B2 (en) Storing data integrity information utilizing dispersed storage
US10180884B2 (en) Dispersed storage network with customized security and methods for use therewith
US11397668B2 (en) Data read/write method and apparatus, and storage server
US20120226667A1 (en) Determining a staleness state of a dispersed storage network local directory
US20160150012A1 (en) Content-based replication of data between storage units
US20140344646A1 (en) Distributed rebuilding of data in a dispersed storage network
WO2017084563A1 (fr) Procédé et dispositif de lecture de fichier reposant sur un système distribué
KR20120018178A (ko) 객체 저장부들의 네트워크상의 스웜-기반의 동기화
CN109144406B (zh) 分布式存储系统中元数据存储方法、系统及存储介质
US20150134672A1 (en) Data Copy Management Apparatus and Data Copy Method Thereof
US20140325260A1 (en) Slice migration in a dispersed storage network
WO2015039569A1 (fr) Dispositif de stockage de copies et procédé de stockage de copies
WO2019071595A1 (fr) Procédé et dispositif de stockage de données dans un système de stockage de blocs distribué et support de stockage lisible par ordinateur
US20240095131A1 (en) Secure Storage Network and Methods for use Therewith
US20130226867A1 (en) Apparatus and method for converting replication-based file into parity-based file in asymmetric clustering file system
US20180356987A1 (en) Information processing system, information processing apparatus, and information processing method
US20210311654A1 (en) Distributed Storage System and Computer Program Product

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17844891

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17844891

Country of ref document: EP

Kind code of ref document: A1