CN106383665B - Date storage method and coordination memory node in data-storage system - Google Patents
Date storage method and coordination memory node in data-storage system Download PDFInfo
- Publication number
- CN106383665B CN106383665B CN201610803072.9A CN201610803072A CN106383665B CN 106383665 B CN106383665 B CN 106383665B CN 201610803072 A CN201610803072 A CN 201610803072A CN 106383665 B CN106383665 B CN 106383665B
- Authority
- CN
- China
- Prior art keywords
- memory node
- data
- node
- memory
- length
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0626—Reducing size or complexity of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Abstract
The invention discloses the date storage method in a kind of data-storage system and coordinate memory node, belong to information technology field.Method includes：Coordinate memory node and the data that length is M are divided into X data fragmentation, and obtain Y verification burst；Coordinate i-th of memory node that the initial address of i-th data fragmentation and length are sent in N number of memory node by memory node；When X is more than N, it is respectively that each burst in the N number of data fragmentations of X and Y verification burst distributes first memory node to coordinate storage section；When X is not more than N, coordinates storage section and distribute second memory node for each burst in Y verification burst.The present invention is when switching to EC memory modules by more copy memory modules by data and being stored, special coordination memory node need not be sent data to, data distribution is carried out by the memory node of data harmonization node can be used as, volume of transmitted data is greatly reduced, saves network resource consumption in system.
Description
Technical field
The present invention relates to information technology field, date storage method and coordination in more particularly to a kind of data-storage system
Memory node.
Background technology
In order to improve the security of data, data are avoided to be lost in transmitting procedure, frequently with superfluous in distributed storage
Remaining mode stores data.Redundant fashion mainly has copy redundancy and EC (Erasure code, error correcting code) redundancy at present
Two kinds.Wherein, more copy redundancies are that initial data is copied into multiple copy datas, and each copy data is stored in a storage
On node.EC redundancies are that initial data is divided into X fragment data, and X fragment data is verified using EC algorithms, is obtained
To Y burst verification data, and then by X+Y burst storage to different memory nodes.
It is data storage method that more copy redundancies and EC redundancies are combined that current application is most commonly used.Specifically stored
Journey is as follows：Client replicates initial data, obtains multiple copy datas, and multiple copy datas are stored in different pairs
On this memory node, any copy data is sent to EC coordinator nodes, EC coordinator nodes the copy data is divided into X points
Sheet data, and X fragment data is verified using EC algorithms, obtain Y burst verification data, EC coordinator nodes storage one
A fragment data, and then X+Y-1 burst is sent to different EC memory nodes and is stored.
Assisted when data storage method is EC redundancies by copy redundant conversion, it is necessary to first send any copy data to EC
Point of adjustment, then multiple bursts including fragment data and burst verification data are sent to different EC by EC coordinator nodes
Memory node is stored, and process transmission mass data, causes network resource consumption in storage system larger.
The content of the invention
When data being switched to EC memory modules by more copy memory modules stored to reduce, network in storage system
Resource consumption, an embodiment of the present invention provides the date storage method in a kind of storage system and coordinates memory node.The skill
Art scheme is as follows：
First aspect, there is provided the date storage method in a kind of storage system, the storage system include multiple storages and save
Point, is stored with the data that length is M on N number of memory node in multiple memory nodes, which includes a coordination
Memory node, wherein, N is the natural number more than 1, and this method includes：
Within the storage system, because of business demand, data are stored by more copy memory module Transformed E C memory modules
When, coordinate memory node and length is divided into X data fragmentation for M data, each data fragmentation has a starting point
Location and length, the initial address of each data fragmentation is position of the data fragmentation in the data that length is M, and each data are divided
The length of piece is M/X.Based on obtained X data fragmentation, coordinate memory node by carrying out school to X data fragmentation
Test, obtain Y verification burst, each length for verifying burst is also M/X.
For i-th of data fragmentation in X data fragmentation, coordinate memory node by the starting point of i-th of data fragmentation
Location and length are sent to i-th of memory node in N number of memory node, wherein, i be more than 1 and less than or equal to D natural number, D
For the minimum value in X and N.
When X is more than N, coordinate memory node respectively by each burst in X-N data fragmentation and Y verification burst
First memory node is distributed, which is any memory node in addition to N in multiple memory nodes.
When X is less than or equal to N, coordinates memory node and deposited for each burst distribution one second in Y verification burst
Node is stored up, which is any memory node in addition to X memory node in multiple memory nodes.
Since there is dual-use function as the data memory node for coordinating memory node, complete data can be not only stored,
But also can to including data fragmentation and verification burst including burst be distributed, because without by complete data sending extremely
Special coordination memory node, substantially reduces the volume of transmitted data in storage system, saves the network money in storage system
Source consumes.
With reference to first aspect, in the first possible implementation of first aspect, the storage system where coordinator node
System further includes view management node, based on the view management node, coordinates memory node by the starting point of i-th of data fragmentation
When location and length are sent to i-th of memory node in N number of memory node, the storage section of view management node transmission can be first received
Point information, the memory node information include the information of the memory node for the data that memory length is M, which includes memory node
Mark etc..Coordinate memory node according to received memory node information, by the initial address and length of i-th of data fragmentation
Degree is targetedly sent to i-th of memory node in N number of memory node.
With reference to first aspect, in second of possible implementation of first aspect, coordinating memory node will also receive
Success message is stored, the storage success message is by storing in X data fragmentation and the Y memory node for verifying burst except coordination is deposited
Other memory nodes beyond node are stored up to send.Success message is stored by receiving, coordinates memory node and would know that each burst
Storage condition so that when any burst stores not successfully, the burst is sent to corresponding memory node deposited in time
Storage, to ensure that data are complete.
The possible implementation of with reference to first aspect the first, the third possible realization side in terms of first aspect
In formula, for enable coordination memory node by the initial address of i-th data fragmentation and length targetedly, in storage system
View management node can carry out Hash calculation to the data that length is M, obtain cryptographic Hash, and according to the cryptographic Hash and advance institute's structure
The hash space made, determines position of data of the length for M in hash space, and then using the position as starting point, will specify
Memory node and storage X data fragmentation and Y verification of the N number of memory node in direction as the data that memory length is M
The memory node of N number of burst in burst.
Second aspect, there is provided one kind coordinates memory node, which is located in storage system, the storage system
Uniting includes multiple memory node and view management nodes, and it is M to be stored with length on N number of memory node in the plurality of memory node
Data, which is contained in N number of memory node, wherein, N is the natural number more than 1, the coordination memory node
For performing the method coordinated in above-mentioned first aspect performed by memory node, which is used to perform above-mentioned first
Method in aspect performed by view management node.
The third aspect, there is provided one kind coordinates memory node, which is located in storage system, the storage system
Uniting includes multiple memory nodes, and the data that length is M, and the association are stored with N number of memory node in the plurality of memory node
Memory node is adjusted to be contained in N number of memory node, wherein, N is the natural data more than 1, which includes storage
Device, processor and bus, and memory and processor are direct-connected by bus；
The memory is used to store computer instruction, and the computer instruction which is used to perform memory storage is used for
Perform the date storage method in the storage system described in above-mentioned first aspect.
Specifically, the coordination memory node of above-mentioned second aspect and the third aspect, passes through specific construction unit or processing
Device, can perform following steps：
The data that the length is M are divided into X data fragmentation, and the X data fragmentation is verified, are obtained
Y verification burst, wherein, X and Y are natural number, and the length of each data fragmentation and each verification burst is M/X；
I-th of the storage section initial address of i-th data fragmentation and length being sent in N number of memory node
Point, wherein, i is the natural number more than 1 and less than or equal to D, and D is the minimum value in X and N；
It is respectively each burst distribution one in X-N data fragmentation and the Y verification burst when X is more than N
First memory node, first memory node are any in addition to N number of memory node in the multiple memory node
Memory node.
Specifically, the coordination memory node of above-mentioned second aspect and the third aspect, passes through specific construction unit or processing
Device, can perform following steps：
The data that the length is M are divided into X data fragmentation, and the X data fragmentation is verified, are obtained
Y verification burst, wherein, X and Y are natural number, and the length of each data fragmentation and each verification burst is M/X；
I-th of the storage section initial address of i-th data fragmentation and length being sent in N number of memory node
Point, wherein, i is the natural number more than 1 and less than or equal to D, and D is the minimum value in X and N；
When X is not more than N, second memory node is distributed for each burst in described Y verification burst, it is described
Second memory node is any memory node in addition to the X memory node in the multiple memory node.
The beneficial effect that technical solution provided in an embodiment of the present invention is brought is：
It is special without sending data to when data being switched to EC memory modules by more copy memory modules being stored
Coordination memory node, and by can be used as data harmonization node memory node carry out data distribution, greatly reduce data
Transmission quantity, saves network resource consumption in system.
Brief description of the drawings
To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment
Attached drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the present invention, for
For those of ordinary skill in the art, without creative efforts, other can also be obtained according to these attached drawings
Attached drawing.
Fig. 1 is that one embodiment of the invention provides a kind of schematic diagram of memory system architecture；
Fig. 2 is the date storage method flow chart in a kind of storage system that another embodiment of the present invention provides；
Fig. 3 is the schematic diagram for the uniformity hash algorithm that another embodiment of the present invention provides；
Fig. 4 is the distribution schematic diagram of memory node in a kind of storage system that another embodiment of the present invention provides；
Fig. 5 is the schematic diagram for the data storage procedure that another embodiment of the present invention provides；
Fig. 6 is the schematic diagram for the data storage procedure that another embodiment of the present invention provides；
Fig. 7 is the structure diagram for the coordination memory node that another embodiment of the present invention provides；
Fig. 8 is the structure diagram for the coordination memory node that another embodiment of the present invention provides；
Fig. 9 is the illustrative Computer Architecture of the coordination memory node used in another embodiment of the present invention.
Embodiment
To make the object, technical solutions and advantages of the present invention clearer, below in conjunction with attached drawing to embodiment party of the present invention
Formula is described in further detail.
Referring to Fig. 1, it illustrates a kind of storage system, which includes view management node 101, client 102
And memory node 103.
Wherein, view management node 101 can be a single computing device, or by more computing device groups
Into computing cluster.The view management node is used to receive the heartbeat message that memory node 103 reports, and according to memory node
103 heartbeat messages reported, update storage the node state of node 103.View management node 101 is additionally operable to as client 102
In data distribution memory node.View management node 101 also safeguards a correspondence between cryptographic Hash and memory node
View, the view inquire about data for client 102 from memory node.
Client 102 can be smart mobile phone, tablet computer, laptop etc., and the present embodiment is not to client 102
Product type make specific limit.Client 102 is used for the view for receiving the push of view management node 101, and is regarded according to this
Figure inquiry data.
Memory node 103 is the physical medium that data are stored in storage system, which can be to every preset duration
View management node reports heartbeat message, so that the state of this memory node of view management node maintenance.
Above-mentioned view management node 101 and client 102 and back end 103 can by cable network or wireless network into
Row communication, client 102 can be communicated with back end 103 by cable network or wireless network.
Based on the Database Systems shown in Fig. 1, an embodiment of the present invention provides a kind of date storage method, the storage system
Including multiple memory nodes and view management node, wherein, being stored with length on N number of memory node in multiple memory nodes is
The data of M, and N number of memory node includes a coordination memory node, referring to Fig. 2, method flow provided in an embodiment of the present invention
Including：
201st, the data that view management node is length M distribute memory node information.
Since the memory node quantity that storage system includes is more, in order to targetedly be used to the data that length is M
Redundant fashion is stored, while reduces in data transmission procedure network resource consumption in system to greatest extent, in this implementation
In example, view management node also distributes the data for using memory node selection algorithm to be M for length to memory node.
Wherein, memory node selection algorithm can include uniformity hash algorithm etc..Uniformity hash algorithm principle is as follows：
Construction one 0~232- 1 hash space, by 0~232- 1 hash space forms a Hash ring, Hash ring is divided into multiple
Equal section, the difference range of the cryptographic Hash in each section are identical.Fig. 3 shows the Hash of a hash space composition
Ring, referring to Fig. 3, which is divided into 20 sections, is respectively the difference range of P0~P19 and the cryptographic Hash of P0~P19
It is identical.
The performance of storage system itself is limited to, storage system stores the data that length is M using redundant fashion
When, the quantity of the copy data data of M (length that duplication obtains be), the quantity of data fragmentation and data fragmentation is verified
To the quantity of verification burst be to determine.Stored for the storage system described in the present embodiment using redundant fashion
When, the quantity of copy data is N number of, and the quantity of data fragmentation is X, and the quantity for verifying burst is Y, wherein, N is more than 1
Natural number, X and Y are natural number.
Specifically, view management node is using memory node selection algorithm, is the data distribution memory node that length is M
When, it may include following steps 2011~2013：
2011st, view management node carries out Hash calculation to the data that length is M, obtains cryptographic Hash.
View management node, which can use, specifies hash algorithm to carry out Hash calculation to length for the data of M, obtains cryptographic Hash.
Wherein, hash algorithm is specified to include addition hash algorithm, multiplication hash algorithm, division hash algorithm, bit arithmetic hash algorithm, look into
Table hash algorithm etc..
2012nd, view management node determines position of data of the length for M in hash space according to cryptographic Hash.
In the present embodiment, each section in hash space corresponds to a cryptographic Hash scope, when cryptographic Hash is positioned at Kazakhstan
, can be using position of the position where the section as the data that length is M in hash space in any section in uncommon space.Example
Such as, length is that the data KEY of M passes through Hash calculation, and obtained cryptographic Hash is k1, which is located in the section in Fig. 3 where PO,
Therefore, can be using position of the position where PO as the data KEY that length is M in hash space.
2013rd, view management node is M using N number of memory node of assigned direction as memory length using position as starting point
Data memory node.
Wherein, assigned direction can be the clockwise direction using cryptographic Hash position as starting point, can also be with Hash
It is worth the counter clockwise direction that position is starting point.By using N number of copy to carry out redundancy to length for the data of M in this present embodiment
Storage, therefore, view management node needs to distribute N number of memory node under copy memory module for the data that length is M.It is based on
The data that length is M are carried out with the obtained cryptographic Hash of Hash calculation, view management node using the position where the cryptographic Hash for
Point, the memory node using N number of memory node on assigned direction as the data that memory length is M, the choosing of N number of memory node
It can be continuous selection to take mode, i.e., will continuously store section by the N number of of starting point of the cryptographic Hash position on assigned direction
Memory node of the point as the data that memory length is M, can also be discontinuous selection, i.e., by assigned direction with the cryptographic Hash institute
Memory node of the N number of discontinuous memory node in place for being set to starting point as the data that memory length is M.
2014th, view management node is using position as starting point, using N number of memory node of assigned direction as X data of storage
The memory node of burst and N number of burst in Y verification burst.
Based on the cryptographic Hash obtained to the data progress Hash calculation that length is M, view management node is with the cryptographic Hash institute
Position be starting point, using N number of memory node of assigned direction as storage X data fragmentation and Y verify burst in it is N number of
The memory node of burst, the selection mode and storage of X data fragmentation of the storage and N number of memory node in Y verification burst
Length is that the selection mode of N number of memory node of the data of M is identical, if N number of memory node that memory length is the data of M is adopted
Chosen with continuation mode, then regard N number of continuous memory node using the cryptographic Hash position as starting point as storage X
The memory node of data fragmentation and N number of burst in Y verification burst, if memory length is N number of memory node of the data of M
Chosen using discontinuous mode, be then N number of discontinuous memory node of starting point as depositing using the cryptographic Hash position
Store up the memory node of X data fragmentation and N number of burst in Y verification burst.Chosen anyway using which kind of mode,
Ensure the N number of memory node for the data that memory length is M and N number of burst in X data fragmentation of storage and Y verification burst
Memory node overlaps to greatest extent.
Exemplified by a manner of the redundant storage of bursts is verified using 3 copy datas, 4 data fragmentations, 2, view management node is
Length is that the data of M distribute the process of memory node, reference can be made to following examples.
(1), Hash meter is carried out to the data KEY that the length is M for the data KEY that length is M, view management node
Calculate, obtain cryptographic Hash k1.
(2), view management node is according to cryptographic Hash k1, the position from hash space where definite cryptographic Hash k1,
And by the position be starting point, the memory node using clockwise 3 memory nodes as the data that memory length is M, ginseng
See Fig. 3, PO, P1, P2 are the memory node for the data that memory length is M.
(3), view management node is according to cryptographic Hash k1, the position from hash space where definite cryptographic Hash k1,
And by the position be starting point, using clockwise 3 memory nodes as in 4 data fragmentations of storage and 2 verification bursts
3 bursts memory node, referring to Fig. 3, P0, P1, P2 are 3 points stored in 4 data fragmentations and 2 verification bursts
The memory node of piece.
The memory node that above-mentioned view management node is distributed by the data that length is M is actually each in hash space
Dummy storage node where section, the dummy storage node veritably can not store data, it is necessary to by dummy storage node with
Physical store node in storage system establishes mapping relations, and the data persistence in corresponding dummy storage node is arrived and physics
In memory node.For the ease of being managed to dummy storage node and physical store node, view management node also safeguards one
The mapping table, the mapping relations between each section of mapping table memory storage hash space and physical store node, leads to
The quantity of normal physical store node is less than the section quantity in hash space, and so each physical store node actually can be right
Multiple sections are answered, that is to say that the data of multiple dummy storage nodes are stored on same physical store node.Referring to Fig. 3, Hash
Space is divided into P0~P19 totally 20 sections, and storage system includes 4 physical store nodes, is respectively physical store node
1st, physical store node 2, physical store node 3 and physical store node 4, wherein, A (P0), E (P4), I (P8), M (P12), Q
(P16) it is mapped on physical node 1, B (P1), F (P5), J (P9), N (P13), R (P17) are mapped on physical node 2, C
(P3), G (P6), K (P10), 0 (P14), S (P18) are mapped on physical node 3, D (P4), H (P7), L (P11), P (P15),
T (P19) is mapped on physical node 4.
Referring to Fig. 4, the mapping table safeguarded based on view management node, when client needs to inquire about number from storage system
According to KEY1, client cocoa first calculates the cryptographic Hash k1 of the KEY1, and determines positions of the cryptographic Hash k1 in hash space, and then
According to the mapping table, the physical store node where the cryptographic Hash is searched, so as to inquire data from the physical store node
KEY1。
202nd, view management node sends memory node information to coordination memory node and client.
Due to N number of in the memory node and X data fragmentation of storage and Y verification burst of the data that memory length is M
The memory node of burst, is that client carries out more copy storages, coordinates the key that memory node carries out EC storages, to make client
It can be stored targeted specifically with memory node is coordinated, view management node can distribute the data for being M by length N number of
The memory node of the memory node information and X data fragmentation of storage of memory node and N number of burst in Y verification burst
Memory node information send to client and coordinate memory node.Wherein, memory node information includes memory node mark etc.
Deng.
203rd, client replicates the data that length is M, and the data after duplication are sent out according to memory node information
Send to N number of memory node and stored.
When the memory node information for receiving the transmission of view management node, data duplication N part of the client to length for M,
Obtain N number of length be M data, and then by N number of length by M data sending to the N number of memory node distributed, deposited by each
Storage node is stored.
In order to improve data access rate, storage system can according to the memory module of the visit capacity change data of client,
When client is more than certain numerical value to the visit capacity of any data, storage system can trigger coordinate memory node by data by
Copy memory module is converted to EC memory modules and is stored.
204th, coordinate memory node and the data that length is M are divided into X data fragmentation, and X data fragmentation is carried out
Verification, obtains Y verification burst.
Deposited by storage system in this present embodiment using the redundant fashion of X data fragmentation, Y verification burst
Storage, therefore, under EC memory modules, coordinating memory node needs the data that the length stored on this node is M being divided into X
A data fragmentation.Based on the X data fragmentation divided, coordinate memory node using checking algorithm is specified, to X data fragmentation
Verified, obtain Y verification burst.Wherein, checking algorithm is specified to include parity arithmetic, the calculation of CRC cyclic redundancy check
Method etc..
205th, coordinate memory node the initial address of i-th data fragmentation and length are sent in N number of memory node
I-th of memory node.
Wherein, i is the natural number more than 1 and less than or equal to D, and D is the minimum value in X and N.Due in N number of memory node
The data that length is M have been stored, therefore, consumed Internet resources has been transmitted in system in order to save data, coordinate storage
Node can send burst information to N number of memory node, according to burst information be M to the length stored by each memory node
Data sheared, obtain a data fragmentation and store.Wherein, burst information includes the initial address and length of data fragmentation
Degree etc..For i-th of data fragmentation in X data fragmentation, coordinate memory node by the initial address of i-th of data fragmentation and
Length is sent to i-th of memory node in N number of memory node, the starting by i-th of memory node according to i-th of data fragmentation
Address and length, the data for being M to the length stored are sheared, and obtain data corresponding with initial address and length point
Piece, and store the data fragmentation.
206th, when X is more than N, it is respectively each point in X-N data fragmentation and Y verification burst to coordinate storage section
Piece distributes first memory node.
Wherein, the first memory node is any memory node in addition to N number of memory node in multiple memory nodes.Work as X
During more than N, illustrate that the quantity of data fragmentation is more than the copy amount for the data that length is M, at this time, coordinating memory node needs is
Each burst in X-N data fragmentation and Y verification burst distributes first memory node, and X-N data are divided
Piece and Y verification burst are sent to the first memory node distributed, are stored by the first memory node.
207th, when X is not more than N, coordination storage section is deposited for each burst distribution one second in Y verification burst
Store up node.
Wherein, the second memory node is any memory node in addition to X memory node in multiple memory nodes.Work as X
During no more than N, illustrate that the quantity of data fragmentation is less than or equal to the copy amount for the data that length is M, coordinating storage section at this time needs
To be each burst one the second memory node of distribution in Y verification burst, and Y verification burst is sent to second and is deposited
Node is stored up, is stored by the second memory node.
For the above process, in order to make it easy to understand, will be illustrated below with a specific example.
Referring to Fig. 5, size of data is set as 4M, and view management node is the storage section for the data distribution that memory length is 4M
Point is N1, N2, N3, and the memory node for 4 data fragmentations and 2 verification burst distribution is N1, N2, N3, N4, N5, N6, is coordinated
The data that length is 4M are divided into the data fragmentation of 4 1M by memory node N1, and 4 data fragmentations are verified, and obtain 2
A verification burst.When needing that the data that length is M are switched to EC memory modules by more copy memory modules and stored, coordinate
The burst information that memory node N1 is 0 according to initial address and length is 1M, the length stored to this node are the data of 4M
Sheared, obtain data fragmentation and store, and sent the burst that initial address is 1M and length is 1M to memory node N2 and believe
Breath, sends the burst information that initial address is 2M and length is 1M, by memory node N2 and N3 to this node to memory node N3
The length stored is that the data of 4M are sheared, and obtains data fragmentation and stores.For memory node N4, N5, N6, coordinate to deposit
The data fragmentation that initial address can be 3M by storage node N1 and length is 1M is stored to memory node N4, is by initial address
0 and verification burst that length is 1M send to memory node N5 and stored, by the verification that initial address is 1M and length is 1M
Burst sends to memory node N6 and is stored.
To enable the client to know the storage request of data under EC memory modules, specify memory node every in successfully storage
When a data fragmentation or verification burst, also storage success message will be sent to memory node is coordinated.Wherein, specified memory node is
Store other memory nodes in addition to memory node is coordinated in the memory node of X data fragmentation and Y verification burst.
Fig. 6 carries out the procedure chart of data storage when showing data switching to EC memory modules by more copy memory modules,
Referring to Fig. 6, view management node is data distribution memory node N1, N2, N3 that length is M, is 4 data fragmentations and 2 schools
The memory node for testing burst distribution is N1, N2, N3, N4, N5, N6.When switching to EC memory modules by more copy memory modules, association
Adjust memory node N1 that the data that the length stored is M are divided into 4 data fragmentations, and 4 data fragmentations are verified, obtain
To 2 verification bursts.Coordinate memory node N1N1 according to burst information<Offset1, len1>, the length that is stored to this node
Sheared for the data of M, obtain data fragmentation and store；And send burst information to memory node N2<Offset2, len2
>, by memory node N2 according to<Offset2, len2>The data for being M to the length stored are sheared, and obtain a data
Burst simultaneously stores；Burst information is sent to N3<Offset3, len3>, by memory node N3 according to<Offset3, len3>To institute
The length of storage is that the data of M are sheared, and obtains a data fragmentation and stores；It is by burst information<Offset4, len4>
Data fragmentation send to memory node N4 and stored, two verification bursts are respectively sent to memory node N5 and N6 carries out
Storage.
Method provided in an embodiment of the present invention, is stored data are switched to EC memory modules by more copy memory modules
When, without sending data to special coordination memory node, and by the memory node of data harmonization node can be used as to carry out
Data distribution, greatly reduces volume of transmitted data, saves network resource consumption in system.
Referring to Fig. 7, an embodiment of the present invention provides one kind to coordinate memory node, which is located at storage system
In, which includes multiple memory nodes, and the number that length is M is stored with N number of memory node in multiple memory nodes
According to, coordinate memory node and be contained in N number of memory node, wherein, N is the natural number more than 1, and coordinating memory node includes：
Division unit 701, for the data that length is M to be divided into X data fragmentation, and carries out X data fragmentation
Verification, obtains Y verification burst, wherein, X and Y are natural number, and each data fragmentation verifies the length of burst with each
It is M/X；
Transmitting element 702, for the initial address of i-th data fragmentation and length to be sent respectively to N number of memory node
In i-th of memory node, wherein, i is more than 1 and the small natural number equal to D, and D is the minimum value in X and N；
Allocation unit 703, for when X is more than N, being respectively each in X-N data fragmentation and Y verification burst
Burst distributes first memory node, and the first memory node is any in addition to N number of memory node in multiple memory nodes
Memory node.
In another embodiment of the present invention, allocation unit 703, for when X is not more than N, being verified for Y in burst
Each burst distribute second memory node, the second memory node be in multiple memory nodes except X memory node with
Outer any memory node.
Optionally, in an alternative embodiment of the invention, allocation unit 703 is used for when X is more than N, is respectively X-N number
First memory node is distributed according to each burst in burst and Y verification burst, the first memory node is multiple storages
Any memory node in node in addition to N number of memory node；When X is not more than N, for each point in Y verification burst
Piece distributes second memory node, and the second memory node is that any in addition to X memory node deposits in multiple memory nodes
Store up node.
Referring to Fig. 8, in another embodiment of the present invention, storage system further includes view management node, coordinates storage
Node further includes receiving unit 704；
Receiving unit 704, for receiving the memory node information of view management node transmission, memory node information includes depositing
Store up the information of the memory node for the data that length is M；
Transmitting element 702, specifically for according to memory node information by the initial address of i-th data fragmentation and length point
I-th of memory node not being sent in N number of memory node.
In another embodiment of the present invention, receiving unit 704, are additionally operable to receive storage success message, store successfully
Other storages in memory node of the message by storing X data fragmentation and Y verification burst in addition to memory node is coordinated save
Point is sent.
In another embodiment of the present invention, view management node, based on carrying out Hash to the data that length is M
Calculate, obtain cryptographic Hash；According to cryptographic Hash, position of data of the length for M in hash space is determined；Using the position as starting point,
Memory node using N number of memory node of assigned direction as the data that memory length is M；Using the position as starting point, will specify
Memory node of the N number of memory node in direction as N number of burst in X data fragmentation of storage and Y verification burst.
In conclusion data are being switched to EC storage moulds by system provided in an embodiment of the present invention by more copy memory modules
When formula is stored, without sending data to special coordination memory node, and by depositing for data harmonization node can be used as
Store up node and carry out data distribution, greatly reduce volume of transmitted data, save network resource consumption in system.
Referring to Fig. 9, it illustrates the illustrative calculating of the coordination memory node 900 used in one embodiment of the present of invention
Machine architecture.The coordination memory node 900 is server or storage device.The coordination memory node 900 includes processing
Device 901, memory 902 and bus 903, wherein, processor 901 and memory 902 are connected by bus 903.Optionally, it is described
Coordinate the basic input/output that memory node 900 further includes transmission information between each device helped in computing device
(I/O systems) 904 and the mass-memory unit 905 for storage program area, application program and other program modules.
Optionally, the mass-memory unit 905 (is not shown by being connected to the bulk memory controller of bus 903
Go out) it is connected to processor 901.The mass-memory unit 905 and its associated computer-readable medium store for coordination
Node 900 provides non-volatile memories.That is, the mass-memory unit 905 can include such as hard disk or CD-
The computer-readable medium (not shown) of ROM drive etc.
Without loss of generality, the computer-readable medium can include computer storage media and communication media.Computer
Storage medium is included for information such as storage computer-readable instruction, data structure, program module or other data
The volatile and non-volatile of any method or technique realization, removable and irremovable medium.Computer-readable storage medium includes
RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storages its technologies, CD-ROM, DVD or other optical storages, tape
Box, tape, disk storage or other magnetic storage apparatus.Certainly, skilled person will appreciate that the computer-readable storage medium
It is not limited to above-mentioned several.
According to various embodiments of the present invention, the coordination memory node 900 can also pass through the networks such as internet
The remote computer operation being connected on network.Namely coordination memory node 900 can be by being connected in the bus 903
Network Interface Unit 906 is connected to network 907, in other words, can also be connected to other types using Network Interface Unit 906
Network or remote computer system (not shown).Wherein, Network Interface Unit 906 can be that (also referred to as network connects network interface card
Mouth card) or host bus adaptor etc..
Computer equipment provided in an embodiment of the present invention, by data by more copy memory modules switch to EC memory modules into
During row storage, without sending data to special coordination memory node, and by the storage section that can be used as data harmonization node
Point carries out data distribution, greatly reduces volume of transmitted data, saves network resource consumption in system.
It should be noted that：The coordination memory node that above-described embodiment provides is when storing data, only with above-mentioned each function
The division progress of module, can be as needed and by above-mentioned function distribution by different function moulds for example, in practical application
Block is completed, and the internal structure that will coordinate memory node is divided into different function modules, with complete it is described above whole or
Person's partial function.In addition, the date storage method coordinated in memory node and data-storage system that above-described embodiment provides is real
Apply example and belong to same design, its specific implementation process refers to embodiment of the method, and which is not described herein again.
One of ordinary skill in the art will appreciate that hardware can be passed through by realizing all or part of step of above-described embodiment
To complete, relevant hardware can also be instructed to complete by program, the program can be stored in a kind of computer-readable
In storage medium, storage medium mentioned above can be read-only storage, disk or CD etc..
The foregoing is merely presently preferred embodiments of the present invention, is not intended to limit the invention, it is all the present invention spirit and
Within principle, any modification, equivalent replacement, improvement and so on, should all be included in the protection scope of the present invention.
Claims (10)
1. the date storage method in a kind of storage system, it is characterised in that the storage system includes multiple memory nodes, institute
The data for being stored with that length is M on N number of memory node in multiple memory nodes are stated, N number of memory node includes an association
Memory node is adjusted, wherein, N is the natural number more than 1, the described method includes：
The data that the length is M are divided into X data fragmentation by the memory node of coordinating, and to the X data fragmentation
Verified, obtain Y verification burst, wherein, X and Y are natural number, and each data fragmentation verifies burst with each
Length is M/X；
The initial address of i-th data fragmentation and length are sent in N number of memory node by the memory node of coordinating
I-th of memory node, wherein, i is the natural number more than 1 and less than or equal to D, and D is the minimum value in X and N；
When X is more than N, the storage section of coordinating is respectively each point in X-N data fragmentation and the Y verification burst
Piece distributes first memory node, and first memory node is to remove N number of memory node in the multiple memory node
Any memory node in addition；
When X is not more than N, the coordination storage section is deposited for each burst distribution one second in described Y verification burst
Node is stored up, second memory node is any storage section in addition to the X memory node in the multiple memory node
Point.
It is 2. described according to the method described in claim 1, it is characterized in that, the storage system further includes view management node
Coordinate i-th of storage that the initial address of i-th data fragmentation and length are sent in N number of memory node by memory node
Node, including：
The memory node of coordinating receives the memory node information that the view management node is sent, the memory node packet
Include the information for the memory node for storing the data that the length is M；
The initial address of i-th data fragmentation and length are sent to by the memory node of coordinating according to the memory node information
I-th of memory node in N number of memory node.
3. according to the method described in claim 1, it is characterized in that, the method further includes：
The coordination memory node receives storage success message, and the storage success message is by storing X data fragmentation and Y school
Other memory nodes in the memory node of burst in addition to the coordination memory node are tested to send.
4. according to the method described in claim 2, it is characterized in that, the method further includes：
The view management node carries out Hash calculation to the data that the length is M, obtains cryptographic Hash；
The view management node determines position of data of the length for M in hash space according to the cryptographic Hash；
The view management node is using the position as starting point, using N number of memory node of assigned direction as the storage length
For the memory node of the data of M；
The view management node is using the position as starting point, using N number of memory node of assigned direction as the storage X number
According to the memory node of N number of burst in burst and the Y verification burst.
5. one kind coordinates memory node, it is characterised in that the coordination memory node is located in storage system, the storage system
Including multiple memory nodes, the data that length is M, the association are stored with N number of memory node in the multiple memory node
Memory node is adjusted to be contained in N number of memory node, wherein, N is the natural number more than 1, and the coordination memory node includes：
Division unit, for the data that the length is M to be divided into X data fragmentation, and carries out the X data fragmentation
Verification, obtains Y verification burst, wherein, X and Y are natural number, and each data fragmentation verifies the length of burst with each
It is M/X；
Transmitting element, for the initial address of i-th data fragmentation and length to be sent respectively in N number of memory node
I-th of memory node, wherein, i is more than 1 and the small natural number equal to D, and D is the minimum value in X and N；
Allocation unit, for when X is more than N, being respectively each point in X-N data fragmentation and the Y verification burst
Piece distributes first memory node, and first memory node is to remove N number of memory node in the multiple memory node
Any memory node in addition.
6. coordination memory node according to claim 5, it is characterised in that the allocation unit, is additionally operable to when X is not more than
During N, second memory node is distributed for each burst in described Y verification burst, second memory node is institute
State any memory node in addition to the X memory node in multiple memory nodes.
7. coordination memory node according to claim 5, it is characterised in that the storage system further includes view management section
Point, the coordination memory node further include receiving unit；
The receiving unit, the memory node information sent for receiving the view management node, the memory node information
Information including the memory node for storing the data that the length is M；
The transmitting element, specifically for according to the memory node information by the initial address and length of i-th of data fragmentation
I-th of memory node being sent respectively in N number of memory node.
8. coordination memory node according to claim 5, it is characterised in that further include receiving unit, stored for receiving
Success message, the storage success message remove the coordination in the memory node by storing X data fragmentation and Y verification burst
Other memory nodes beyond memory node are sent.
9. coordination memory node according to claim 7, it is characterised in that the view management node, for described
The data that length is M carry out Hash calculation, obtain cryptographic Hash；According to the cryptographic Hash, determine that the data that the length is M are being breathed out
Position in uncommon space；It is M's using N number of memory node of assigned direction as the length is stored using the position as starting point
The memory node of data；Using the position as starting point, divide using N number of memory node of assigned direction as the X data are stored
The memory node of piece and N number of burst in the Y verification burst.
10. one kind coordinates memory node, it is characterised in that the coordination memory node is located in storage system, the storage system
Uniting includes multiple memory nodes, and the data that length is M are stored with N number of memory node in the multiple memory node, described
Coordinate memory node to be contained in N number of memory node, wherein, N is the natural number more than 1, the coordination memory node bag
Memory, processor and bus are included, and the memory and the processor are direct-connected by the bus；
The memory is used to store computer instruction, and the processor is used for the computer instruction for performing the memory storage
The date storage method in 1 to 4 any one of them storage system is required for perform claim.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610803072.9A CN106383665B (en) | 2016-09-05 | 2016-09-05 | Date storage method and coordination memory node in data-storage system |
| PCT/CN2017/081073 WO2018040583A1 (en) | 2016-09-05 | 2017-04-19 | Data storage method in data storage system and coordinating storage node |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610803072.9A CN106383665B (en) | 2016-09-05 | 2016-09-05 | Date storage method and coordination memory node in data-storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106383665A CN106383665A (en) | 2017-02-08 |
| CN106383665B true CN106383665B (en) | 2018-05-11 |
Family
ID=57938089
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610803072.9A Active CN106383665B (en) | 2016-09-05 | 2016-09-05 | Date storage method and coordination memory node in data-storage system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN106383665B (en) |
| WO (1) | WO2018040583A1 (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106383665B (en) * | 2016-09-05 | 2018-05-11 | 华为技术有限公司 | Date storage method and coordination memory node in data-storage system |
| CN112328168A (en) * | 2017-06-29 | 2021-02-05 | 华为技术有限公司 | Fragment management method and fragment management device |
| CN107491361B (en) * | 2017-07-21 | 2019-04-02 | 山东九州信泰信息科技股份有限公司 | The method for being classified other redundant storage is carried out to column in tables of data |
| CN108780386B (en) * | 2017-12-20 | 2020-09-04 | 华为技术有限公司 | Data storage method, device and system |
| CN108647523B (en) * | 2018-04-28 | 2020-01-17 | 华南理工大学 | Block chain-based electronic certification system and certificate storage and file recovery method |
| CN109062512B (en) * | 2018-07-26 | 2022-02-18 | 郑州云海信息技术有限公司 | Distributed storage cluster, data read-write method, system and related device |
| CN111309801A (en) * | 2018-12-11 | 2020-06-19 | 航天信息股份有限公司 | Method and device for fragmenting alliance chain |
| CN110149373B (en) * | 2019-04-28 | 2022-01-07 | 平安科技（深圳）有限公司 | Data storage method, device, medium and server based on peer-to-peer network |
| CN113691581B (en) * | 2021-07-08 | 2023-05-09 | 杭州又拍云科技有限公司 | Efficient CDN refreshing and slicing method |
| CN113986944B (en) * | 2021-12-29 | 2022-03-25 | 天地伟业技术有限公司 | Writing method and system of fragment data and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102110154A (en) * | 2011-02-21 | 2011-06-29 | 中国科学院计算技术研究所 | File redundancy storage method in cluster file system |
| US8458287B2 (en) * | 2009-07-31 | 2013-06-04 | Microsoft Corporation | Erasure coded storage aggregation in data centers |
| CN103631539A (en) * | 2013-12-13 | 2014-03-12 | 百度在线网络技术（北京）有限公司 | Distributed storage system and distributed storage method based on erasure coding mechanism |
| CN103984607A (en) * | 2013-02-08 | 2014-08-13 | 华为技术有限公司 | Distributed storage method, device and system |
| CN105573680A (en) * | 2015-12-25 | 2016-05-11 | 北京奇虎科技有限公司 | Storage method and device for replicated data |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106383665B (en) * | 2016-09-05 | 2018-05-11 | 华为技术有限公司 | Date storage method and coordination memory node in data-storage system |
-
2016
- 2016-09-05 CN CN201610803072.9A patent/CN106383665B/en active Active
-
2017
- 2017-04-19 WO PCT/CN2017/081073 patent/WO2018040583A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8458287B2 (en) * | 2009-07-31 | 2013-06-04 | Microsoft Corporation | Erasure coded storage aggregation in data centers |
| CN102110154A (en) * | 2011-02-21 | 2011-06-29 | 中国科学院计算技术研究所 | File redundancy storage method in cluster file system |
| CN103984607A (en) * | 2013-02-08 | 2014-08-13 | 华为技术有限公司 | Distributed storage method, device and system |
| CN103631539A (en) * | 2013-12-13 | 2014-03-12 | 百度在线网络技术（北京）有限公司 | Distributed storage system and distributed storage method based on erasure coding mechanism |
| CN105573680A (en) * | 2015-12-25 | 2016-05-11 | 北京奇虎科技有限公司 | Storage method and device for replicated data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106383665A (en) | 2017-02-08 |
| WO2018040583A1 (en) | 2018-03-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106383665B (en) | Date storage method and coordination memory node in data-storage system | |
| CN110059494B (en) | Privacy protection method for block chain transaction data and block chain system | |
| CN109471744B (en) | Main chain and parallel multi-sub-chain system architecture based on block chain | |
| CN111681003B (en) | Resource cross-chain transfer method and device, computer equipment and storage medium | |
| US11108630B2 (en) | Hierarchical configuration of networked communication devices | |
| US20210083856A1 (en) | Improved hardware security module management | |
| EP3785204A1 (en) | Maintaining blocks of a blockchain in a partitioned blockchain network | |
| CN107657438A (en) | A kind of block chain generation method, data verification method, node and system | |
| CN107807951A (en) | A kind of block chain generation method, data verification method, node and system | |
| CN110149373A (en) | Date storage method, device, medium and server based on point to point network | |
| CN110442652A (en) | A kind of across chain data processing method and device based on block chain | |
| EP3779760B1 (en) | Blockchain-based data processing method and apparatus, and electronic device | |
| US20230316273A1 (en) | Data processing method and apparatus, computer device, and storage medium | |
| CN109510876A (en) | A kind of alliance's chain sharding method and corresponding alliance's chain based on PBFT | |
| CN109445839A (en) | A kind of module management packet dissemination method, device and terminal device | |
| CN108156232A (en) | Data sharing method and device | |
| CN108833610B (en) | Information updating method, device and system | |
| CN104142871B (en) | Data backup method and device and distributed file system | |
| CN109583899A (en) | Data storage and management method and system based on cloud storage service device | |
| CN109462508A (en) | Node deployment method, device and storage medium | |
| CN110599175A (en) | Block processing method and related equipment | |
| CN109739684A (en) | The copy restorative procedure and device of distributed key value database based on vector clock | |
| CN109348434A (en) | A kind of sending method of scene information, sending device and terminal device | |
| CN108494828A (en) | A kind of update method of node data, medium, device and computing device | |
| CN105162869A (en) | Data backup management method and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |