CN106383665A - Data storage method in data storage system and coordinate storage node - Google Patents
Data storage method in data storage system and coordinate storage node Download PDFInfo
- Publication number
- CN106383665A CN106383665A CN201610803072.9A CN201610803072A CN106383665A CN 106383665 A CN106383665 A CN 106383665A CN 201610803072 A CN201610803072 A CN 201610803072A CN 106383665 A CN106383665 A CN 106383665A
- Authority
- CN
- China
- Prior art keywords
- memory node
- data
- node
- memory
- length
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0626—Reducing size or complexity of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data storage method in a data storage system and a coordinate storage node, and belongs to the technical field of information. The method comprises the following steps of: equally dividing a data with a length of M into X data fragments by the coordinate storage node, and obtaining Y verification fragments; sending the initial address and length of the ith data fragment to the ith storage node in N storage nodes by the coordinate storage node; when X is greater than N, distributing a first storage node for each fragment in X-N data fragments and the Y verification fragments by the coordinate storage node; and when X is not greater than N, distributing a second storage node for each fragment in the Y verification fragments by the coordinate storage node. According to the data storage method, the data does not need to be sent to a special coordinate storage node when the data is converted from a multi-copy storage mode to an EC storage mode to store, and data distribution is carried out by means of the storage nodes which can be used as data coordination nodes, so that the data transmission quantity is greatly decreased and the network resource consumption in the system is saved.
Description
Technical field
The present invention relates to areas of information technology, particularly to the date storage method in a kind of data-storage system and coordination
Memory node.
Background technology
In order to improve the security of data, it is to avoid data is lost in transmitting procedure, frequently with superfluous in distributed storage
Remaining mode stores to data.Redundant fashion mainly has copy redundancy and EC (Erasure code, error correcting code) redundancy at present
Two kinds.Wherein, many copies redundancy is that initial data is copied into multiple copy datas, and each copy data is stored in a storage
On node.EC redundancy is that initial data is divided into X fragment data, using EC algorithm, X fragment data is verified, obtains
To Y burst verification data, and then X+Y burst is stored on different memory nodes.
It to be the data storage method that many copies redundancy and EC redundancy combine that current application is most commonly used.Specifically stored
Journey is as follows：Client replicates to initial data, obtains multiple copy datas, and multiple copy datas are stored in different pairs
On this memory node, arbitrary copy data is sent to EC coordinator node, this copy data is divided into X point by EC coordinator node
Sheet data, and using EC algorithm, X fragment data is verified, obtain Y burst verification data, EC coordinator node storage one
Individual fragment data, and then X+Y-1 burst is sent to different EC memory nodes is stored.
When data storage method is EC redundancy by copy redundant conversion, need first to send arbitrary copy data to EC association
Point of adjustment, the more multiple bursts including fragment data and burst verification data are sent to by different EC by EC coordinator node
Memory node is stored, and this process transmits mass data, leads to network resource consumption in storage system larger.
Content of the invention
When data being switched to EC memory module by many copies memory module stored to reduce, network in storage system
Resource consumption, embodiments provides the date storage method in a kind of storage system and coordinates memory node.Described skill
Art scheme is as follows：
A kind of first aspect, there is provided the date storage method in storage system, this storage system includes multiple storage sections
Point, length that the N number of memory node in multiple memory nodes is stored with is the data of M, and this N number of memory node includes a coordination
Memory node, wherein, N is the natural number more than 1, and the method includes：
Within the storage system, because of business demand, data is stored by many copies memory module Transformed E C memory module
When, coordinate memory node and length is divided into X data fragmentation for M data, each data fragmentation has a starting point
Location and length, the initial address of each data fragmentation is this data fragmentation is the position in the data of M in length, and each data is divided
The length of piece is M/X.Based on X obtained data fragmentation, coordinate memory node by school is carried out to X data fragmentation
Test, obtain Y verification burst, the length of each verification burst is also M/X.
For i-th data fragmentation in X data fragmentation, coordinate memory node by the starting point of i-th data fragmentation
Location and length are sent to i-th memory node in N number of memory node, and wherein, i is the natural number more than 1 and less than or equal to D, D
For the minimum of a value in X and N.
When X is more than N, coordinate memory node respectively by each of X-N data fragmentation and Y verification burst burst
One the first memory node of distribution, this first memory node is the arbitrary memory node in multiple memory nodes in addition to N.
When X is less than or equal to N, coordinating memory node is that each of Y verification burst burst distribution one second is deposited
Storage node, this second memory node is the arbitrary memory node in multiple memory nodes in addition to X memory node.
Due to there is dual-use function as the data memory node coordinating memory node, not only complete data can be stored,
But also the burst including data fragmentation and verification burst can be distributed, thus need not by complete data is activation extremely
Special coordination memory node, substantially reduces the volume of transmitted data in storage system, saves the network money in storage system
Source consumes.
In conjunction with a first aspect, in the first possible implementation of first aspect, the storage system that coordinator node is located
System also includes view management node, based on this view management node, coordinates memory node by the starting point of i-th data fragmentation
When location and length are sent to i-th memory node in N number of memory node, can first receive the storage section of view management node transmission
Point information, this memory node information includes the information of the memory node of data that memory length is M, and this information includes memory node
Mark etc..Coordinate memory node according to received memory node information, by the initial address of i-th data fragmentation and length
Degree is targetedly sent to i-th memory node in N number of memory node.
In conjunction with a first aspect, in the possible implementation of the second of first aspect, coordinating memory node and also will receive
Storage success message, this storage success message is by storing X data fragmentation and Y and verify in the memory node of burst except coordination is deposited
Other memory nodes beyond storage node send.By receiving storage success message, coordinate memory node and would know that each burst
Storage condition, thus in the unsuccessful storage of arbitrary burst, this burst being sent deposited to corresponding memory node in time
Storage, complete to guarantee data.
In conjunction with the first possible implementation of first aspect, the third the possible realization side in terms of first aspect
In formula, for enabling coordination memory node by the initial address of i-th data fragmentation and length targetedly, in storage system
View management node can carry out Hash calculation to the data for M for the length, obtains cryptographic Hash, and according to this cryptographic Hash and institute's structure in advance
The hash space made, determines data that length the is M position in hash space, and then with this position as starting point, will specify
N number of memory node in the direction memory node of the data for M and X data fragmentation of storage and Y verification as memory length
The memory node of the N number of burst in burst.
A kind of second aspect, there is provided coordination memory node, this coordination memory node is located in storage system, this storage system
System includes multiple memory nodes and view management node, and length that the N number of memory node in the plurality of memory node is stored with is M
Data, this coordination memory node is contained in N number of memory node, and wherein, N is the natural number more than 1, this coordination memory node
For executing the method coordinated in above-mentioned first aspect performed by memory node, this view management node is used for executing above-mentioned first
Method performed by view management node in aspect.
A kind of third aspect, there is provided coordination memory node, this coordination memory node is located in storage system, this storage system
System includes multiple memory nodes, and length that the N number of memory node in the plurality of memory node is stored with is the data of M, and this association
Memory node is adjusted to be contained in N number of memory node, wherein, N is the natural data more than 1, this coordination memory node includes storing
Device, processor and bus, and memory and processor to pass through bus direct-connected；
This memory is used for storing computer instruction, and the computer instruction that this processor is used for executing memory storage is used for
Execute the date storage method in the storage system described in above-mentioned first aspect.
Specifically, the coordination memory node of above-mentioned second aspect and the third aspect, by specific construction unit or process
Device, executable following steps：
The data that described length is M is divided into X data fragmentation, and described X data fragmentation is verified, and obtains
Y verification burst, wherein, X and Y is natural number, and the length of each data fragmentation and each verification burst is M/X；
The initial address of i-th data fragmentation and length are sent to i-th storage section in described N number of memory node
Point, wherein, i be more than 1 and less than or equal to D natural number, D be X and N in minimum of a value；
When X is more than N, each of respectively X-N data fragmentation and described Y verification burst burst distributes one
First memory node, described first memory node is arbitrary in addition to described N number of memory node in the plurality of memory node
Memory node.
Specifically, the coordination memory node of above-mentioned second aspect and the third aspect, by specific construction unit or process
Device, executable following steps：
The data that described length is M is divided into X data fragmentation, and described X data fragmentation is verified, and obtains
Y verification burst, wherein, X and Y is natural number, and the length of each data fragmentation and each verification burst is M/X；
The initial address of i-th data fragmentation and length are sent to i-th storage section in described N number of memory node
Point, wherein, i be more than 1 and less than or equal to D natural number, D be X and N in minimum of a value；
When X is not more than N, it is that each of described Y verification burst burst distributes second memory node, described
Second memory node is the arbitrary memory node in the plurality of memory node in addition to described X memory node.
The beneficial effect that technical scheme provided in an embodiment of the present invention is brought is：
When data being switched to EC memory module by many copies memory module being stored, need not send data to special
Coordination memory node, and by data distribution can be carried out as the memory node of data harmonization node, greatly reduce data
Transmission quantity, saves network resource consumption in system.
Brief description
For the technical scheme being illustrated more clearly that in the embodiment of the present invention, will make to required in embodiment description below
Accompanying drawing be briefly described it should be apparent that, drawings in the following description are only some embodiments of the present invention, for
For those of ordinary skill in the art, on the premise of not paying creative work, other can also be obtained according to these accompanying drawings
Accompanying drawing.
Fig. 1 is that one embodiment of the invention provides a kind of schematic diagram of memory system architecture；
Fig. 2 is the date storage method flow chart in a kind of storage system that another embodiment of the present invention provides；
Fig. 3 is the schematic diagram of the uniformity hash algorithm that another embodiment of the present invention provides；
Fig. 4 is the distribution schematic diagram of memory node in a kind of storage system that another embodiment of the present invention provides；
Fig. 5 is the schematic diagram of the data storage procedure that another embodiment of the present invention provides；
Fig. 6 is the schematic diagram of the data storage procedure that another embodiment of the present invention provides；
Fig. 7 is the structural representation of the coordination memory node that another embodiment of the present invention provides；
Fig. 8 is the structural representation of the coordination memory node that another embodiment of the present invention provides；
Fig. 9 is the illustrative Computer Architecture coordinating memory node used in another embodiment of the present invention.
Specific embodiment
For making the object, technical solutions and advantages of the present invention clearer, below in conjunction with accompanying drawing to embodiment party of the present invention
Formula is described in further detail.
Referring to Fig. 1, it illustrates a kind of storage system, this storage system includes view management node 101, client 102
And memory node 103.
Wherein, view management node 101 can be for a single computing device it is also possible to be by multiple stage computing device group
The computing cluster becoming.This view management node is used for receiving the heartbeat message that memory node 103 reports, and according to memory node
103 heartbeat messages reporting, update the node state of memory node 103.View management node 101 is additionally operable to as client 102
In data distribution memory node.View management node 101 also safeguards a corresponding relation between cryptographic Hash and memory node
View, this view inquires about data for client 102 from memory node.
Client 102 can be smart mobile phone, panel computer, notebook computer etc., and the present embodiment is not to client 102
Product type make specific restriction.Client 102 is used for receiving the view of view management node 101 push, and is regarded according to this
Figure inquiry data.
Memory node 103 is the physical medium of data storage in storage system, and this memory node can be to every preset duration
View management node reports heartbeat message, so that the state of view management this memory node of node maintenance.
Above-mentioned view management node 101 can be entered by cable network or wireless network with client 102 and back end 103
Row communication, client 102 can be communicated by cable network or wireless network with back end 103.
Based on the Database Systems shown in Fig. 1, embodiments provide a kind of date storage method, this storage system
Including multiple memory nodes and view management node, wherein, length that the N number of memory node in multiple memory nodes is stored with is
The data of M, and N number of memory node includes a coordination memory node, referring to Fig. 2, method flow provided in an embodiment of the present invention
Including：
201st, view management node is the data distribution memory node information of length M.
Because the memory node quantity that storage system includes is more, in order to targetedly to length, the data for M adopts
Redundant fashion is stored, and reduces network resource consumption in system in data transmission procedure to greatest extent, in this enforcement simultaneously
Example in, view management node also by using memory node selection algorithm be length for M data distribution memory node.
Wherein, memory node selection algorithm can include uniformity hash algorithm etc..Uniformity hash algorithm principle is as follows：
Construction one 0～232- 1 hash space, by 0～232- 1 hash space forms a Hash ring, Hash ring is divided into multiple
Equal interval, the difference range of the cryptographic Hash in each interval is identical.Fig. 3 shows the Hash of a hash space composition
Ring, referring to Fig. 3, this Hash ring is divided into 20 intervals, the difference range of the cryptographic Hash of respectively P0～P19 and P0～P19
Identical.
It is limited to the performance of storage system itself, the data for M is stored storage system using redundant fashion to length
When, the quantity of the copy data data of M (length that duplication obtains be), the quantity of data fragmentation and data fragmentation is verified
To the quantity of verification burst be to determine.Storage system described in the present embodiment is stored using redundant fashion
When, the quantity of copy data is N number of, and the quantity of data fragmentation is X, and the quantity of verification burst is Y, and wherein, N is more than 1
Natural number, X and Y be natural number.
Specifically, view management node using memory node selection algorithm, is the data distribution memory node that length is M
When, it may include following steps 2011～2013：
2011st, to length, the data for M carries out Hash calculation to view management node, obtains cryptographic Hash.
View management node can to length, the data for M carries out Hash calculation using specifying hash algorithm, obtains cryptographic Hash.
Wherein it is intended that hash algorithm includes addition hash algorithm, multiplication hash algorithm, division hash algorithm, bit arithmetic hash algorithm, looks into
Table hash algorithm etc..
2012nd, view management node is according to cryptographic Hash, determines data that length the is M position in hash space.
In the present embodiment, each interval in hash space corresponds to a cryptographic Hash scope, breathes out when cryptographic Hash is located at
Uncommon space arbitrary interval in, can using the position at this interval place as the data for M for the length position in hash space.Example
As, through Hash calculation, the cryptographic Hash obtaining is k1 to data KEY for M for the length, and the interval that this k1 is located at PO place in Fig. 3 is interior,
Therefore, position PO being located position in hash space as data KEY for M for the length.
2013rd, view management node is with position as starting point, and N number of memory node of assigned direction is M as memory length
Data memory node.
Wherein it is intended that direction can be the clockwise direction with cryptographic Hash position as starting point, can also be with Hash
Value position is the counter clockwise direction of starting point.Because in the present embodiment, using N number of copy, to length, the data for M carries out redundancy
Storage, therefore, view management node needs the data for M for length to distribute N number of memory node under copy memory module.It is based on
The cryptographic Hash that Hash calculation obtains is carried out to the data for M for the length, the position that view management node is located with this cryptographic Hash is for rising
Point, using the memory node of the N number of memory node data for M as memory length on assigned direction, the choosing of this N number of memory node
Take mode can choose for continuous, N number of continuous storage section that will be on assigned direction with this cryptographic Hash position as starting point
The memory node of point data for M as memory length, also can choose for discontinuous, will be with this cryptographic Hash institute on assigned direction
It is set to the memory node of N number of discontinuous memory node data for M as memory length of starting point in place.
2014th, view management node is with position as starting point, using N number of memory node of assigned direction as X data of storage
The memory node of the N number of burst in burst and Y verification burst.
The cryptographic Hash that Hash calculation obtains is carried out based on the data for M to length, view management node is with this cryptographic Hash institute
Position be starting point, using N number of memory node of assigned direction as storage X data fragmentation and Y verify burst in N number of
The memory node of burst, the selection mode of N number of memory node in X data fragmentation of this storage and Y verification burst and storage
Length is that the selection mode of N number of memory node of data of M is identical, if N number of memory node that memory length is the data of M is adopted
Chosen with continuation mode, then by the N number of continuous memory node using this cryptographic Hash position as starting point as storage X
The memory node of the N number of burst in data fragmentation and Y verification burst, if memory length is N number of memory node of the data of M
Chosen using discontinuous mode, then the N number of discontinuous memory node using this cryptographic Hash position as starting point is as depositing
The memory node of the N number of burst in X data fragmentation of storage and Y verification burst.Chosen using which kind of mode anyway,
Guarantee N number of memory node and X data fragmentation of storage and the Y N number of burst verifying in burst of the data that memory length is M
Memory node overlaps to greatest extent.
Taking using 3 copy datas, 4 data fragmentations, the redundant storage mode of 2 verification bursts as a example, view management node is
Length is the process of the data distribution memory node of M, can be found in following examples.
(1), data KEY for M for length, data KEY that view management node is M to this length carries out Hash meter
Calculate, obtain cryptographic Hash k1.
(2), view management node, according to this cryptographic Hash k1, determines the position that this cryptographic Hash k1 is located from hash space,
And be starting point by this position, the memory node of clockwise 3 memory node data for M as memory length is joined
See Fig. 3, PO, P1, P2 are the memory node of the data that memory length is M.
(3), view management node, according to this cryptographic Hash k1, determines the position that this cryptographic Hash k1 is located from hash space,
And by this position be starting point, will clockwise 3 memory nodes as storage 4 data fragmentations and 2 verify bursts in
3 bursts memory node, referring to Fig. 3, P0, P1, P2 are 3 points storing in 4 data fragmentations and 2 verification bursts
The memory node of piece.
The above-mentioned view management node memory node that the data by M is distributed by length is actually each in hash space
The interval dummy storage node being located, this dummy storage node cannot veritably data storage, need by dummy storage node with
Physical store node in storage system sets up mapping relations, by the data persistence in corresponding dummy storage node to physics
In memory node.For the ease of being managed to dummy storage node and physical store node, view management node also safeguards one
This mapping table, this mapping table memory storage each interval mapping relations and physical store node between of hash space, lead to
Often the quantity of physical store node is less than the interval quantity in hash space, and so each physical store node actually can be right
Answer multiple intervals, the data storage that is to say multiple dummy storage node is on Same Physical memory node.Referring to Fig. 3, Hash
Space is divided into P0～P19 totally 20 intervals, and storage system includes 4 physical store nodes, respectively physical store node
1st, physical store node 2, physical store node 3 and physical store node 4, wherein, A (P0), E (P4), I (P8), M (P12), Q
(P16) it is mapped on physical node 1, B (P1), F (P5), J (P9), N (P13), R (P17) are mapped on physical node 2, C
(P3), G (P6), K (P10), 0 (P14), S (P18) are mapped on physical node 3, D (P4), H (P7), L (P11), P (P15),
T (P19) is mapped on physical node 4.
Referring to Fig. 4, the mapping table safeguarded based on view management node, when client needs to inquire about number from storage system
According to KEY1, client cocoa first calculates cryptographic Hash k1 of this KEY1, and determines position in hash space for cryptographic Hash k1, and then
According to this mapping table, search the physical store node at this cryptographic Hash place, thus inquiring data from this physical store node
KEY1.
202nd, view management node sends memory node information to coordination memory node and client.
Due to the data for M for the memory length memory node and storage X data fragmentation and Y verify burst in N number of
The memory node of burst, is that client carries out the storage of many copies, coordinates the key that memory node carries out EC storage, for making client
With coordinate memory node can be stored targeted specifically, view management node can by the data by M by length distributed N number of
The memory node of the N number of burst in the memory node information of memory node and X data fragmentation of storage and Y verification burst
Memory node information send to client and coordinate memory node.Wherein, memory node information includes memory node mark etc.
Deng.
203rd, client, according to memory node information, replicates to the data for M for the length, and the data after replicating is sent out
Deliver to N number of memory node to be stored.
When receive view management node transmission memory node information, client data duplication N part for M to length,
Obtain the data that N number of length is M, and then by the data is activation by M for N number of length to the N number of memory node being distributed, deposited by each
Storage node is stored.
In order to improve data access rate, storage system can according to the memory module of the visit capacity change data of client,
When client is more than certain numerical value to the visit capacity of any data, storage system can trigger coordination memory node by data by
Copy memory module is converted to EC memory module and is stored.
204th, the data for M is divided into X data fragmentation by length to coordinate memory node, and X data fragmentation is carried out
Verification, obtains Y verification burst.
Because the storage system in the present embodiment is deposited using the redundant fashion of X data fragmentation, Y verification burst
Storage, therefore, under EC memory module, coordinating memory node needs the data being M by the length being stored on this node to be divided into X
Individual data fragmentation.Based on the X data fragmentation being divided, coordinate memory node using specifying checking algorithm, to X data fragmentation
Verified, obtained Y verification burst.Wherein it is intended that checking algorithm includes parity arithmetic, CRC CRC is calculated
Method etc..
205th, coordinate memory node the initial address of i-th data fragmentation and length are sent in N number of memory node
I-th memory node.
Wherein, i be more than 1 and less than or equal to D natural number, D be X and N in minimum of a value.Due in N number of memory node
Store the data that length is M, therefore, transmitted consumed Internet resources in system to save data, coordinate storage
Node can send burst information to N number of memory node, is M according to burst information to the length being stored by each memory node
Data sheared, obtain a data fragmentation and store.Wherein, burst information includes initial address and the length of data fragmentation
Degree etc..For i-th data fragmentation in X data fragmentation, coordinate memory node by the initial address of i-th data fragmentation and
Length is sent to i-th memory node in N number of memory node, initial according to i-th data fragmentation by i-th memory node
Address and length, the data being M to the length being stored is sheared, and obtains data corresponding with initial address and length and divides
Piece, and store this data fragmentation.
206th, when X is more than N, coordinate storage section and be respectively each of X-N data fragmentation and Y verification burst point
Piece distributes first memory node.
Wherein, the first memory node is the arbitrary memory node in multiple memory nodes in addition to N number of memory node.Work as X
During more than N, illustrate data fragmentation quantity be more than length be M data copy amount, now, coordinate memory node need be
Each of X-N data fragmentation and Y verification burst burst distributes first memory node, and X-N data is divided
Piece and Y verification burst send to the first memory node being distributed, and are stored by the first memory node.
207th, when X is not more than N, coordination storage section is deposited for one second for the burst distribution of each of Y verification burst
Storage node.
Wherein, the second memory node is the arbitrary memory node in multiple memory nodes in addition to X memory node.Work as X
When being not more than N, illustrate that the quantity of data fragmentation is less than or equal to the copy amount of the data that length is M, now coordinating storage section needs
Y verification each of burst burst to be distributes second memory node, and Y is verified burst sends and deposit to second
Storage node, is stored by the second memory node.
For said process, in order to make it easy to understand, will be illustrated with a specific example below.
Referring to Fig. 5, setting data size is 4M, the storage section of the data distribution that view management node is 4M for memory length
Point be N1, N2, N3, be 4 data fragmentations and 2 verification burst distribution memory nodes be N1, N2, N3, N4, N5, N6, coordinate
By length, the data for 4M is divided into the data fragmentation of 4 1M to memory node N1, and 4 data fragmentations are verified, and obtains 2
Individual verification burst.When being stored when needing the data for M by length to switch to EC memory module by many copies memory module, coordinate
Memory node N1 is 0 according to initial address and length is the burst information of 1M, and the length that this node is stored is the data of 4M
Sheared, obtain data fragmentation and store, and sent, to memory node N2, the burst letter that initial address is 1M and length is 1M
Breath, sending initial address to memory node N3 is the 2M and length burst information for 1M, by memory node N2 and N3 to this node
The length being stored is that the data of 4M is sheared, and obtains data fragmentation and stores.For memory node N4, N5, N6, coordinate to deposit
Storage node N1 can for 3M and length, the data fragmentation for 1M be stored to memory node N4 by initial address, and initial address is
0 and verification burst that length is 1M send and stored to memory node N5, by initial address be 1M and length is the verification of 1M
Burst sends and is stored to memory node N6.
For enabling the client to know the storage request of data under EC memory module it is intended that memory node is successfully storing often
When individual data fragmentation or verification burst, also will send storage success message to coordinating memory node.Wherein it is intended that memory node is
Other memory nodes in addition to coordinating memory node in the memory node of X data fragmentation of storage and Y verification burst.
Fig. 6 shows the procedure chart that data is switched to carry out during EC memory module data storage by many copies memory module,
Referring to Fig. 6, view management node is data distribution memory node N1, N2, N3 of M for length, is 4 data fragmentations and 2 schools
The memory node testing burst distribution is N1, N2, N3, N4, N5, N6.When EC memory module is switched to by many copies memory module, association
Adjust memory node N1 that the data that the length being stored is M is divided into 4 data fragmentations, and 4 data fragmentations are verified, obtain
To 2 verification bursts.Coordinate memory node N1N1 according to burst information<Offset1, len1>, length that this node is stored
Data for M is sheared, and obtains data fragmentation and stores；And send burst information to memory node N2<Offset2, len2
>, by memory node N2 according to<Offset2, len2>The data being M to the length being stored is sheared, and obtains a data
Burst simultaneously stores；Send burst information to N3<Offset3, len3>, by memory node N3 according to<Offset3, len3>To institute
The length of storage is that the data of M is sheared, and obtains a data fragmentation and stores；Burst information is<Offset4, len4>
Data fragmentation send and stored to memory node N4, two verification bursts are respectively sent to memory node N5 and N6 and carry out
Storage.
Method provided in an embodiment of the present invention, is stored data is switched to EC memory module by many copies memory module
When, special coordination memory node need not be sent data to, and by can carry out as the memory node of data harmonization node
Data distribution, greatly reduces volume of transmitted data, saves network resource consumption in system.
Referring to Fig. 7, embodiments provide a kind of coordination memory node, this coordination memory node is located at storage system
In, this storage system includes multiple memory nodes, and length that the N number of memory node in multiple memory nodes is stored with is the number of M
According to coordination memory node is contained in N number of memory node, and wherein, N is the natural number more than 1, coordinates memory node and includes：
Division unit 701, is divided into X data fragmentation for the data for M by length, and X data fragmentation is carried out
Verification, obtains Y verification burst, and wherein, X and Y is natural number, and the length of each data fragmentation and each verification burst
It is M/X；
Transmitting element 702, for being sent respectively to N number of memory node by the initial address of i-th data fragmentation and length
In i-th memory node, wherein, i is more than 1 and the little natural number equal to D, D be X and N in minimum of a value；
Allocation unit 703, for when X is more than N, respectively X-N data fragmentation and Y verify each of burst
Burst distributes first memory node, and the first memory node is arbitrary in addition to N number of memory node in multiple memory nodes
Memory node.
In another embodiment of the present invention, allocation unit 703, for when X is not more than N, being in Y verification burst
Each burst distribute second memory node, the second memory node be in multiple memory nodes except X memory node with
Outer arbitrary memory node.
Optionally, in an alternative embodiment of the invention, allocation unit 703 is used for when X is more than N, respectively X-N number
Distribute first memory node according to each of burst and Y verification burst burst, the first memory node is multiple storages
Arbitrary memory node in addition to N number of memory node in node；When X is not more than N, it is each of Y verification burst point
Piece distributes second memory node, and the second memory node is that in multiple memory nodes, arbitrary in addition to X memory node deposits
Storage node.
Referring to Fig. 8, in another embodiment of the present invention, storage system also includes view management node, coordinates storage
Node also includes receiving unit 704；
Receiving unit 704, for receiving the memory node information of view management node transmission, memory node information includes depositing
Storage length is the information of the memory node of data of M；
Transmitting element 702, specifically for dividing the initial address of i-th data fragmentation and length according to memory node information
It is not sent to i-th memory node in N number of memory node.
In another embodiment of the present invention, receiving unit 704, are additionally operable to receive storage success message, store successfully
Message by store X data fragmentation and Y verification burst memory node in except coordinate memory node in addition to other storage section
Point sends.
In another embodiment of the present invention, view management node, based on to length, the data for M carries out Hash
Calculate, obtain cryptographic Hash；According to cryptographic Hash, determine data that length the is M position in hash space；With this position as starting point,
Memory node using N number of memory node data for M as memory length of assigned direction；With this position as starting point, will specify
N number of memory node in direction is as the memory node of the N number of burst in X data fragmentation of storage and Y verification burst.
In sum, system provided in an embodiment of the present invention, data is being switched to EC storage mould by many copies memory module
When formula is stored, special coordination memory node need not be sent data to, and by can depositing as data harmonization node
Storage node carries out data distribution, greatly reduces volume of transmitted data, saves network resource consumption in system.
Referring to Fig. 9, it illustrates the illustrative calculating coordinating memory node 900 used in one embodiment of the present of invention
Machine architecture.Described coordination memory node 900 is server or storage device.Described coordination memory node 900 includes processing
Device 901, memory 902 and bus 903, wherein, processor 901 and memory 902 are connected by bus 903.Optionally, described
Coordinate memory node 900 also to include helping the basic input/output of transmission information between each device in computing device
(I/O system) 904 and the mass-memory unit 905 for storage program area, application program and other program modules.
Optionally, described mass-memory unit 905 (is not shown by being connected to the bulk memory controller of bus 903
Go out) it is connected to processor 901.Described mass-memory unit 905 and its associated computer-readable medium are to coordinate storage
Node 900 provides non-volatile memories.That is, described mass-memory unit 905 can include such as hard disk or CD-
The computer-readable medium (not shown) of ROM drive etc.
Without loss of generality, described computer-readable medium can include computer-readable storage medium and communication media.Computer
Storage medium is included for information such as storage computer-readable instruction, data structure, program module or other data
Volatibility and non-volatile, removable and irremovable medium that any method or technique is realized.Computer-readable storage medium includes
RAM, ROM, EPROM, EEPROM, flash memory or its technology of other solid-state storage, CD-ROM, DVD or other optical storage, tape
Box, tape, disk storage or other magnetic storage apparatus.Certainly, skilled person will appreciate that described computer-readable storage medium
It is not limited to above-mentioned several.
According to various embodiments of the present invention, described coordination memory node 900 can also pass through the networks such as internet
The remote computer being connected on network runs.Namely coordination memory node 900 can be by being connected in described bus 903
NIU 906 is connected to network 907, in other words, it is possible to use NIU 906 is being connected to other types
Network or remote computer system (not shown).Wherein, NIU 906 can (also referred to as network connects for network interface card
Mouth card) or host bus adaptor etc..
Computer equipment provided in an embodiment of the present invention, enters data is switched to EC memory module by many copies memory module
During row storage, special coordination memory node need not be sent data to, and by can be used as the storage section of data harmonization node
Point carries out data distribution, greatly reduces volume of transmitted data, saves network resource consumption in system.
It should be noted that：Above-described embodiment provide coordination memory node in data storage, only with above-mentioned each function
The division of module is illustrated, and in practical application, can distribute above-mentioned functions by different function moulds as desired
Block completes, and the internal structure that will coordinate memory node is divided into different functional modules, with complete described above all or
Person's partial function.In addition, the date storage method coordinating in memory node and data-storage system that above-described embodiment provides is real
Apply example and belong to same design, it implements process and refers to embodiment of the method, repeats no more here.
One of ordinary skill in the art will appreciate that all or part of step realizing above-described embodiment can pass through hardware
To complete it is also possible to the hardware being instructed correlation by program is completed, described program can be stored in a kind of computer-readable
In storage medium, storage medium mentioned above can be read-only storage, disk or CD etc..
The foregoing is only presently preferred embodiments of the present invention, not in order to limit the present invention, all spirit in the present invention and
Within principle, any modification, equivalent substitution and improvement made etc., should be included within the scope of the present invention.
Claims (10)
1. the date storage method in a kind of storage system is it is characterised in that described storage system includes multiple memory nodes, institute
The data that length is M that is stored with is stated on the N number of memory node in multiple memory nodes, described N number of memory node includes an association
Adjust memory node, wherein, N is the natural number more than 1, and methods described includes：
The data that described length is M is divided into X data fragmentation by described memory node of coordinating, and to described X data fragmentation
Verified, obtained Y verification burst, wherein, X and Y is natural number, and each data fragmentation and each verification burst
Length is M/X；
The initial address of i-th data fragmentation and length are sent in described N number of memory node described coordination memory node
I-th memory node, wherein, i be more than 1 and less than or equal to D natural number, D be X and N in minimum of a value；
When X is more than N, described coordination storage section is respectively each of X-N data fragmentation and described Y verification burst point
Piece distributes first memory node, and described first memory node is to remove described N number of memory node in the plurality of memory node
Arbitrary memory node in addition；
When X is not more than N, described coordination storage section distributes one second for each of described Y verification burst burst and deposits
Storage node, described second memory node is the arbitrary storage section in the plurality of memory node in addition to described X memory node
Point.
2. method according to claim 1 is it is characterised in that described storage system also includes view management node, described
Coordinate i-th storage that the initial address of i-th data fragmentation and length are sent in described N number of memory node by memory node
Node, including：
Described memory node of coordinating receives the memory node information that described view management node sends, described memory node packet
Include the information storing the memory node of data that described length is M；
The initial address of i-th data fragmentation and length are sent to by described coordination memory node according to described memory node information
I-th memory node in described N number of memory node.
3. method according to claim 1 is it is characterised in that methods described also includes：
Described coordination memory node receives storage success message, and described storage success message is by storing X data fragmentation and Y school
Test other memory nodes in addition to described coordination memory node in the memory node of burst to send.
4. method according to claim 2 is it is characterised in that methods described also includes：
The data that described view management node is M to described length carries out Hash calculation, obtains cryptographic Hash；
Described view management node, according to described cryptographic Hash, determines position in hash space for the data for M for the described length；
Described view management node with described position as starting point, using N number of memory node of assigned direction as store described length
The memory node of the data for M；
Described view management node with described position as starting point, using N number of memory node of assigned direction as store described X number
Memory node according to the N number of burst in burst and described Y verification burst.
5. one kind coordinates memory node it is characterised in that described coordination memory node is located in storage system, described storage system
Including multiple memory nodes, length that the N number of memory node in the plurality of memory node is stored with is the data of M, described association
Memory node is adjusted to be contained in described N number of memory node, wherein, N is the natural number more than 1, and described coordination memory node includes：
Division unit, is divided into X data fragmentation for the data for M by described length, and described X data fragmentation is carried out
Verification, obtains Y verification burst, and wherein, X and Y is natural number, and the length of each data fragmentation and each verification burst
It is M/X；
Transmitting element, for being sent respectively to the initial address of i-th data fragmentation and length in described N number of memory node
I-th memory node, wherein, i is more than 1 and the little natural number equal to D, and D is the minimum of a value in X and N；
Allocation unit, for when X is more than N, each of respectively X-N data fragmentation and described Y verification burst divide
Piece distributes first memory node, and described first memory node is to remove described N number of memory node in the plurality of memory node
Arbitrary memory node in addition.
6. coordination memory node according to claim 5, it is characterised in that described allocation unit, is additionally operable to be not more than as X
During N, it is that each of described Y verification burst burst distributes second memory node, described second memory node is institute
State the arbitrary memory node in addition to described X memory node in multiple memory nodes.
7. coordination memory node according to claim 5 is it is characterised in that described storage system also includes view management section
Point, described coordination memory node also includes receiving unit；
Described receiving unit, for receiving the memory node information that described view management node sends, described memory node information
Information including the memory node storing the data that described length is M；
Described transmitting element, specifically for according to described memory node information by the initial address of i-th data fragmentation and length
It is sent respectively to i-th memory node in described N number of memory node.
8. coordination memory node according to claim 5, it is characterised in that described receiving unit, is additionally operable to receive storage
Success message, described storage success message by store X data fragmentation and Y verification burst memory node in remove described coordination
Other memory nodes beyond memory node send.
9. coordination memory node according to claim 7 is it is characterised in that described view management node, for described
Length is that the data of M carries out Hash calculation, obtains cryptographic Hash；According to described cryptographic Hash, determine that the data that described length is M is being breathed out
Position in uncommon space；With described position as starting point, N number of memory node of assigned direction is M as storing described length
The memory node of data；With described position as starting point, N number of memory node of assigned direction is divided as storing described X data
The memory node of the N number of burst in piece and described Y verification burst.
10. it is characterised in that described coordination memory node is located in storage system, described storage is a kind of coordination memory node
System includes multiple memory nodes, and length that the N number of memory node in the plurality of memory node is stored with is the data of M, described
Coordinate memory node to be contained in described N number of memory node, wherein, N is the natural number more than 1, described coordination memory node bag
Include memory, processor and bus, and described memory and described processor direct-connected by described bus；
Described memory is used for depositing computer instruction, and described processor is used for executing the computer instruction of described memory storage
Require the date storage method in the storage system described in 1 to 4 any one for perform claim.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610803072.9A CN106383665B (en) | 2016-09-05 | 2016-09-05 | Date storage method and coordination memory node in data-storage system |
| PCT/CN2017/081073 WO2018040583A1 (en) | 2016-09-05 | 2017-04-19 | Data storage method in data storage system and coordinating storage node |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610803072.9A CN106383665B (en) | 2016-09-05 | 2016-09-05 | Date storage method and coordination memory node in data-storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106383665A true CN106383665A (en) | 2017-02-08 |
| CN106383665B CN106383665B (en) | 2018-05-11 |
Family
ID=57938089
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610803072.9A Active CN106383665B (en) | 2016-09-05 | 2016-09-05 | Date storage method and coordination memory node in data-storage system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN106383665B (en) |
| WO (1) | WO2018040583A1 (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107436733A (en) * | 2017-06-29 | 2017-12-05 | 华为技术有限公司 | Management by district method and management by district device |
| CN107491361A (en) * | 2017-07-21 | 2017-12-19 | 山东九州信泰信息科技股份有限公司 | The method for being classified other redundant storage is carried out to row in tables of data |
| WO2018040583A1 (en) * | 2016-09-05 | 2018-03-08 | 华为技术有限公司 | Data storage method in data storage system and coordinating storage node |
| CN108647523A (en) * | 2018-04-28 | 2018-10-12 | 华南理工大学 | A kind of electronic identification system based on block chain and deposit card, file access pattern method |
| CN108780386A (en) * | 2017-12-20 | 2018-11-09 | 华为技术有限公司 | A kind of methods, devices and systems of data storage |
| CN109062512A (en) * | 2018-07-26 | 2018-12-21 | 郑州云海信息技术有限公司 | A kind of distributed storage cluster, data read-write method, system and relevant apparatus |
| CN113986944A (en) * | 2021-12-29 | 2022-01-28 | 天地伟业技术有限公司 | Writing method and system of fragment data and electronic equipment |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111309801A (en) * | 2018-12-11 | 2020-06-19 | 航天信息股份有限公司 | Method and device for fragmenting alliance chain |
| CN110149373B (en) * | 2019-04-28 | 2022-01-07 | 平安科技（深圳）有限公司 | Data storage method, device, medium and server based on peer-to-peer network |
| CN113691581B (en) * | 2021-07-08 | 2023-05-09 | 杭州又拍云科技有限公司 | Efficient CDN refreshing and slicing method |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8458287B2 (en) * | 2009-07-31 | 2013-06-04 | Microsoft Corporation | Erasure coded storage aggregation in data centers |
| CN102110154B (en) * | 2011-02-21 | 2012-12-26 | 中国科学院计算技术研究所 | File redundancy storage method in cluster file system |
| CN103984607A (en) * | 2013-02-08 | 2014-08-13 | 华为技术有限公司 | Distributed storage method, device and system |
| CN103631539B (en) * | 2013-12-13 | 2016-08-24 | 百度在线网络技术（北京）有限公司 | Distributed memory system based on erasure codes mechanism and storage method thereof |
| CN105573680A (en) * | 2015-12-25 | 2016-05-11 | 北京奇虎科技有限公司 | Storage method and device for replicated data |
| CN106383665B (en) * | 2016-09-05 | 2018-05-11 | 华为技术有限公司 | Date storage method and coordination memory node in data-storage system |
-
2016
- 2016-09-05 CN CN201610803072.9A patent/CN106383665B/en active Active
-
2017
- 2017-04-19 WO PCT/CN2017/081073 patent/WO2018040583A1/en active Application Filing
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018040583A1 (en) * | 2016-09-05 | 2018-03-08 | 华为技术有限公司 | Data storage method in data storage system and coordinating storage node |
| CN107436733B (en) * | 2017-06-29 | 2020-11-06 | 华为技术有限公司 | Fragment management method and fragment management device |
| US11243706B2 (en) | 2017-06-29 | 2022-02-08 | Huawei Technologies Co., Ltd. | Fragment management method and fragment management apparatus |
| CN107436733A (en) * | 2017-06-29 | 2017-12-05 | 华为技术有限公司 | Management by district method and management by district device |
| CN112328168A (en) * | 2017-06-29 | 2021-02-05 | 华为技术有限公司 | Fragment management method and fragment management device |
| WO2019000950A1 (en) * | 2017-06-29 | 2019-01-03 | 华为技术有限公司 | Fragment management method and fragment management apparatus |
| CN107491361A (en) * | 2017-07-21 | 2017-12-19 | 山东九州信泰信息科技股份有限公司 | The method for being classified other redundant storage is carried out to row in tables of data |
| CN107491361B (en) * | 2017-07-21 | 2019-04-02 | 山东九州信泰信息科技股份有限公司 | The method for being classified other redundant storage is carried out to column in tables of data |
| US10915397B2 (en) | 2017-12-20 | 2021-02-09 | Huawei Technologies Co., Ltd. | Data storage method, apparatus, and system |
| CN108780386B (en) * | 2017-12-20 | 2020-09-04 | 华为技术有限公司 | Data storage method, device and system |
| CN108780386A (en) * | 2017-12-20 | 2018-11-09 | 华为技术有限公司 | A kind of methods, devices and systems of data storage |
| US11487618B2 (en) | 2017-12-20 | 2022-11-01 | Huawei Cloud Computing Technologies Co., Ltd. | Data storage method, apparatus, and system |
| CN108647523B (en) * | 2018-04-28 | 2020-01-17 | 华南理工大学 | Block chain-based electronic certification system and certificate storage and file recovery method |
| CN108647523A (en) * | 2018-04-28 | 2018-10-12 | 华南理工大学 | A kind of electronic identification system based on block chain and deposit card, file access pattern method |
| CN109062512A (en) * | 2018-07-26 | 2018-12-21 | 郑州云海信息技术有限公司 | A kind of distributed storage cluster, data read-write method, system and relevant apparatus |
| CN109062512B (en) * | 2018-07-26 | 2022-02-18 | 郑州云海信息技术有限公司 | Distributed storage cluster, data read-write method, system and related device |
| CN113986944A (en) * | 2021-12-29 | 2022-01-28 | 天地伟业技术有限公司 | Writing method and system of fragment data and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2018040583A1 (en) | 2018-03-08 |
| CN106383665B (en) | 2018-05-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106383665B (en) | Date storage method and coordination memory node in data-storage system | |
| CN109471744B (en) | Main chain and parallel multi-sub-chain system architecture based on block chain | |
| CN110059494B (en) | Privacy protection method for block chain transaction data and block chain system | |
| US11973869B2 (en) | Maintaining blocks of a blockchain in a partitioned blockchain network | |
| US11469949B2 (en) | Hierarchical configuration of networked communication devices | |
| CN110490305B (en) | Machine learning model processing method based on block chain network and node | |
| CN111681003A (en) | Resource cross-chain transfer method and device, computer equipment and storage medium | |
| EP3779760A1 (en) | Blockchain-based data processing method and apparatus, and electronic device | |
| CN108712488A (en) | A kind of data processing method based on block chain, device, block catenary system | |
| CN108833610B (en) | Information updating method, device and system | |
| CN109493051B (en) | Main chain and parallel multi-subchain system architecture capable of dynamically allocating and migrating accounts | |
| US20210089517A1 (en) | Communication device and communication system | |
| CN109445839A (en) | A kind of module management packet dissemination method, device and terminal device | |
| CN108156232A (en) | Data sharing method and device | |
| CN108494828A (en) | A kind of update method of node data, medium, device and computing device | |
| CN110290168A (en) | Data transmission method for uplink, device, server and storage medium | |
| CN109583899A (en) | Data storage and management method and system based on cloud storage service device | |
| Jin et al. | Cost-effective data placement in edge storage systems with erasure code | |
| CN112910870A (en) | Collaborative privacy computation data communication method based on block chain | |
| CN109348434A (en) | A kind of sending method of scene information, sending device and terminal device | |
| CN110673966B (en) | Data interaction method based on block chain up-link down cooperation | |
| EP3780488A1 (en) | Digital certificate processing method and device, blockchain node and storage medium | |
| CN113626765A (en) | Energy data processing method, device, equipment and storage medium | |
| CN109445973A (en) | Position the method and device of application crash | |
| WO2023207529A1 (en) | Data processing method and apparatus, device, medium, and product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |