CN110413694A - Metadata management method and relevant apparatus - Google Patents
Metadata management method and relevant apparatus Download PDFInfo
- Publication number
- CN110413694A CN110413694A CN201910709228.0A CN201910709228A CN110413694A CN 110413694 A CN110413694 A CN 110413694A CN 201910709228 A CN201910709228 A CN 201910709228A CN 110413694 A CN110413694 A CN 110413694A
- Authority
- CN
- China
- Prior art keywords
- file
- stored
- metadata
- memory node
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The embodiment provides a kind of metadata management method and relevant apparatus, it is related to field of distributed storage, the described method includes: obtaining the mark of file to be stored and storing the first object memory node of file to be stored, wherein, first object memory node is the size determination based on file to be stored；The metadata of file to be stored is generated according to the mark and first object memory node of file to be stored；Mark according to file to be stored determines the second target storage node according to preset algorithm；The metadata of file to be stored is stored to the second target storage node.Compared with the existing technology, the embodiment of the present invention provides a kind of metadata management method and relevant apparatus, by the way that meta-data distribution formula is stored in each memory node, avoids metadata node from being managed collectively metadata bring access bottleneck, influence the entirety ability of distributed memory system.
Description
Technical field
The present invention relates to field of distributed storage, in particular to a kind of metadata management method and relevant apparatus.
Background technique
In field of distributed storage, most important metadata is to describe the information of data storage position, that is, needs to store
Corresponding relationship between file and the memory node for storing this document.Distributed memory system would generally use a metadata section
Point is to save the corresponding relationship.Metadata node due to the logic of processing and the data of preservation and other back end not
It together, therefore in hardware configuration can different from.Because all service requests require to interact with metadata node, a side
Face, metadata node can become the bottleneck that distributed memory system carries out data access, influence the entirety of distributed memory system
Processing capacity.On the other hand, once metadata node failure, entire distributed memory system are all unavailable.Therefore, how effectively
Management metadata to influence distributed memory system performance minimum as those skilled in the art's urgent problem to be solved.
Summary of the invention
The embodiment of the present invention has been designed to provide a kind of metadata management method and relevant apparatus, by by metadata
Distributed storage avoids metadata node from being managed collectively metadata bring access bottleneck, influences distribution in each memory node
The entirety ability of storage system.
To achieve the goals above, technical solution used in the embodiment of the present invention is as follows:
In a first aspect, the embodiment of the present invention provides a kind of metadata management method, applied in distributed memory system
Client, distributed memory system further include multiple memory nodes, which comprises obtain the mark of file to be stored and deposit
Store up the first object memory node of file to be stored, wherein first object memory node is that the size based on file to be stored is true
Fixed；The metadata of file to be stored is generated according to the mark and first object memory node of file to be stored；According to wait store
The mark of file determines the second target storage node according to preset algorithm；The metadata of file to be stored is stored to the second target
Memory node.
Second aspect, the embodiment of the present invention provides a kind of meta data management device, applied in distributed memory system
Client, distributed memory system further include multiple memory nodes, and described device includes obtaining module, generation module, determining mould
Block and memory module.Wherein, module is obtained, for obtaining the mark of file to be stored and storing the first object of file to be stored
Memory node, wherein first object memory node is the size determination based on file to be stored；Generation module is used for foundation
The mark and first object memory node of file to be stored generate the metadata of file to be stored；Determining module, for according to
The mark of storage file determines the second target storage node according to preset algorithm；Memory module, for by the member of file to be stored
Data are stored to the second target storage node.
The third aspect, the embodiment of the present invention provide a kind of client, and the client includes: one or more processors；
Memory, for storing one or more programs, when one or more of programs are executed by one or more of processors
When, so that one or more of processors realize above-mentioned metadata management method.
Fourth aspect, the embodiment of the present invention provide a kind of computer readable storage medium, are stored thereon with computer program,
The computer program realizes above-mentioned metadata management method when being executed by processor.
Compared with the existing technology, the embodiment of the present invention provides a kind of metadata management method and relevant apparatus, by will be first
Data distribution formula is stored in each memory node, and metadata node is avoided to be managed collectively metadata bring access bottleneck, influences to divide
The entirety ability of cloth storage system.
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below will be to needed in the embodiment attached
Figure is briefly described, it should be understood that the following drawings illustrates only certain embodiments of the present invention, therefore is not construed as pair
The restriction of range for those of ordinary skill in the art without creative efforts, can also be according to this
A little attached drawings obtain other relevant attached drawings.
Fig. 1 shows application scenarios schematic diagram provided in an embodiment of the present invention.
Fig. 2 shows metadata management method flow charts provided in an embodiment of the present invention.
Fig. 3 shows the method flow diagram provided in an embodiment of the present invention for reading file.
Fig. 4 shows the method flow diagram of metadata migration provided in an embodiment of the present invention.
Fig. 5 shows the block diagram of meta data management device provided in an embodiment of the present invention.
Fig. 6 shows the block diagram of client provided in an embodiment of the present invention.
Icon: 10- client；101- memory；102- communication interface；103- processor；104- bus；20- storage section
Point；200- meta data management device；201- obtains module；202- generation module；203- determining module；204- memory module；
205- read module；206- transferring module.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.The present invention being usually described and illustrated herein in the accompanying drawings is implemented
The component of example can be arranged and be designed with a variety of different configurations.
Therefore, the detailed description of the embodiment of the present invention provided in the accompanying drawings is not intended to limit below claimed
The scope of the present invention, but be merely representative of selected embodiment of the invention.Based on the embodiments of the present invention, this field is common
Technical staff's every other embodiment obtained without creative efforts belongs to the model that the present invention protects
It encloses.
It should also be noted that similar label and letter indicate similar terms in following attached drawing, therefore, once a certain Xiang Yi
It is defined in a attached drawing, does not then need that it is further defined and explained in subsequent attached drawing.
In the description of the present invention, it should be noted that if there is the side of the instructions such as term " on ", "lower", "inner", "outside"
Position or positional relationship be based on the orientation or positional relationship shown in the drawings or the invention product using when the side usually put
Position or positional relationship, are merely for convenience of description of the present invention and simplification of the description, rather than the device or member of indication or suggestion meaning
Part must have a particular orientation, be constructed and operated in a specific orientation, therefore be not considered as limiting the invention.
In addition, being only used for distinguishing description if there is term " first ", " second " etc., it is not understood to indicate or imply phase
To importance.
It should be noted that in the absence of conflict, the feature in the embodiment of the present invention can be combined with each other.
Currently, distributed memory system would generally use a metadata node to unify Preservation Metadata.For example, In
In Hadoop distributed file system HDFS (Hadoop Distributed File System, HDFS), by metadata management
Node (also referred to as NameNode) is managed collectively metadata, since all nodes require to interact with metadata management node,
Metadata management node easily becomes the performance bottleneck of entire distributed memory system, leads to entire distributed memory system performance
Decline, on the other hand, there is Single Point of Faliure in metadata management node, once i.e. metadata management node goes wrong, then
Cause entire distributed memory system unavailable, greatly affected the reliability of distributed memory system.
For both of these problems, existing solution is by configuring multiple metadata management nodes, on the one hand by negative
Balanced, entire distributed memory system performance decline caused by reduction metadata management joint behavior bottleneck is carried, on the other hand
Master-slave back-up can be done, the reliability of metadata management node is improved.
But since metadata management node and the memory node of storing data have certain difference to configuration requirement always
It is different, therefore increase distributed memory system extension storage node bring processing complexity.
In order to avoid the place of extension storage node caused by the configuration variance between metadata management node and memory node
The increase of complexity is managed, the scheme generallyd use is: calculating the memory node of storage file to be stored, the program by algorithm
Due to being not necessarily to special Preservation Metadata (relationship i.e. between file to be stored and corresponding memory node), thus, avoid member
Data management node is managed collectively the problem of performance bottleneck and Single Point of Faliure of metadata.But in this scenario, due to storage
The memory node of file to be stored is determined by algorithm, therefore, when any one memory node in distributed memory system
State when changing (from normally to failure, or normal from fault recovery), the file stored in distributed memory system
It requires to re-start arrangement, thus the migration of heap file can be related to, greatly affected entire distributed memory system
Overall performance.
In view of this, inventor is furtherd investigate by the realization principle to current various distributed memory systems, it is right
The realization mechanism of advantage existing for various distributed memory systems and defect behind is found after carrying out analyzing in detail: existing to pass through
The scheme that algorithm determines memory node, abandons metadata completely will lead to distribution when the state of memory node changes and deposit
The reason of performance of storage system is decreased obviously, and analyses in depth its behind is: when the state of memory node changes, in order to full
The data balancing and data reliability of sufficient distributed memory system need to determine the stored file of storage again through algorithm
New memory node, then by stored file migration to new memory node, and the migration of heap file results in distribution
The decline of formula storage system overall performance.
Defect present in the above scheme in the prior art, is that inventor is obtaining after practicing and carefully studying
As a result, therefore, the solution that the discovery procedure of the above problem and the hereinafter embodiment of the present invention are proposed regarding to the issue above
Scheme all should be the contribution that inventor makes the present invention in process of the present invention.Below in conjunction in the embodiment of the present invention
Attached drawing, technical scheme in the embodiment of the invention is clearly and completely described.
Fig. 1 is please referred to, Fig. 1 shows application scenarios schematic diagram provided in an embodiment of the present invention.In Fig. 1, distributed storage
System includes at least one client 10 and multiple memory nodes 20, each client 10 and the equal communication link of multiple memory nodes 20
It connects, after client 10 receives the write operation order for file to be stored, client 10 is primarily based on the size of file to be stored
Determine the first object memory node of storage file to be stored, first object memory node can be any in memory node 20
One or more memory node, secondly, client 10 is generated according to the mark and first object memory node of file to be stored
The metadata of file to be stored, next, client 10 determines the second mesh according to preset algorithm according to the mark of file to be stored
Memory node is marked, the second target storage node can be any one or multiple memory nodes in memory node 20, finally,
Client 10 stores the metadata of file to be stored to the second target storage node.
It should be noted that the state of memory node 20 has normal and abnormal, the memory node 20 in normal condition can
Normally externally to provide storage service, the memory node 20 in abnormality then normally cannot externally provide storage service, lead
Cause memory node 20 the reason of being abnormal including but not limited to memory node 20 to break down, 20 powered-off fault of memory node,
The network failure of memory node 20, can not be communicated to connect with other memory nodes 20 or client 10 etc..
In embodiments of the present invention, multiple memory nodes 20 constitute topological structure, and topological structure changes, can be and open up
The state for flutterring memory node 20 in structure changes, i.e., the state of memory node 20 is from normally becoming abnormal or by exception
Become normal.Topological structure, which changes, is also possible to have new memory node 20 to be added into distributed memory system, for example, working as
It include 4 memory nodes: 1# memory node -4# memory node in preceding topological structure, at this point, 5# memory node is added to distribution
It include 1# memory node -5# memory node in the topological structure after variation in storage system.
Referring to figure 2., Fig. 2 shows metadata management method flow chart provided in an embodiment of the present invention, this method includes
Following steps:
Step S101 obtains the mark of file to be stored and stores the first object memory node of file to be stored, wherein
First object memory node is the size determination based on file to be stored.
In embodiments of the present invention, the mark of file to be stored can with but be not limited to the filename of file to be stored, to
Number of storage file etc. can uniquely indicate the mark of file to be stored, and the mark of file to be stored can use character, character
String or number etc. indicate.
In embodiments of the present invention, first object memory node is used to store the file content of file to be stored, the first mesh
It is more from distributed memory system by default load-balancing algorithm that mark memory node can be the size based on file to be stored
It is determined in a memory node 20, load-balancing algorithm is accounted for according to the residual capacity of multiple memory nodes 20, cpu load, memory
With etc. indexs determine first object memory node, to guarantee that file to be stored can be balancedly stored in distributed memory system
Multiple memory nodes 20 in.For example, load-balancing algorithm can be maximum selection residual capacity and CPU and memory load not
Memory node 20 more than 80%.
It should be noted that load-balancing algorithm can also according to other indexs, such as the network load of memory node 20,
The present invention not limits responsible equalization algorithm and the index item used.
Step S102 generates first number of file to be stored according to the mark and first object memory node of file to be stored
According to.
In embodiments of the present invention, the metadata of file to be stored is used to characterize the mark and first object of file to be stored
Mapping relations between memory node, i.e., in the case where getting the mark of file to be stored, according to the member of file to be stored
Data can determine the first object memory node of the content of storage file to be stored.It, can as a kind of specific embodiment
Using by the mark of file to be stored as keyword, and using first object memory node as value corresponding with keyword, composition
One key-value pair, using the key-value pair as the metadata of file to be stored.It should be noted that key-value pair is the one of metadata
Kind specific implementation form, in fact, metadata can also carry out tissue using relevant database in the form of structuring.
Step S103, the mark according to file to be stored determine the second target storage node according to preset algorithm.
In embodiments of the present invention, preset algorithm can be any distributed algorithm that can guarantee consistency, consistency
Refer to identical input energy return it is identical as a result, i.e. input file to be stored mark it is identical, then by preset algorithm into
The second target storage node that row processing obtains is centainly identical, and preset algorithm can be consistency hash algorithm, such as CRUSH is calculated
Method etc..
Step S104 stores the metadata of file to be stored to the second target storage node.
In embodiments of the present invention, client 10 is after determining the second target storage node, by first number of file to be stored
It is stored according to the second target storage node is sent to, the content of file to be stored is also sent to first object and deposited by client 10
Storage node is stored.
It should be noted that the reliability in order to guarantee the content of file to be stored and the metadata of file to be stored, leads to
The content of file to be stored and the metadata of file to be stored are stored frequently with preset redundancy strategy, preset redundancy
Strategy can be replication policy either correcting and eleting codes strategy.Under same redundant effect, the memory space ratio of replication policy consumption
Correcting and eleting codes strategy is big, and the reliability of replication policy is higher than correcting and eleting codes strategy.User can be determine according to actual needs wait store
The redundancy strategy that the content of file and the metadata of file to be stored use, the redundancy strategy that the two uses can be identical, can also
With difference.It should be noted that under normal conditions, the content of the data volume of the metadata of file to be stored than file to be stored
Data volume is much smaller, also relatively high to the reliability requirement of the metadata of file to be stored, therefore, as a kind of preferably real
Mode is applied, file to be stored is stored in first object memory node by client 10 in the form of correcting and eleting codes, and will be wait store
The metadata of file is stored in the second target storage node in the form of copy.
In embodiments of the present invention, when being stored in the form of correcting and eleting codes when file to be stored, the content of file to be stored is first
Correcting and eleting codes coding is first carried out, corresponding m data piece and n check sheet are obtained, when in m data piece or n check sheet
After one is destroyed, the data for the on piece being destroyed can be restored by remaining m+n-1 piece, to improve data
Reliability.Therefore, client 10 is according to load-balancing algorithm from the memory node 20 for being in normal condition in topological structure
M+n memory node is selected, to store m data piece and n check sheet.For example, distributed memory system shares 10 states
Memory node in normal condition: 1# memory node -10# memory node, the mark of file to be stored are as follows: file.txt entangles
Deleting code coding m value be 4, n value is 2, to the content progress correcting and eleting codes coding of file f ile.txt, obtains 4 data slices and 2
A check sheet, client 10 are determined according to load-balancing algorithm from the normal 1# memory node -10# memory node of state
Totally 6 memory nodes are stored as first object and are saved for 1# memory node -3# memory node, 5# memory node -7# memory node
Point stores 4 data slices and 2 check sheets respectively to this 6 memory nodes.
It should be noted that the value of m and n can be configured in advance according to the demand of actual scene.
In embodiments of the present invention, when the metadata of file to be stored is stored in the form of copy, at this point, text to be stored
The metadata of part can have k copy, and each copy is respectively stored in different k memory nodes 20, when any copy quilt
When destruction, metadata can be normally accessed by any copy from remaining copy.For example, k value is 3, then file to be stored
After metadata generates, client 10 determines 3 memory nodes according to preset algorithm from the memory node 20 in normal condition,
This 3 memory nodes as the second target storage node, by 3 copies of the metadata of file to be stored store respectively to this 3
A memory node.
It should be noted that the value of k can be configured in advance according to the demand of actual scene.
Being stored with copy version for file to be stored is similar with copy version storage with the metadata of file to be stored, wait deposit
The metadata of storage file is stored in the form of correcting and eleting codes to be stored in the form of correcting and eleting codes similar with file to be stored, and details are not described herein again.
In embodiments of the present invention, file to be stored is stored to first object memory node and by file to be stored
Metadata store to the second target storage node, so far, file to be stored by storage with distributed memory system in, user
It also needs to read file stored in distributed memory system, therefore, the embodiment of the invention also includes step S201-
S205。
Referring to figure 3., Fig. 3 show it is provided in an embodiment of the present invention read file method flow diagram, this method include with
Lower step:
Step S201 is received for the read operation request of file of continuing, wherein read operation request includes the mark of file of continuing
Know.
Step S202, the mark according to the file that continues determine that first continues memory node according to preset algorithm.
In embodiments of the present invention, preset algorithm herein must be identical with the preset algorithm in step S103, thus
It can guarantee, for identical file identification, the memory node 20 of the metadata of the storage this document determined is identical.
For example, obtaining the second target storage node according to preset algorithm according to this file identification of file.txt is 1# in step S103
Memory node -5# memory node, the i.e. metadata of storage file.txt are 1# memory node -5# memory node.When needing to read
When file f ile.txt, similarly, step S202 obtains according to preset algorithm according to this file identification of file.txt first
One continues memory node, i.e. the memory node 20 of the metadata of storage file.txt, due to using identical with step S103 pre-
Imputation method, and file identification is file.txt, therefore, first obtained herein continues memory node as 1# memory node -5#
Memory node.
Step S203 reads the metadata of file of continuing from first memory node that continues.
In embodiments of the present invention, be before the metadata for the file that continues when being written in the form of correcting and eleting codes, at this time from
First, which continues, reads data fragmentation in memory node and will read in the memory node 20 of the corresponding data fragmentation of Preservation Metadata
To data fragmentation merge to obtain the metadata for the file that continues, for example, the metadata for the file that continues with 4 data fragmentations and
2 verification fragment storages, 1# memory node -4# memory node store 4 data fragmentations, 5# memory node -6# storage section respectively
Point stores 2 verification fragments respectively, then determines that first continues memory node as 1# memory node -6# memory node in step S202
Afterwards, it reads the data fragmentation in 1# memory node -4# memory node and merges, the metadata for the file that continues must be beaten.
It in embodiments of the present invention, is when being written in the form of copy before the metadata for the file that continues, first continues
Memory node include storage continue file metadata all copies memory node 20, continue in memory node from first
Any memory node 20 may be read into the metadata for the file that continues.For example, the metadata for the file that continues is deposited using 3 copy versions
It is stored in 1# memory node -3# memory node, the first of step S202 determination continues memory node as 1# memory node -3# storage section
Point, can be from the metadata for reading the file that continues in 1# memory node -3# memory node in any memory node 20.
Step S204, the metadata according to the file that continues determine that second continues memory node.
Step S205 reads the content that continues to continue in file from second memory node that continues.
In embodiments of the present invention, the storage for storing the content for the file that continues can be determined according to the metadata for the file that continues
Node 20, i.e., second continues memory node.This second continue memory node be before be written this continue file when, in step S101
In based on the file that continues size determine first object memory node.
In embodiments of the present invention, the topological structure that multiple memory nodes 20 are constituted in by distributed memory system occurs
When variation, in order to be evenly distributed among the metadata of storage file more in distributed memory system in memory node 20,
The metadata of storage file can be migrated, therefore, the embodiment of the invention also includes step S301-S303.
Referring to figure 4., Fig. 4 shows the method flow diagram of metadata migration provided in an embodiment of the present invention, this method packet
Include following steps:
Step S301 obtains the file of storage file when the topological structure that multiple memory nodes are constituted changes
Mark and the initial storage node for storing the metadata of storage file.
Step S302, the file identification according to storage file according to the preset algorithm, again after variation described in
The newest storage section for storing the metadata of storage file is determined in memory node in topological structure in normal condition
Point.
Step S303 migrates the metadata of storage file to newest memory node from initial storage node.
In embodiments of the present invention, carrying out migration to the metadata of storage file includes two kinds of situations: (1) when topology is tied
When the state of any memory node 20 is from normally becoming abnormal in structure, at this time, it may be necessary to again from the topological structure after variation
The newest memory node for storing the metadata of storage file is determined in the memory node 20 of normal condition, for example, working as
There are 9 memory node 20:1# memory node -9# memory nodes in preceding topological structure, 9 memory nodes 20 are in normal shape
State, store 3 storage files altogether in distributed memory system: 1.txt, 2.txt and 3.txt, the metadata of 1.txt is with copy
Form is stored in 1# memory node -3# memory node, and the metadata of 2.txt is stored in 4# memory node -6# memory node,
The metadata of 3.txt is stored in 7# memory node -9# memory node.When 6# memory node is from normally becoming abnormal, according to pre-
Imputation method recalculates the newest memory node of the metadata of storage 2.txt are as follows: 4# memory node, 5# memory node and 7# are deposited
Store up node, then need to migrate the initial storage node (i.e. 4# memory node -6# memory node) for storing the metadata of 2.txt to
The newest memory node (i.e. 4# memory node, 5# memory node and 7# memory node) for storing the metadata of 2.txt, as one kind
Specific embodiment can be from 4# memory node or 5# memory node by 2.txt since 6# memory node has been changed to exception
Metadata migration to 7# memory node.It should be noted that newest memory node can have part phase with initial storage node
Together, can also be entirely different, depending on the preset algorithm used.(2) when there is new memory node 20 to be added to distributed storage system
When system, topological structure can also change, at this time, it may be necessary to be in the storage of normal condition from the topological structure after variation again
The newest memory node for storing the metadata of storage file is determined in node 20, to guarantee the metadata of storage file
It is substantially evenly distributed among each memory node 20, in this scenario, stores the memory node 20 of the content of storage file not
It is affected, therefore, the content of storage file will not migrate, since under normal conditions, the data volume of metadata is remote
Less than the data volume of the content of file, therefore, in this scenario, the data volume for needing to migrate is smaller, can greatly reduce because
Influence of the Data Migration to distributed memory system overall performance.For example, there is 8 memory nodes in current topological structure: 1# is deposited
Node -8# memory node is stored up, stores 3 storage files: 1.txt, 2.txt and 3.txt altogether in distributed memory system,
The metadata of 1.txt is stored in 1# memory node -3# memory node, the metadata of 2.txt is stored in 4# memory node -6# storage
Node, 3.txt metadata be stored in 6# memory node -8# memory node, when 9# memory node be added after, store 1.txt,
The memory node of the metadata of 2.txt and 3.txt can be recalculated according to preset algorithm, after recalculating, the metadata of 1.txt
It is stored in 1# memory node -3# memory node, the metadata of 2.txt is stored in 4# memory node -6# memory node, 3.txt
Metadata is stored in 7# memory node -9# memory node, the metadata migration of the 3.txt for needing to store 6# memory node to 9#
Memory node.
It should be noted that being in abnormality when the state of memory node 20 becomes abnormality from normal condition
Memory node 20 on the content of file that stores can also migrate to the memory node 20 for being in normal condition, as a kind of specific
Embodiment can determine this content migration based on the size of the content of the file stored on the memory node 20 of abnormality
Target storage node 20, then by the content migration of the file stored on the memory node 20 of abnormality to the target store save
On point 20, the content of the file stored on the memory node 20 of abnormality can pass through replication policy or correcting and eleting codes strategy root
It is obtained according to the content of the file stored on the memory node 20 of corresponding normal condition.
Compared with prior art, the embodiment of the present invention has the advantages that
Firstly, client 10 determines the memory node 20 of storage metadata with preset algorithm, so that metadata is distributed
It is stored in each memory node 20 likes, avoids and metadata bring performance bottleneck and single-point event is managed collectively by metadata node
Barrier.
Secondly, metadata is stored in the form of correcting and eleting codes by copy version storage, file to be stored, that is, meet metadata
High reliability request, and make effective use of the file to be stored to the space occupied.
Third, when new memory node 20 is added to distributed memory system, it is only necessary to migrate metadata, not need pair
Storage file is migrated, and the migration for avoiding a large amount of storage files reduces the overall performance of distributed memory system.
Finally, all memory nodes 20 configure identical in distributed memory system, function does not have difference, significantly reduces
The complexity of deployment and the complexity of extension storage node 20, reduce the cost of extension storage node 20.
Based on the same inventive concept, the embodiment of the invention also provides a kind of meta data management devices, referring to figure 5., Fig. 5
Show the block diagram of meta data management device 200 provided in an embodiment of the present invention.Meta data management device 200 includes obtaining
Modulus block 201, generation module 202, determining module 203, memory module 204, read module 205 and transferring module 206.
Module 201 is obtained, for obtaining the mark of file to be stored and storing the first object storage section of file to be stored
Point, wherein first object memory node is the size determination based on file to be stored.
Generation module 202, for the mark and first object memory node generation file to be stored according to file to be stored
Metadata.
Specifically, generation module 202 is used for using the mark of file to be stored as keyword, and first object is stored and is saved
Point forms a key-value pair, using key-value pair as the metadata of file to be stored as value corresponding with keyword.
Determining module 203 determines the second target storage node according to preset algorithm for the mark according to file to be stored.
Memory module 204, for storing the metadata of file to be stored to the second target storage node.
Specifically, memory module 204, for file to be stored to be stored in first object storage section in the form of correcting and eleting codes
Point, and the metadata of file to be stored is stored in the second target storage node in the form of copy.
Read module 205, for receiving the read operation request for being directed to the file that continues, wherein read operation request includes continuing
The mark of file；Mark according to the file that continues determines that first continues memory node according to preset algorithm；It continues storage from first
Node reads the metadata for the file that continues；Metadata according to the file that continues determines that second continues memory node；It continues from second
Memory node reads the content that continues in the file that continues.
Transferring module 206 when the topological structure for constituting when multiple memory nodes changes, obtains storage file
File identification and the storage metadata of storage file initial storage node；According to storage file file identification according to
Preset algorithm is determined from the memory node in the topological structure after variation in normal condition for having stored text again
The newest memory node of the metadata of part；The metadata of storage file is migrated from initial storage node to newest storage and is saved
Point.
It is apparent to those skilled in the art that for convenience and simplicity of description, first number of foregoing description
It according to the specific work process of managing device 200, can refer to corresponding processes in the foregoing method embodiment, details are not described herein.
Fig. 6 is please referred to, Fig. 6 shows the block diagram of client 10 provided in an embodiment of the present invention, and client 10 can
To be any one client in Fig. 1, client 10 be may be, but not limited to, and host, property server etc. can provide and institute
State server or virtual machine have identical function entity server-side.The operating system of client 10 may be, but not limited to,
Windows system, linux system etc..The client 10 includes memory 101, communication interface 102, processor 103 and bus
104, the memory 101, communication interface 102 and processor 103 are connected by bus 104, and processor 103 is for executing storage
The executable module stored in device 101, such as computer program.
Wherein, memory 101 may include high-speed random access memory (RAM:Random Access Memory),
It may further include non-labile memory (non-volatile memory), for example, at least a magnetic disk storage.By extremely
A few communication interface 102 (can be wired or wireless) realizes the client 10 and at least memory node 20 and outside
Store the communication connection between equipment.
Bus 104 can be isa bus, pci bus or eisa bus etc..It is only indicated with a four-headed arrow in Fig. 6, but
It is not offered as only a bus or a type of bus.
Wherein, memory 101, can be in the form of software or firmware (firmware) including at least one for storing program
The software for being stored in memory 101 or being solidificated in the operating system (operating system, OS) of the client 10
Functional module, for example, meta data management device shown in fig. 5 200, meta data management device 200 can be received in processor 103
To after executing instruction, the corresponding program of meta data management device 200 is executed, to realize above-mentioned metadata management method.
The embodiment of the present invention further discloses a kind of computer readable storage medium, is stored thereon with computer program, the meter
Calculation machine program realizes above-mentioned metadata management method when being executed by processor.
In conclusion being applied to distribution the embodiment of the invention provides a kind of metadata management method and relevant apparatus
Client in storage system, distributed memory system further include multiple memory nodes, which comprises obtain text to be stored
The mark of part and the first object memory node for storing file to be stored, wherein first object memory node is based on wait store
What the size of file determined；First number of file to be stored is generated according to the mark and first object memory node of file to be stored
According to；Mark according to file to be stored determines the second target storage node according to preset algorithm；By the metadata of file to be stored
It stores to the second target storage node.Compared with the existing technology, the embodiment of the present invention provides a kind of metadata management method and phase
Device is closed, by the way that meta-data distribution formula is stored in each memory node, metadata node is avoided to be managed collectively metadata bring
Access bottleneck influences the entirety ability of distributed memory system.
The above description is merely a specific embodiment, but scope of protection of the present invention is not limited thereto, any
In the technical scope disclosed by the present invention, any changes or substitutions that can be easily thought of by those familiar with the art, all answers
It is included within the scope of the present invention.Therefore, protection scope of the present invention should be with the scope of protection of the claims
It is quasi-.
Claims (10)
1. a kind of metadata management method, which is characterized in that applied to the client in distributed memory system, the distribution
Storage system further includes multiple memory nodes, which comprises
Obtain the mark of file to be stored and the first object memory node of the storage file to be stored, wherein described first
Target storage node is the size determination based on the file to be stored；
The metadata of the file to be stored is generated according to the mark of the file to be stored and the first object memory node；
Mark according to the file to be stored determines the second target storage node according to preset algorithm；
The metadata of the file to be stored is stored to second target storage node.
2. metadata management method as described in claim 1, which is characterized in that the mark according to the file to be stored
And first object memory node the step of generating the metadata of the file to be stored, includes:
Using the mark of the file to be stored as keyword, and using the first object memory node as with the keyword
Corresponding value forms a key-value pair, using the key-value pair as the metadata of the file to be stored.
3. metadata management method as described in claim 1, which is characterized in that the method also includes:
Receive the read operation request for the file that continues, wherein the read operation request includes the mark of the file that continues；
The mark of file of continuing according to described in determines that first continues memory node according to the preset algorithm；
From described first continue memory node read described in continue the metadata of file；
The metadata of file of continuing according to described in determines that second continues memory node；
From described second continue memory node read described in the content that continues that continues in file.
4. metadata management method as described in claim 1, which is characterized in that the method also includes:
When the topological structure that the multiple memory node is constituted changes, the file identification of storage file and storage is obtained
The initial storage node of the metadata of the storage file；
File identification according to the storage file is according to the preset algorithm, again out of, the topological structure after variation
The newest memory node for storing the metadata of storage file is determined in memory node in normal condition；
The metadata of the storage file is migrated from the initial storage node to the newest memory node.
5. metadata management method as described in claim 1, which is characterized in that the client by the file to be stored with
The form of correcting and eleting codes is stored in the first object memory node, and by the metadata of the file to be stored with the shape of copy
Formula is stored in second target storage node.
6. a kind of meta data management device, which is characterized in that applied to the client in distributed memory system, the distribution
Storage system further includes multiple memory nodes, and described device includes:
Module is obtained, for obtaining the mark of file to be stored and the first object memory node of the storage file to be stored,
Wherein, the first object memory node is the size determination based on the file to be stored；
Generation module, for according to the file to be stored mark and the first object memory node generate it is described wait store
The metadata of file；
Determining module determines the second target storage node according to preset algorithm for the mark according to the file to be stored；
Memory module, for storing the metadata of the file to be stored to second target storage node.
7. meta data management device as claimed in claim 6, which is characterized in that described device further includes read module, described
Read module is used for:
Receive the read operation request for the file that continues, wherein the read operation request includes the mark of the file that continues；
The mark of file of continuing according to described in determines that first continues memory node according to the preset algorithm；
From described first continue memory node read described in continue the metadata of file；
The metadata of file of continuing according to described in determines that second continues memory node；
From described second continue memory node read described in the content that continues that continues in file.
8. meta data management device as claimed in claim 6, which is characterized in that described device further includes transferring module, described
Transferring module is used for:
When the topological structure that the multiple memory node is constituted changes, the file identification of storage file and storage is obtained
The initial storage node of the metadata of the storage file；
File identification according to the storage file is according to the preset algorithm, again out of, the topological structure after variation
The newest memory node for storing the metadata of storage file is determined in memory node in normal condition；
The metadata of the storage file is migrated from the initial storage node to the newest memory node.
9. a kind of client, which is characterized in that the client includes:
One or more processors；
Memory, for storing one or more programs, when one or more of programs are by one or more of processors
When execution, so that one or more of processors realize metadata management method according to any one of claims 1 to 5.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program quilt
Processor realizes metadata management method according to any one of claims 1 to 5 when executing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910709228.0A CN110413694A (en) | 2019-08-01 | 2019-08-01 | Metadata management method and relevant apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910709228.0A CN110413694A (en) | 2019-08-01 | 2019-08-01 | Metadata management method and relevant apparatus |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110413694A true CN110413694A (en) | 2019-11-05 |
Family
ID=68365401
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910709228.0A Pending CN110413694A (en) | 2019-08-01 | 2019-08-01 | Metadata management method and relevant apparatus |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110413694A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110825698A (en) * | 2019-11-07 | 2020-02-21 | 重庆紫光华山智安科技有限公司 | Metadata management method and related device |
| CN111209263A (en) * | 2020-01-14 | 2020-05-29 | 中国建设银行股份有限公司 | Data storage method, device, equipment and storage medium |
| CN111240899A (en) * | 2020-01-10 | 2020-06-05 | 北京百度网讯科技有限公司 | State machine copying method, device, system and storage medium |
| CN111857603A (en) * | 2020-07-31 | 2020-10-30 | 重庆紫光华山智安科技有限公司 | Data processing method and related device |
| CN113051221A (en) * | 2021-03-31 | 2021-06-29 | 网易（杭州）网络有限公司 | Data storage method, device, medium, equipment and distributed file system |
| CN116150280A (en) * | 2023-04-04 | 2023-05-23 | 之江实验室 | Mimicry redis database synchronization method, system, equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102594852A (en) * | 2011-01-04 | 2012-07-18 | 中国移动通信集团公司 | Data access method, node and system |
| CN102821138A (en) * | 2012-07-09 | 2012-12-12 | 广州鼎鼎信息科技有限公司 | Metadata distributed storage method applicable to cloud storage system |
| CN103067461A (en) * | 2012-12-18 | 2013-04-24 | 曙光信息产业(北京)有限公司 | Metadata management system of document and metadata management method thereof |
| CN103310000A (en) * | 2013-06-25 | 2013-09-18 | 曙光信息产业(北京)有限公司 | Metadata management method |
| CN104965845A (en) * | 2014-12-30 | 2015-10-07 | 浙江大华技术股份有限公司 | Small file positioning method and system |
| EP3454231A1 (en) * | 2017-09-07 | 2019-03-13 | Cohesity, Inc. | Remotely mounted file system with stubs |
| US20190114223A1 (en) * | 2017-10-13 | 2019-04-18 | Cisco Technology, Inc. | Accelerating erasure code replication in distributed systems |
-
2019
- 2019-08-01 CN CN201910709228.0A patent/CN110413694A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102594852A (en) * | 2011-01-04 | 2012-07-18 | 中国移动通信集团公司 | Data access method, node and system |
| CN102821138A (en) * | 2012-07-09 | 2012-12-12 | 广州鼎鼎信息科技有限公司 | Metadata distributed storage method applicable to cloud storage system |
| CN103067461A (en) * | 2012-12-18 | 2013-04-24 | 曙光信息产业(北京)有限公司 | Metadata management system of document and metadata management method thereof |
| CN103310000A (en) * | 2013-06-25 | 2013-09-18 | 曙光信息产业(北京)有限公司 | Metadata management method |
| CN104965845A (en) * | 2014-12-30 | 2015-10-07 | 浙江大华技术股份有限公司 | Small file positioning method and system |
| EP3454231A1 (en) * | 2017-09-07 | 2019-03-13 | Cohesity, Inc. | Remotely mounted file system with stubs |
| US20190114223A1 (en) * | 2017-10-13 | 2019-04-18 | Cisco Technology, Inc. | Accelerating erasure code replication in distributed systems |
Non-Patent Citations (2)
| Title |
|---|
| S. ANJANADEVI ET AL: "An efficient dynamic indexing and metadata based storage in cloud environment", 《2014 INTERNATIONAL CONFERENCE ON RECENT TRENDS IN INFORMATION TECHNOLOGY 》 * |
| 杨东日 等: "一种副本复制和纠错码融合的云存储文件系统容错机制", 《清华大学学报（自然科学版）》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110825698A (en) * | 2019-11-07 | 2020-02-21 | 重庆紫光华山智安科技有限公司 | Metadata management method and related device |
| CN110825698B (en) * | 2019-11-07 | 2021-02-09 | 重庆紫光华山智安科技有限公司 | Metadata management method and related device |
| CN111240899A (en) * | 2020-01-10 | 2020-06-05 | 北京百度网讯科技有限公司 | State machine copying method, device, system and storage medium |
| CN111209263A (en) * | 2020-01-14 | 2020-05-29 | 中国建设银行股份有限公司 | Data storage method, device, equipment and storage medium |
| CN111857603A (en) * | 2020-07-31 | 2020-10-30 | 重庆紫光华山智安科技有限公司 | Data processing method and related device |
| CN111857603B (en) * | 2020-07-31 | 2022-12-02 | 重庆紫光华山智安科技有限公司 | Data processing method and related device |
| CN113051221A (en) * | 2021-03-31 | 2021-06-29 | 网易（杭州）网络有限公司 | Data storage method, device, medium, equipment and distributed file system |
| CN113051221B (en) * | 2021-03-31 | 2023-06-30 | 网易（杭州）网络有限公司 | Data storage method, device, medium, equipment and distributed file system |
| CN116150280A (en) * | 2023-04-04 | 2023-05-23 | 之江实验室 | Mimicry redis database synchronization method, system, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11757795B2 (en) | Resolving mediator unavailability | |
| US11716385B2 (en) | Utilizing cloud-based storage systems to support synchronous replication of a dataset | |
| US11086555B1 (en) | Synchronously replicating datasets | |
| US20210397359A1 (en) | Storing Data For Machine Learning And Artificial Intelligence Applications In A Decentralized Storage Network | |
| US11803492B2 (en) | System resource management using time-independent scheduling | |
| US11120152B2 (en) | Dynamic quorum membership changes | |
| US10831614B2 (en) | Visualizing restoration operation granularity for a database | |
| CN110413694A (en) | Metadata management method and relevant apparatus | |
| JP4809040B2 (en) | Storage apparatus and snapshot restore method | |
| US11507597B2 (en) | Data replication to meet a recovery point objective | |
| US20110282834A1 (en) | Continuous replication in a distributed computer system environment | |
| US11789780B1 (en) | Preserving quality-of-service (‘QOS’) to storage system workloads | |
| US11003554B2 (en) | RAID schema for providing metadata protection in a data storage system | |
| WO2022220940A1 (en) | Ensuring timely restoration of an application | |
| US11947968B2 (en) | Efficient use of zone in a storage device | |
| US20200195717A1 (en) | Managing remote replication in storage systems | |
| US10671494B1 (en) | Consistent selection of replicated datasets during storage system recovery | |
| US11334456B1 (en) | Space efficient data protection | |
| CN113641298A (en) | Data storage method, device and computer program product | |
| US20230300143A1 (en) | Adaptive Distributed Backoff Scheme | |
| US20230353635A1 (en) | Replication Utilizing Cloud-Based Storage Systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191105 |