CN103207916A - Metadata processing method and device - Google Patents
Metadata processing method and device Download PDFInfo
- Publication number
- CN103207916A CN103207916A CN201310145878XA CN201310145878A CN103207916A CN 103207916 A CN103207916 A CN 103207916A CN 201310145878X A CN201310145878X A CN 201310145878XA CN 201310145878 A CN201310145878 A CN 201310145878A CN 103207916 A CN103207916 A CN 103207916A
- Authority
- CN
- China
- Prior art keywords
- metadata
- index
- internal memory
- data structure
- generate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention discloses a metadata processing method and device. The method comprises the steps of: storing the to-be-stored metadata into an internal memory, and establishing a data structure in the internal memory when the metadata stored in the internal memory exceeds a default value; and generating an index of the new metadata for the to-be-stored new metadata, and storing the generated index into the data structure. According to the metadata processing method and device, the to-be-stored metadata is stored into the internal memory, the data structure is established in the internal memory when the metadata stored in the internal memory exceeds the default value, the index of the new metadata is generated for the to-be-stored new metadata, and the generated index is stored into the data structure, so that the storage space of the metadata in the internal memory can be saved, and the reliability of a system is guaranteed.
Description
Technical field
The present invention relates to computer realm, particularly, relate to the method and apparatus that a kind of metadata is handled.
Background technology
Usually the method that adopts metadata and data to store respectively in the distributed system, for the metadata in the distributed file system, usually adopt the method for writing many parts to back up, such one is in order to improve system reliability, the 2nd, for reduce when certain metadata node unusual and reduce the calculating of data.If metadata node is because hardware fault may need the long time to carry out fault recovery, after reaching the standard grade again through fault eliminating after a while, the metadata node of then just having reached the standard grade and the node of operate as normal have very large difference, these differences are generally guaranteed by daily record, so, after a unusual node was reached the standard grade, system had very many Journaling File System and can not in time delete.At this moment, may there be two problems: the first, owing to the limited storage space of metadata, the situation of memory space inadequate might appear, in this case, if can not in time handle, then have the possibility of stopping doing business and being engaged in, influenced the operate as normal of system.Second, if the space of metadata is enough big, admit abundant daily record, because the number of metadata is limited, that is to say, the capacity of storing metadata is limited, recover a large amount of journal file might be only corresponding few metadata amount, at this moment, will waste a large amount of time to the recovery of the metadata of fault, if and can not be timely metadata be recovered to finish, the reliability of system is reduced.So, how to make metadata node recover more rapidly to finish fast, be focus and the difficult point problem of current research.
The method that traditional metadata is recovered is exactly log, when daily record dish capacity is unable to hold, will be with the dilatation of daily record dish, or the dilatation daily record dish of break-in, such as, the journal file of metadata is removed; When daily record recovers, or be that daily record is used one by one, or be when using, daily record to be merged, no matter which kind of method all exists release time long, the risk that system reliability is reduced.
It is excessive to take up room in internal memory at metadata in the correlation technique, and the problem that causes system reliability to reduce does not propose effective solution at present as yet.
Summary of the invention
In internal memory, take up room excessive at metadata in the correlation technique, the problem that causes system reliability to reduce, the present invention proposes the method and apparatus that a kind of metadata is handled, and can save the storage space of metadata in internal memory, and guarantee the reliability of system.
Technical scheme of the present invention is achieved in that
According to an aspect of the present invention, proposed the method that a kind of metadata is handled, this method comprises:
The metadata store of needs storages to internal memory, when the metadata of having stored in the internal memory reaches under the situation of preset value, is set up data structure in internal memory;
For the new metadata of needs storages, generate the index of new metadata and with the index stores that generates to data structure.
Wherein, the index stores that generates is further comprised to data structure:
Generate index according to the metadata of having stored in the internal memory, if do not have index in the data structure, then with the index stores that generates to data structure, and the index metadata corresponding that will generate is deleted from internal memory.
And, the index stores that generates is further comprised to data structure:
Generate index according to the metadata of having stored in the internal memory, if had index in the data structure, then the index metadata corresponding that generates is deleted from internal memory.
And, when needs are carried out the operation of metadata correspondence, the index that is recorded in the data structure is read from internal memory, the index of reading is reverted to metadata and carry out the corresponding operation of this metadata.
In addition, if in the process of generator data directory the generation systems fault, then stop to generate index according to new metadata, and after the system failure is eliminated, continue according to new metadata generate index and with the index stores of generation to data structure.
In addition, after the system failure is eliminated, if there are not the difference data in the metadata of memory, then when needs are carried out the operation of metadata correspondence, the index that is recorded in the data structure is read from internal memory, the index of reading is reverted to metadata and carry out the corresponding operation of this metadata.
And, after the system failure is eliminated, if there are the difference data in the metadata of memory, then the difference data carried out synchronously and generate index according to the data synchronously, with the index stores of the data correspondence after synchronously to data structure;
When needs are carried out the operation of metadata correspondence, the index that is recorded in the data structure is read from internal memory, the index of reading is reverted to metadata and carry out the corresponding operation of this metadata.
According to another aspect of the present invention, proposed the device that a kind of metadata is handled, this device comprises:
Memory module, the metadata store that is used for will needing to store when the metadata of having stored in the internal memory reaches under the situation of preset value, is set up data structure to internal memory in internal memory;
Generation module, for the new metadata of needs storages, be used for generating the index of new metadata and with the index stores that generates to data structure.
Wherein, this device further comprises:
Removing module, after the index stores that will generate is to data structure, generation module generates index according to the metadata of having stored in the internal memory, if do not have index in the data structure, then with the index stores that generates to data structure, the index metadata corresponding that removing module will generate is deleted from internal memory.
And after the index stores that will generate was to data structure, generation module generated index according to the metadata of having stored in the internal memory, if had index in the data structure, then removing module is deleted the index metadata corresponding that generates from internal memory.
The present invention by the metadata store that will need to store to internal memory, when the metadata of having stored in the internal memory reaches under the situation of preset value, in internal memory, set up data structure, new metadata for the needs storage, generate the index of new metadata and with the index stores that generates to data structure, can save the storage space of metadata in internal memory, and guarantee the reliability of system.
Description of drawings
In order to be illustrated more clearly in the embodiment of the invention or technical scheme of the prior art, to do to introduce simply to the accompanying drawing of required use among the embodiment below, apparently, accompanying drawing in describing below only is some embodiments of the present invention, for those of ordinary skills, under the prerequisite of not paying creative work, can also obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is the process flow diagram according to the method for the metadata processing of the embodiment of the invention;
Fig. 2 is the block diagram according to the device of the metadata processing of the embodiment of the invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the invention, the technical scheme in the embodiment of the invention is clearly and completely described, obviously, described embodiment only is the present invention's part embodiment, rather than whole embodiment.Based on the embodiment among the present invention, the every other embodiment that those of ordinary skills obtain belongs to the scope of protection of the invention.
According to embodiments of the invention, the method that provides a kind of metadata to handle.
As shown in Figure 1, the method for handling according to the metadata of the embodiment of the invention comprises:
Step S101 to internal memory, when the metadata of having stored in the internal memory reaches under the situation of preset value, sets up data structure with the metadata store of needs storages in internal memory;
Step S103, for the new metadata of needs storages, generate the index of new metadata and with the index stores that generates to data structure.
For example, can in internal memory, set up a B+ tree (corresponding to above-mentioned data structure), the information that in this B+ tree, has comprised whole key words in all leaf node, and sensing contains the pointer of these keyword record, and leaf node can link in proper order according to the arrangement requirement of key word, therefore, B+ tree canned data is exactly the index of metadata.
Wherein, index stores to the processing procedure in the data structure that generates is further comprised: generate index according to the metadata of having stored in the internal memory, if do not have index in the data structure, then with the index stores that generates to data structure, and the index metadata corresponding that will generate is deleted from internal memory.
And, index stores to the processing procedure in the data structure that generates be may further include: generate index according to the metadata of having stored in the internal memory, if had index in the data structure, then the index metadata corresponding that generates is deleted from internal memory.By generating index stores to data structure according to the metadata of having stored, and metadata deleted, can effectively save the space of internal memory, and avoid both having preserved metadata for same object in the internal memory and also preserved index, can further optimize the utilization of memory headroom.
And, when needs are carried out the operation of metadata correspondence, the index that is recorded in the data structure is read from internal memory, the index of reading is reverted to metadata and carry out the corresponding operation of this metadata.
In addition, if in the process of generator data directory the generation systems fault, then stop to generate index according to new metadata, and after the system failure is eliminated, continue according to new metadata generate index and with the index stores of generation to data structure.
In addition, after the system failure is eliminated, if there are not the difference data in the metadata of memory, then when needs are carried out the operation of metadata correspondence, the index that is recorded in the data structure is read from internal memory, the index of reading is reverted to metadata and carry out the corresponding operation of this metadata.
And, after the system failure is eliminated, if there are the difference data in the metadata of memory, then the difference data carried out synchronously and generate index according to the data synchronously, with the index stores of the data correspondence after synchronously to data structure;
When needs are carried out the operation of metadata correspondence, the index that is recorded in the data structure is read from internal memory, the index of reading is reverted to metadata and carry out the corresponding operation of this metadata.
Because the file size of metadata is all smaller, so, repeatedly identical file is operated and the data volume of log, it might be the multiple of meta data file, so, daily record can be compressed, i.e. log not, and just record the minimizing (that is, only preserving the index of metadata) which file guarantees the daily record amount unusually.Therefore, concrete method is:
Step 1 when the metadata fault, can still record the operation of metadata according to the mode of daily record.If unusual metadata node can recover normal in the relatively shorter time, then metadata is still recovered according to the method for using daily record by system.
Step 2, if the daily record amount of the metadata of record reaches certain threshold values (being preset value), then begin to set up a B+ tree in the metadata node that service is provided, when system receives Operation Log, log no longer, but will be recorded in the B+ tree by operand (i.e. the index that is generated by metadata).If object has been recorded in the B+ tree, then the metadata of serving is normally write in success.
Step 3, when the newly-increased metadata operation object of record was set to B+, the data that also will be recorded as daily record were resolved, and the operand of daily record correspondence is recorded in the B+ tree, if exist, then not at record, finish if be recorded to the step of B+ tree, then delete the daily record of having recorded.
Step 4, if journal file is being generated index and depositing in the process of B+ tree, the node of serving breaks down, at this moment, system should stop service, waits for and continues reconstruct B+ tree after unusual node is reached the standard grade again, continues to provide professional.
Step 5 after unusual node is reached the standard grade, for the node of serving, has the disposal route of following three kinds of situations:
Method one, if do not create the B+ tree in the system, then the metadata of illustrative system storage does not also reach threshold values (being preset value), at this moment, only needs that daily record is applied to the meta data server of just having reached the standard grade and gets final product.
Method two, if successfully created the B+ tree in the system, and there is not the existence of difference daily record (being the difference data), the content with the object of the metadata that records on the B+ tree that then can be concurrent is read (read be for the data that guarantee to read be up-to-date) from internal memory from internal memory, and and the new operation mutual exclusion to these data, and the metadata node of reaching the standard grade after sending to data unusually, after the other side writes the metadata store space with the data of object, delete the data item (being the index of metadata) in the B+ tree again.
Method three if created successful B+ tree in the system, and has the existence of difference daily record (being the difference data), at this moment, can be divided into two kinds of situations and handle:
Situation one continues daily record is converted to the B+ tree, waits and changes successfully, handles by method two.
Situation two, residual quantity daily record and B+ tree are recovered to the other side simultaneously, at this moment, need handle the recovery of metadata in residual quantity daily record and the B+ tree in sequence, increase the concurrency of operation, and, owing to reduced the link of daily record to the conversion of B+ tree, daily record simultaneously and B+ tree recover simultaneously, and the fast recovery of systematic comparison is finished, and improve the reliability of system.
According to embodiments of the invention, the device that provides a kind of metadata to handle.
As shown in Figure 2, the device of handling according to the metadata of the embodiment of the invention comprises:
Wherein, this device further comprises:
The removing module (not shown), after the index stores that will generate is to data structure, generation module 22 generates index according to the metadata of having stored in the internal memory, if do not have index in the data structure, then with the index stores that generates to data structure, the index metadata corresponding that removing module will generate is deleted from internal memory.
And after the index stores that will generate was to data structure, generation module 22 generated index according to the metadata of having stored in the internal memory, if had index in the data structure, then removing module is deleted the index metadata corresponding that generates from internal memory.
In sum, by means of technique scheme of the present invention, on the basis of the method that traditional metadata is recovered, disposal route when disclosing metadata fault in a kind of distributed system, by the metadata store that will need to store to internal memory, when the metadata of having stored in the internal memory reaches under the situation of preset value, in internal memory, set up data structure, new metadata for the needs storage, generate the index of new metadata and with the index stores that generates to data structure, can save the storage space of metadata in internal memory, and guarantee the reliability of system.
In addition, the present invention can also make system when breaking down, by log at first, by the time default threshold values, carry out daily record compression (setting up data structure) more automatically, thereby reduce the time that metadata is recovered, by this method, log no longer, saved log space, and, when unusual node is reached the standard grade again, improve the sales volume that recovers, guaranteed the reliability of system.
Below only be preferred embodiment of the present invention, in order to limit the present invention, within the spirit and principles in the present invention not all, any modification of doing, be equal to replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1. the method handled of a metadata is characterized in that described method comprises:
The metadata store of needs storages to internal memory, when the metadata of having stored in the described internal memory reaches under the situation of preset value, is set up data structure in described internal memory;
For the new metadata of needs storages, generate the index of described new metadata and the described index stores that will generate extremely in the described data structure.
2. method according to claim 1 is characterized in that, the described index stores that generates is further comprised to described data structure:
Generate index according to the metadata of having stored in the described internal memory, if do not have described index in the described data structure, then with the index stores that generates to described data structure, and the index metadata corresponding that will generate is deleted from described internal memory.
3. method according to claim 1 is characterized in that, the described index stores that generates is further comprised to described data structure:
Generate index according to the metadata of having stored in the described internal memory, if had described index in the described data structure, then the index metadata corresponding that generates is deleted from described internal memory.
4. method according to claim 1 is characterized in that, when needs are carried out the operation of metadata correspondence, the index that is recorded in the described data structure is read from internal memory, the index of reading is reverted to metadata and carry out the corresponding operation of this metadata.
5. method according to claim 1, it is characterized in that, if generation systems fault in the process of generator data directory, then stop to generate index according to new metadata, and after the system failure is eliminated, continue to generate index and with the index stores that generates extremely in the described data structure according to new metadata.
6. method according to claim 5, it is characterized in that, after the system failure is eliminated, if there are not the difference data in the metadata of described memory, then when needs are carried out the operation of metadata correspondence, the index that is recorded in the described data structure is read from internal memory, the index of reading is reverted to metadata and carry out the corresponding operation of this metadata.
7. method according to claim 5, it is characterized in that, after the system failure is eliminated, if there are the difference data in the metadata of described memory, then the difference data are carried out synchronously and generate index according to the data synchronously, with the index stores of the data correspondence after synchronously to described data structure;
When needs are carried out the operation of metadata correspondence, the index that is recorded in the described data structure is read from internal memory, the index of reading is reverted to metadata and carry out the corresponding operation of this metadata.
8. the device handled of a metadata is characterized in that described device comprises:
Memory module, the metadata store that is used for will needing to store when the metadata of having stored in the described internal memory reaches under the situation of preset value, is set up data structure to internal memory in described internal memory;
Generation module for the new metadata of needs storages, is used for generating the index of described new metadata and the described index stores that will generate described data structure extremely.
9. device according to claim 8 is characterized in that, described device further comprises:
Removing module, after the described index stores that will generate is to the described data structure, described generation module generates index according to the metadata of having stored in the described internal memory, if do not have described index in the described data structure, then with the index stores that generates to described data structure, the index metadata corresponding that described removing module will generate is deleted from described internal memory.
10. device according to claim 8, it is characterized in that, after the described index stores that will generate is to the described data structure, described generation module generates index according to the metadata of having stored in the described internal memory, if had described index in the described data structure, then described removing module is deleted the index metadata corresponding that generates from described internal memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310145878.XA CN103207916B (en) | 2013-04-24 | 2013-04-24 | The method and apparatus of metadata processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310145878.XA CN103207916B (en) | 2013-04-24 | 2013-04-24 | The method and apparatus of metadata processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103207916A true CN103207916A (en) | 2013-07-17 |
| CN103207916B CN103207916B (en) | 2017-09-19 |
Family
ID=48755137
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310145878.XA Active CN103207916B (en) | 2013-04-24 | 2013-04-24 | The method and apparatus of metadata processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103207916B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105488047A (en) * | 2014-09-16 | 2016-04-13 | 华为技术有限公司 | Metadata read-write method and device |
| CN108121303A (en) * | 2016-11-30 | 2018-06-05 | 沈阳中科博微科技股份有限公司 | A kind of log recording method applied to manufacturing equipment statistical analysis process |
| CN108900337A (en) * | 2018-06-29 | 2018-11-27 | 郑州云海信息技术有限公司 | A kind of fault recovery method of Metadata Service, server, client and system |
| CN111435331A (en) * | 2019-01-14 | 2020-07-21 | 杭州宏杉科技股份有限公司 | Data writing method and device for storage volume, electronic equipment and machine-readable storage medium |
| CN113901293A (en) * | 2021-09-30 | 2022-01-07 | 苏州浪潮智能科技有限公司 | Metadata management method, electronic device, and computer-readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007272588A (en) * | 2006-03-31 | 2007-10-18 | Kddi Corp | Information retrieval method |
| CN102483714A (en) * | 2009-07-24 | 2012-05-30 | 苹果公司 | Restore Index Page |
| CN102567427A (en) * | 2010-12-30 | 2012-07-11 | 中国移动通信集团公司 | Method and device for processing object data |
| CN102831240A (en) * | 2012-09-05 | 2012-12-19 | 曙光信息产业(北京)有限公司 | Storage method and storage structure of extensible metadata documents |
| CN102024020B (en) * | 2010-11-04 | 2013-02-06 | 曙光信息产业(北京)有限公司 | Efficient metadata memory access method in distributed file system |
-
2013
- 2013-04-24 CN CN201310145878.XA patent/CN103207916B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007272588A (en) * | 2006-03-31 | 2007-10-18 | Kddi Corp | Information retrieval method |
| CN102483714A (en) * | 2009-07-24 | 2012-05-30 | 苹果公司 | Restore Index Page |
| CN102024020B (en) * | 2010-11-04 | 2013-02-06 | 曙光信息产业(北京)有限公司 | Efficient metadata memory access method in distributed file system |
| CN102567427A (en) * | 2010-12-30 | 2012-07-11 | 中国移动通信集团公司 | Method and device for processing object data |
| CN102831240A (en) * | 2012-09-05 | 2012-12-19 | 曙光信息产业(北京)有限公司 | Storage method and storage structure of extensible metadata documents |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105488047A (en) * | 2014-09-16 | 2016-04-13 | 华为技术有限公司 | Metadata read-write method and device |
| CN105488047B (en) * | 2014-09-16 | 2019-01-18 | 华为技术有限公司 | Metadata reading/writing method and device |
| CN108121303A (en) * | 2016-11-30 | 2018-06-05 | 沈阳中科博微科技股份有限公司 | A kind of log recording method applied to manufacturing equipment statistical analysis process |
| CN108900337A (en) * | 2018-06-29 | 2018-11-27 | 郑州云海信息技术有限公司 | A kind of fault recovery method of Metadata Service, server, client and system |
| CN111435331A (en) * | 2019-01-14 | 2020-07-21 | 杭州宏杉科技股份有限公司 | Data writing method and device for storage volume, electronic equipment and machine-readable storage medium |
| CN113901293A (en) * | 2021-09-30 | 2022-01-07 | 苏州浪潮智能科技有限公司 | Metadata management method, electronic device, and computer-readable storage medium |
| CN113901293B (en) * | 2021-09-30 | 2024-01-16 | 苏州浪潮智能科技有限公司 | Metadata management method, electronic device, and computer-readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103207916B (en) | 2017-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102662992B (en) | Method and device for storing and accessing massive small files | |
| CN103116661B (en) | A kind of data processing method of database | |
| CN101577735B (en) | Method, device and system for taking over fault metadata server | |
| CN102521072B (en) | Virtual tape library equipment and data recovery method | |
| CN101256561B (en) | Method, apparatus and system for storing and accessing database data | |
| CN102135963B (en) | Data transfer method and system | |
| CN101777017B (en) | Rapid recovery method of continuous data protection system | |
| CN101582076A (en) | Data de-duplication method based on data base | |
| US20150193473A1 (en) | Database Storage System based on Optical Disk and Method Using the System | |
| CN107111460A (en) | Use the data de-duplication of block file | |
| CN107391306A (en) | A kind of isomeric data library backup file access pattern method | |
| CN101916290B (en) | Managing method of internal memory database and device | |
| CN102012933A (en) | Distributed file system and method for storing data and providing services by utilizing same | |
| CN103207916A (en) | Metadata processing method and device | |
| CN107957920A (en) | Database backup system | |
| CN103377100B (en) | A kind of data back up method, network node and system | |
| CN103617277A (en) | Method for restoring data table content deleted mistakenly | |
| CN101707633A (en) | Message-oriented middleware persistent message storing method based on file system | |
| CN104965835B (en) | A kind of file read/write method and device of distributed file system | |
| CN107885616A (en) | A kind of mass small documents back-up restoring method based on file system parsing | |
| CN102541691A (en) | Log check point recovery method applied to memory data base OLTP (online transaction processing) | |
| CN104461773A (en) | Backup deduplication method of virtual machine | |
| CN103034592A (en) | Data processing method and device | |
| CN103268270A (en) | Method and device for managing snapshot | |
| CN104199963A (en) | Method and device for HBase data backup and recovery |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220208 Address after: 300450 3 / F, No. 15, Haitai Huake street, Huayuan Industrial Zone (outer ring), Tianjin Binhai New Area, Tianjin Patentee after: Tianjin Zhongke Shuguang Storage Technology Co.,Ltd. Address before: 100193 No. 36 Building, No. 8 Hospital, Wangxi Road, Haidian District, Beijing Patentee before: Dawning Information Industry (Beijing) Co.,Ltd. |