CN102508891B - Consistency method based on discarded multi-metadata server metadata log - Google Patents
Consistency method based on discarded multi-metadata server metadata log Download PDFInfo
- Publication number
- CN102508891B CN102508891B CN 201110328292 CN201110328292A CN102508891B CN 102508891 B CN102508891 B CN 102508891B CN 201110328292 CN201110328292 CN 201110328292 CN 201110328292 A CN201110328292 A CN 201110328292A CN 102508891 B CN102508891 B CN 102508891B
- Authority
- CN
- China
- Prior art keywords
- data server
- request
- daily record
- metadata
- meta data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention provides a consistency method based on a discarded multi-metadata server metadata log, which comprises the following steps: after a master metadata server receives a metadata request, storing the metadata request in a memory, and then, transmitting the metadata request to a slave metadata server; after the slave metadata server receives the transmitted request and stores the transmitted request to the memory, responding to the master metadata server; after the master metadata server receives the response, submitting the request to a logging device; after the request is submitted to the logging device, submitting the request to the slave metadata server; after the slave metadata server receives the submitting request, submitting the submitting request to the logging device, meanwhile, responding to the submitting request, applying to a disc and synchronizing; after the master metadata server receives a submitting request response and the submitting request response is applied to the disc and is synchronized, sending a discarding command to the slave metadata server; and according to a discarding condition, storing a local copy log. Communication mechanisms are introduced in for three times, so that a logging system can be restored by copying a small quantity of log files under the condition that the logging device is damaged, and thereby the restoring time is greatly shortened.
Description
Technical field
The present invention relates to the metadata log approach of multidata device, specifically, relate to a kind of method based on the multivariate data server metadata log consistency that abandons.
Background technology
In distributed file system, there is correlativity between the metadata, this shows that a lot of operations will revise the metadata of several parts simultaneously, and when having only partial data to revise, system is inconsistent, and namely this correlativity is damaged.When whole operation was finished, system transferred to another consistent state from a consistent state.When system was in inconsistent state, affected metadata and relevant data can not correctly be used, even become rubbish.If this problem is not corrected by system, and continue operation, will cause bigger infringement.
For the multivariate data server, each metadata store is in different nodes, and when collapse took place node, the metadata on these nodes will be in an inconsistent state, causes the metadata service unavailable.Therefore, how to guarantee that the metadata consistance on a plurality of meta data servers is the key factor that influences the metadata reliability.
In order to guarantee the consistance of metadata, some distributed file systems have adopted log system.A plurality of relevant metadata operations are encapsulated as affairs.Log system all adopts writes the strategy of using disk after the daily record earlier, even when it takes place to collapse like this, the affairs that are not applied to disk also can guarantee its consistance by using daily record again.Yet when being damaged as if daily record equipment, it is inconsistent to recover to collapse the system that causes by application daily record equipment, traditional method is to use the method for diskcopy, copy is carried out data recover, in the system of TB level and even PB level, this expense is flagrant.
Summary of the invention
The present invention relates to a kind of metadata log approach based on the multidata device that abandons, purpose is the required time of recovery when being reduced in the daily record device damage.
A kind of method based on the multivariate data server metadata log consistency that abandons, after the pivot data server is received metadata request, it is saved in internal memory after, the request of propagation is given from meta data server;
After being saved in internal memory from the meta data server request of receive propagating, reply the pivot data server;
After the pivot data server is received and replied request is submitted to daily record equipment, after finishing, submits to request to give from meta data server;
From meta data server receive submit request to after, be submitted to daily record equipment, reply simultaneously and submit request to and be applied to disk and synchronously;
After the pivot data server is received and is submitted request-reply to, be applied to disk and synchronously after, send and abandon order to from meta data server, and preserve the local replica daily record according to abandoning situation.
Preferably, described basis abandons situation and preserves the process of local replica daily record and be:
If corresponding internal memory and daily record equipment are then reclaimed in the request that abandons of receiving primary copy from copy, and return and abandon acknowledgement command and give primary copy, otherwise be saved on the daily record equipment;
If primary copy is received the acknowledgement command that abandons from copy, then reclaim corresponding internal memory and daily record equipment, otherwise be saved on the daily record equipment.
Preferably, when described daily record equipment was damaged, the daily record that will not abandon sent to the replica node of the daily record equipment of damage.
Preferably, the described record that abandons message from copy uses a hash table to manage, with the transaction number of first number operation as hash key if from copy that transactional applications is intact, check then in this hash table whether these affairs are arranged, if these affairs then do not join these affairs in the hash table; If this affairs are arranged, then send to abandon and reply.
Preferably, described from the copy to when request of abandoning of these affairs, inquire about at first in the hash table whether these affairs are arranged, have then to send and reply, otherwise add in the hash table.
The present invention makes log system under the situation of daily record device damage by introducing three communication mechanisms, can realize recovering by a small amount of copy journal file, very big has reduced the time of recovering.
Description of drawings
Fig. 1 is process flow diagram of the present invention
Embodiment
Technical scheme in the invention specifically describes as follows:
In order to realize the consistance on a plurality of meta data servers, the reliable operation of metadata operation is divided into several stages:
● be saved in internal memory
● write daily record equipment
● write disk
● be synchronized to disk
● preserve the local replica daily record according to abandoning situation
The message communicating flow process of the metadata of many copies service is as shown in Figure 1:
After the pivot data server is received metadata request, it is saved in internal memory after, the request of propagation is given from meta data server; After being saved in internal memory from the meta data server request of receive propagating, reply the pivot data server; After the pivot data server is received and replied request is submitted to daily record equipment, after finishing, submits to request to give from meta data server; From meta data server receive submit request to after, be submitted to daily record equipment, reply simultaneously and submit request to and be applied to disk and synchronously; After the pivot data server is received and is submitted request-reply to, be applied to disk and synchronously after, send and abandon order to from meta data server, and preserve the local replica daily record according to abandoning situation.
If namely receive the request that abandons of primary copy from copy, then reclaim corresponding internal memory and daily record equipment, and return and abandon acknowledgement command and give primary copy, otherwise be saved on the daily record equipment; If primary copy is received the acknowledgement command that abandons from copy, then reclaim corresponding internal memory and daily record equipment, otherwise be saved on the daily record equipment.
By sending three-message, control the metadata operation of many copies metadata.Wherein, spread news and submit to message to guarantee to write daily record earlier after write disk, abandon message control and can not abandon the daily record that is not applied to disk.
By above-mentioned method, when the daily record equipment of system was damaged, the replica node that a daily record that need will not abandon sends to the occurrence log device damage got final product, and need not to copy all data in magnetic disk.
The record that abandons message from copy uses a hash table to manage, the transaction number of metadata operation as hash key, if from copy that transactional applications is intact, is checked then in this hash table whether these affairs are arranged, if these affairs then do not join these affairs in the hash table; If this affairs are arranged, then send to abandon and reply.In like manner, this node is received abandoning when asking of these affairs, inquires about at first in the hash table whether these affairs are arranged, and has then to send and replys, otherwise add in the hash table.Inquiry and insertion action need use lock to come mutual exclusion.
Simultaneously, this message that abandons can arrange dynamically according to system, if the reliability of the daily record equipment of system is better, the message flow that abandons can be removed, and directly uses two stage message communicating, guarantees the conformance requirement of many copies.Be highly susceptible to realizing.
Claims (2)
1. method based on the multivariate data server metadata log consistency that abandons is characterized in that:
After the pivot data server is received metadata request, it is saved in the internal memory of pivot data server after, the request of propagation is given from meta data server;
Receive the request of propagation from meta data server and be saved in behind the internal memory of meta data server, reply the pivot data server;
After the pivot data server is received and replied request is submitted to the daily record equipment of pivot data server, after finishing, submits to request to give from meta data server;
From meta data server receive submit request to after, be submitted to the daily record equipment from meta data server, reply simultaneously and submit request to and be applied to from the disk of meta data server and synchronously;
After the pivot data server is received and is submitted request-reply to, be applied to the disk of pivot data server and synchronously after, send and abandon order to from meta data server, and preserve the local replica daily record according to abandoning situation;
The process that described basis abandons the daily record of situation preservation local replica is:
If corresponding internal memory and daily record equipment are then reclaimed in the request that abandons of receiving primary copy from copy, and return and abandon acknowledgement command and give primary copy, otherwise be saved in from the daily record equipment of meta data server;
If primary copy is received the acknowledgement command that abandons from copy, then reclaim corresponding internal memory and daily record equipment, otherwise be saved on the daily record equipment of pivot data server;
The described record that abandons message from copy uses a hash table to manage, the transaction number of metadata operation as hash key, if from copy that transactional applications is intact, is checked then in this hash table whether these affairs are arranged, if these affairs then do not join these affairs in the hash table; If this affairs are arranged, then send to abandon and reply.
2. the method for claim 1 is characterized in that: described pivot data server and when being damaged from the daily record equipment of meta data server, the daily record that will not abandon sends to the replica node of the daily record equipment of damage.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110328292 CN102508891B (en) | 2011-10-25 | 2011-10-25 | Consistency method based on discarded multi-metadata server metadata log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110328292 CN102508891B (en) | 2011-10-25 | 2011-10-25 | Consistency method based on discarded multi-metadata server metadata log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102508891A CN102508891A (en) | 2012-06-20 |
| CN102508891B true CN102508891B (en) | 2013-08-28 |
Family
ID=46220977
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110328292 Active CN102508891B (en) | 2011-10-25 | 2011-10-25 | Consistency method based on discarded multi-metadata server metadata log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102508891B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103019886B (en) * | 2012-12-11 | 2016-03-30 | 曙光信息产业(北京)有限公司 | The restoration methods of log system in multivariate data server and device |
| CN103049351B (en) * | 2012-12-13 | 2016-06-08 | 曙光信息产业(北京)有限公司 | The log processing method of multivariate data server and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101706805A (en) * | 2009-10-30 | 2010-05-12 | 中国科学院计算技术研究所 | Method and system for storing object |
| CN102024022A (en) * | 2010-11-04 | 2011-04-20 | 曙光信息产业(北京)有限公司 | Method for copying metadata in distributed file system |
| CN102033786A (en) * | 2010-11-04 | 2011-04-27 | 天津曙光计算机产业有限公司 | Method for repairing consistency of copies in object storage system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8935382B2 (en) * | 2009-03-16 | 2015-01-13 | Microsoft Corporation | Flexible logging, such as for a web server |
-
2011
- 2011-10-25 CN CN 201110328292 patent/CN102508891B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101706805A (en) * | 2009-10-30 | 2010-05-12 | 中国科学院计算技术研究所 | Method and system for storing object |
| CN102024022A (en) * | 2010-11-04 | 2011-04-20 | 曙光信息产业(北京)有限公司 | Method for copying metadata in distributed file system |
| CN102033786A (en) * | 2010-11-04 | 2011-04-27 | 天津曙光计算机产业有限公司 | Method for repairing consistency of copies in object storage system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102508891A (en) | 2012-06-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10795911B2 (en) | Apparatus and method for replicating changed-data in source database management system to target database management system in real time | |
| CN103297268B (en) | Based on the distributed data consistency maintenance system and method for P2P technology | |
| TWI777935B (en) | Business processing method, device and system | |
| CN103226502B (en) | A kind of data calamity is for control system and data reconstruction method | |
| CN102098342B (en) | Transaction level-based data synchronizing method, device thereof and system thereof | |
| CN101755257B (en) | Managing the copying of writes from primary storages to secondary storages across different networks | |
| US20150347250A1 (en) | Database management system for providing partial re-synchronization and partial re-synchronization method of using the same | |
| WO2016107220A1 (en) | Remote replication method and apparatus based on duplicated data deletion | |
| CN103548011A (en) | Asynchronous replication in a distributed storage environment | |
| US8255360B1 (en) | Synchronization of database changes among multiple devices | |
| WO2016026306A1 (en) | Data backup method, system, node and computer storage media | |
| CN101809558A (en) | System and method for remote asynchronous data replication | |
| CN102368267A (en) | Method for keeping consistency of copies in distributed system | |
| GB2472484A (en) | Error reduction in archiving of electronic documents using two phase commit process | |
| CN112988883A (en) | Database data synchronization method and device and storage medium | |
| US7197519B2 (en) | Database system including center server and local servers | |
| US8612799B2 (en) | Method and apparatus of backing up subversion repository | |
| JP2011210107A (en) | Message queue management system, lock server, message queue management method, and message queue management program | |
| CN102508891B (en) | Consistency method based on discarded multi-metadata server metadata log | |
| JP2006323663A (en) | Information processing system, replication method, and difference information holding device and program | |
| CN101841425B (en) | Network backup method, device and system without proxy | |
| US20110246424A1 (en) | Automated relocation of in-use multi-site protected data storage | |
| WO2013091183A1 (en) | Method and device for key-value pair operation | |
| US8799211B1 (en) | Cascaded replication system with remote site resynchronization after intermediate site failure | |
| CN103092533A (en) | Method and system for data remote synchronization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220727 Address after: 100193 No. 36 Building, No. 8 Hospital, Wangxi Road, Haidian District, Beijing Patentee after: Dawning Information Industry (Beijing) Co.,Ltd. Patentee after: DAWNING INFORMATION INDUSTRY Co.,Ltd. Address before: 100084 Beijing Haidian District City Mill Street No. 64 Patentee before: Dawning Information Industry (Beijing) Co.,Ltd. |