CN103049351B - The log processing method of multivariate data server and device - Google Patents
The log processing method of multivariate data server and device Download PDFInfo
- Publication number
- CN103049351B CN103049351B CN201210541441.3A CN201210541441A CN103049351B CN 103049351 B CN103049351 B CN 103049351B CN 201210541441 A CN201210541441 A CN 201210541441A CN 103049351 B CN103049351 B CN 103049351B
- Authority
- CN
- China
- Prior art keywords
- copy
- daily record
- primary copy
- metadata
- primary
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses the log processing method of a kind of multivariate data server, including: after primary copy preserves daily record in internal memory, Log Shipping is given from copy; Primary copy and be recorded respectively from copy logging device and application metadata disk operation; And when primary copy and complete from copy application metadata disk operation after, send application end to the other side. Pass through technical scheme, it is possible to increase the availability of system and reliability.
Description
Technical field
Present invention relates in general to multivariate data server, more particularly, to log processing method and the device of multivariate data server.
Background technology
In order to ensure concordance and the atomicity of multivariate data server, have employed log system and carried out the data storage of metadata. In order to ensure the atomicity of data, namely the rule that modern log system adopts first write the strategy of write magnetic dish after daily record. In order to prevent metadata from losing, only all of copy could be deleted daily record, therefore, how immediately reclaim logging device, be a very important problem in multivariate data server after all completing application.
Existing log system such as hadoop file system (HDFS) adopts timing to apply daily record, after application terminates, is reclaimed by logging device. Under this approach, when primary copy (providing the copy of data, services in meta data server group) occurs to lose efficacy, owing to (meta data server not providing service from copy, but accepting the copy of data of primary copy transmission) node needs to be undertaken just being provided that service by the daily record of logging device, cannot take over (during metadata primary copy node failure in time, replace primary copy server to provide from replica node and change data, services) Metadata Service, therefore, availability during inefficacy is poor. Meanwhile, if the logging device of primary copy damages, if these daily records are not transferred to from copy, the meta data file corresponding by abandoning this partial log, therefore, the Reliability comparotive of system is poor.
For the problem in correlation technique, effective solution is not yet proposed at present.
Summary of the invention
For the problem in correlation technique, the present invention proposes log processing method and the device of multivariate data server, it is possible to solve the problem that availability and reliability in prior art are poor.
According to an aspect of the invention, it is provided the log processing method of a kind of multivariate data server, including: after primary copy preserves daily record in internal memory, Log Shipping is given from copy; Primary copy and be recorded respectively from copy logging device and application metadata disk operation; And when primary copy and complete from copy application metadata disk operation after, send application end to the other side.
Preferably, primary copy utilizes internet message to transmit described daily record to from copy.
Preferably, receiving from after the application end of copy at primary copy, acknowledged client end.
Preferably, running background one reclaims daily record thread, when receiving primary copy and the application from copy transmission receives message, reclaims daily record thread and starts reclaimer operation.
Preferably, starting reclaimer operation and include: whether regular check exists recuperable daily record, if existed, carrying out reclaimer operation.
According to a further aspect in the invention, it is provided that the log processing device of a kind of multivariate data server, including delivery module, for after primary copy preserves daily record in internal memory, make primary copy by Log Shipping to from copy; Record and application module, for making primary copy and being recorded logging device and the operation of application metadata disk from copy respectively; And sending module, for when primary copy and complete from copy application metadata disk operation after, make primary copy and send application end from copy to the other side.
Preferably, delivery module utilizes internet message to transmit daily record to described from copy from primary copy.
Preferably, this device also includes responder module, for receiving from after the described application end of copy at primary copy, and acknowledged client end.
Preferably, this device also includes recycling module, reclaims daily record thread for running background one, when receiving primary copy and the application from copy transmission receives message, reclaims daily record thread and starts reclaimer operation.
Preferably, whether recycling module regular check exists recuperable daily record, if existed, carries out reclaimer operation.
By the present invention in that primary copy in time by Log Shipping to from copy, make when primary copy occurs to lose efficacy, from copy can the Metadata Service of adapter primary copy, and due to just acknowledged client end after the equal Preservation Metadata daily record of principal and subordinate's copy, so will not the metadata of lost customer end.
Accompanying drawing explanation
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, the accompanying drawing used required in embodiment will be briefly described below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the premise not paying creative work, it is also possible to obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is the flow chart of the log processing method of multivariate data server according to embodiments of the present invention;
Fig. 2 is the block diagram of the log processing device of multivariate data server according to embodiments of the present invention; And
Fig. 3 is the diagram being shown specifically log processing device according to embodiments of the present invention.
Detailed description of the invention
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is only a part of embodiment of the present invention, rather than whole embodiments. Based on the embodiment in the present invention, the every other embodiment that those of ordinary skill in the art obtain, broadly fall into the scope of protection of the invention.
Fig. 1 is the flow chart of the log processing method of multivariate data server according to embodiments of the present invention.
With reference to Fig. 1, include according to the log processing method of the multivariate data server of the present invention: S102, after primary copy preserves daily record in internal memory, Log Shipping is given from copy; S104, primary copy and be recorded respectively from copy logging device and application metadata disk operation;And S106, when primary copy and complete from copy application metadata disk operation after, send application end to the other side.
Here, primary copy can utilize internet message to transmit daily record to from copy. Certainly, it should be appreciated by those skilled in the art that other message modes can also be adopted to transmit daily record.
In an embodiment of the present invention, primary copy is receiving from after the application end of copy, acknowledged client end (that is, the node of meta data server). This makes it possible to when primary copy lost efficacy, not acknowledged client end so that client can be retransmitted from copy, thus without the metadata of lost customer end.
It addition, running background one reclaims daily record thread, when receiving primary copy and the application from copy transmission receives message, reclaim daily record thread and start reclaimer operation. Wherein, starting reclaimer operation and include: whether regular check exists recuperable daily record, if existed, carrying out reclaimer operation.
Fig. 2 is the block diagram of the log processing device of multivariate data server according to embodiments of the present invention.
With reference to Fig. 2, include according to the log processing device of the multivariate data server of the present invention: delivery module 22, for after primary copy preserves daily record in internal memory, make primary copy by Log Shipping to from copy; Record and application module 24, for making primary copy and being recorded logging device and the operation of application metadata disk from copy respectively; And sending module 26, for when primary copy and complete from copy application metadata disk operation after, make primary copy and send application end from copy to the other side.
Here, delivery module 22 can utilize internet message to transmit daily record from primary copy to from copy.
Additionally, this device can also include responder module, for receiving from after the application end of copy at primary copy, acknowledged client end.
It addition, this device can also include recycling module, reclaim daily record thread for running background one, when receiving primary copy and the application from copy transmission receives message, reclaim daily record thread and start reclaimer operation. Whether recycling module regular check exists recuperable daily record, if existed, carries out reclaimer operation.
Fig. 3 is the diagram being shown specifically log processing device according to embodiments of the present invention.
With reference to Fig. 3, metadata copy adopts message (such as, internet message) to transmit daily record. After primary copy receives daily record, preserve after daily record in primary copy internal memory, daily record is passed to from copy.
Then, primary copy and be recorded respectively from copy logging device and application metadata disk operation. When either party applies after the operation of metadata disk terminates, notify this result to the other side. When primary copy receives after the recovery daily record response of copy, acknowledged client end (node of meta data server).
Additionally, after the application end receiving all copies, reclaim daily record. Specifically, start one on backstage and reclaim daily record thread, when receiving primary copy and the application from copy transmission receives message, start reclaimer operation, be specially whether regular check has callable daily record, if passable, then carry out recovery action.
Although it should be appreciated by those skilled in the art that and present invention employs the strategy that timing is reclaimed, it is also possible to adopt other modes, such as triggering mode.
By adopting this method of the present invention, it is possible to make daily record be passed to from copy in time, so, when primary copy occurs to lose efficacy, the daily record of primary copy transmission only need to be applied from copy, namely can the Metadata Service of adapter primary copy.
Additionally, due to after principal and subordinate's copy all saves metadata daily record, primary copy is acknowledged client end. Therefore, when primary copy lost efficacy, not acknowledged client end, client can be retransmitted from copy so that will not the metadata of lost customer end.
It addition, the present invention additionally uses asynchronous garbage collection take-back strategy, greatly simplifie the reclaim mechanism of whole log system. Owing to being that timing carries out, it is possible to the data that this stage produces are uniformly carried out recovery, are conducive to system to adopt the optimisation strategy such as data aggregate, improve performance.
In sum, by means of the technique scheme of the present invention, by ensure the daily record between principal and subordinate's copy be all saved in internal memory after just acknowledged client end carry out repeatedly message communicating and notify that the application of other replica node terminates to make the log system of the present invention compared with prior art improve availability and reliability. And by the asynchronous recovery daily record in backstage, adopt data aggregation technique, the daily record of a time period is reclaimed, further increases the performance of log system.
The foregoing is only presently preferred embodiments of the present invention, not in order to limit the present invention, all within the spirit and principles in the present invention, any amendment of making, equivalent replacement, improvement etc., should be included within protection scope of the present invention.
Claims (8)
1. the log processing method of a multivariate data server, it is characterised in that described method includes:
Primary copy preserves after daily record in internal memory, is given by described Log Shipping from copy;
Metadata is successively write corresponding logging device and disk by described primary copy, meanwhile, described also successively described metadata is write corresponding logging device and disk from copy; And
After described primary copy and the described operation completed disk corresponding for the write of described metadata from copy, send application end to the other side;
Then, described primary copy receive described after the described application end of copy, acknowledged client end.
2. method according to claim 1, it is characterised in that described primary copy utilizes internet message to transmit described daily record to described from copy.
3. method according to claim 1, it is characterised in that running background one reclaim daily record thread, when receive described primary copy and described from copy send described application end time, described recovery daily record thread start reclaimer operation.
4. method according to claim 3, it is characterised in that starting reclaimer operation and include: whether regular check exists recuperable daily record, if existed, carrying out described reclaimer operation.
5. the log processing device of a multivariate data server, it is characterised in that described device includes:
Delivery module, for after primary copy preserves daily record in internal memory, makes described primary copy by described Log Shipping to from copy;
Writing module, successively writes the operation of corresponding logging device and disk for making described primary copy carry out by metadata, also makes the described operation carrying out successively writing described metadata corresponding logging device and disk from copy simultaneously; And
Sending module, for after described primary copy and the described operation completed disk corresponding for the write of described metadata from copy, makes described primary copy and described sends application end from copy to the other side;
Responder module, for described primary copy receive described after the described application end of copy, acknowledged client end.
6. device according to claim 5, it is characterised in that described delivery module utilizes internet message to transmit described daily record to described from copy from described primary copy.
7. device according to claim 5, it is characterized in that, described device also includes recycling module, reclaims daily record thread for running background one, when receive described primary copy and described from copy send described application end time, described recovery daily record thread start reclaimer operation.
8. device according to claim 7, it is characterised in that whether described recycling module regular check exists recuperable daily record, if existed, carries out described reclaimer operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210541441.3A CN103049351B (en) | 2012-12-13 | 2012-12-13 | The log processing method of multivariate data server and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210541441.3A CN103049351B (en) | 2012-12-13 | 2012-12-13 | The log processing method of multivariate data server and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103049351A CN103049351A (en) | 2013-04-17 |
| CN103049351B true CN103049351B (en) | 2016-06-08 |
Family
ID=48062000
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210541441.3A Active CN103049351B (en) | 2012-12-13 | 2012-12-13 | The log processing method of multivariate data server and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103049351B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103516549B (en) * | 2013-09-27 | 2018-03-27 | 浪潮电子信息产业股份有限公司 | A kind of file system metadata log mechanism based on shared object storage |
| CN103544081B (en) * | 2013-10-23 | 2015-08-12 | 曙光信息产业(北京)有限公司 | The management method of double base data server and device |
| CN111190874B (en) * | 2019-12-27 | 2023-10-03 | 天津中科曙光存储科技有限公司 | High-reliability data log module for distributed storage and construction method thereof |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6205449B1 (en) * | 1998-03-20 | 2001-03-20 | Lucent Technologies, Inc. | System and method for providing hot spare redundancy and recovery for a very large database management system |
| CN102368267A (en) * | 2011-10-25 | 2012-03-07 | 曙光信息产业(北京)有限公司 | Method for keeping consistency of copies in distributed system |
| CN102508891A (en) * | 2011-10-25 | 2012-06-20 | 曙光信息产业(北京)有限公司 | Consistency method based on discarded multi-metadata server metadata log |
-
2012
- 2012-12-13 CN CN201210541441.3A patent/CN103049351B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6205449B1 (en) * | 1998-03-20 | 2001-03-20 | Lucent Technologies, Inc. | System and method for providing hot spare redundancy and recovery for a very large database management system |
| CN102368267A (en) * | 2011-10-25 | 2012-03-07 | 曙光信息产业(北京)有限公司 | Method for keeping consistency of copies in distributed system |
| CN102508891A (en) * | 2011-10-25 | 2012-06-20 | 曙光信息产业(北京)有限公司 | Consistency method based on discarded multi-metadata server metadata log |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103049351A (en) | 2013-04-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10831741B2 (en) | Log-shipping data replication with early log record fetching | |
| CN111200624B (en) | Cross-gatekeeper data transmission method and system | |
| CN101997823B (en) | Distributed file system and data access method thereof | |
| US9053123B2 (en) | Mirroring file data | |
| US11868324B2 (en) | Remote durable logging for journaling file systems | |
| CN102955720B (en) | A kind of method improving ext file system stability | |
| US8195613B2 (en) | Transactional archiving of an electronic document | |
| US11003621B2 (en) | Scalable enterprise content management | |
| CN103942252B (en) | A kind of method and system for recovering data | |
| US20120303761A1 (en) | Breakpoint continuous transmission method | |
| US20160132410A1 (en) | Kernel state and user state data exchange method for disaster recovery of virtual container system | |
| CN104092717A (en) | Message processing method and system, message destination end device and distributed message system | |
| CN101216794A (en) | Improved Server-Less backup method, memory apparatus and backup server | |
| CN103049351B (en) | The log processing method of multivariate data server and device | |
| CN106293954A (en) | A kind of High Availabitity service management based on distributed lock | |
| CN104965835B (en) | A kind of file read/write method and device of distributed file system | |
| US20160139996A1 (en) | Methods for providing unified storage for backup and disaster recovery and devices thereof | |
| CN109165112A (en) | A kind of fault recovery method, system and the associated component of metadata cluster | |
| CN108205482A (en) | File mount restoration methods | |
| CN106843760A (en) | It is a kind of based on the asynchronous remote copy system deleted and method again | |
| CN103530203A (en) | System and method for remote backups of remote sensing satellite data | |
| CN102523205A (en) | Determination method and device for content checksum | |
| CN116633948A (en) | Wide area network data copying and transmitting method and device | |
| TW201308946A (en) | Method for resuming downloading data from interruptive-point | |
| CN102868753B (en) | Adaptive null response restoration methods and device after content-based distributing network optimization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220721 Address after: 100089 building 36, courtyard 8, Dongbeiwang West Road, Haidian District, Beijing Patentee after: Dawning Information Industry (Beijing) Co.,Ltd. Patentee after: DAWNING INFORMATION INDUSTRY Co.,Ltd. Address before: 100193 No. 36 Building, No. 8 Hospital, Wangxi Road, Haidian District, Beijing Patentee before: Dawning Information Industry (Beijing) Co.,Ltd. |