CN103049351A - Log processing method and device of multivariate data server - Google Patents
Log processing method and device of multivariate data server Download PDFInfo
- Publication number
- CN103049351A CN103049351A CN2012105414413A CN201210541441A CN103049351A CN 103049351 A CN103049351 A CN 103049351A CN 2012105414413 A CN2012105414413 A CN 2012105414413A CN 201210541441 A CN201210541441 A CN 201210541441A CN 103049351 A CN103049351 A CN 103049351A
- Authority
- CN
- China
- Prior art keywords
- copy
- daily record
- primary copy
- log
- data server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a log processing method of a multivariate data server. The log processing method of the multivariate data server includes that a log is transmitted to a secondary copy after a primary copy saves the log in an internal memory; the primary copy and the secondary copy respectively perform operation of logging equipment and metadata disc application; and an application ending message is sent to an opposite side after the primary copy and the secondary copy complete the operation of the metadata disc application. According to the technical scheme of the log processing method and a device of the multivariate data server, the availability and reliability of the system can be improved.
Description
Technical field
Present invention relates in general to the multivariate data server, more specifically, relate to log processing method and the device of multivariate data server.
Background technology
For consistance and the atomicity that guarantees the multivariate data server, adopted log system to finish the data storage of metadata.In order to guarantee the atomicity of data, the rule that modern log system adopts is namely write first the strategy of writing disk after the daily record.In order to prevent that metadata from losing, only have all copies all to finish after the application, could delete daily record, therefore, how immediately to reclaim daily record equipment, be very important problem in the multivariate data server.
Existing log system such as hadoop file system (HDFS) adopt regularly uses daily record, after using end, daily record equipment is reclaimed.Under this method, when primary copy (copy of data, services is provided in the meta data server group) occurs to lose efficacy, owing to (do not provide service the meta data server from copy, but accept the copy of the data that primary copy transmits) node need to carry out the daily record of daily record equipment just can provide service, can't in time take over (during metadata primary copy node failure, replace the primary copy server to provide from replica node and change data, services) Metadata Service, therefore, the availability during inefficacy is poor.Simultaneously, if during the daily record device damage of primary copy, if these daily records do not pass to from copy, will abandon meta data file corresponding to this part daily record, therefore, the reliability ratio of system is relatively poor.
For the problem in the correlation technique, effective solution is proposed not yet at present.
Summary of the invention
For the problem in the correlation technique, the present invention proposes log processing method and the device of multivariate data server, can solve availability and the relatively poor problem of reliability in the prior art.
According to an aspect of the present invention, provide a kind of log processing method of multivariate data server, having comprised: primary copy is preserved daily record in internal memory after, Log Shipping has been given from copy; Primary copy and carry out respectively the operation of log equipment and apply metadata disk from copy; And when primary copy with finish the operation of apply metadata disk from copy after, send the application end to the other side.
Preferably, primary copy utilizes internet message to transmit described daily record from copy.
Preferably, receiving after the application end of copy the acknowledged client end at primary copy.
Preferably, one of running background reclaims the daily record thread, when the application receipt message that receives primary copy and send from copy, reclaims the daily record thread and starts reclaimer operation.
Preferably, start reclaimer operation and comprise: whether quantitative check exists recuperable daily record, if exist then carry out reclaimer operation.
According to a further aspect in the invention, provide a kind of log processing device of multivariate data server, having comprised: delivery module, be used at primary copy after internal memory is preserved daily record, make primary copy with Log Shipping to from copy; Record and application module are used for making primary copy and carrying out respectively the operation of log equipment and apply metadata disk from copy; And sending module, be used for when primary copy with finish the operation of apply metadata disk from copy after, make primary copy and send the application end from copy to the other side.
Preferably, delivery module utilizes internet message to transmit daily record to described from copy from primary copy.
Preferably, this device also comprises responder module, is used for receiving after the described application end of copy the acknowledged client end at primary copy.
Preferably, this device also comprises recycling module, is used for one of running background and reclaims the daily record thread, when the application receipt message that receives primary copy and send from copy, reclaims the daily record thread and starts reclaimer operation.
Preferably, whether the recycling module quantitative check exists recuperable daily record, if exist then carry out reclaimer operation.
The present invention is by in time giving Log Shipping from copy primary copy, so that when primary copy occurs to lose efficacy, can take over the Metadata Service of primary copy from copy, and since after the equal Preservation Metadata daily record of principal and subordinate's copy acknowledged client end, so metadata that can the lost customer end.
Description of drawings
In order to be illustrated more clearly in the embodiment of the invention or technical scheme of the prior art, the below will do to introduce simply to the accompanying drawing of required use among the embodiment, apparently, accompanying drawing in the following describes only is some embodiments of the present invention, for those of ordinary skills, under the prerequisite of not paying creative work, can also obtain according to these accompanying drawings other accompanying drawing.
Fig. 1 is the process flow diagram according to the log processing method of the multivariate data server of the embodiment of the invention;
Fig. 2 is the block diagram according to the log processing device of the multivariate data server of the embodiment of the invention; And
Fig. 3 is the diagram that is shown specifically according to the log processing device of the embodiment of the invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the invention, the technical scheme in the embodiment of the invention is clearly and completely described, obviously, described embodiment only is the present invention's part embodiment, rather than whole embodiment.Based on the embodiment among the present invention, the every other embodiment that those of ordinary skills obtain belongs to the scope of protection of the invention.
Fig. 1 is the process flow diagram according to the log processing method of the multivariate data server of the embodiment of the invention.
With reference to Fig. 1, comprise according to the log processing method of multivariate data server of the present invention: S102, primary copy is preserved daily record in internal memory after, give Log Shipping from copy; S104, primary copy and carry out respectively the operation of log equipment and apply metadata disk from copy; And S106, when primary copy with finish the operation of apply metadata disk from copy after, send the application end to the other side.
Here, primary copy can utilize internet message to transmitting daily record from copy.Certainly, it should be appreciated by those skilled in the art that and to adopt other message modes to transmit daily record.
In an embodiment of the present invention, primary copy is receiving after the application end of copy, acknowledged client end (that is, the node of meta data server).So just can be when primary copy lose efficacy, acknowledged client end not, so that client can retransmit from copy, metadata that therefore can the lost customer end.
In addition, one of running background reclaims the daily record thread, when the application receipt message that receives primary copy and send from copy, reclaims the daily record thread and starts reclaimer operation.Wherein, start reclaimer operation and comprise: whether quantitative check exists recuperable daily record, if exist then carry out reclaimer operation.
Fig. 2 is the block diagram according to the log processing device of the multivariate data server of the embodiment of the invention.
With reference to Fig. 2, comprise according to the log processing device of multivariate data server of the present invention: delivery module 22, be used at primary copy after internal memory is preserved daily record, make primary copy with Log Shipping to from copy; Record and application module 24 are used for making primary copy and carrying out respectively the operation of log equipment and apply metadata disk from copy; And sending module 26, be used for when primary copy with finish the operation of apply metadata disk from copy after, make primary copy and send the application end from copy to the other side.
Here, delivery module 22 can utilize internet message from primary copy to transmitting daily record from copy.
In addition, this device can also comprise responder module, is used for receiving after the application end of copy the acknowledged client end at primary copy.
In addition, this device can also comprise recycling module, is used for one of running background and reclaims the daily record thread, when the application receipt message that receives primary copy and send from copy, reclaims the daily record thread and starts reclaimer operation.Whether the recycling module quantitative check exists recuperable daily record, if exist then carry out reclaimer operation.
Fig. 3 is the diagram that is shown specifically according to the log processing device of the embodiment of the invention.
With reference to Fig. 3, the metadata copy adopts message (for example, internet message) to transmit daily record.After primary copy is received daily record, in the primary copy internal memory, preserve after the daily record, daily record is passed to from copy.
Then, primary copy and carry out respectively the operation of log equipment and apply metadata disk from copy.After the EO of either party apply metadata disk, notify this result to the other side.When primary copy receive reply from the recovery daily record of copy after, acknowledged client end (node of meta data server).
In addition, after the application end of receiving all copies, reclaim daily record.Whether specifically, start one on the backstage and reclaim the daily record thread, when the application receipt message that receives primary copy and send from copy, start reclaimer operation, being specially quantitative check has callable daily record, if can, then reclaim action.
Adopt the strategy that regularly reclaims although it should be appreciated by those skilled in the art that the present invention, can also adopt other modes, such as triggering mode.
By adopting this method of the present invention, can like this, when primary copy occurs to lose efficacy, only need use the daily record that primary copy transmits from copy so that daily record is in time passed to from copy, namely can take over the Metadata Service of primary copy.
In addition, because after principal and subordinate's copy had all been preserved the metadata daily record, primary copy is the acknowledged client end.Therefore, when primary copy lost efficacy, acknowledged client end not, client can be retransmitted from copy, so that metadata that can the lost customer end.
In addition, the present invention has also adopted the asynchronous garbage collection take-back strategy, has greatly simplified the reclaim mechanism of whole log system.Owing to be regularly to carry out, the data unification that this one-phase produces can be reclaimed, be conducive to system and adopt the optimisation strategy such as data aggregate, improve performance.
In sum, by means of technique scheme of the present invention, by guaranteeing after the daily record between principal and subordinate's copy all is saved in internal memory just the acknowledged client end and carrying out repeatedly message communicating and notify other replica node to use to finish so that log system of the present invention has compared with prior art improved availability and reliability.And by the asynchronous recovery daily record in backstage, adopt data aggregation technique, the daily record of a time period is reclaimed, further improved the performance of log system.
The above only is preferred embodiment of the present invention, and is in order to limit the present invention, within the spirit and principles in the present invention not all, any modification of doing, is equal to replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1. the log processing method of a multivariate data server is characterized in that, described method comprises:
Primary copy is preserved in internal memory after the daily record, and described Log Shipping is given from copy;
Described primary copy and the described operation of carrying out respectively log equipment and apply metadata disk from copy; And
After described primary copy and the described operation of finishing the apply metadata disk from copy, send the application end to the other side.
2. method according to claim 1 is characterized in that, described primary copy utilizes internet message to transmit described daily record to described from copy.
3. method according to claim 1 is characterized in that, receiving at described primary copy described after the described application end of copy, the acknowledged client end.
4. method according to claim 1 is characterized in that, one of running background reclaims the daily record thread, and when receiving described primary copy and the described application receipt message that sends from copy, described recovery daily record thread starts reclaimer operation.
5. method according to claim 4 is characterized in that, start reclaimer operation and comprise: whether quantitative check exists recuperable daily record, if exist then carry out reclaimer operation.
6. the log processing device of a multivariate data server is characterized in that, described device comprises:
Delivery module is used at primary copy after internal memory is preserved daily record, make described primary copy with described Log Shipping to from copy;
Record and application module are used for making described primary copy and the described operation of carrying out respectively log equipment and apply metadata disk from copy; And
Sending module is used for making described primary copy and the described application end that sends from copy to the other side after described primary copy and the described operation of finishing the apply metadata disk from copy.
7. device according to claim 6 is characterized in that, described delivery module utilizes internet message to transmit described daily record to described from copy from described primary copy.
8. device according to claim 6 is characterized in that, described device also comprises responder module, and is described after the described application end of copy for receiving at described primary copy, the acknowledged client end.
9. device according to claim 6, it is characterized in that described device also comprises recycling module, be used for one of running background and reclaim the daily record thread, when receiving described primary copy and the described application receipt message that sends from copy, described recovery daily record thread starts reclaimer operation.
10. device according to claim 9 is characterized in that, whether described recycling module quantitative check exists recuperable daily record, if exist then carry out reclaimer operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210541441.3A CN103049351B (en) | 2012-12-13 | 2012-12-13 | The log processing method of multivariate data server and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210541441.3A CN103049351B (en) | 2012-12-13 | 2012-12-13 | The log processing method of multivariate data server and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103049351A true CN103049351A (en) | 2013-04-17 |
| CN103049351B CN103049351B (en) | 2016-06-08 |
Family
ID=48062000
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210541441.3A Active CN103049351B (en) | 2012-12-13 | 2012-12-13 | The log processing method of multivariate data server and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103049351B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103516549A (en) * | 2013-09-27 | 2014-01-15 | 浪潮电子信息产业股份有限公司 | File system metadata log mechanism based on shared object storage |
| CN103544081A (en) * | 2013-10-23 | 2014-01-29 | 曙光信息产业(北京)有限公司 | Management method and device for double metadata servers |
| CN111190874A (en) * | 2019-12-27 | 2020-05-22 | 天津中科曙光存储科技有限公司 | Distributed storage high-reliability data log module and construction method thereof |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6205449B1 (en) * | 1998-03-20 | 2001-03-20 | Lucent Technologies, Inc. | System and method for providing hot spare redundancy and recovery for a very large database management system |
| CN102368267A (en) * | 2011-10-25 | 2012-03-07 | 曙光信息产业(北京)有限公司 | Method for keeping consistency of copies in distributed system |
| CN102508891A (en) * | 2011-10-25 | 2012-06-20 | 曙光信息产业(北京)有限公司 | Consistency method based on discarded multi-metadata server metadata log |
-
2012
- 2012-12-13 CN CN201210541441.3A patent/CN103049351B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6205449B1 (en) * | 1998-03-20 | 2001-03-20 | Lucent Technologies, Inc. | System and method for providing hot spare redundancy and recovery for a very large database management system |
| CN102368267A (en) * | 2011-10-25 | 2012-03-07 | 曙光信息产业(北京)有限公司 | Method for keeping consistency of copies in distributed system |
| CN102508891A (en) * | 2011-10-25 | 2012-06-20 | 曙光信息产业(北京)有限公司 | Consistency method based on discarded multi-metadata server metadata log |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103516549A (en) * | 2013-09-27 | 2014-01-15 | 浪潮电子信息产业股份有限公司 | File system metadata log mechanism based on shared object storage |
| CN103516549B (en) * | 2013-09-27 | 2018-03-27 | 浪潮电子信息产业股份有限公司 | A kind of file system metadata log mechanism based on shared object storage |
| CN103544081A (en) * | 2013-10-23 | 2014-01-29 | 曙光信息产业(北京)有限公司 | Management method and device for double metadata servers |
| CN103544081B (en) * | 2013-10-23 | 2015-08-12 | 曙光信息产业(北京)有限公司 | The management method of double base data server and device |
| CN111190874A (en) * | 2019-12-27 | 2020-05-22 | 天津中科曙光存储科技有限公司 | Distributed storage high-reliability data log module and construction method thereof |
| CN111190874B (en) * | 2019-12-27 | 2023-10-03 | 天津中科曙光存储科技有限公司 | High-reliability data log module for distributed storage and construction method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103049351B (en) | 2016-06-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103309767B (en) | A kind of client log disposal route and device | |
| KR102167613B1 (en) | Message push method and device | |
| US10831741B2 (en) | Log-shipping data replication with early log record fetching | |
| US20110246763A1 (en) | Parallel method, machine, and computer program product for data transmission and reception over a network | |
| CN104092717A (en) | Message processing method and system, message destination end device and distributed message system | |
| CN102571635A (en) | Message transmission method and equipment | |
| CN110460412B (en) | Method and RDMA network card for data transmission | |
| CN103095843A (en) | Method and client of data backup based on version vectors | |
| CN102752283A (en) | Method and system for transmitting files with large data sizes | |
| CN108989432B (en) | User-mode file sending method, user-mode file receiving method and user-mode file receiving and sending device | |
| CN104503863A (en) | Kernel mode and user mode data exchange method used for virtual container system disaster recovery | |
| CN108347463A (en) | A kind of terminal document method for uploading | |
| US20210334791A1 (en) | Method and device for blockchain-based data traffic calculation | |
| CN104462562A (en) | Data migration system and method based on data warehouse automation | |
| CN103049351A (en) | Log processing method and device of multivariate data server | |
| US8149830B2 (en) | Efficient light-weight multicasting communication protocol | |
| CN107172112B (en) | Computer file transmission method and device | |
| CN103269376A (en) | Enterprise network large file transmission system authorized and managed in centralized mode and method | |
| CN103220343B (en) | Big data multi-system retransmission method | |
| CN203574673U (en) | Message-processing system | |
| KR101465891B1 (en) | Traffic redundancy elimination method and apparatus in wireless network | |
| CN105471616A (en) | Cache system management method and system | |
| CN101001213A (en) | Short message transmission system and method | |
| CN103092932A (en) | Distributed document transcoding system | |
| CN116015871A (en) | Data transmission system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220721 Address after: 100089 building 36, courtyard 8, Dongbeiwang West Road, Haidian District, Beijing Patentee after: Dawning Information Industry (Beijing) Co.,Ltd. Patentee after: DAWNING INFORMATION INDUSTRY Co.,Ltd. Address before: 100193 No. 36 Building, No. 8 Hospital, Wangxi Road, Haidian District, Beijing Patentee before: Dawning Information Industry (Beijing) Co.,Ltd. |
|
| TR01 | Transfer of patent right |