CN103856534A - Method and system for collecting logs based on peer-to-peer network - Google Patents
Method and system for collecting logs based on peer-to-peer network Download PDFInfo
- Publication number
- CN103856534A CN103856534A CN201210516908.9A CN201210516908A CN103856534A CN 103856534 A CN103856534 A CN 103856534A CN 201210516908 A CN201210516908 A CN 201210516908A CN 103856534 A CN103856534 A CN 103856534A
- Authority
- CN
- China
- Prior art keywords
- peer
- log
- system server
- rpc message
- log system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention discloses a method and system for collecting logs based on a peer-to-peer network. The method comprises the steps that after a current log system server receives an RPC protocol message sent by a corresponding service module server, the RPC message is processed according to the state of the current log system server and the state of a peer-to-peer log system server, and according to the step of processing the RPC message, log information in the RPC message is recorded into files to be log files, or the RPC message is forwarded to the peer-to-peer log system server; all log system servers in the peer-to-peer network upload the log files recorded by the log system servers to a specified file storage server. Through the method and system, service logic and log logic are decoupled, and meanwhile performance and reliability of a log search system are improved.
Description
Technical field
The present invention relates to file search technical field, relate in particular to a kind of daily record gathering method and system based on peer-to-peer network.
Background technology
At present, daily record gathering system is mainly divided into two large classes: one is traditional star Nonseparation daily record gathering system, as shown in Figure 1, in this system, produce log information and be recorded as journal file by business module server (as the request response server of service), upload to again certain or some addresses (ftp file server or NFS), gather journal file.A kind of is separate type daily record gathering system based on hierarchical network, this type systematic be typically represented as the Scribe system that Facebook increases income, this type systematic mainly contains following characteristics: 1, log system server is server for business module server, remote procedure call protocol (RPC, the Remote Procedure Call Protocol) request that the module server of accepting business sends.Business module server is passed to log system server by the log information of generation by RPC mode.2, log system server can be journal file, also log information can be sent to next stage log system server the log information recording of receiving, the processing procedure of daily record is based on configuration.
Mainly there is following problem in above-mentioned daily record gathering system:
In traditional star Nonseparation daily record gathering system, business module server produces log information and log information recording is arrived to file, the interrelated logic of log recording and service logic are on same server like this, there is following shortcoming in this mode: the logic that 1, service logic is relevant with log recording intercouples, and performance can only be got both low values.Log recording generally can take a large amount of IO and consume, and can affect business service performance.2, another benefit of log recording and service detach is: log recording is more flexible.Service logic does not need to pay close attention to the logic that log recording is relevant (as journal format, be recorded to local or NFS etc.), and traditional star Nonseparation daily record gathering system does not really possess these features, log recording is dumb comparatively speaking, service logic need to be paid close attention to the logic that log recording is relevant, can affect business service performance.3, from log recording to daily record, uploading is all single line logic, and some links are made mistakes and all can be caused log recording unsuccessful, there is no reliable fault tolerant mechanism.
Separate type daily record gathering system based on hierarchical network, has following shortcoming: the network configuration of this type systematic is single, and log recording efficiency and reliability are low.
Summary of the invention
In view of this, main purpose of the present invention is to provide a kind of daily record gathering method and system based on peer-to-peer network, to solve the problem that log recording efficiency and reliability in existing daily record gathering system are low, affect business service performance.
For achieving the above object, technical scheme of the present invention is achieved in that
The invention provides a kind of daily record gathering method based on peer-to-peer network, set up the peer-to-peer network being made up of multiple log system servers, the method also comprises:
Current log system server receives after the remote procedure call protocol RPC message of corresponding business module server transmission, according to the state of the state of self and reciprocity log system server, described RPC message is processed, comprise: by the log information recording in described RPC message in file, become journal file, or, described RPC message is transmitted to reciprocity log system server;
Each log system server in described peer-to-peer network uploads to each self-recording journal file the document storage server of appointment.
Preferably, the state of log system server comprises: the ability information of type, ability and the place network of reaction log system server, the performance information of reaction log system server load and the range information of reaction and reciprocity log system server spacing.
Preferably, described log information recording is comprised to file: by log information recording in local file or network file.
Preferably, described current log system server or reciprocity log system server receive after RPC message, RPC message are put in the task queue of self.
Preferably, when described RPC message processing according to following rule:
According to the RPC message in priority sequential processes task queue from high to low; And/or, the type of RPC message in statistics task queue, RPC message corresponding to type that priority treatment message number is maximum; And/or for the RPC message of same type, priority treatment enters task queue RPC message the earliest.
The present invention also provides a kind of daily record gathering system based on peer-to-peer network, comprising:
Business module server, for generation of log information, and sends to corresponding log system server by RPC message;
Log system server, for receiving after the RPC message of corresponding business module server transmission, according to the state of the state of self and reciprocity log system server, described RPC message is processed, comprise: by the log information recording in described RPC message in file, become journal file, or, described RPC message is transmitted to described reciprocity log system server;
Log system server, also for uploading to the journal file of self record the document storage server of appointment.
Preferably, the state of log system server comprises: the ability information of type, ability and the place network of reaction log system server, the performance information of reaction log system server load and the range information of reaction and reciprocity log system server spacing.
Preferably, described log system server, also for by log information recording to local file or network file.
Preferably, described log system server or described reciprocity log system server, also for being put into the task queue of self after the RPC message receiving.
Preferably, described log system server, also for processing RPC message according to following rule: according to the RPC message of priority sequential processes task queue from high to low; And/or, the type of RPC message in statistics task queue, RPC message corresponding to type that priority treatment message number is maximum; And/or for the RPC message of same type, priority treatment enters task queue RPC message the earliest.
The present invention is based on the method and system of the daily record collection of peer-to-peer network: the peer-to-peer network that model is made up of multiple log system servers, in peer-to-peer network, current log system server receives the RPC message that corresponding business module server sends, and according to the state of the state of self and reciprocity log system server, RPC message is processed, comprise: by the log information recording in RPC message in file, become journal file, or, RPC message is transmitted to reciprocity log system server.Pass through the method, the interrelated logic of log recording is carried out by log system server, service logic is carried out by business module server, like this, service logic and daily record logic solution are coupled, and have improved the performance of business module server, simultaneously, utilize high efficiency and the reliability of peer-to-peer network, also promoted widely the Performance And Reliability of daily record search system.
Brief description of the drawings
Fig. 1 is traditional star Nonseparation daily record gathering system schematic diagram;
Fig. 2 is the separate type daily record gathering system schematic diagram based on hierarchical network;
Fig. 3 is the daily record gathering system schematic diagram that the present invention is based on peer-to-peer network.
Embodiment
Below in conjunction with the drawings and specific embodiments, the technical solution of the present invention is further elaborated.
The basic thought that the present invention is based on the daily record collection of peer-to-peer network is: the peer-to-peer network that model is made up of multiple log system servers, in peer-to-peer network, current log system server receives the RPC message that corresponding business module server sends, and according to the state of the state of self and reciprocity log system server, RPC message is processed, comprise: by the log information recording in RPC message in file, become journal file, or, RPC message is transmitted to reciprocity log system server.Pass through the method, the interrelated logic of log recording is carried out by log system server, service logic is carried out by business module server, like this, service logic and daily record logic solution are coupled, and have improved the performance of business module server, simultaneously, utilize high efficiency and the reliability of peer-to-peer network, also promoted widely the Performance And Reliability of daily record search system.
The daily record that the present invention is based on peer-to-peer network is collected as shown in Figure 3, finds the other side, and set up peer-to-peer network between log system server, and wherein, reciprocity networking can adopt P2P mechanism to set up.In peer-to-peer network, intercommunication state periodically between the log system server of peer node each other, preferably, the state of log system server comprises: ability information, performance information and range information, wherein: ability information refers to the information of type, ability and the place network of log system server; Performance information refers to the load information of log system server, the length of such as task queue etc.; Range information has reacted the distance between reciprocity log system server.
Log system server receives the RPC message that one or more business module servers send, and then, RPC message is put into successively to the task queue of self to late order by morning according to the time receiving.In RPC message, carry the log information that business module server produces.
In the time that RPC message is processed, log system server need to be carried out according to the state of the state of self and reciprocity log system server, preferably, result comprises two kinds of situations, the one, by the log information recording in RPC message to file, become journal file; The one, directly RPC message is transmitted to reciprocity log system server.
Log system server, can be according to following rule in the time that the RPC message in self task queue is processed:
According to the RPC message in priority sequential processes task queue from high to low; And/or, the type of RPC message in statistics task queue, RPC message corresponding to type that priority treatment message number is maximum; And/or priority treatment enters task queue RPC message the earliest.
Above-mentioned rule can combination in any be used, for example: for multiple RPC message of same type, can be according to priority sequential processes from high to low, also can priority treatment enter task queue the earliest; For multiple RPC message of same priority, can priority treatment enter task queue the earliest; For entering the task queue time identical, can be according to priority sequential processes from high to low.
First log system server judges whether to process RPC message according to the state of self, if so, the log information recording in RPC message, to file, here, is recorded to file and comprises: be recorded to local file or network file.If not, RPC message is transmitted to reciprocity log system server by log system server.A log system server may have multiple reciprocity log system servers, in the time selecting, can be according to the state of reciprocity log system server, and select the reciprocity log system server that is applicable to, for example, select least-loaded or chosen distance is nearest etc.Concrete, when log system server first-selection judges whether to process RPC message according to the state of self, and when the reciprocity log system server being applicable to according to the condition selecting of reciprocity log system server, can formulate corresponding one or more rules according to the state of log system server.
Equity log system server receives after the one or more RPC message that forward, and also it is put into the task queue of self to late order by morning according to time of reception, and the operation of follow-up execution as previously mentioned, repeats no more herein.
So, in reciprocity networking, each log system server uploads to each self-recording journal file the document storage server of appointment.
In order to realize said method, in the daily record gathering system based on peer-to-peer network as shown in Figure 3 that the embodiment of the present invention provides, business module server and log system server function are separately as follows:
Business module server, for generation of log information, and sends to corresponding log system server by RPC message;
Log system server, for receiving after the RPC message of corresponding business module server transmission, according to the state of the state of self and reciprocity log system server, RPC message is processed, comprise: by the log information recording in RPC message in file, become journal file, or, RPC message is transmitted to reciprocity log system server;
Log system server, also for uploading to the journal file of self record the document storage server of appointment.
The state of log system server comprises: the ability information of type, ability and the place network of reaction log system server, the performance information of reaction log system server load and the range information of reaction and reciprocity log system server spacing.
Log system server, also for by log information recording to local file or network file.
Log system server or reciprocity log system server, also for being put into the task queue of self after the RPC message receiving.
Log system server, also for processing RPC message according to following rule: according to the RPC message of priority sequential processes task queue from high to low; And/or, the type of RPC message in statistics task queue, RPC message corresponding to type that priority treatment message number is maximum; And/or for the RPC message of same type, priority treatment enters task queue RPC message the earliest.
The above, be only preferred embodiment of the present invention, is not intended to limit protection scope of the present invention.
Claims (10)
1. the daily record gathering method based on peer-to-peer network, is characterized in that, sets up the peer-to-peer network being made up of multiple log system servers, and the method also comprises:
Current log system server receives after the remote procedure call protocol RPC message of corresponding business module server transmission, according to the state of the state of self and reciprocity log system server, described RPC message is processed, comprise: by the log information recording in described RPC message in file, become journal file, or, described RPC message is transmitted to reciprocity log system server;
Each log system server in described peer-to-peer network uploads to each self-recording journal file the document storage server of appointment.
2. the daily record gathering method based on peer-to-peer network according to claim 1, it is characterized in that, the state of log system server comprises: the ability information of type, ability and the place network of reaction log system server, the performance information of reaction log system server load and the range information of reaction and reciprocity log system server spacing.
3. the daily record gathering method based on peer-to-peer network according to claim 1, is characterized in that, described log information recording is comprised to file: by log information recording in local file or network file.
4. according to the arbitrary described daily record gathering method based on peer-to-peer network of claims 1 to 3, it is characterized in that, described current log system server or reciprocity log system server receive after RPC message, RPC message are put in the task queue of self.
5. the daily record gathering method based on peer-to-peer network according to claim 4, is characterized in that, when described RPC message is processed according to following rule:
According to the RPC message in priority sequential processes task queue from high to low; And/or, the type of RPC message in statistics task queue, RPC message corresponding to type that priority treatment message number is maximum; And/or for the RPC message of same type, priority treatment enters task queue RPC message the earliest.
6. the daily record gathering system based on peer-to-peer network, is characterized in that, comprising:
Business module server, for generation of log information, and sends to corresponding log system server by RPC message;
Log system server, for receiving after the RPC message of corresponding business module server transmission, according to the state of the state of self and reciprocity log system server, described RPC message is processed, comprise: by the log information recording in described RPC message in file, become journal file, or, described RPC message is transmitted to described reciprocity log system server;
Log system server, also for uploading to the journal file of self record the document storage server of appointment.
7. the daily record gathering system based on peer-to-peer network according to claim 6, it is characterized in that, the state of log system server comprises: the ability information of type, ability and the place network of reaction log system server, the performance information of reaction log system server load and the range information of reaction and reciprocity log system server spacing.
8. the daily record gathering system based on peer-to-peer network according to claim 6, is characterized in that, described log system server, also for by log information recording to local file or network file.
9. according to the arbitrary described daily record gathering system based on peer-to-peer network of claim 6 to 8, it is characterized in that, described log system server or described reciprocity log system server, also for being put into the task queue of self after the RPC message receiving.
10. the daily record gathering system based on peer-to-peer network according to claim 9, it is characterized in that, described log system server, also for processing RPC message according to following rule: according to the RPC message of priority sequential processes task queue from high to low; And/or, the type of RPC message in statistics task queue, RPC message corresponding to type that priority treatment message number is maximum; And/or for the RPC message of same type, priority treatment enters task queue RPC message the earliest.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210516908.9A CN103856534B (en) | 2012-12-05 | 2012-12-05 | A kind of daily record gathering method and system based on peer-to-peer network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210516908.9A CN103856534B (en) | 2012-12-05 | 2012-12-05 | A kind of daily record gathering method and system based on peer-to-peer network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103856534A true CN103856534A (en) | 2014-06-11 |
| CN103856534B CN103856534B (en) | 2018-02-13 |
Family
ID=50863741
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210516908.9A Active CN103856534B (en) | 2012-12-05 | 2012-12-05 | A kind of daily record gathering method and system based on peer-to-peer network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103856534B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106357454A (en) * | 2016-10-08 | 2017-01-25 | 武汉斗鱼网络科技有限公司 | Log system control device and method |
| CN108733698A (en) * | 2017-04-19 | 2018-11-02 | 腾讯科技（深圳）有限公司 | A kind of processing method and background service system of log information |
| CN108847989A (en) * | 2018-06-29 | 2018-11-20 | 杭州安恒信息技术股份有限公司 | Log processing method, business service system and electronic equipment based on micro services framework |
| CN109710433A (en) * | 2018-12-30 | 2019-05-03 | 浙江蓝卓工业互联网信息技术有限公司 | Data processing method, apparatus and system based on metadata driven |
| CN109918282A (en) * | 2019-03-16 | 2019-06-21 | 河南工学院 | A kind of computer log monitoring method and monitor supervision platform |
| CN110166298A (en) * | 2019-05-24 | 2019-08-23 | 和记奥普泰通信技术有限公司 | A kind of packetizing OTN, PTN, IPRAN and SPN control with management log realization system and method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101361054A (en) * | 2005-11-15 | 2009-02-04 | 蒂弗萨公司 | System for identifying the presence of peer-to-peer network software applications |
| CN101729581A (en) * | 2008-10-28 | 2010-06-09 | 华为技术有限公司 | Method, system and device for realizing re-orientation in P2P network |
| CN101800776A (en) * | 2010-02-26 | 2010-08-11 | 中国电信股份有限公司 | Network fusion and control method and system of CDN (Content Delivery Network) and P2P (Peer-to-Peer) |
| WO2012047461A2 (en) * | 2010-09-27 | 2012-04-12 | Ebay, Inc. | Instant financial account verification using direct connect data communication protocol and open financial exchange data-stream format |
-
2012
- 2012-12-05 CN CN201210516908.9A patent/CN103856534B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101361054A (en) * | 2005-11-15 | 2009-02-04 | 蒂弗萨公司 | System for identifying the presence of peer-to-peer network software applications |
| CN101729581A (en) * | 2008-10-28 | 2010-06-09 | 华为技术有限公司 | Method, system and device for realizing re-orientation in P2P network |
| CN101800776A (en) * | 2010-02-26 | 2010-08-11 | 中国电信股份有限公司 | Network fusion and control method and system of CDN (Content Delivery Network) and P2P (Peer-to-Peer) |
| WO2012047461A2 (en) * | 2010-09-27 | 2012-04-12 | Ebay, Inc. | Instant financial account verification using direct connect data communication protocol and open financial exchange data-stream format |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106357454A (en) * | 2016-10-08 | 2017-01-25 | 武汉斗鱼网络科技有限公司 | Log system control device and method |
| CN106357454B (en) * | 2016-10-08 | 2020-06-16 | 武汉斗鱼网络科技有限公司 | Log system control device and control method thereof |
| CN108733698A (en) * | 2017-04-19 | 2018-11-02 | 腾讯科技（深圳）有限公司 | A kind of processing method and background service system of log information |
| CN108733698B (en) * | 2017-04-19 | 2023-08-08 | 腾讯科技（深圳）有限公司 | Log message processing method and background service system |
| CN108847989A (en) * | 2018-06-29 | 2018-11-20 | 杭州安恒信息技术股份有限公司 | Log processing method, business service system and electronic equipment based on micro services framework |
| CN109710433A (en) * | 2018-12-30 | 2019-05-03 | 浙江蓝卓工业互联网信息技术有限公司 | Data processing method, apparatus and system based on metadata driven |
| CN109918282A (en) * | 2019-03-16 | 2019-06-21 | 河南工学院 | A kind of computer log monitoring method and monitor supervision platform |
| CN110166298A (en) * | 2019-05-24 | 2019-08-23 | 和记奥普泰通信技术有限公司 | A kind of packetizing OTN, PTN, IPRAN and SPN control with management log realization system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103856534B (en) | 2018-02-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107749840B (en) | One-way data secure transmission and cooperative processing system and method based on one-way gatekeeper | |
| CN103856534A (en) | Method and system for collecting logs based on peer-to-peer network | |
| CN106790572B (en) | Distributed log collection system and method | |
| US10931479B2 (en) | Inband data gathering with dynamic intermediary route selections | |
| CN106470123B (en) | Log collecting method, client, server and electronic equipment | |
| CN102611642A (en) | System for processing nonsynchronous message and method for system to send message and monitor processing task | |
| CN101588350A (en) | Method and system for realizing file sharing | |
| CN101834765A (en) | System and method for remote file transfer | |
| CN114338650A (en) | File transmission method and device, electronic equipment and readable storage medium | |
| CN1327653C (en) | Routing method of pub/sub system on structural P2P network | |
| CN104618466A (en) | System for balancing load and controlling overload based on message transfer and control method of system | |
| CN110086759A (en) | A kind of method and apparatus for realizing message transmission between heterogeneous system | |
| WO2010101038A1 (en) | Destination node selection method, node, and recording medium | |
| CN104618320A (en) | Systems and methods for secure file transfers | |
| CN106648922A (en) | Data collecting method based on XMPP | |
| CN101188574B (en) | An information transmission method and system between instant communication users | |
| JP2010287987A (en) | Mail server system, and convergence control method | |
| CN105740435A (en) | On-line preview design method of document on the basis of distribution | |
| CN101420357B (en) | Reflected route processing method and route reflecting device | |
| CN1937553B (en) | Reciprocal network data dispatching method based on flow media data frame | |
| CN103580951B (en) | Output comparative approach, test migration householder method and the system of multiple information systems | |
| JP4884520B2 (en) | Data collection method and system | |
| JP6364727B2 (en) | Information processing system, distributed processing method, and program | |
| US20130282846A1 (en) | System and method for processing similar emails | |
| CN104503866A (en) | Data backup system, data backup method and backup data recovery method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |