WO2010099754A1

WO2010099754A1 - Log information transmission method and apparatus

Info

Publication number: WO2010099754A1
Application number: PCT/CN2010/070876
Authority: WO
Inventors: 樊滑翔
Original assignee: 成都市华为赛门铁克科技有限公司
Priority date: 2009-03-06
Filing date: 2010-03-04
Publication date: 2010-09-10
Also published as: CN101505245A; CN101505245B

Abstract

The embodiments of present invention provide a log information transmission method, including: detecting whether the load on the buffer queue of a log transmitter reaches a preset abnormity threshold (10); generating and transmitting the log information including a pre-treatment event starting message by the log transmitter to a log receiver when the load on the buffer queue of the log transmitter reaches the preset abnormity threshold (12); generating and transmitting the log information including a pre-treatment event ending message by the log transmitter to the log receiver if it is detected that the load on the buffer queue does not reach the preset abnormity threshold when the log transmitter performs pre-treatment operation for the log in its buffer queue (14). The embodiments of present invention also provide a log information transmission apparatus. By generating the log information including a pre-treatment event starting message or a pre-treatment event ending message and transmitting it to the log receiver, the embodiments of present invention can increase the subsequent accuracy of auditing and analyzing the log information.

Description

Method and device for transmitting log information This application claims to be filed on March 6, 2009, the Chinese Patent Office, the application number is 200910105892.0, the priority of the Chinese patent application entitled "A method and device for transmitting log information" The entire contents of which are incorporated herein by reference. Technical field

The present invention relates to the field of communications technologies, and in particular, to a method and an apparatus for transmitting log information. Background technique

Log J! An important means of controlling computer and network security, and an important way to assess computer and network security. In practical applications, the log system often adopts a distributed deployment, which collects logs generated by multiple services and/or devices for analysis and processing. In this case, the party that generates the ambition is called the log sender. The party collecting the log is called the log receiver, the log sender transmits the log message to the log receiver, and the log receiver receives the log sender from different geographical locations.

Syslog is a widely accepted log standard in the industry. The new Syslog standard developed by the IETF allows for larger volume log messages. While the Syslog/TLS standard and the Syslog-sign standard increase the security and reliability of Syslog, they also increase the number of data transmitted by certificates and signatures. At the same time, some applications, such as the pharmaceutical industry, require large log volumes. All of this increases the amount of Syslog log traffic and increases the likelihood of log latency and network congestion. How to deal with various events that occur during Syslog log transmission in a timely manner is an effective means to ensure the reliability and security of the log system.

In extreme cases (such as network congestion or traffic control or bursts of log generation, this will cause a large load or even overflow of the buffer queue), and some pre-processing measures may be taken, such as sending important logs or Filter the secondary logs to ensure timely alarms for important logs. These measures will inevitably result in the loss or out of order of the log, which will damage the integrity of the log.

In the process of implementing the present invention, the inventors have found that at least the following problems exist in the prior art: The log receiver cannot detect the occurrence of log transmission events (such as filtering or disorder) in time, When the collected logs are audited in the future, they do not know the loss and out-of-order conditions of the logs in the above scenarios, and have certain influence on the correctness of the audit and analysis of the logs.

Summary of the invention

The embodiment of the invention provides a method and a device for transmitting log information, which can enable the log receiver to timely detect the pre-processing event information of the sender on the log, and improve the accuracy of subsequent auditing and analysis of the log.

The embodiment of the invention provides a method for sending log information, including:

Check whether the load of the buffer queue of the log sender reaches a preset abnormal threshold;

When the load of the buffer queue of the log sender reaches a preset abnormal threshold, the log sender generates log information including a pre-process event start message and sends the log information to the log receiver.

When the log sender performs a pre-processing operation on the buffer queue, if it detects that the load of the buffer queue does not reach the preset abnormal threshold, the log information including the pre-processing event end message is generated and sent to the log receiver. The pre-processing operation is performed by the log sender on the log of the buffer queue when the load of the buffer queue of the log sender reaches a preset abnormal threshold.

The embodiment of the invention further provides an apparatus for sending log information, including:

a buffer unit, configured to store a log queue to be sent;

a detecting unit, configured to check whether a load of the log queue of the buffer unit reaches a preset abnormal threshold;

a first log information generating unit, configured to: when the detecting unit detects that the load of the buffer queue of the buffer unit reaches a preset abnormal threshold, generate log information including a pre-process event start message; and send log information When the device performs a pre-processing operation on the log information of the buffer unit, when the detecting unit detects that the load of the log queue of the buffer unit does not reach a preset abnormal threshold, the first information generation The unit generates the information including the pre-processing event end message, where the pre-processing operation is that the device that sends the log information detects that the load of the buffer queue of the buffer unit reaches a preset abnormal threshold when the detecting unit detects that the load of the buffer queue of the buffer unit reaches a preset abnormal threshold , made to the log information of the buffer unit;

a sending unit for collecting the log information including the pre-process event start message and the package The log information including the pre-processing event end message is sent to the log receiver.

In the embodiment of the present invention, the log sender generates the log information including the pre-processing event start message or the pre-processing event end message, and sends the log information to the log receiver, so that the log receiver can obtain the load of the log sender in the buffer queue to reach the preset time. The pre-processing information to be sent to the log when the abnormal value is abnormal, and the loss or out-of-order situation when the load of the buffer queue reaches the preset abnormal threshold is improved, which improves the correctness of subsequent auditing and analysis of the log information. . DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings in the following description are only It is a certain embodiment of the present invention, and other drawings can be obtained from those skilled in the art without any inventive labor.

Figure 1 is a schematic structural diagram of a packet header of a Syslog protocol packet;

2 is a schematic flowchart of a method for sending log information according to Embodiment 1 of the present invention;

3 is a schematic flowchart of a method for sending log information according to Embodiment 2 of the present invention;

4 is a schematic structural diagram of an apparatus for transmitting log information according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of an apparatus for transmitting log information according to Embodiment 2 of the present invention. detailed description

BRIEF DESCRIPTION OF THE DRAWINGS The technical solutions in the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without departing from the inventive scope are the scope of the present invention.

In the embodiment of the present invention, the log information is described by using a Syslog protocol packet as an example.

The Syslog protocol uses the client/server (client/server) mode of communication, and the client is the receiver of event packets. The client can be the originator of the event, such as a device or process, or a relay entity. The relayed entity processes the Syslog time received from other senders (event generators or other relay entities). Send it to other recipients.

The Syslog protocol packet consists of three parts: packet header and structured data (Structured Data) Element , SDE ) and the message body. As shown in FIG. 1, the header includes the following fields: PRI, VERSION, TIMESTAMP, HOSTNAME, APP-NAME, PROCID, and MSGID.

PRI indicates the priority, VERSION indicates the Syslog protocol version number, TIMESTAMP indicates the timestamp generated by the log, HOSTNAME indicates the host domain name or address of the log sender, APP-NAME indicates the application name that generated the log, and PROCID indicates the process indicating the log system. No., MSGID indicates the message category of the log.

The structured data is composed of a series of structured elements, each of which includes a structured element name and a series of parameter names and parameter value pairs.

The message body is free-form text information, and a Syslog log message may not include the message body.

Referring to FIG. 2, it is a schematic flowchart of a method for sending log information according to an embodiment of the present invention, and the steps include:

Step S10: Detect whether the load of the buffer queue of the log sender reaches a preset abnormal threshold. The log sender stores the log to be sent in its buffer queue. The log sender detects whether the load of the buffer queue reaches the preset abnormal threshold in real time. For example, if the length of the buffer queue reaches the preset threshold, Then, the process proceeds to step S12.

If the load of the buffer queue of the log sender has reached the preset abnormal threshold, the log sender will perform pre-processing operations on the log information in the buffer queue, such as filtering the secondary log, prioritizing the important log, or overwriting the old one. Logs, etc.

Step S12: The log sender generates log information including a pre-process event start message and sends the message to the recipient, the pre-process event start message including a pre-processed start time message and a pre-processing policy message.

After detecting that the load of the buffer queue of the log sender reaches the preset abnormal threshold in step S10, the sender sends a log information including a pre-process event start message, and the log information including the pre-process event start message is The format of the log information in the buffer queue of the log sender is the same. The Syslog log information is used as an example for description.

The log information including the pre-processing event start message can be implemented by setting structured data (SDE) in the Syslog log information. An SDE consists of a structured element name and a series of parameter names and parameter value pairs. The syntax is as follows: [SD-ID NAME1: "VALUE 1" NAMEn= "VALUEn"]

For example, the SD-ID of the structured data is set to preprocess, and the parameter name and its attributes are described as follows:

The "pre-processing action" is used to describe the pre-processing actions of the log sender to the log to be sent in the buffer queue: filtering, out-of-order transmission, etc., wherein filtering may include filtering the secondary log, and the out-of-order transmission may include prioritizing the important log transmission. Or overwrite old logs, etc. "Time Type" indicates that the subsequent "time point" is the timestamp of the earliest preprocessed log or the timestamp of the log that was preprocessed at the latest, "policy type" and "thickness," indicating the condition of the important log, such as Criteria = "severity", Threshold="3" means that the log corresponding to the severity of the header field PRI is less than 3 is an important log.

E.g:

[preprocess Type= " filter" timeType= " start" timeValue= " 2009-02-13 15:00:00" Criteria= "severity" Threshold= "3"] , indicating that a pre-processing event has occurred, the pre-processing strategy is All logs with a severity greater than 3 (minor log) are filtered out. The first filtered log is generated at "2009-02-13 15:00:00".

After receiving the log information including the pre-processing event start message, the receiver can obtain the pre-processing information of the sender in real time after being parsed, and the sender pre-processes the log that can be learned during subsequent auditing and analysis of the log. The time when the time occurs, the pre-processing strategy, by analyzing the time when the pre-processing event occurs, and the pre-processing strategy can learn from what time the log sender has processed the log, which can improve the correctness of the audit and analysis; After obtaining the log information including the pre-process event start message, you can obtain the extreme situation such as network congestion or a large number of log bursts in time, so as to respond in time. Step S14: When the log sender performs a pre-processing operation on the buffer of the buffer queue, if it is detected that the load of the buffer queue does not reach the preset abnormal threshold, the log information including the pre-processing event end message is generated and sent to the log. The receiver, the pre-processing event end message includes a pre-processed end time message and a pre-processing policy message.

When the log sender performs pre-processing operations on the logs of its buffer queue, for example, all logs with a severity greater than 3 (secondary logs) are filtered out. In the pre-processing operation, the logs in the buffer queue will be less and less. After the time has elapsed, it is detected that the load of the buffer queue (for example, the length of the log in the buffer queue) falls below the preset abnormal threshold, that is, if the preset abnormal threshold is not reached, the log sender will end the pre-processing operation. Generate log information including the pre-processing event end message and send it to the log receiver.

The log information including the pre-processing event end message can also be implemented by setting structured data (SDE) in the Syslog log information. E.g:

[preprocess Type= "filter" timeType= "end" timeValue= " 2009-02-13 15:20:00"], indicating that the current pre-processing event (filtering) has ended, and the latest filtered log is "2009- 02-13 15 :20:00" is generated.

After receiving the log information including the pre-processing event end message, the receiving party can obtain the pre-processing information of the sender in real time after being parsed, and the sender pre-processes the log that can be learned during subsequent auditing and analysis of the log. The time at the end of the time, the pre-processing strategy, the time when the pre-processing event ends, and the pre-processing strategy can learn from the time when the log sender has finished pre-processing the log, which can improve the correctness of the audit and analysis.

In the embodiment of the present invention, the log sender generates the log information including the pre-processing event start message or the pre-processing event end message, and sends the log information to the log receiver, so that the log receiver can obtain the load of the log sender in the buffer queue to reach the preset time. The pre-processing information to be sent to the log when the abnormal value is abnormal, and the loss or out-of-order situation when the load of the buffer queue reaches the preset abnormal threshold is improved, which improves the correctness of subsequent auditing and analysis of the log information. In addition, the log receiver can timely know the occurrence of extreme situations such as network congestion or a large number of log bursts through the log system itself, so as to respond in time.

Referring to FIG. 3, it is a schematic flowchart of a method for sending log information according to Embodiment 2 of the present invention, which is similar to the method for sending log information according to Embodiment 1 of the present invention, and the difference is that the packet is further encapsulated before step S10. Includes:

Step S8: The log sender generates log information including a message describing the importance of the log and sends it to the log receiver. The log information including the message describing the importance of the log is used by the log receiver to analyze the importance of the log according to the description. The message is processed accordingly.

The log information including the description of the log importance message can also be implemented by setting structured data (SDE) in the Syslog log information. E.g:

[preprocess Criteria= "severity" Threshold="3"] , indicating that the log sent by the sender of the log is less than 3 is an important log.

After receiving the log information including the message describing the importance of the log, the log receiver obtains the message describing the importance of the log, and the log receiver identifies which log is an important log by using the message describing the importance of the log. For example, if the log sender sends a log with a severity of less than 3, it is an important log. When the log receiver processes a large number of logs at the same time, the important logs with less severity of 3 can be preferentially processed to ensure the safe and reliable reception and processing of important logs.

In the embodiment of the present invention, the log sender generates the log information including the pre-processing event start message or the pre-processing event end message and sends the log information to the log receiver, so that the log receiver can obtain the log sender's load in the buffer queue to reach the preset abnormality in time. The pre-processing information to be sent to the log when the value is wide, knowing whether the log party loses or the out-of-order situation when the load of the buffer queue reaches the preset abnormal threshold, which improves the subsequent auditing and analysis of the log information. Correctness; In addition, the log receiver can timely know the occurrence of extreme conditions such as network congestion or a large number of log bursts through the log system itself, and respond in time; and the log sender generates log information including a message describing the importance of the log and sends it to The log receiver can enable the log receiver to identify important logs and ensure the safe and reliable reception and processing of important logs.

Referring to FIG. 4, which is a schematic structural diagram of an apparatus for sending log information according to an embodiment of the present invention, the apparatus for sending log information may include a client in a Client/Server mode in a Syslog protocol.

The apparatus for transmitting log information includes a buffer unit 52, a detecting unit 54, a first log information generating unit 56, and a log transmitting unit 58.

The buffer unit 52 is configured to store a log queue to be sent;

The detecting unit 54 is configured to detect whether the load of the log queue of the buffer unit 52 reaches a preset. The detection unit 54 detects in real time whether the load of the buffer unit 52 reaches a preset abnormal value, for example, whether the length of the log queue in the buffer unit 52 reaches a preset threshold.

The first log information generating unit 56 is configured to generate log information including a pre-process event start message when the detecting unit 54 detects that the load of the log queue of the buffer unit 52 reaches a preset abnormal threshold. The pre-processing event start message includes a pre-processed start time message and a pre-processing policy message; the log sending unit 58 is configured to send the log information including the pre-process event start message generated by the first log information generating unit 56. To the log receiver.

The log information including the pre-processing event start message is consistent with the format of the log information in the log queue of the buffer unit 52. The Syslog log information is taken as an example for description.

The log information including the pre-processing event start message can be implemented by setting structured data (SDE) in the Syslog log information. An SDE consists of a structured element name and a series of parameter names and parameter value pairs. The syntax is as follows:

[SD-ID NAME1: "VALUE 1" NAMEn= "VALUEn"]

E.g: [preprocess Type= "filter" timeType= "start" timeValue= " 2009-02-13 15:00:00" Criteria= "severity" Threshold= "3"] , indicating that a pre-processing event has occurred, the pre-processing strategy is All logs with a severity greater than 3 (minor log) are filtered out. The first filtered log is generated at "2009-02-13 15:00:00".

After receiving the log information including the pre-processing event start message, the log receiver can obtain the pre-processing information of the sender in real time after being parsed, and the pre-processing time of the sender to the log can be learned during subsequent auditing and analysis of the log. The time of occurrence, the pre-processing strategy, by analyzing the time when the pre-processing event occurs, and the pre-processing strategy can learn from what time the log sender has processed the log, which can improve the correctness of the audit and analysis; After the log information including the pre-processing event start message is obtained, the extreme situation such as network congestion or a large number of log bursts can be timely acquired, thereby responding in time.

The first log information generating unit 56, when the device for transmitting the log information performs a pre-processing operation on the log information of the buffer unit 52, when the detecting unit 54 detects the log queue of the buffer unit 52. When the load does not reach the preset abnormal threshold, generating the information including the pre-processing event end message, the pre-processing event end message includes the pre-processed end time message and the pre-processing policy message; the log sending unit 58 It is further configured to send log information including the pre-process event end message generated by the first log information generating unit 56 to the log recipient. Similar to the previous method embodiment, in one embodiment, the device that transmits the log information performs a pre-processing operation on the log information of the buffer unit 52, and the device that transmits the log information detects the buffer unit at the detecting unit 54. When the load of the log queue of 52 reaches the preset abnormal threshold, the log information of the buffer unit 52 is made.

When the device that sends the log information performs a pre-processing operation on the log information of the buffer unit 52, for example, all logs with a severity greater than 3 (secondary logs) are filtered out, and the logs in the buffer queue will be in the pre-processing operation. The less the device detects the load of the buffer queue after a period of time (for example, the length of the log in the buffer queue) drops below the preset abnormal threshold, that is, the device does not reach the preset abnormal threshold. The pre-processing operation before the end will be completed, and log information including the pre-processing event end message is generated and sent to the log receiver.

The log information including the pre-processing event end message can also be implemented by setting structured data (SDE) in the Syslog log information. E.g: [preprocess Type= "filter" timeType= "end" timeValue= " 2009-02-13 15:20:00"], indicating that the current pre-processing event (filtering) has ended, and the latest filtered log is "2009- 02-13 15 :20:00" is generated.

After receiving the log information including the pre-processing event end message, the receiving party can obtain the pre-processing information of the sender in real time after being parsed, and the sender pre-processes the log that can be learned during subsequent auditing and analysis of the log. The time at the end of the time, the pre-processing strategy, the time to analyze the end of the pre-processing event, and the pre-processing strategy can be learned from the time when the log sender has completed the pre-processing of the log, which can improve the correctness of the audit and analysis, improve the audit and analysis. The correctness.

The embodiment of the present invention generates the log information including the pre-processing event start message or the pre-processing event end message to be sent to the log receiver by the first log information generating unit 56, so that the log receiver can acquire the load of the buffer unit 52 in time. The pre-processing information to be sent to the log when the preset abnormal threshold is reached, and the loss or out-of-order situation when the load of the buffer unit 52 reaches the preset abnormal threshold is improved, and the subsequent auditing of the log information is improved. The correctness of the analysis; in addition, the receiver can timely know the occurrence of extreme situations such as network congestion or a large number of log surges through the log system itself, so as to respond in time.

Referring to FIG. 5, it is a schematic structural diagram of an apparatus for transmitting log information according to Embodiment 2 of the present invention, which is similar to the structure of an apparatus for transmitting log information according to Embodiment 1 of the present invention, and further includes: a second log information generating unit 60.

The second log information generating unit 60 is configured to generate log information including a message describing the importance of the log and send the log information to the log receiver, where the log information including the message describing the importance of the log is used by the receiver after parsing The received log is processed accordingly according to the message describing the importance of the log.

After receiving the log information including the message describing the importance of the log, the log receiver obtains the message describing the importance of the log, and the log receiver identifies which log is an important log by using the message describing the importance of the log. For example, the log sender sends a display severity less than 3. The log is an important log. When the log receiver processes a large number of logs at the same time, it can preferentially process these important logs with a severity less than 3 to ensure the safe and reliable reception and processing of important logs.

The embodiment of the present invention generates the log information including the pre-processing event start message or the pre-processing event end message to be sent to the log receiver by the first log information generating unit 56, so that the log receiver can acquire the load of the buffer unit 52 in time. The pre-processing information to be sent to the log when the preset abnormal threshold is reached, and the loss or out-of-order situation when the load of the buffer unit 52 reaches the preset abnormal threshold is improved, and the subsequent auditing of the log information is improved. The correctness of the analysis; in addition, the receiving party can timely know the occurrence of extreme situations such as network congestion or a large number of log bursts through the log system itself, thereby responding in time; and the second log information generating unit 60 generates a description log including The log information of the importance message is sent to the log receiver, which enables the log receiver to identify the important log and ensure the safe and reliable reception and processing of the important log.

A person skilled in the art can understand that all or part of the steps of implementing the foregoing embodiments can be completed by a program instructing related hardware, and the program can be stored in a computer readable storage medium, where the storage medium is ROM/RAM, disk, CD, etc.

The above is only a preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think of changes or within the technical scope disclosed by the present invention. Alternatives are intended to be covered by the scope of the present invention. Therefore, the scope of the invention should be determined by the scope of the claims.

Claims

Rights request

1. A method for sending log information, including:

2. The method of claim 1, wherein: the pre-processing event start message includes a pre-processed start time message and a pre-processing policy message.

3. The method of claim 1, wherein: the pre-processing event end message includes a pre-processed end time message and a pre-processing policy message.

4. The method according to claim 1, wherein: the log information including the pre-processing event start message or the log information including the pre-processing event end message is implemented by setting structured data of the Syslog log information.

5. The method of claim 1, further comprising: the log sender generating log information including a message describing the importance of the log and transmitting to the log recipient, the log including a message describing the importance of the log The information is used by the log receiver to parse the received log according to the message describing the importance of the log.

6. A device for transmitting log information, comprising:

a buffer unit, configured to store a log queue to be sent;

a first log information generating unit, configured to: when the detecting unit detects that the load of the log queue of the buffer unit reaches a preset abnormal threshold, generate log information including a pre-process event start message; and send the log information When the pre-processing operation is performed on the log information of the buffer unit, when the detecting unit detects that the load of the log queue of the buffer unit does not reach a preset abnormal threshold The first information generating unit generates the information including the pre-processing event end message, and the pre-processing operation is that the device that sends the log information detects the log queue of the buffer unit in the detecting unit. When the load reaches a preset abnormal threshold, the log information of the buffer unit is made;

And a log sending unit, configured to send the log information including the pre-process event start message and the log information including the pre-process event end message to the log receiver.

7. The apparatus of claim 6, wherein: the pre-processing event start message includes a pre-processed start time message and a pre-processing policy message.

8. The apparatus according to claim 6, wherein: the pre-processing event end message includes a pre-processed end time message and a pre-processing policy message.

9. The apparatus according to claim 6, wherein: the log information including the pre-processing event start message or the log information including the pre-processing event end message is implemented by setting structured data SDE of Syslog log information. .

The apparatus according to claim 6, further comprising: a second log information generating unit, configured to generate log information including a message describing the importance of the log and send the log information to the log receiver, wherein the description includes the log important The log information of the sexual message is used by the log receiver to parse the received log according to the message describing the importance of the log.