KR100937329B1 - Method and system for sampling log - Google Patents

Abstract

The present invention relates to a log sampling method and system. A log sampling method in a log generation system for generating a log according to an event occurrence of the present invention and transmitting the log to a log analysis system includes: a first step of determining whether a predetermined time has elapsed from the time of generating the log, and As a result of the determination, if the predetermined time has elapsed from the time of generation of the log, a second step of excluding the log having elapsed from the predetermined time from the object of processing. This prevents delays in log analysis and reduces CPU resource load. Therefore, it is possible to analyze a large amount of logs in real time, which can be usefully used in security system management.

Log, sampling

Description

Log sampling method and system {METHOD AND SYSTEM FOR SAMPLING LOG}

The present invention relates to a log sampling method and system, and to a method and system for limiting the number of logs for each log generation time, excluding a delayed log from processing, or sampling a log according to a CPU resource load (CPU usage). It is to offer.

A log is an execution record that records a situation of sending and receiving a message in an online system. The log is used for long-term archiving, management work, and usage measurement of a main file such as a system log. In particular, the log generated according to the network connection event occurs to record the connection, so it can be utilized in security system management.

For example, the log may be used in the process of granting access to the network from the network firewall and restricting access. A firewall is a piece of software or hardware that blocks unauthorized people from connecting to the internal network from the outside network. A firewall is a network that blocks unauthorized access according to an access control list (ACL) to protect the resources of the internal network. It is a kind of security equipment. The firewall generates logs to record the connection contents according to the network connection event occurrence, and analyzes the generated logs and uses them for various purposes. The analysis of the log can be used as an important data to prove the responsibility of the breach in case of an infringement incident, and can also be used as a data for establishing a security policy after identifying a route such as an illegal access attempt. In addition, through real-time analysis of the log, you can take immediate action against illegal access attempts.

The problem is that when a large log is generated, the CPU resource load is increased by the log transmission and analysis process, and the log analysis process is delayed. For example, in a large ISP environment or Internet banking, thousands of connections can be made at the same time, and these connection events can generate thousands of logs per second. In particular, when an infection such as a worm or virus is infected or an emergency situation such as a denial of service attack occurs, hundreds of thousands of logs, which are several times as usual, are generated. Sending and analyzing such large logs can increase the CPU resource load and disrupt the system's own business services. In addition, there is a problem that the real-time analysis becomes impossible because the log transmission, analysis, and the like are delayed.

Accordingly, there is a need for a log sampling method and system that prevents the load of CPU resources due to the generation and processing of large logs and prevents delays in log transmission and analysis.

Accordingly, an object of the present invention is to propose a log sampling method and system for sampling a log file to be analyzed in a log generated according to an event occurrence, preventing a heavy CPU resource load, and preventing a delay in log transmission and analysis. .

delete

In order to achieve the above object, the present invention provides a log sampling method in a log generation system that generates a log according to an event occurrence and transmits the log to a log analysis system, wherein the log determination method determines whether a predetermined time has elapsed from the log generation time point. And a second step of excluding the log, in which the predetermined time has elapsed, from the object of processing when the predetermined time has elapsed from the time of generating the log as a result of the determination in the first step. do.

According to another aspect of the present invention, there is provided a log sampling method in a log generation system that transmits a log generated according to an event to a log analysis system according to a CPU usage rate. And a second step of limiting the number of analysis target logs to be transmitted to the analysis system among the generated logs when the threshold value is greater than or equal to the threshold value, and a third step of transmitting the log restricted in the second step to the analysis system. It is another feature.

delete

The present invention also provides a log generating system for generating a log according to an event occurrence and transmitting the log to a log analysis system, comprising: a generation unit generating a log according to the event occurrence, and determining whether a predetermined time has elapsed from the time of generating the log. In addition, as a result of the determination, if the predetermined time has elapsed from the time of generating the log, another feature is to include a sampling unit for excluding the log from the processing target.

In addition, the present invention provides a log generating system for transmitting a log generated in accordance with the occurrence of the event to the log analysis system according to the CPU utilization rate, the generation unit for generating the file in response to the occurrence of the event, and checks the CPU utilization, When the CPU utilization rate is greater than or equal to the threshold value, and including a sampling unit for limiting the number of analysis target logs to be transmitted to the analysis system from the generated log, and a transmission unit for transmitting the log restricted by the sampling unit to the analysis system; It is another feature.

According to the sampling method and system of the present invention, the amount of logs to be processed can be reduced. By limiting the log to a predetermined number for each log generation point, excluding the log that has been delayed from processing, or sampling the log according to the utilization of the CPU allocated to the log generation system, the log analysis delay is prevented and the CPU resources The load can be reduced. Therefore, it is possible to analyze a large amount of logs in real time log, which can be usefully used in security system management.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

The present invention relates to a method of generating a log according to an event occurrence in a log generation system and sampling a log in a process of transmitting the log to a log analysis system. Logs should be sent to the analysis system immediately after creation to be analyzed. However, as described above, when a large log is generated or CPU utilization is high, a log is sent to the analysis unit and a delay occurs in the analysis. Therefore, in the present invention, a sampling procedure is performed before the log is transmitted from the sampling unit of the log generation system to the log analysis system. Preferred embodiments in this regard will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating the configuration of a log generation system and a log analysis system to which the present invention is applied. As illustrated, the log generating system 110 includes a log generating unit 112, a log sampling unit 114, and a log transmitting unit 116. The log analyzing system 150 includes a log collecting unit 152, The log analyzer 154 is provided.

The log generator 112 of the log generation system 110 generates a log according to an event occurrence. If multiple events occur at the same time, multiple logs are generated accordingly. Therefore, a plurality of logs having the same generation point may exist, and the generation point of the log may be confirmed by a time stamp.

The log sampling unit 114 may select an analysis target log to be transmitted to the log analysis system from the generated logs based on the generation time of the log. Alternatively, a log delayed for more than a predetermined time may be discarded, and sampling may be performed according to the utilization of the CPU. The sampling method will be described in detail with reference to FIGS. 3 and 5 to 7.

The log transmitter 116 transmits the log to the analysis system. At this time, the transmission may be performed in consideration of the utilization of the CPU. Related contents will be described in detail with reference to FIG. 4.

The log collector 152 of the log analysis system 150 receives a log from the log generation system. The log analyzer 154 analyzes the logs collected by the log collector 152.

In the above-described embodiment of the present invention, the log is generated, sampled, and transmitted in the log generation system, and the log analysis system is illustrated as collecting and analyzing the log, but this is only one embodiment. A series of processes such as log generation and analysis may be performed in the same system, and the design of the system may be modified and changed by those skilled in the art to which the present invention pertains within the spirit of the present invention. Can be. In addition, the sampling method according to the present invention can be applied not only to log sampling but also to all sampling.

2 is a diagram illustrating a flow of processes in a log generation system to which the present invention is applied.

When an event such as a network connection event occurs (S210), a log is generated accordingly (S220). The generated log is sampled (S230), and the sampled log is transmitted to the log analysis system (S240). This series of steps is repeated according to the log generation. The sampling step can always be applied to the generated log, or can only be applied if a delay occurs in the transmission and analysis of the log. In other words, the sampling process may be performed only when a predetermined time elapses from the generation time confirmed through the log time stamp. In the log sampling method of the present invention, a method of selecting based on a generation time, a method of excluding a delayed log from a processing target, and sampling a log to be analyzed according to CPU usage. Each method is described in detail with reference to the drawings below.

3 is a view illustrating a flow of the log selection method according to the present invention, and the log sampling step S230 of FIG. 2 is described in detail.

The conventional selection process is performed based on the probability. For example, in the case of selecting 1: 100, one log is selected for every 100 logs. Since this method probabilistically selects without considering the generation time of the log, it is impossible to select the log by reflecting all events evenly.

Therefore, the present invention selects a log in consideration of the generation time. First, the number of logs for each generation point is checked (S310). Through the check result, it is determined whether the number of logs is greater than or equal to a reference value for each log generation time (S320). As a result of the determination, when the number of logs is greater than or equal to the reference value by generation time, only a part of the logs is selected and the number of logs is limited to the reference value (S330). Here, the reference value refers to a value in which the operator designates how many of a plurality of logs having the same generation time point. The reference value may be specified as a fixed value, or may be specified as a percentage of the number of logs having the same generation point.

Table 1 shows a comparison of the selection results by the conventional selection method and the selection method of the present invention.

Creation time t t + 1 t + 2 Number of logs generated 1000 20000 20 Selection by prior art 100 200 0 Selection by fixed reference value 100 100 20 Selection by a certain ratio 100 2000 2

For example, it is assumed that only 100 logs are selected for the same generation time point by the selection method of the present invention. Here, the reference value is 100. The sampling unit checks the generation time through the log time stamp. Since there are 1000 logs generated at generation time t, 20000 logs generated at generation time t + 1, and 20 logs generated at generation time t + 2, the selection method is as follows. First, only 100 logs generated at t are selected and the remaining 900 are excluded from processing. At this time, the 100 selections may be selected based on the fact that the creation time is fast, and may be selected by various other methods. Only 100 logs generated at t + 1 are selected and the remaining 19900 are excluded from processing. All twenty logs generated at t + 2 are selected. Here, the unselected log may be stored in the storage unit.

It is assumed that only 10% of logs are selected for the same generation time point by the selection method of the present invention. The reference value here is 10%. Only 100, 10% of the logs generated at t, are selected and the remaining 900 are excluded from processing. Only 2000 logs generated at t + 1 are selected and the remaining 18000 are excluded from processing. Two logs generated at t + 2 are selected. Here, the unselected log may be stored in the storage unit.

If 1:10 is selected by the conventional selection method in the same case, 100 logs generated at t, 200 logs generated at t + 1, and 0 logs generated at t + 2 are selected. Therefore, we cannot analyze the information about the events occurring at t + 2.

Logs that occur at the same time have similar records. Therefore, when a large log occurs, it is important to select which log to analyze. In particular, security systems require analysis of various events. Therefore, the selection method based on the creation time point according to the present invention enables efficient log analysis.

4 is a diagram illustrating a flow of a transmission method to a log analysis system of the present invention, and the step (S240) of transmission to the log analysis system of FIG. 2 will be described in detail.

Prior to transmitting the log to the analysis system, the log generation system may determine whether the limit value is exceeded by checking the utilization of the CPU (S410). If the CPU utilization is too high, the whole system (where the whole system refers to the whole system including the log generation system and the log analysis system) may not perform its original function. Here, the limit value may mean the utilization of the CPU allocated to the log system to the extent that the entire system does not burden the original function, and may mean the CPU utilization of the entire system. Alternatively, it may be assigned to an appropriate value by the user.

delete

Through this, log generation and transmission can be automatically controlled, so maximum real time can be guaranteed with minimum CPU resources. Therefore, the present invention can be applied to all cases, such as the case where a large amount of logs are transmitted using a large number of CPU resources, but the real-time properties are inferior but a minimum CPU resource is desired.

If the utilization rate of the CPU exceeds the threshold value, the step of transmitting the log is performed after a predetermined time (S430). If the utilization rate of the CPU does not exceed the limit value, the log is transmitted to the log analysis system (S420).

5 is a diagram illustrating a flow of a sampling method for excluding a log whose transmission is delayed according to the present invention from being processed.

The generation time point at which the log is generated may be checked by checking the time stamp of the log (S510). It is determined whether a predetermined time has elapsed from the generation time point identified through the time stamp (S520). If the predetermined time has elapsed, that is, when the transmission of the log is delayed, the log is excluded from the processing target (S530). ). Here, the predetermined time becomes a criterion for determining whether the log transmission is delayed or not. If a predetermined time elapses from the time of creation, it is determined that the transmission is delayed. Logs are generated based on event occurrences, which can generate thousands or tens of thousands of logs per second. Therefore, when real-time analysis of the log is required, such as security system management, it is not meaningful to analyze the log that has been delayed for a considerable time. Therefore, the delayed log is excluded from the processing. Excluding from processing may be performed by moving a file pointer. Since the log is processed sequentially according to the creation order, the delayed files can be excluded from processing by moving the file pointer to the last file, that is, the most recently created file.

 For example, when one minute has elapsed since the log is generated, a predetermined time for determining whether or not to delay may be specified as one minute when it is considered to be a meaningless log and excluded from processing. Therefore, when the log has been checked for time stamps, if a minute has already passed since the time of creation, all the logs that have not been transmitted to the current analysis system, that is, the delayed logs, are excluded from processing. At this time, the delayed log can be stored in the storage and used later for analysis.

6 is a diagram illustrating a flow of a method of sampling a log in which a transmission of the present invention is delayed.

A procedure of additionally performing a sampling step according to the selection method described in FIG. 3 with respect to the newly generated log except the log accumulated due to delay in transmission to the analysis system is shown. A significant delay in log transfer means that there is a burden on the current CPU resource load. Therefore, the log that has been delayed for a long time is not only excluded from processing, but also a sampling method is performed on newly generated logs.

The generation point of the log is checked (S610), and it is determined whether a predetermined time has elapsed from the generation point (S620). If it is determined that the log transmission is delayed, the accumulated log is excluded from the processing target (S630). As a result of the determination, if the log transmission is not delayed, the number of logs for each generation time is checked (S640). Through the check result, it is determined whether the number of logs is greater than or equal to the reference value for each log generation time (S650). As a result of the determination, when the number of logs is greater than or equal to the reference value for each generation time, only a part of the logs are selected and the number of logs is limited to the reference value (S660), and the limited number of logs are transmitted to the analysis system (S670). The transmitting step may be performed after a predetermined time has elapsed according to the utilization rate of the CPU.

7 is a diagram illustrating a flow of a sampling method according to the CPU utilization rate of the present invention.

The analysis result of the log generated according to the occurrence of the event can be used for various purposes such as checking network access contents and preventing illegal access attempts. However, allocating an excessive amount of CPU resources for log transmission and analysis may hinder the performance of the entire system.

Therefore, the sampling method according to the CPU utilization rate of the present invention first checks the CPU utilization rate (S710), and determines whether or not the threshold value is exceeded (S720). Here, the limit value may mean a value when the CPU utilization allocated to the log generating system does not interfere with the performance of the entire system. As a result of the determination, if the limit value is not exceeded, the log is transmitted to the analysis system (S740). As a result of the determination, if the limit value is exceeded, the number of logs to be analyzed is limited. The restriction may be performed by a selection method based on the generation time described above, or a method of excluding a delayed log from processing. The restricted log is transmitted to the analysis system (S740).

8 is a flowchart illustrating a log generation and sampling method according to an access event generation according to an embodiment of the present invention.

Firewalls use logs to record their connections and can be used for a variety of purposes. At this time, if more users access the network, a large capacity log is generated, which may delay the process of transmission and analysis. Therefore, the log sampling method and system of the present invention can be used for real-time processing of such a large log.

When the user connects to the network (S810), a connection event occurs (S815), and an event log for recording the connection contents according to the connection event is generated (S820). By checking the time stamp of the log (S830), it is determined whether there is a log transmission delay. At this time, the time for determining whether the log transmission is delayed is regarded as the first limit time. If it is determined that the first limit time has not been exceeded, it is determined that the log transmission has not been delayed yet and the CPU utilization rate is checked (S880). If the CPU usage exceeds the threshold, wait a predetermined time (S890). This is to prevent the load on the system due to log transfer. If the CPU utilization does not exceed the limit, the event log is immediately transmitted to the analysis system (S895).

 When it is determined that there is a delay in log transmission as a result of the time stamp check (S840) In other words, when it is determined that the log has passed the first limit time from the time of creation, it is determined whether the log is processed. At this time, the time for determining whether the log is processed is regarded as the second limit time. If the analysis of a log that is delayed for too long a transmission is not meaningful, the log is not processed. Therefore, when the second limit time is exceeded, the delayed log is excluded from the processing target (S860). Move the file pointer to the end of file (EOF), that is, the most recently created log. Therefore, the log which is stuck due to transmission delay is excluded from processing. If the log has some delay but needs to be processed, that is, if the first time limit is exceeded but the second time has not yet exceeded the second time, the log is sampled to process only a part of the log whose transmission is delayed.

For example, the first limit time may be set to 10 seconds, and the second limit time may be set to 1 minute. In this case, 10 seconds is a determination criterion for delay, and 1 minute is a determination criterion for processing. Therefore, the log within 10 seconds from the time of creation is not delayed, so it is processed normally. The log between 10 seconds and 1 minute has a slight delay, but it is not so severe that the log is first processed and sampled. Logs exceeding 1 minute are considered to be delayed for too long and are excluded from processing.

1 is a block diagram of a log generation system and log analysis system to which the present invention is applied.

2 is a flowchart of a process in a log generation system to which the present invention is applied.

3 is a flow chart of the log selection method of the present invention.

4 is a flowchart of a transmission method to a log analysis system of the present invention.

5 is a flowchart of a sampling method for excluding a log whose transmission is delayed according to the present invention from being processed.

6 is a flow chart of a method for sampling a log of delayed transmission of the present invention.

7 is a flowchart of a sampling method according to the CPU utilization rate of the present invention.

8 is a flowchart illustrating a log generation and sampling method according to an access event occurrence according to an embodiment of the present invention.

Claims (15)

delete delete delete In the log sampling method in the log generation system for generating a log according to the occurrence of the event and transmitting to the log analysis system, A first step of determining whether a predetermined time has elapsed from the time of generating the log; And As a result of the determination in the first step, when the predetermined time has elapsed from the time of generating the log, the second step of excluding the log after the predetermined time has not been processed Log sampling method comprising a. The method of claim 4, wherein As a result of the determination in the first step, If a predetermined time has not elapsed since the log generation time, A third step of determining whether the number of generated logs is equal to or greater than a reference value for each generation time of the logs in order to select an analysis target log to be transmitted from the generated logs to the analysis system; A fourth step of limiting the number of logs to the reference value for each generation time when the number of logs is greater than or equal to the reference value for each generation time as a result of the determination in the third step; And A fifth step of transmitting the log restricted in the fourth step to the analysis system Log sampling method characterized in that it further comprises. The method of claim 5, wherein The fifth step, If the CPU utilization is more than the predetermined reference value, the operation is performed after the predetermined time. Characteristic log sampling method. In the log sampling method in the log generation system for transmitting the log generated according to the event occurrence to the log analysis system according to the CPU utilization, A first step of checking a corresponding CPU utilization rate; A second step of determining whether a predetermined time has elapsed from a log generation time when the identified CPU usage rate is greater than or equal to a threshold value; And As a result of the determination in the second step, when the predetermined time has elapsed from the time of generating the log, the third step of excluding the log after the predetermined time has not been processed Log sampling method comprising a. The method of claim 7, wherein As a result of the judgment at the second stage, If a predetermined time has not elapsed since the log generation time, A fourth step of determining whether the number of generated logs is equal to or greater than a reference value for each generation time of the logs in order to select an analysis target log to be transmitted from the generated logs to the analysis system; A fifth step of limiting the number of logs to the reference value for each generation point when the number of logs is greater than or equal to the reference value for each generation point as a result of the determination in the fourth step; And A sixth step of transmitting the log of which the number of generation points by the fifth step is limited to the analysis system; Log sampling method characterized in that it further comprises. The method of claim 8, The reference value is, A fixed percentage or a percentage of the number of logs with the same generation Log sampling method. The method of claim 8, The sixth step If the CPU utilization is above the threshold, Characteristic log sampling method. delete In the log generation system for generating a log according to the occurrence of the event and transmitting to the log analysis system, A generation unit generating a log according to the event occurrence; And It is determined whether or not a predetermined time has elapsed from the time of generating the log, and as a result of the determination, when the predetermined time has elapsed from the time of generating the log, a sampling unit which excludes the log whose time has elapsed from the processing target. Log generation system comprising a. In the log generation system for transmitting the log generated according to the event occurrence to the log analysis system according to the CPU utilization, A generating unit generating the log according to the event occurrence; Checking the CPU utilization rate, and if the identified CPU utilization rate is greater than or equal to the threshold value, the sampling that limits the number of analysis target logs to be transmitted to the analysis system by excluding a log that has elapsed a predetermined time since the log generation time is not processed. part; And A transmission unit for transmitting the log of which the number is limited by the sampling unit to the analysis system Log generation system comprising a. The method of claim 13, The sampling unit Logs that have not elapsed for a predetermined time since the log is generated, It is determined whether the number of logs generated by each generation time is greater than or equal to the reference value. If the number of logs is greater than or equal to the reference value by generation time, limiting the number of logs by the generation time to the reference value is determined. Characterized in the log generation system. The method of claim 14, The reference value is, A fixed percentage or a percentage of the number of logs with the same generation Log generation system.

Images