JP2005196675A - Log analysis device, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a log analysis system, a log analysis method and a log analysis program capable of objectively estimating the transition of the number of future events. <P>SOLUTION: A log collecting part 101 collects logs output from each device of networks 20-22, and outputs the same to a log saving part 102. The log saving part 102 extracts the parameter used in analysis from the logs output from the log collecting part 101, and stores the parameter in a storage part 103. The log analysis part 104 reads out the parameter to be analyzed from the storage part 103, and calculates a ratio D by ratio analysis. Continuously, the log analysis part 104 calculates conditional probability of occurrence of specific event in a case when specific conditions relating to the ratio D are satisfied, saves a result of the calculation in the storage part 103, and outputs the same to a Web browser 301 through an interface part 105. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a log analysis device, a log analysis method, and a log analysis program that analyze logs output from network devices such as IDS (Intrusion Detection System), Router, and Firewall. .

  In recent years, more and more sites have introduced IDS for monitoring attacks on network systems. In general, IDS simply compares a packet flowing on a network with an attack pattern file called Attack Signature, and outputs a log if there is a match.

  In order to efficiently analyze a redundant log output from the IDS, there is a system that visualizes the log and notifies an abnormality (see Non-Patent Document 1). However, in the system described in Non-Patent Document 1, the judgment of abnormality is left to the subjectivity of the operator, and the reliability of the results is not uniform.

  On the other hand, Non-Patent Document 2 proposes a method for evaluating abnormality using objective numerical values by ratio analysis and rare rate analysis. In the ratio analysis, the ratio of the number of events per unit time of interest to the average value per unit time of the number of events detected in the past long time recorded in the log is obtained. Further, in the rare rate analysis, the anomaly of an event whose output number fluctuates in various ways is evaluated using the average and standard deviation of the number of events detected in the past for a long time. In the statistical analysis, an index of 95% confidence interval is often used. The rare rate analysis here is to calculate a complement of this confidence interval.

Traditionally, new security holes are discovered on the Internet every day, and viruses and worms are born. FIG. 15 is a reference diagram showing the transition of the attack frequency against Port 1434 detected by IDS installed in various parts of the world. From FIG. 15, it can be seen that the attack on Port 1434 suddenly becomes active around January 24, 2003. This is an attack by the Slammer worm and is a cyber terror that has caused global problems. In FIG. 15, when attention is paid before January 24, 2003, it can be seen that a predictable event was actually detected by IDS. If we were able to maintain a warning posture at the time of this sign, it is expected that the subsequent spread of damage could be suppressed. In addition, it is desirable to estimate the transition after March 7 as of March 7.
Tetsuji Takada, 1 other person, “Appearance Log: Log Information Browser Using Information Visualization and Text Mining”, Journal of Information Processing Society of Japan, December 2000, Vol 41, No. 12, p. 3265-3275 Keisuke Takemori, 4 others, “Application of Security Device Log Analysis Support System to Wide Area Monitoring”, Computer Security Symposium (CSS 2003), October 2003, p. 397-402

  However, in the conventional log analysis method, the current abnormal log can be analyzed, but the future transition cannot be predicted. In recent attacks by Slammer worms and Blaster worms, the Internet has become impossible 10 minutes after the attacks start to become popular (attack reaches criticality and starts to spread rapidly), and the threat of cyber terrorism is highlighted became. Therefore, it is an urgent task to predict the transition of the number of future events by statistically learning the transition of the number of past and current events recorded in the log.

  The present invention has been made in view of the above-described problems, and an object thereof is to provide a log analysis system, a log analysis method, and a log analysis program that can objectively predict the transition of the number of future events. And

The present invention has been made to solve the above problems, and the invention according to claim 1 is output from the network device in a log analysis device that performs analysis processing based on a log collected from the network device. A collecting unit that collects a log; an abnormal value calculating unit that calculates an abnormal value related to a degree of abnormality of the network based on the number of recorded events belonging to the parameter in the log collected by the collecting unit; and A log analysis apparatus comprising: probability calculation means for calculating a conditional probability that a predetermined event occurs when a predetermined condition is satisfied.
Examples of the parameters in the previous period include Attack Signature, Source / Destination Port, and Source / Destination IP recorded in logs output from network devices such as IDS, Router, and Firewall. The abnormal value calculating means performs a ratio analysis or a rare rate analysis based on the number of recorded events, and calculates an abnormal value related to the degree of abnormality of the network to be analyzed. Examples of the abnormal value include a ratio in a ratio analysis, an upper rare rate in a rare rate analysis, and a lower rare rate.

  According to a second aspect of the present invention, in the log analysis device according to the first aspect, the abnormal value calculating unit is configured to select a predetermined time unit from events belonging to the parameters in the log collected by the collecting unit. And an abnormal value related to the degree of network abnormality is calculated based on the number of recorded events.

  According to a third aspect of the present invention, in the log analysis device according to the first or second aspect, the abnormal value calculating means includes an event belonging to a parameter in the log collected by the collecting means. A specific event is selected based on public information related to the vulnerability, and an abnormal value related to the degree of network abnormality is calculated based on the number of recorded events.

  According to a fourth aspect of the present invention, in a log analysis apparatus that performs analysis processing based on a log collected from a network device, a collection unit that collects a log output from the network device, and the collection unit that collects the log Grouping means for grouping events belonging to parameters in the log into a plurality of groups, and calculating an abnormal value related to the degree of network abnormality based on the number of recorded events grouped by the grouping means A log analysis apparatus comprising: an abnormal value calculating means for calculating a conditional probability that a predetermined event occurs when a predetermined condition is satisfied with respect to the abnormal value; .

  According to a fifth aspect of the present invention, in the log analysis device according to the fourth aspect, the grouping unit selects a predetermined time unit from events belonging to the parameters in the log collected by the collecting unit. An event is selected, and grouping is performed to divide the event into a plurality of groups.

  According to a sixth aspect of the present invention, in the log analysis device according to the fourth or fifth aspect, the grouping unit is vulnerable from events belonging to the parameters in the log collected by the collecting unit. A specific event is selected based on public information regarding sex, and grouping is performed by dividing the event into a plurality of groups.

  The invention according to claim 7 is the log analysis device according to any one of claims 1 to 6, wherein the predetermined event is an event in which the number of recorded events belonging to a parameter in the log increases. Alternatively, it is an event in which the number of recorded events belonging to the parameter in the log decreases.

  The invention according to claim 8 is a log analysis method for performing an analysis process based on a log collected from a network device, the step of collecting a log output from the network device, and belonging to a parameter in the collected log Calculating an abnormal value related to the degree of abnormality of the network based on the number of recorded events, and calculating a conditional probability that a predetermined event occurs when a predetermined condition related to the abnormal value is satisfied. It is the log analysis method characterized by comprising.

  According to a ninth aspect of the present invention, in the log analysis method according to the eighth aspect, in the step of calculating an abnormal value related to the degree of abnormality of the network based on the number of recorded events belonging to the parameter in the collected log. Selects an event in a predetermined time unit from among the events belonging to the collected parameters of the log, and calculates an abnormal value related to the degree of network abnormality based on the number of recorded events. .

  According to a tenth aspect of the present invention, in the log analysis method according to the eighth or ninth aspect, the abnormal value related to the degree of abnormality of the network is calculated based on the number of recorded events belonging to the collected parameters of the log. In the calculating step, a specific event is selected from the collected events belonging to the parameters in the log based on public information about the vulnerability, and the network abnormality level is determined based on the number of recorded events. An abnormal value is calculated.

  The invention according to claim 11 is a log analysis method for performing analysis processing based on a log collected from a network device, the step of collecting a log output from the network device, and belonging to a parameter in the collected log A step of grouping the event into a plurality of groups, a step of calculating an abnormal value related to the degree of abnormality of the network based on the number of recorded events of the group, and a predetermined condition for the abnormal value is satisfied And a step of calculating a conditional probability that a predetermined event occurs in the event of occurrence of a predetermined event.

  According to a twelfth aspect of the present invention, in the log analysis method according to the eleventh aspect, in the step of grouping the events belonging to the parameter in the collected log into a plurality of groups, the collected log An event of a predetermined time unit is selected from among the events belonging to the middle parameter, and grouping is performed to divide the event into a plurality of groups.

  According to a thirteenth aspect of the present invention, in the log analysis method according to the eleventh or twelfth aspect, in the step of performing grouping to divide events belonging to the parameters in the collected logs into a plurality of groups, A particular event is selected from the collected events belonging to the parameters in the log based on public information related to the vulnerability, and grouping is performed to divide the event into a plurality of groups.

  The invention according to claim 14 is the log analysis method according to any one of claims 8 to 13, wherein the predetermined event is an event in which the number of recorded events belonging to the parameter in the log increases. Alternatively, it is an event in which the number of recorded events belonging to the parameter in the log decreases.

  According to a fifteenth aspect of the present invention, there is provided a log analysis program for causing a computer to execute analysis processing based on a log collected from a network device, collecting a log output from the network device, and collecting the collected log A step of calculating an abnormal value related to the degree of abnormality of the network based on the number of recorded events belonging to the internal parameters, and a conditional probability that a predetermined event occurs when a predetermined condition related to the abnormal value is satisfied. A log analysis program for causing a computer to execute a calculating step.

  According to a sixteenth aspect of the present invention, in the log analysis program according to the fifteenth aspect, in the step of calculating an abnormal value related to the degree of abnormality of the network based on the number of recorded events belonging to the parameter in the collected log. Selects an event in a predetermined time unit from among the events belonging to the collected parameters of the log, and calculates an abnormal value related to the degree of network abnormality based on the number of recorded events. .

  According to a seventeenth aspect of the present invention, in the log analysis program according to the fifteenth or sixteenth aspect, an abnormal value related to the degree of abnormality of the network is calculated based on the number of recorded events belonging to the parameters in the collected log. In the calculating step, a specific event is selected from the collected events belonging to the parameters in the log based on public information about the vulnerability, and the network abnormality level is determined based on the number of recorded events. An abnormal value is calculated.

  The invention according to claim 18 is a log analysis program for causing a computer to execute an analysis process based on a log collected from a network device, a step of collecting a log output from the network device, and the collected log A step of grouping the events belonging to the internal parameters into a plurality of groups, a step of calculating an abnormal value related to the degree of abnormality of the network based on the number of recorded events of the group, and a predetermined value related to the abnormal value A log analysis program for causing a computer to execute a step of calculating a conditional probability that a predetermined event occurs when the above condition is satisfied.

  According to a nineteenth aspect of the present invention, in the log analysis program according to the eighteenth aspect, in the step of grouping the events belonging to the parameters in the collected logs into a plurality of groups, the collected logs An event of a predetermined time unit is selected from among the events belonging to the middle parameter, and grouping is performed to divide the event into a plurality of groups.

  According to a twentieth aspect of the present invention, in the log analysis program according to the eighteenth or nineteenth aspect, in the step of performing grouping that divides events belonging to the parameters in the collected logs into a plurality of groups, A particular event is selected from the collected events belonging to the parameters in the log based on public information related to the vulnerability, and grouping is performed to divide the event into a plurality of groups.

  According to a twenty-first aspect of the present invention, in the log analysis program according to any one of the fifteenth to twentieth aspects, the predetermined event is an event in which the number of recorded events belonging to the parameter in the log increases. Alternatively, it is an event in which the number of recorded events belonging to the parameter in the log decreases.

  The invention described in claim 22 is a computer-readable recording medium on which the log analysis program according to any one of claims 15 to 21 is recorded.

  According to the present invention, based on the number of events belonging to parameters in logs output from various network devices, an abnormal value related to the degree of network abnormality is calculated, and a predetermined condition related to the abnormal value is satisfied. In this case, since the conditional probability that the predetermined event occurs is calculated, an effect that the transition of the number of events in the future can be predicted objectively can be obtained.

  The best mode for carrying out the present invention will be described below with reference to the drawings. FIG. 1 is a configuration diagram showing the configuration of a network including a log analysis device according to an embodiment of the present invention. In the figure, 10 is a log analyzer. The log analysis device 10 collects and analyzes logs output from the networks 20 to 22 to be analyzed, and outputs the analysis results to the web browser 301. The log analysis device 10 may output the analysis result to another device having a display unit. When the log analysis device 10 includes a display unit, the log analysis device 10 displays the analysis result. May be.

  Further, the log analysis device 10 may be a device independent of the security operation center 30 including the web browser 301, and the device including the web browser 301 in the security operation center 30 has the function of the log analysis device 10. You may do it. The Router, Firewall, and IDS of the networks 20 to 22 generate logs, and output the logs to the log analysis device 10 by a communication method such as syslogd. The security operation center 30 has a web browser 301, and the operator can determine the degree of abnormality of the networks 20 to 22 based on the analysis result displayed on the web browser 301.

  In the log analysis device 10, reference numeral 101 denotes a log collection unit that periodically collects logs output from devices in the networks 20 to 22 and outputs the logs to the log storage unit 102. The log storage unit 102 extracts parameters used for analysis from the log output by the log collection unit 101 and stores them in the storage unit 103. When the operator requests a parameter desired for analysis via the Web browser 301, instruction information for instructing analysis is input to the log analysis unit 104 via the interface unit 105.

  The log analysis unit 104 reads out an analysis parameter from the storage unit 103 in accordance with the instruction information, analyzes the parameter (details will be described later), saves the analysis result in the storage unit 103, and outputs it to the interface unit 105 To do. The interface unit 105 outputs the analysis result output from the log analysis unit 104 to the Web browser 301 via a communication line. Based on the analysis result displayed on the Web browser 301, the operator determines the status of the attack on the networks 20-22.

  Next, parameters to be analyzed in this embodiment will be described. FIG. 2 shows parameters to be analyzed in the log in this embodiment. In the present embodiment, events belonging to parameters of Attack Signature, Source / Destination Port, and Source / Destination IP are grouped, and a statistical anomaly value of each parameter group is calculated. Source / Destination Port indicates a port number of a transmission source / destination device. Source / Destination IP indicates the IP address of the source / destination device. Note that the Source / Destination Port indicates one of the two parameters of the Source Port and the Destination Port, and the same applies to the Source / Destination IP.

  The Attack Signature is a parameter included in the log output from the IDS. By analyzing the Attack Signature, it is possible to specify the type of attack being performed on the analysis target network. All network type IDSs and some host type IDSs can output an attack signature. Source / Destination Port and Source / Destination IP are parameters included in logs output from IDS, Router, and Firewall. By analyzing the Source / Destination Port, the port number of the attack source / attack target can be specified. Further, by analyzing the Source / Destination IP, the IP address of the attack source / attack target can be specified. The above parameters consist of a plurality of events as shown in FIG. For example, there are events such as Ping sweep and HTTP Port Probe in Attack Signature. When a plurality of the same events are recorded in the log, the total number is defined as the number of events of the event.

  Next, the operation of the log analysis apparatus 10 according to the present embodiment will be described. In the present embodiment, as an example, an operation in the case where the transition of the number of events is predicted using the ratio D by the ratio analysis will be described. FIG. 3 is a flowchart showing the operation of the log analysis apparatus 10 according to the present embodiment. The log collection unit 101 collects logs output from the devices of the networks 20 to 22 and outputs them to the log storage unit 102 (step S300). The log storage unit 102 extracts parameters used for analysis from the log output by the log collection unit 101 and stores them in the storage unit 103 (step S310). When an analysis request regarding a specific parameter is made from the operator via the Web browser 301, the instruction information output from the Web browser 301 is input to the log analysis unit 104 via the interface unit 105. Then, the log analysis unit 104 reads the analysis target parameter from the storage unit 103 based on the instruction information (step S320).

  The log analysis unit 104 performs an analysis process to be described later on the analysis target parameter, and stores the analysis result in the storage unit 103 (step S330). Subsequently, the log analysis unit 104 outputs the analysis result to the web browser 301 via the interface unit 105 (step S340). An analysis result is displayed on the Web browser 301, for example, “the future number of XX events is a 30% increase probability and a 70% decrease probability.” Based on this analysis result, the operator The transition of future attacks on 20-22 is determined.

  Next, details of the analysis processing in step S330 in FIG. 3 will be described. FIG. 4 is a flowchart showing analysis processing by the log analysis unit 104. The log analysis unit 104 performs ratio analysis or rare rate analysis based on the number of events in the parameters read from the storage unit 103 in step S320 of FIG. 3 (step S3301). Subsequently, the log analysis unit 104 calculates a predicted value based on the result of the ratio analysis or the rare rate analysis, and stores the predicted value in the storage unit 103 (step S3302).

Next, details of the ratio analysis and the rare rate analysis will be described. FIG. 5 shows an example of the ratio analysis model. In the figure, E _{1 to} E ₁₁ are bar graphs showing the number of events per unit time of a specific event recorded in the log. In this example, the unit time is one day. E _l represents the average of the number of events in a plurality of past unit times (E _{1 to} E ₁₀ ). E _s indicates the number of events of the unit time E ₁₁ of interest. The ratio D as an index in the ratio analysis is expressed by the following formula.

FIG. 6 shows an example of a rare rate analysis model. _El is an average of the number of events in a plurality of past unit times (E ₁ , E ₂ ,... E _k ), and SD is a standard deviation. The abscissa represents the distance from the average value _El , with the standard deviation SD as a unit. The vertical axis represents the number of unit times (for example, the number of days) in which the number of events of interest is the value indicated by the horizontal axis in a plurality of past unit times. The standard deviation SD is obtained from [Equation 2]. The upper rare rate R _u when the number of events per unit time of interest is larger than the long-term profile E _l is defined as [Equation 3], and the lower rare rate R _l when it is smaller is defined as [Equation 4]. In [Equation 3] and [Equation 4], f (E) is a density function of a normal distribution and is expressed by the following [Equation 5]. Note that a density function such as an exponential distribution or a gamma distribution may be appropriately selected and used as f (E).

The determination regarding the ratio D, the upper rare ratio R _u and the lower rare ratio R ₁ calculated by the ratio analysis and the rare ratio analysis is as follows. First, if D> 1.0 or R _u ≈0.0%, a new attack is circulating on the Internet, the internal host is infected with a worm, a DDoS (Destroyed Denial of Service) attack (An attacker can invade multiple websites that are vulnerable to security on the Internet and use them as attack bases, and simultaneously run programs installed at multiple attack bases to send a large number of packets to the attack target server. This shows that the number of short-term events has increased rapidly due to attacks that stop the function.

When D≈0.0 or R ₁ ≈0.0%, the alarms that have been continuously output are suddenly reduced or eliminated, or the device is stopped due to an attack. This is the stage where the attack is converging, or the system is stopped. Further, when D≈1.0 or R _u , R _l ≈50.0%, it is possible to distinguish a log that has been continuously detected. This is an attack for which countermeasures have already been taken, or an event that is likely to be erroneously detected, and there is no need to pay particular attention.

Next, the calculation method of the predicted value in this embodiment is demonstrated. In the present embodiment, the predicted value is calculated by Bayesian inference. In the following, description will be made using the ratio D by the ratio analysis as an example, but the same applies to the case where the upper rare ratio R _u and the lower rare ratio R _l by the rare ratio analysis are used. The following [Formula 6] is a general formula of Bayesian inference.

  In [Formula 6], X represents a state, and P (X) is its appearance probability. a represents an observation result, and P (a) is its appearance probability. P (a | X) represents a conditional probability that a occurs when X is a condition. In order to obtain a predicted value using Bayesian inference, for example, parameters as shown in FIG. 7 are applied. FIG. 7 shows conditional probabilities that a (D> 1.0) or b (D ≦ 1.0) will occur when X (increase) or Y (decrease) is a condition.

Hereinafter, specific examples of X, Y, a, and b will be shown.
(Event example 1)
X: Event that the number of events (detected number) on the next day has increased from the current day Y: Event number that the number of events (detected number) on the next day has decreased from the current day a: Ratio D related to the number of events on the current day The event that D> 1.0 with respect to D b: the event that D ≦ 1.0 with respect to the ratio D related to the number of events on the current day The conditional probability P (a | X) of being is obtained by [Equation 7]. P (b | X), P (a | Y), and P (b | Y) can be obtained in the same manner.

  Further, P (X) and P (Y) are obtained by [Equation 8] and [Equation 9], respectively.

With respect to the ratio D, when D> 1.0 (event a), the probability that the number of detections will increase in the future (event X occurs) is obtained by [Equation 6]. Here, the denominator on the right side in [Expression 6] is expressed by the following equation.
P (a) = P (a | X) P (X) + P (a | Y) P (Y) (1)
By substituting the value obtained by [Equation 7] to [Equation 9] and the equation (1) into [Equation 6], it is possible to calculate the probability that the detected number of the next day will increase. P (X | b) can also be obtained by the following equation (2) or [Equation 10]. The log analysis unit 104 calculates the above-described probability based on the number of events belonging to the parameters read from the log storage unit 103.
P (Y | a) = 1−P (X | a) (2)

Other specific examples of X, Y, a, b, etc. are given below.
(Event example 2)
X: Event that D> 1.0 with respect to the ratio D related to the number of events on the next day Y: Event that D ≦ 1.0 with respect to the ratio D related to the number of events on the next day a: Number of events on the current day That the number of events increased from the previous day's number of events b: The number of events of the current day decreased from the previous day's number of events (Event Example 3)
X: Event that the number of events on the next day has increased from the current day Y: Event that the number of events on the next day has decreased from the current day a: Event that the number of events on the current day has increased from the number of events on the previous day b : Event that the number of events on the current day is less than the number of events on the previous day

(Event example 4)
X: Event that D> 1.0 with respect to the ratio D related to the number of events on the next day Y: Event that D ≦ 1.0 with respect to the ratio D related to the number of events on the next day a: Number of events on the current day An event that D> 1.0 with respect to the ratio D related to b: an event that D ≦ 1.0 with respect to the ratio D related to the number of events on the day (event example 5)
X: Event that the number of events on the next day has increased from the current day Y: Event that the number of events on the next day has decreased from the current day a: D> 10.0 for the ratio D related to the number of events on the current day B: an event that D> 1.0 and D ≦ 10.0 regarding the ratio D related to the number of events on the day c: D ≦ 1.0 and D> 0.1 regarding the ratio D related to the number of events on the current day D: an event that D ≦ 0.1 with respect to the ratio D related to the number of events on the current day. Note that the value of D (eg, 10.0) is used as a variable, and the optimal value is always confirmed. You may do it. That is, verification regarding the predicted value may be performed from the usual time, D that is most likely to be predicted is obtained, and prediction may be performed using the value.

(Event example 6)
X: Event that the number of events on the next day has increased from the current day Y: Event that the number of events on the next day has decreased from the current day a: D> 1.0 with respect to the ratio D related to the number of events on the current day An event that the number of events on the current day has increased from the previous day b: An event that D> 1.0 with respect to the ratio D related to the number of events on the current day, and a decrease in the number of events on the current day than the previous day c: D ≦ 1.0 with respect to the ratio D related to the number of events of the day, and an event that the number of events of the day has increased from the previous day d: D ≦ 1.0 with respect to the ratio D related to the number of events of the day, And the event that the number of events on that day decreased from the previous day (event example 7)
X: event that the number of events in the next time interval (unit time) has increased from the current time interval Y: event that the number of events in the next time interval has decreased from the current time interval a: in the previous time interval event that for the percentage D ₂ according to the number of events the ratio D ₁ and the current time interval according to the number of events is D _{1 <D 2} b: the number of events before the time interval the number of events in accordance ratio D ₁ and the current time interval The event that D ₁ ≧ D ₂ with respect to such a ratio D ₂

(Event 8)
X: Event that the number of events on the next day has increased from the current day Y: Event that the number of events on the next day has decreased from the current day a: Number of events on the current day more than double the number of events on the previous day Increasing event b: The number of events on the current day increased to 1 to 2 times the number of events on the previous day c: The number of events on the current day more than half the number of events on the previous day 1 D: Event that the number of events on the current day decreased to less than one-half the number of events on the previous day Note that the above two times and half times are more general Further, m times (1 <m) and n times (0 ≦ n <1) may be used.

  Next, application of the above-described predicted value calculation method will be described with reference to FIG. FIG. 8 is a graph showing changes in the number of detected packets detected on a daily basis at Port 27374 of the Internet Storm Center. The Internet Storm Center is the world's largest IDS that monitors traffic on the Web, and information such as the transition of the number of attacks detected by Ports with many attacks and the tendency of attacks is disclosed on the Web.

  In the graph of FIG. 8, the number of detections increases rapidly from October 13 to October 14. If the number of detections is recorded not in units of days but in units of time, when the number of detections increases rapidly, it can be predicted that the number of detections will increase rapidly thereafter. When a sudden increase is predicted, the number of detections is recorded in finer units such as minutes.

  In addition, there is a need to predict a decrease in attacks. In the graph of FIG. 8, it can be seen that the number of detections rapidly decreases from October 14 to October 15. When detecting the tendency that the increasing rate of the detection number gradually decreases, it is possible to predict the start of the decrease in the detection number.

  More specifically, the above phenomenon can be predicted by a combination of parameters in the event example 7 described above. For example, predicting whether the number of detections will continue to increase by calculating P (X | a) when the number of detections starts to increase rapidly from October 13 to October 14. Can do. Subsequently, P (X | b) is calculated after the point when the number of detections starts to increase suddenly, and whether or not the increase in the number of detections becomes moderate (whether the number of detections approaches the peak). Can be predicted.

  Further, by calculating P (Y | b) at the time point when the number of detected peaks is recorded, it can be predicted whether the number of detected decreases in the future. Subsequently, it is possible to predict whether or not the end of the attack is near by calculating P (Y | a) after the point when the number of detections starts to decrease rapidly.

  Next, a prediction method for an event that has hardly been detected in the past will be described. For example, there are events in Attack Signature that are detected only about a few in a month, and it is difficult to statistically estimate such an event with a low frequency of occurrence. Therefore, the prediction is performed taking into consideration the situation of other events that follow a similar transition. For example, the number of detected events in the long term is close to the number of detected events in the long term (more specifically, the number of detected events in the long term is the number of detected events in the long term That are within ± 5% of the above) are extracted, and the total number of those events is predicted. Thereby, even if it is an event with a low frequency, prediction with respect to the event group which collected the similar event can be performed.

  Next, event grouping will be described. In the prediction value calculation method described above, the prediction value is calculated individually for each event belonging to the parameter to be analyzed. However, the detection frequency is small in the event alone, and prediction is difficult as described above. There is a case. In addition, for example, there are about 2000 types of events in Attack Signature, and calculating predicted values for each of these events requires work man-hours and time, and it is difficult to make predictions in real time. .

  Therefore, by grouping a plurality of events, it is possible to increase the detection frequency, improve the estimation accuracy, and reduce the work man-hours and the calculation time. Hereinafter, a specific method of grouping will be described.

(Static grouping 1)
Attack Signatures are grouped by semantic content in consideration of the communication layer, attack purpose, and service. FIG. 9 and FIG. 10 are examples in which Attack Signatures in the log output from Snort IDS are grouped into 10 semantic units (classes). The number is the number of Attack Signatures included in each group. The Snort rule file is a file in which Attack Signatures are collected in a certain semantic unit. Each Attack Signature is originally assigned a key relating to vulnerability, and grouping is performed for each semantic content based on this key. The storage unit 103 stores in advance a table in which the key and the group are associated with each other, and the log analysis unit 104 performs event grouping with reference to the table.

(Static grouping 2)
Source / Destination IP is grouped by country or domain. FIG. 11 shows a conversion table for converting an IP address into a country name to which the IP address belongs. In the figure, for example, IP addresses of 0.0.0.0 to 0.255.255.255 belong to the US (United States), and 24.42.0.0 to 24.43.255.255 belong to the CA (Canada). I understand that

  This conversion table can be created from an IP address and a domain table managed by the IANA (The Internet Assigned Numbers Authority) or the like. A conversion table indicating conversion from an IP address to a domain can be created in the same manner. Also in this case, the log analysis unit 104 groups each event based on the conversion table in the storage unit 103.

  When performing the static grouping as described above, the log analysis unit 104 groups events for each predetermined unit, and stores information for identifying each group and information indicating the total number of events in the group. To store. The log analysis unit 104 groups events in the past unit time and events in the unit time of interest for each predetermined unit. Subsequently, the log analysis unit 104 regards each group after grouping as one event, performs the ratio analysis described above, and calculates a predicted value based on the results.

(Dynamic grouping 1)
With respect to the parameters to be analyzed, the events are rearranged in the order of the number of events, and grouping is performed so that the total number of events in each group is substantially equal based on the total number of events. FIG. 12 shows a state of grouping in this case. In the figure, each bar graph represents an event (HTTP Port Probe, Smurf Attack...), And the length of the bar in the vertical axis direction represents the number of events. In the figure, as an example, each event is arranged from the left in order of the number of events, and the group names G1, G2,... Are assigned from the left so that the total number of events in each group is almost equal. ing.

  In this case, the log analysis unit 104 groups events, and stores in the storage unit 103 information for identifying each group and information indicating the total amount of events in the group. In dynamic grouping 1, the log analysis unit 104 performs grouping as follows. For example, the events are divided into 6 groups, and the events are grouped so that the number of events in each group is about 1/6 of the total number of events. In FIG. 12, the number of events is added in order from the left event, and when the amount becomes 1/6 or more of the total number of events, a group is formed. At this time, for example, the lowest digit of the number of events is rounded off.

  Alternatively, the number of events is added from the left event, and one group is formed when the amount exceeds one-sixth of the total number of events. By the above method, the first grouping is performed, and for the next group, the second grouping is performed in the same manner as described above so that the number of events is about one fifth of the remaining number of events. Do. Further, for the next group as well, the third grouping is performed so that the number of events is about one-fourth of the number of remaining events. By repeating this, the event is divided into 6 groups. Note that the above-described method is an example, and any method can be used as long as it can be grouped so that the number of events in each group is substantially equal.

(Dynamic grouping 2)
With respect to the parameters to be analyzed, the events are rearranged in the order of event IDs, and grouping is performed so that the number of events in each group is almost equal based on the total number of events. Here, the ID is an Attack Signature number determined for each product in the case of Attack Signature, a Port number in the case of Source / Destination Port, and an IP address order in the case of Source / Destination IP. FIG. 13 shows the state of grouping in this case. The grouping method so that the total number of events in each group is almost equal is the same as that of dynamic grouping 1.

  In the figure, each bar graph represents a parameter (HTTP Port Probe, Smurf Attack...). In the figure, the group names G1, G2,... Are assigned from the left so that the total number of events in each group is substantially equal. The operation of the log analysis apparatus 10 in this case is the same as that of the dynamic grouping 1. This method is suitable for application to analysis of Attack Signature, Source / Destination Port, and Source / Destination IP.

  When performing dynamic grouping, the log analysis unit 104 performs dynamic grouping of events based on data of a certain unit time in the past, information for identifying each group after grouping, and the total in the group. Information indicating the number of events and information indicating a division form (such as the number of divisions, correspondence between event types and group identification information) are stored in the storage unit 103. Then, based on the information indicating the division form, the log analysis unit 104 groups the remaining events in each unit time and the events in the current unit time of interest. Subsequently, the log analysis unit 104 regards each group after grouping as one event, performs the ratio analysis described above, and calculates a predicted value based on the results.

  As described above, by grouping events, the number of variables to be calculated can be reduced, so that the time required to calculate an abnormal value can be reduced. In addition, it is possible to greatly grasp the log tendency of the entire network by grouping, and it is possible to improve the reliability of analysis by reducing false detection regarding abnormality.

  FIG. 14 is a flowchart illustrating an example of analysis processing when the log analysis unit 104 groups events in the analysis processing in step S330 of FIG. The log analysis unit 104 groups events in the parameters read from the storage unit 103 in step S320 of FIG. 3 (step S3303). Subsequently, the log analysis unit 104 performs ratio analysis or rare rate analysis based on the total number of events in each group (step S3304). Subsequently, the log analysis unit 104 calculates a predicted value based on the result of the ratio analysis or the rare rate analysis, and stores the predicted value in the storage unit 103 (step S3305).

  In the grouping described above, the number of groups may be selectable by the user. In addition, all events may be grouped together, in which case, the total number of all events belonging to a specific parameter will be predicted, and an overview of predicting increase / decrease of attacks without specifying attacks Suitable for grasping.

  In the analysis process described above, a method is used in which the network to be analyzed is fixed, and how abnormal the number of events per specific unit time is relative to the past number of events per unit time. Although shown, the following analysis processing can also be performed. That is, when the period to be analyzed is fixed, the number of events related to a specific parameter detected in a specific network (defined as the local network profile) is the same specific parameter detected in other networks. This is a method for evaluating how abnormal the event is with respect to the average number of events per network (defined as other network profile).

For example, in the ratio analysis, if the local network profile in the network to be analyzed is E _m and the other network profile in each of the other networks not including this network is E _o , the ratio D is expressed by the following [Equation 11]. expressed. Similarly, the upper and lower rare rates in the rare rate analysis can be calculated. By calculating the predicted value using these values, it is possible to know the probability that the number of packets from other networks to the network of interest will increase or decrease in the future.

  In the above embodiment, a method for narrowing the prediction range is also effective. Many cyber terrorism occurs 2 to 4 weeks after the vulnerabilities of OS (Operating System) and protocols are released on the Internet. For example, the MS-Blaster worm spread on the Internet in mid-August 2003 after information on OS vulnerabilities was released in mid-July 2003. Therefore, by considering the vulnerability public information, it is possible to narrow the prediction range and improve the prediction accuracy.

  As public information of this vulnerability, for example, a specific Port number can be cited. The user confirms the Port number on the Internet, and inputs the Port number from an operation unit (not shown) of the log analysis apparatus 10. This Port number is stored in the storage unit 103, and the log analysis unit 104 selects only an event related to the Port number and performs the above-described analysis processing. The above-described vulnerability public information includes the OS type and version, the application type and version, the communication protocol type and version, the hardware type and version, etc., and the analysis processing described above for only the events related thereto. May be performed.

  In each event example in the present embodiment, the unit time is a day, but the unit time may be seconds, minutes, hours, weeks, months, years, or the like. Further, the unit time may be set as follows. For example, if the day of interest is Monday and you want to obtain a predicted value for the number of events on the next day (Tuesday), you can use the past Monday as past unit time data for ratio analysis and rare rate analysis. Detected event data. In addition, when the time of interest is 12:00 and it is desired to obtain a predicted value related to the number of events at 1 o'clock, as past unit time data in ratio analysis or rare rate analysis, the past 11:00 to 12:00 Detected event data.

  In this case, the log analysis unit 104 reads an event in a predetermined time unit based on the time information of the event stored in the storage unit 103 (for example, only the event recorded on Monday, or 11:00 to 12:00 The above-described analysis processing is performed based on the number of events of the event). The event time information is information related to the time when the event was detected by IDS or the like and recorded in the log. Many of the recent attacks are due to viruses or worms having a latent period, onset / propagation period, and the accuracy of the predicted value is improved by the above-described method taking into account the periodicity of the attack.

As described above, according to the present embodiment, based on the number of events belonging to parameters in logs output from various network devices, an abnormal value related to the degree of network abnormality (ratio D in ratio analysis, rare When the upper rare rate R _u and lower rare rate R ₁ in the rate analysis are calculated, and a predetermined condition regarding the abnormal value (D> 1.0 etc. with respect to the ratio D related to the number of events on the day) is satisfied In addition, since the conditional probability that a predetermined event (the event that the number of events of the next day has increased from the current day) occurs is calculated by Bayesian inference, the transition of the number of future events can be predicted objectively be able to.

  As described above, the embodiments of the present invention have been described in detail with reference to the drawings, but the specific configuration is not limited to these embodiments, and includes design changes and the like within a scope not departing from the gist of the present invention. It is. For example, the log analysis device in the above-described embodiment records a program for realizing the operation and function on a computer-readable recording medium, and causes the computer to read and execute the program recorded on the recording medium. May be realized.

  Here, the “computer” includes a homepage providing environment (or display environment) if the WWW system is used. The “computer-readable recording medium” refers to a storage device such as a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a hard disk built in the computer. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The log analysis program described above may be transmitted from a computer storing the program in a storage device or the like to another computer via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the log analysis program described above may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer, what is called a difference file (difference program) may be sufficient.

It is a block diagram which shows the structure of the network provided with the log analyzer 10 by one Embodiment of this invention. 5 is a reference diagram showing parameters to be analyzed in the embodiment. FIG. It is a flowchart which shows operation | movement of the log analyzer 10 in the same embodiment. It is a flowchart which shows the analysis process of the log analyzer 10 in the embodiment. It is a reference diagram showing a ratio analysis model in the same embodiment. It is a reference figure showing a rare rate analysis model in the embodiment. It is a reference figure showing a parameter used for calculation of a predicted value in the embodiment. It is a graph which shows the change of the detection number of the packet detected in Port27374 of Internet Storm Center. It is a reference figure for demonstrating the grouping of the event of Attack Signature in the embodiment. It is a reference figure for demonstrating the grouping of the event of Attack Signature in the embodiment. It is a reference figure which shows the content of the conversion table for converting Source / Destination IP into a country name in the same embodiment. It is a reference figure which shows the method of the dynamic grouping 1 of the event in the embodiment. It is a reference figure which shows the method of the dynamic grouping 2 of the event in the embodiment. It is a flowchart which shows the other analysis processing of the log analyzer 10 in the embodiment. It is a reference figure showing change of attack frequency to Port1434.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Log analyzer, 20, 21, 22 ... Network, 30 ... Security operation center, 101 ... Log collection part, 102 ... Log storage part, 103 ... Storage part, 104 ... Log analysis part, 105 ... Interface part, 301 ... Web browser.

Claims (22)

In a log analyzer that performs analysis processing based on logs collected from network devices,
Collecting means for collecting logs output from the network device;
An abnormal value calculating means for calculating an abnormal value related to the degree of abnormality of the network based on the number of recorded events belonging to the parameters in the log collected by the collecting means;
A probability calculating means for calculating a conditional probability that a predetermined event occurs when a predetermined condition regarding the abnormal value is satisfied;
A log analyzer characterized by comprising:   The abnormal value calculating means selects an event of a predetermined time unit from events belonging to the parameters in the log collected by the collecting means, and relates to the degree of network abnormality based on the number of recorded events. The log analysis apparatus according to claim 1, wherein an abnormal value is calculated.   The abnormal value calculating means selects a specific event from events belonging to the parameters in the log collected by the collecting means, based on public information related to the vulnerability, and based on the number of recorded events. The log analysis apparatus according to claim 1, wherein an abnormal value related to a degree of network abnormality is calculated. In a log analyzer that performs analysis processing based on logs collected from network devices,
Collecting means for collecting logs output from the network device;
Grouping means for performing grouping to divide events belonging to the parameters in the log collected by the collecting means into a plurality of groups;
An abnormal value calculating means for calculating an abnormal value related to the degree of abnormality of the network based on the number of recorded events grouped by the grouping means;
A probability calculating means for calculating a conditional probability that a predetermined event occurs when a predetermined condition is satisfied with respect to the abnormal value;
A log analyzer characterized by comprising:   The grouping means selects an event of a predetermined time unit from events belonging to the parameters in the log collected by the collecting means, and performs grouping to divide the event into a plurality of groups. The log analyzer according to claim 4, wherein   The grouping unit selects a specific event from events belonging to the parameters in the log collected by the collecting unit based on public information related to the vulnerability, and divides the event into a plurality of groups. 6. The log analyzer according to claim 4, wherein grouping is performed.   The predetermined event is an event in which the number of recorded events belonging to a parameter in the log increases or an event in which the number of recorded events belonging to a parameter in the log decreases. 6. The log analysis device according to any one of items 6. In a log analysis method that performs analysis processing based on logs collected from network devices,
Collecting logs output from the network device;
Calculating an abnormal value related to the degree of network abnormality based on the number of recorded events belonging to the parameter in the collected log;
Calculating a conditional probability that a predetermined event occurs when a predetermined condition regarding the abnormal value is satisfied;
A log analysis method comprising:   In the step of calculating an abnormal value related to the degree of network abnormality based on the number of recorded events belonging to the parameter in the collected log, a predetermined time unit is selected from the collected events belonging to the parameter in the log. The log analysis method according to claim 8, further comprising: selecting an event and calculating an abnormal value related to the degree of network abnormality based on the number of recorded events.   In the step of calculating an abnormal value relating to the degree of network abnormality based on the number of recorded events belonging to the parameter in the collected log, disclosure of vulnerability is made from among the events belonging to the collected parameter in the log. The log analysis method according to claim 8 or 9, wherein a specific event is selected based on the information, and an abnormal value related to the degree of network abnormality is calculated based on the number of recorded events. In a log analysis method that performs analysis processing based on logs collected from network devices,
Collecting logs output from the network device;
Performing a grouping to divide events belonging to the parameters in the collected logs into a plurality of groups;
Calculating an abnormal value related to the degree of abnormality of the network based on the number of records of the grouped events;
Calculating a conditional probability that a predetermined event occurs when a predetermined condition is satisfied with respect to the abnormal value;
A log analysis method comprising:   In the step of grouping to divide events belonging to the parameters in the collected logs into a plurality of groups, an event of a predetermined time unit is selected from the events belonging to the parameters in the collected logs, The log analysis method according to claim 11, wherein the event is divided into a plurality of groups.   In the step of grouping to divide events belonging to the parameters in the collected logs into a plurality of groups, identification is made based on public information related to the vulnerability from among the events belonging to the parameters in the collected logs. The log analysis method according to claim 11 or 12, wherein the event is selected and grouping is performed to divide the event into a plurality of groups.   9. The predetermined event is an event in which the number of recorded events belonging to a parameter in the log is increased or an event in which the number of recorded events belonging to a parameter in the log is decreased. 14. The log analysis method according to any one of items 13. In a log analysis program for causing a computer to execute analysis processing based on logs collected from network devices,
Collecting logs output from the network device;
Calculating an abnormal value related to the degree of network abnormality based on the number of recorded events belonging to the parameter in the collected log;
Calculating a conditional probability that a predetermined event occurs when a predetermined condition regarding the abnormal value is satisfied;
Log analysis program to make a computer execute.   In the step of calculating an abnormal value related to the degree of network abnormality based on the number of recorded events belonging to the parameter in the collected log, a predetermined time unit is selected from the collected events belonging to the parameter in the log. The log analysis program according to claim 15, wherein the event is selected, and an abnormal value related to the degree of network abnormality is calculated based on the number of recorded events.   In the step of calculating an abnormal value relating to the degree of network abnormality based on the number of recorded events belonging to the parameter in the collected log, disclosure of vulnerability is made from among the events belonging to the collected parameter in the log. The log analysis program according to claim 15 or 16, wherein a specific event is selected based on the information, and an abnormal value related to the degree of network abnormality is calculated based on the number of recorded events. In a log analysis program for causing a computer to execute analysis processing based on logs collected from network devices,
Collecting logs output from the network device;
Performing a grouping to divide events belonging to the parameters in the collected logs into a plurality of groups;
Calculating an abnormal value related to the degree of abnormality of the network based on the number of records of the grouped events;
Calculating a conditional probability that a predetermined event occurs when a predetermined condition is satisfied with respect to the abnormal value;
Log analysis program to make a computer execute.   In the step of grouping to divide events belonging to the parameters in the collected logs into a plurality of groups, an event of a predetermined time unit is selected from the events belonging to the parameters in the collected logs, The log analysis program according to claim 18, wherein the event is divided into a plurality of groups.   In the step of grouping to divide events belonging to the parameters in the collected logs into a plurality of groups, identification is made based on public information related to the vulnerability from among the events belonging to the parameters in the collected logs. The log analysis program according to claim 18 or 19, wherein the event is selected and grouping is performed to divide the event into a plurality of groups.   16. The predetermined event is an event in which the number of recorded events belonging to a parameter in the log is increased or an event in which the number of recorded events belonging to a parameter in the log is decreased. The log analysis program according to any one of 20 items. A computer-readable recording medium in which the log analysis program according to any one of claims 15 to 21 is recorded.

Images