JP4060263B2 - Log analysis apparatus and log analysis program - Google Patents

Description

  The present invention relates to a log analysis apparatus, a log analysis method, and a log analysis program for analyzing logs output from network devices such as IDS (Intrusion Detection System), Router, and Firewall.

  In recent years, more and more sites have introduced IDS for monitoring attacks on network systems. In general, the IDS simply compares a packet flowing on the network with an attack pattern file called Attack Signature, and outputs a log if there is a match.

Conventionally, it reduces the time required for analyzing log information indicating the operating status of the computer, integrates log information having various data formats and uneven distribution on the file system, and detects abnormalities that are not known abnormalities. A log information analysis device has been devised for the purpose of extracting the log information shown. In this log information analysis apparatus, a system administrator displays an enormous amount of log information in characters like a bar graph so as to quickly grasp information to be noted. (For example, see Patent Document 1)
JP 2001-356939 A

  However, the log output from the conventional IDS outputs a large amount of redundant logs such as false detection, multiple detection, and attack detection for a system with security countermeasures. Since such a log is output every time the same attack is repeated, it becomes more redundant. A large amount of logs is suitable for grasping the trend, but it tends to overlook logs with low frequency of appearance, making it difficult to accurately extract new attack signs. By monitoring the logs output from Router and Firewall, more diverse analysis becomes possible, but there is a problem in that there is no system for integrated management of these logs.

  The increase or decrease in traffic on the network depends on the time zone and day of the week of PC use by the user, and the tendency of attacks is roughly similar to the PC use frequency. However, many of the recent attacks are due to viruses and worms having a latent period, onset / propagation period, and some appear with periodicity other than the frequency of PC use. Conventionally, in consideration of the periodicity of these attacks, there has been a problem that there is no method for detecting an attack tendency to an analysis target or a sign of a new attack.

  The present invention has been made in view of the above-described problems, and is a log analysis apparatus and log that can perform log analysis in consideration of the periodicity of an attack based on logs output from various network devices The purpose is to provide an analysis program.

The present invention has been made to solve the above problems, and the invention according to claim 1 is output from the network device in a log analysis device that performs analysis processing based on a log collected from the network device. A collecting means for collecting logs, and a first distribution for obtaining a distribution on the time axis of events belonging to the parameters in the log collected by the collecting means, and converting the distribution on the time axis to a distribution on the frequency axis. A log analysis apparatus comprising a conversion unit.
The parameters for the previous period include packet volume, traffic volume, protocol volume, port volume, Attack Signature, Source / Destination Port, and Source / Destination IP recorded in logs output from network devices such as IDS, Router, and Firewall. Etc.

  According to a second aspect of the present invention, in the log analyzing apparatus according to the first aspect, grouping means for performing grouping to divide the distribution on the frequency axis generated by the first conversion means into a plurality of groups. And a second conversion means for converting the distribution on the frequency axis grouped by the grouping means into a distribution in which each event is arranged in order of the amount of the event.

  According to a third aspect of the present invention, in the log analyzing apparatus according to the first aspect, grouping means for performing grouping to divide the distribution on the frequency axis generated by the first conversion means into a plurality of groups. And an abnormal value calculating means for calculating a value related to the degree of network abnormality based on the amount of events in each group in the distribution on the frequency axis grouped by the grouping means. And

  According to a fourth aspect of the present invention, in the log analyzing apparatus according to the first aspect, grouping means for performing grouping to divide the distribution on the frequency axis generated by the first conversion means into a plurality of groups. Entropy calculating means for calculating an entropy value based on the amount of events in each group in the distribution on the frequency axis grouped by the grouping means, and the entropy value calculated by the entropy calculating means And an abnormal value calculating means for calculating a value relating to the degree of network abnormality.

According to a fifth aspect of the present invention, in the log analysis device according to the third aspect, the distribution on the frequency axis belongs to a parameter in the log and is a distribution related to events within a predetermined period, and the abnormal value calculation The means includes a first value relating to the distribution in a specific predetermined period grouped by the grouping means, and a second value relating to a plurality of the distributions in the past predetermined period grouped by the grouping means. The ratio is calculated.
The first value includes the amount of events in a specific group of the distribution belonging to the parameters in the log and relating to events within a specific predetermined period. The second value is preferably an average value of the amount of events in a specific group of the distribution related to events in a plurality of past predetermined periods. Accordingly, the degree of abnormality can be determined based on 1 as the ratio value.

According to a sixth aspect of the present invention, in the log analysis device according to the third aspect, the distribution on the frequency axis belongs to a parameter in the log, the first distribution relating to an event of a specific network, and the log And a plurality of second distributions related to events of a plurality of other networks excluding the specific network, wherein the abnormal value calculation means is the first grouped by the grouping means. A ratio between a first value related to the distribution and a second value related to the plurality of second distributions grouped by the grouping unit is calculated.
The first value includes the amount of events in a specific group of the distribution belonging to the parameters in the log and relating to events in a specific network. The second value may be an average value of the amount of events in a specific group of the distribution related to events of a plurality of other networks excluding the specific network. Accordingly, the degree of abnormality can be determined based on 1 as the ratio value.

According to a seventh aspect of the present invention, in the log analysis device according to the third aspect, the distribution on the frequency axis belongs to a parameter in the log and is a distribution related to an event within a predetermined period, and the abnormal value calculation The means obtains a probability distribution based on the amount of events of the plurality of distributions related to the past predetermined period grouped by the grouping means, and the distribution of the distribution in the specific predetermined period grouped by the grouping means The upper rare rate or the lower rare rate relating to the amount of events is calculated based on the probability distribution.
Regarding the probability distribution, the distribution of a plurality of values related to the plurality of distributions related to a predetermined period in the past (for example, the amount of events in a specific group) is one of probability distributions such as a normal distribution, an exponential distribution, and a gamma distribution. The probability distribution may be obtained as appropriate. The upper rare rate is a probability that a value in the obtained probability distribution is equal to or greater than a value related to the distribution in a specific predetermined period (for example, the amount of events in a specific group), and the lower rare rate is obtained. The probability that the value in the probability distribution is less than or equal to the value for the particular distribution.

According to an eighth aspect of the present invention, in the log analyzer according to the third aspect, the distribution on the frequency axis belongs to a parameter in the log, the first distribution relating to an event of a specific network, and the log And the second distribution relating to events of a plurality of other networks excluding the specific network, and the abnormal value calculation means is configured to output the second distribution grouped by the grouping means. A probability distribution is obtained based on the amount of events, and an upper rare rate or a lower rare rate related to the event amount of the first distribution grouped by the grouping unit is calculated based on the probability distribution. And
Regarding the probability distribution, the distribution of a plurality of values (for example, the amount of events in a specific group) related to the second distribution for each of a plurality of other networks excluding the specific network is a normal distribution, an exponential distribution, a gamma Assuming that one of the probability distributions such as the distribution is followed, the probability distribution may be obtained as appropriate. The upper rare rate is a probability that a value in the obtained probability distribution is not less than a value related to the first distribution (for example, the amount of events in the specific group), and the lower rare rate is the obtained probability. It is the probability that the value in the distribution will be less than or equal to the value for the first distribution.

  According to a ninth aspect of the present invention, in the log analysis device according to the fourth aspect, the distribution on the frequency axis belongs to a parameter in the log and is a distribution related to events within a predetermined period, and the abnormal value calculation The means calculates a ratio between a first entropy value related to the distribution in a specific predetermined period calculated by the entropy calculating means and a second entropy value related to the plurality of distributions related to a past predetermined period. And

  According to a tenth aspect of the present invention, in the log analysis device according to the fourth aspect, the distribution on the frequency axis belongs to a parameter in the log, the first distribution regarding events of a specific network, and the log And the second distribution relating to events of a plurality of other networks excluding the specific network, wherein the abnormal value calculating means is a second distribution relating to the first distribution calculated by the entropy calculating means. A ratio between an entropy value of 1 and a second entropy value relating to the second distribution is calculated.

  According to an eleventh aspect of the present invention, in the log analyzer according to the fourth aspect, the distribution on the frequency axis belongs to a parameter in the log and is a distribution relating to an event within a predetermined period, and the entropy calculating unit Is a first entropy value related to the distribution in a specific predetermined period grouped by the grouping means, and a second entropy value related to a plurality of the distributions related to past predetermined periods grouped by the grouping means. The abnormal value calculation means obtains a probability distribution of the second entropy value, and calculates an upper rare rate or a lower rare rate related to the first entropy value based on the probability distribution. Features.

  According to a twelfth aspect of the present invention, in the log analysis device according to the fourth aspect, the distribution on the frequency axis belongs to a parameter in the log, the first distribution relating to an event of a specific network, and the log And a second distribution relating to events of other networks excluding the specific network, wherein the entropy calculating means is a first distribution relating to the first distribution grouped by the grouping means. An entropy value and a second entropy value related to the second distribution grouped by the grouping means are calculated, the abnormal value calculation means obtains a probability distribution of the second entropy value, and An upper rare rate or a lower rare rate for one entropy value is calculated based on the probability distribution.

  The invention according to claim 13 is the log analysis apparatus according to any one of claims 1 to 12, wherein the grouping unit is on the frequency axis generated by the first conversion unit. The distribution is divided into a plurality of groups for each predetermined frequency.

  In a fourteenth aspect of the present invention, in the log analysis apparatus according to any one of the third to twelfth aspects, the grouping unit is on the frequency axis generated by the first conversion unit. The events are grouped so that the total amount of events in each group approaches the event of each frequency in the distribution equally.

  The invention according to claim 15 is a log analysis method for performing analysis processing based on a log collected from a network device, wherein the log output from the network device is collected, and belongs to the parameters in the collected log A log analysis program for causing a computer to obtain a distribution on the time axis of events and converting the distribution on the time axis to a distribution on the frequency axis.

  According to a sixteenth aspect of the present invention, in the log analysis program according to the fifteenth aspect, a step of grouping the distribution on the frequency axis into a plurality of groups, and the grouped distribution on the frequency axis And a step of converting each event into a distribution arranged in the order of the amount of the event.

  According to a seventeenth aspect of the present invention, in the log analysis program according to the fifteenth aspect, a step of grouping the distribution on the frequency axis into a plurality of groups, and the grouped distribution on the frequency axis And a step of converting each event into a distribution arranged in the order of the amount of the event.

  According to an eighteenth aspect of the present invention, in the log analysis program according to the fifteenth aspect, a step of grouping the distribution on the frequency axis into a plurality of groups, and a grouped distribution on the frequency axis The method further includes the step of calculating an entropy value based on the amount of events in each group, and the step of calculating a value related to the degree of network abnormality based on the calculated entropy value.

  According to a nineteenth aspect of the present invention, in the log analysis program according to the seventeenth aspect, the distribution on the frequency axis belongs to a parameter in the log and is a distribution related to events within a predetermined period, and the grouping is performed. In the step of calculating the value related to the degree of network abnormality based on the amount of events in each group in the distribution on the frequency axis, the first value related to the distribution in the specific grouped period and the grouping The ratio with the 2nd value regarding the said some distribution in the past predetermined period was calculated.

  According to a twentieth aspect of the present invention, in the log analysis program according to the seventeenth aspect, the distribution on the frequency axis belongs to a parameter in the log, the first distribution relating to an event of a specific network, and the log And a second distribution relating to events of a plurality of other networks excluding the specific network, based on the amount of events in each group in the grouped distribution on the frequency axis, In the step of calculating a value related to the degree of network abnormality, a ratio between a first value related to the grouped first distribution and a second value related to the grouped second distribution is calculated. And

  According to a twenty-first aspect of the present invention, in the log analysis program according to the seventeenth aspect, the distribution on the frequency axis belongs to the parameter in the log and is a distribution related to events within a predetermined period, and the grouped Based on the amount of events in each group in the distribution on the frequency axis, in the step of calculating a value related to the degree of network anomaly, based on the amount of events in the plurality of distributions related to a predetermined past period grouped A probability distribution is obtained, and an upper rare rate or a lower rare rate related to the amount of events of the distribution in a specific grouped period is calculated based on the probability distribution.

  According to a twenty-second aspect of the present invention, in the log analysis program according to the seventeenth aspect, the distribution on the frequency axis belongs to a parameter in the log, the first distribution relating to an event of a specific network, and the log And a second distribution relating to events of a plurality of other networks excluding the specific network, based on the amount of events in each group in the grouped distribution on the frequency axis, In the step of calculating the value related to the degree of network abnormality, the probability distribution is obtained based on the amount of events of the second distribution grouped, and the upper rare rate regarding the amount of events of the grouped first distribution or The lower rare rate is calculated based on the probability distribution.

  According to a twenty-third aspect of the present invention, in the log analysis program according to the eighteenth aspect, the distribution on the frequency axis belongs to a parameter in the log and is a distribution related to an event within a predetermined period, and the calculated In the step of calculating the value related to the degree of abnormality of the network based on the entropy value, the calculated first entropy value related to the distribution in the specific predetermined period and second entropy related to the plurality of distributions related to the past predetermined period. A ratio with the value is calculated.

  According to a twenty-fourth aspect of the present invention, in the log analysis program according to the eighteenth aspect, the distribution on the frequency axis belongs to a parameter in the log, the first distribution relating to an event of a specific network, and the log And a second distribution relating to events of a plurality of other networks excluding the specific network, and calculating a value related to the degree of network abnormality based on the calculated entropy value. A ratio between the calculated first entropy value related to the first distribution and the second entropy value related to the second distribution is calculated.

  According to a 25th aspect of the present invention, in the log analysis program according to the 18th aspect, the distribution on the frequency axis belongs to the parameter in the log, and is a distribution related to events within a predetermined period, and the grouping is performed. In the step of calculating an entropy value based on the amount of events in each group in the distribution on the frequency axis, a first entropy value related to the distribution in a specific grouped period and a grouped past In the step of calculating a second entropy value related to a plurality of the distributions related to a predetermined period and calculating a value related to the degree of network abnormality based on the calculated entropy value, a probability distribution of the second entropy value And determine the upper or lower rare rate for the first entropy value as the certainty. And calculating on the basis of the distribution.

  According to a twenty-sixth aspect of the present invention, in the log analysis program according to the eighteenth aspect, the distribution on the frequency axis belongs to a parameter in the log, the first distribution relating to an event of a specific network, and the log And a second distribution relating to events of a plurality of other networks excluding the specific network, based on the amount of events in each group in the grouped distribution on the frequency axis, In the step of calculating an entropy value, a first entropy value related to the grouped first distribution and a second entropy value related to the grouped second distribution are calculated, and the calculated entropy value is calculated. In the step of calculating the value relating to the degree of network abnormality based on the second entry, A probability distribution of a ropey value is obtained, and an upper rare rate or a lower rare rate related to the first entropy value is calculated based on the probability distribution.

  According to a twenty-seventh aspect of the present invention, in the log analysis program according to any one of the fifteenth to twenty-sixth aspects, in the step of performing grouping to divide the distribution on the frequency axis into a plurality of groups, The distribution on the frequency axis is divided into a plurality of groups for each predetermined frequency.

  According to a twenty-eighth aspect of the present invention, in the log analysis program according to any one of the seventeenth to twenty-sixth aspects, in the step of performing grouping to divide the distribution on the frequency axis into a plurality of groups, The events are grouped so that the total amount of events in each group approaches the event of each frequency in the distribution on the frequency axis evenly.

  A twenty-ninth aspect of the invention is a computer-readable recording medium on which the log analysis program according to any one of the fifteenth to twenty-eighth aspects is recorded.

  According to the present invention, the distribution on the time axis of the events belonging to the parameters in the log output from various network devices is obtained, and the distribution on the time axis is converted to the distribution on the frequency axis. It is possible to perform log analysis in consideration of the periodicity.

  The best mode for carrying out the present invention will be described below with reference to the drawings. FIG. 1 is a configuration diagram showing the configuration of a network including a log analysis device according to an embodiment of the present invention. In the figure, 10 is a log analyzer. The log analysis device 10 collects and analyzes logs output from the networks 20 to 22 to be analyzed, and outputs the analysis results to the web browser 301. The log analysis device 10 may output the analysis result to another device having a display unit. When the log analysis device 10 includes a display unit, the log analysis device 10 displays the analysis result. May be.

  Further, the log analysis device 10 may be a device independent of the security operation center 30 including the web browser 301, and the device including the web browser 301 in the security operation center 30 has the function of the log analysis device 10. You may do it. The Router, Firewall, and IDS of the networks 20 to 22 generate logs, and output the logs to the log analysis device 10 by a communication method such as syslogd. The security operation center 30 has a web browser 301, and the operator can determine the degree of abnormality of the networks 20 to 22 based on the analysis result displayed on the web browser 301.

  In the log analysis device 10, reference numeral 101 denotes a log collection unit that periodically collects logs output from devices in the networks 20 to 22 and outputs the logs to the log storage unit 102. The log storage unit 102 extracts parameters used for analysis from the log output by the log collection unit 101 and stores them in the storage unit 103. When the operator requests a parameter desired for analysis via the Web browser 301, instruction information for instructing analysis is input to the log analysis unit 104 via the interface unit 105.

  The log analysis unit 104 reads out an analysis parameter from the storage unit 103 in accordance with the instruction information, analyzes the parameter (details will be described later), saves the analysis result in the storage unit 103, and outputs the analysis result to the interface unit 105. To do. The interface unit 105 outputs the analysis result output from the log analysis unit 104 to the Web browser 301 via a communication line. Based on the analysis result displayed on the Web browser 301, the operator determines the status of the attack on the networks 20-22.

Next, parameters to be analyzed in this embodiment will be described. In this embodiment,
(A) Number of packets (b) Traffic volume (c) Protocol volume (d) Port volume (e) Attack Signature
(F) Source / Destination Port
(G) Source / Destination IP
These parameters are subject to analysis.

  (A)-(d) are recorded in the log output from Router and Firewall. (E) is recorded in the log output from the IDS. (F) to (g) are recorded in logs output from IDS, Router, and Firewall. By analyzing the number of packets or the amount of traffic, it is possible to detect whether an attack on the network is occurring. The protocol amount indicates the number of packets (or the amount of traffic) related to a protocol such as TCP (Transmission Control Protocol) / UDP (User Datagram Protocol) / ICMP (Internet Control Message Protocol).

  By analyzing the protocol amount, various attacks can be detected. For example, Ping, Traceroute, and the like related to ICMP are a preliminary check before an attack, so that it is possible to pay attention in advance to a full-scale attack by detecting these. The Port amount indicates the number of packets (or traffic amount) for each port number of the transmission source / destination device of a packet related to a protocol such as TCP / UDP. By analyzing the amount of Port, it is possible to detect access or intrusion to a specific Port.

  The Attack Signature is a parameter included in the log output from the IDS. By analyzing the Attack Signature, it is possible to specify the type of attack being performed on the analysis target network. All network type IDSs and some host type IDSs can output an attack signature. Source / Destination Port indicates a port number of a transmission source / destination device. Source / Destination IP indicates the IP address of the source / destination device. Note that the Source / Destination Port indicates one of the two parameters of the Source Port and the Destination Port, and the same applies to the Source / Destination IP.

  By analyzing the Source / Destination Port, the port number of the attack source / attack target can be specified. Further, by analyzing the Source / Destination IP, the IP address of the attack source / attack target can be specified. The above parameters consist of multiple events. When a plurality of the same events are recorded in the log, the total number is defined as the event amount of the event.

  Next, the operation of the log analysis apparatus 10 in this embodiment will be described. FIG. 2 is a flowchart showing the operation of the log analysis apparatus 10. The log collection unit 101 collects logs output from the devices of the networks 20 to 22 and outputs them to the log storage unit 102 (step S200). The log storage unit 102 extracts parameters used for analysis from the log output by the log collection unit 101 and stores them in the storage unit 103 (step S210). When an analysis request regarding a specific parameter is made from the operator via the Web browser 301, the instruction information output from the Web browser 301 is input to the log analysis unit 104 via the interface unit 105. Then, the log analysis unit 104 reads parameters from the storage unit 103 based on the instruction information (step S220).

  The log analysis unit 104 performs an analysis process to be described later on the analysis target parameter, and stores the analysis result in the storage unit 103 (step S230). Subsequently, the log analysis unit 104 outputs the analysis result to the web browser 301 via the interface 105 (step S240). The analysis result is displayed on the Web browser 301, and the operator determines whether an attack is being performed on the networks 20 to 22 based on the analysis result.

  Next, the analysis process performed by the log analysis unit 104 in step S23 will be described. First, the log analysis unit 104 obtains a distribution of events on the time axis based on the event amount of the event related to the parameter read from the storage unit 103. FIG. 3 shows the distribution on the time axis of events related to the Attack Signature, taking the Attack Signature recorded in the log output from the IDS as an example. In the figure, the horizontal axis indicates the time, and the vertical axis indicates the event amount of the event at each time. Since the time at which the event was detected is also recorded in the log, the total amount of events at each time can be obtained.

  The Attack Signature has event types such as HTTP port probe, DNS port probe,..., And the log analysis unit 104 pays attention to an event of a specific event type (for example, HTTP port probe), and at a certain time (or a predetermined time) The total number of events detected in (time range) is obtained, and a distribution on the time axis as shown in FIG. 3 is obtained. This distribution is stored in the storage unit 103 as data in a format such as table data in which a time is associated with an event amount at that time. Note that the log analysis unit 104 may pay attention to a plurality of specific event types, regard them as a group of event types, and obtain the distribution on the time axis.

  In FIG. 3, the broken line a represents the average of the event amount at each time on weekdays over a predetermined period (for example, one month). A broken line b represents the average of the event amount at each time of holidays and holidays over a predetermined period. In FIG. 3, the event amount is displayed in units of one hour, but may be displayed in units of minutes or seconds. The log analysis unit 104 obtains a distribution on the frequency axis from the distribution on the time axis by discrete Fourier transform. If the event amount at the nth observation point on the time axis is e (n) and the kth frequency component amount on the frequency axis is E (k) (where n and k are natural numbers), N (N Is a natural number) discrete time signal {e (n)}, n = 0, 1,..., N−1, the discrete Fourier transform is expressed as [Equation 1]. In addition, you may perform FFT (Fast Fourier Transform) which performs a discrete Fourier transform at high speed.

  FIG. 4 shows an example in which the distribution on the time axis is converted to the distribution on the frequency axis. The horizontal axis indicates the time interval (hour, day, week, month, year) corresponding to each frequency, and the vertical axis indicates the event amount. Subsequently, the log analysis unit 104 divides the distribution on the frequency axis into predetermined groups (grouping) so that a plurality of adjacent frequencies are in the same group. Examples of grouping include static grouping and dynamic grouping. In the static grouping, grouping is performed in fixed time units of time / day / week / month / year. In this case, the log analysis unit 104 groups events for each predetermined frequency, and stores information for identifying each group and information indicating the total amount of events in the group in the storage unit 103.

  In the dynamic grouping, neighboring frequency components are grouped together into the same group so that the total amount of events in each group is substantially equal. FIG. 5 shows how the distribution of FIG. 4 is grouped into six groups. Also in this case, the log analysis unit 104 groups events for each predetermined frequency, and stores in the storage unit 103 information for identifying each group and information indicating the total amount of events in the group. In the dynamic grouping, the log analysis unit 104 performs grouping as follows. For example, the event is divided into 6 groups, and the event amount in each group is grouped so that it is about 1/6 of the total event amount. In FIG. 5, the event amount is added in order from the left event (low frequency region event), and when the amount becomes 1/6 or more of the total event amount, one group is formed. At this time, for example, the lowest digit of the event amount is rounded off.

  Alternatively, the event amount is added from the left event, and one group is formed when the amount exceeds 1/6 of the total event amount. By the above method, the first grouping is performed, and for the next group, the second grouping is performed in the same manner as described above so that the event amount is about one fifth of the remaining event amount. Do. Further, for the next group, similarly, the third grouping is performed so that the event amount is about one fourth of the remaining event amount. By repeating this, the event is divided into 6 groups. Note that the above-described method is an example, and any method can be used as long as it can be grouped so that the amount of events in each group is substantially equal.

Subsequently, the log analysis unit 104 performs the following analysis based on the total amount of events in each group. The analysis performed by the log analysis unit 104 includes:
(A) Frequency analysis (b) Ratio analysis (c) Rare rate analysis.

  When performing frequency analysis, the log analysis unit 104 uses a distribution obtained by converting a distribution on the time axis on the frequency axis (for example, FIG. 3) as an analysis result, or a distribution obtained by converting the distribution on the time axis on the frequency axis. Then, static grouping is performed, and frequency amounts (event total amounts) in each group are arranged in order of frequency amounts, and output to the interface unit 105 as analysis results. The analysis result is output from the interface unit 105 to the web browser 301. The frequency distribution as shown in FIG. 3 is displayed on the web browser 301, or the frequency amount for each time / day / week / month / year is represented by a graph or numerical value. They are displayed in order.

  In the ratio analysis, attention is paid to one group among a plurality of groups generated by grouping, and analysis based on the total amount of events in the group is performed. The group of interest was originally obtained based on the average amount of events over a predetermined period, as shown in FIG. 2, and the total amount of events obtained based on the data over the predetermined period is defined as a short-term profile. . In addition, for a plurality of past predetermined periods, an average of a plurality of short-term profiles obtained by the same operation as described above is defined as a long-term profile. In the ratio analysis, the ratio between the short-term profile and the long-term profile is evaluated as an abnormal value. FIG. 6 shows an example of the ratio analysis model.

In the figure, E _l indicates the average of the total event amount (E _{1 to} E ₁₀ ) obtained for a specific group in a plurality of past predetermined periods. With short-term profile and E _s, the ratio D of long-term profile for short-term profile is expressed as [Equation 2]. The short-term profile is stored in the storage unit 103, and the log analysis unit 104 reads a plurality of past short-term profiles from the storage unit 103 and creates a long-term profile. Alternatively, a plurality of past logs may be stored in the storage unit 103, and a long-term profile may be created from these logs.

  When grouping is performed by static grouping, division is performed in units of frequency such as time / day / week / month / year, and the frequency that is the boundary of division is fixed. On the other hand, when the grouping is performed by dynamic grouping, if the grouping is performed as it is by the above-described method, the frequency serving as the boundary of division may be different among a plurality of past short-term profiles. Therefore, when grouping is performed by dynamic grouping, it is as follows.

  First, when obtaining the long-term profile, the log analysis unit 104 obtains the total event amount of a specific group from the data in a predetermined period in the past by the dynamic grouping described above. Subsequently, the log analysis unit 104 stores the division form (number of divisions and division boundary frequency for each group) in the dynamic grouping in the storage unit 103 as information. The log analysis unit 104 reads out information related to the division form from the storage unit 103 and refers to the information while grouping frequency distributions in other past predetermined periods in the same division form. Then, the log analysis unit 104 obtains the total event amount of a specific group for each of these frequency distributions, calculates the average of the plurality of event total amounts, and obtains a long-term profile. Further, the log analysis unit 104 performs grouping of frequency distributions in a predetermined period to be analyzed according to the above-described division form to obtain a short-term profile.

Next, in the rare rate analysis, the anomaly of an event in which the number of outputs varies variously is evaluated using the average and standard deviation of the total amount of events. In the statistical analysis, an index of 95% confidence interval is often used. The rare rate analysis here is to calculate a complement of this confidence interval. FIG. 7 shows an example of a rare rate analysis model in a predetermined period. _El is the average (long-term profile) of the past total amount of events, and SD is the standard deviation. The abscissa represents the distance from the average value _El , with the standard deviation SD as a unit, and the ordinate represents the number of groups whose total event amount is the value indicated on the abscissa. The standard deviation SD is obtained from [Equation 3]. The upper rare rate R _u when the short-term profile is larger than the long-term profile E _l is defined as [Equation 4], and the lower rare rate R _l when the short-term profile is smaller is defined as [Equation 5]. E ₁ , E ₂ ,... E _k are the total amount of events obtained for a specific group in a plurality of past predetermined periods.

  In [Equation 4] and [Equation 5], f (E) is a density function of a normal distribution, and is expressed by the following [Equation 6]. In the above example, it is assumed that the attack distribution for calculating the rare rate is a normal distribution, but a function such as an exponential distribution or a gamma distribution may be appropriately selected as f (E).

The log analysis unit 104 outputs the abnormal values D, R _u , and R ₁ calculated by the above-described ratio analysis and rare rate analysis to the interface unit 105 as analysis results. The analysis result is output from the interface unit 105 to the Web browser 301 of the security operation center 30, and the analysis result is displayed on the Web browser 301. The network operator may make a determination regarding D, R _u , and R _l as follows.

First, if D> 1.0 or R _u ≈0.0%, new attacks with a specific periodicity are circulating on the Internet, and internal hosts are periodically infected with worms. , DDoS (Destroyed Denial of Service) attack (intruder enters multiple websites vulnerable to security on the Internet to make it an attack base, and simultaneously operates the programs prepared in multiple attack bases to serve as the attack target server This shows that the amount of short-term events, such as attacks that send a large number of packets and stop the server's function, has increased rapidly in a specific cycle.

When D≈0.0 or R _l ≈0.0%, the alarms that have been output periodically are suddenly reduced or eliminated, or the device is stopped due to an attack. Further, when D≈1.0 or R _u , R _l ≈50.0%, it is possible to distinguish a log that has been continuously detected. This is an attack for which countermeasures have already been taken, or an event that is easily misdetected, and does not require special attention. The network operator may perform the above-described evaluation based on values such as D, R _u , and R _l output from the log analysis device 10.

  Regarding the determination of the result by the analysis processing described above, the network operator may determine the state of the network as follows. Some viruses and worms are active at specific times, and such are frequency dependent. Even with viruses and worms that do not have frequency dependency, periodicity may appear in the activity due to the periodic operation of the computer on the network (for example, ON / OFF). Virus and worm activity (infection) date calendars are published by companies that handle Internet security, and by referring to the activity date, the type of attack on the network to be analyzed can be determined.

  In the analysis process described above, a technique for fixing the network to be analyzed and evaluating how abnormal the short-term profile is with respect to the long-term profile has been shown, but the following analysis process can also be performed. That is, when the period to be analyzed is fixed, the total amount of events of a specific group (defined as the own network profile) in the frequency distribution of events related to specific parameters detected in the network to be analyzed is a plurality of other networks. This is a method for evaluating how abnormal the average total event amount (defined as other network profile) of a specific group in the frequency distribution of events related to the same specific parameter detected in (1) is.

For example, in the ratio analysis, if the local network profile in the network to be analyzed is E _m and the other network profile in each of the other networks not including this network is E _o , the ratio D is expressed by the following [Equation 7]. expressed. Similarly, the upper and lower rare rates in the rare rate analysis can be calculated.

  As described above, log analysis focusing on the periodicity of attacks can be performed by converting the distribution on the time axis to the distribution on the frequency axis and performing frequency analysis, ratio analysis, and rare rate analysis. In addition, by grouping events obtained as a frequency distribution, the number of variables to be calculated can be reduced, so that the time required to calculate an abnormal value can be reduced. In addition, it is possible to greatly grasp the log tendency of the entire network by grouping, and it is possible to improve the reliability of analysis by reducing false detection regarding abnormality.

  Next, another process performed by the log analysis unit 104 in the analysis process will be described. A network abnormality can be detected by evaluating the ambiguity (entropy) of logs output from various network devices. For example, in the normal state where there is no attack, the log ambiguity is large, but in the state where a DDoS attack or virus infection is prevalent, the events output to the log are biased, Log ambiguity is reduced. Therefore, if the ambiguity of the log is analyzed, the state of the network can be grasped.

In the following, a method for detecting the degree of network abnormality by calculating entropy will be described. The log analysis unit 104 calculates entropy from the event amount in each frequency band in the grouped distribution on the frequency axis as shown in FIG. First, the total event amount E is calculated by [Equation 8] (k is the number of frequency bands). Then, the event the amount in a certain frequency band and e _i, is calculated by the relative frequency of each frequency band [Equation 9]. Then, the entropy of the parameter to be analyzed is calculated by [Equation 10].

  Similarly, the log analysis unit 104 calculates entropy for each distribution based on the distribution on the frequency axis of a plurality of past events, and performs the above-described ratio analysis and rare rate analysis. For example, in the case of ratio analysis, the log analysis unit 104 calculates an entropy (short-term profile) obtained based on a distribution in a predetermined period to be analyzed and an average of a plurality of entropies obtained based on distributions in a plurality of past predetermined periods ( The ratio to the long-term profile is calculated.

  Further, in the case of rare rate analysis, the log analysis unit 104 calculates entropy (short-term profile) based on the distribution of the analysis target for a predetermined period, and based on the distribution on the frequency axis of a plurality of past events, Calculate entropies and their standard deviations. Then, the probability distribution of the entropy of each distribution calculated based on the distribution on the frequency axis of a plurality of past events is obtained, and the entropy value in the probability distribution is lower than the upper rare rate or the short-term profile below the short-term profile. Calculate the side rare rate.

  Performing ratio analysis or rare rate analysis after calculating entropy has the following advantages. When performing ratio analysis or rare rate analysis focusing on a specific event type in the analysis target parameter, detailed data on some frequencies (groups) of the analysis target can be obtained, but for all frequencies It takes a lot of work to perform ratio analysis and rare rate analysis, and it is not suitable for grasping the outline of the entire analysis target.

  On the other hand, when entropy is calculated based on the total event amount of the distribution of events to be analyzed on the frequency axis, and ratio analysis and rare rate analysis are performed, entropy indicating the ambiguity over the entire analysis target is used, so the analysis target Suitable for getting an overview of the whole. Furthermore, by grouping the events described above, the number of variables used for entropy calculation can be greatly reduced, so the time required to calculate outliers can be greatly reduced, and attacks can be performed in real time. Can be detected.

  Further, in the event grouping, the above-described dynamic grouping has the following advantages. The log analysis unit 104 performs dynamic grouping in calculating a long-term profile (or other network profile). At this time, long-term profiles are in a state of high ambiguity by averaging the long-term analysis results. When entropy is calculated in this state, the entropy increases.

  Subsequently, the log analysis unit 104 groups events using the division form used for calculating the long-term profile, and calculates entropy based on the total amount of events in each group. When an attack with periodicity is performed in the short term (or against the own network), the total amount of events in a specific group increases, resulting in a state of low ambiguity. When entropy is calculated in this state, the entropy is smaller than the long-term profile (or other network profile). Therefore, by detecting this entropy change using ratio analysis or rare rate analysis, the degree of network abnormality can be objectively detected. Can be evaluated.

  8 to 12 are flowcharts showing the operation of the log analysis unit 104 in the analysis processing described above. Hereinafter, the operation of the log analysis unit 104 in each case will be described using a flowchart. FIG. 8 is a flowchart showing the operation of the log analysis unit 104 when performing the frequency analysis described above. The log analysis unit 104 obtains a distribution on the time axis of the event based on the event amount related to the parameter read from the storage unit 103 in step S22 in FIG. 2, performs a discrete Fourier transform on the distribution, An upper distribution is obtained, and this distribution is stored in the storage unit 103 as an analysis result (step S2301).

  FIG. 9 is a flowchart showing the operation of the log analysis unit 104 when frequency analysis is performed after grouping. The operation in step S2302 is the same as the operation in step S2301 in FIG. Subsequently, the log analysis unit 104 groups (static group) events belonging to neighboring frequency bands for each predetermined frequency (step S2303). The log analysis unit 104 stores the frequency amount (event total amount) in each group in order of frequency amount in the storage unit 103 as an analysis result (step S2304).

  FIG. 10 is a flowchart showing the operation of the log analysis unit 104 when grouping is performed and ratio analysis or rare rate analysis is performed. The operation in step S2305 is the same as the operation in step S2301 in FIG. Subsequently, the log analysis unit 104 groups events for each predetermined frequency (step S2306). The log analysis unit 104 performs ratio analysis or rare rate analysis based on the total amount of events in each group, and stores the analysis result in the storage unit 103 (step S2307). The log analysis unit 104 may perform both ratio analysis and rare rate analysis.

  FIG. 12 is a flowchart showing the operation of the log analysis unit 104 when entropy is calculated after grouping and ratio analysis or rare rate analysis is performed on the obtained entropy. The operations in steps S2308 to S2309 are the same as the operations in steps S2302 to S2303 in FIG. In step S2310, the log analysis unit 104 calculates entropy based on the event amount of each frequency band in the distribution on the frequency axis as shown in FIG. Subsequently, the log analysis unit 104 performs ratio analysis or rare rate analysis, and stores the analysis result in the storage unit 103 (step S2311).

  As described above, the embodiments of the present invention have been described in detail with reference to the drawings, but the specific configuration is not limited to these embodiments, and includes design changes and the like within a scope not departing from the gist of the present invention. It is. For example, the log analysis device in the above-described embodiment records a program for realizing the operation and function on a computer-readable recording medium, and causes the computer to read and execute the program recorded on the recording medium. May be realized.

  Here, the “computer” includes a homepage providing environment (or display environment) if the WWW system is used. The “computer-readable recording medium” refers to a storage device such as a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a hard disk built in the computer. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The log analysis program described above may be transmitted from a computer storing the program in a storage device or the like to another computer via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the log analysis program described above may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer, what is called a difference file (difference program) may be sufficient.

It is a block diagram which shows the structure of the network provided with the log analyzer 10 by one Embodiment of this invention. It is a flowchart which shows operation | movement of the log analyzer 10 in the same embodiment. It is the graph which represented the event amount on the time axis in the same embodiment. It is the graph which represented the event amount on the frequency axis in the same embodiment. It is a graph which shows the example which grouped distribution on a frequency axis in the embodiment. It is a figure which shows the ratio analysis model in the same embodiment. It is a figure which shows the rare rate analysis model in the embodiment. It is a flowchart which shows operation | movement of the log analyzer 10 in the same embodiment. It is a flowchart which shows operation | movement of the log analyzer 10 in the same embodiment. It is a flowchart which shows operation | movement of the log analyzer 10 in the same embodiment. It is a flowchart which shows operation | movement of the log analyzer 10 in the same embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Log analyzer, 20, 21, 22 ... Network, 30 ... Security operation center, 101 ... Log collection part, 102 ... Log storage part, 103 ... Storage part, 104 ... Log analysis part, 105 ... Interface part, 301 ... Web browser.

Claims (11)

In a log analyzer that performs analysis processing based on logs collected from network devices,
Collecting means for collecting logs output from the network device;
Conversion means for obtaining a distribution on the time axis of events belonging to the parameters in the log collected by the collecting means, and converting the distribution on the time axis to a distribution on the frequency axis;
The have line grouping to divide the distribution on the frequency axis generated in the plurality of groups by the conversion unit, for each frequency of the event in the distribution on the frequency axis generated by the conversion means, each group Grouping means for grouping the events so that the total amount of events in the
Entropy calculating means for calculating an entropy value based on the amount of events in each group in the distribution on the frequency axis grouped by the grouping means;
Based on the entropy value calculated by the entropy calculating means, an abnormal value calculating means for calculating a value related to the degree of network abnormality,
A log analyzer characterized by comprising: The distribution on the frequency axis belongs to the parameter in the log, and is a distribution related to events within a predetermined period,
The abnormal value calculating unit calculates a ratio between a first entropy value related to the distribution in a specific predetermined period calculated by the entropy calculating unit and second entropy values related to the plurality of distributions related to a predetermined period in the past. The log analyzer according to claim 1 , wherein: The distribution on the frequency axis belongs to a parameter in the log and relates to a first distribution relating to an event of a specific network, and relates to an event of a plurality of other networks belonging to the parameter in the log and excluding the specific network A second distribution,
The abnormal value calculating means calculates a ratio between a first entropy value related to the first distribution calculated by the entropy calculating means and a second entropy value related to the second distribution. The log analysis device according to claim 1 . The distribution on the frequency axis belongs to the parameter in the log, and is a distribution related to events within a predetermined period,
The entropy calculating means includes a first entropy value relating to the distribution in a specific predetermined period grouped by the grouping means, and a plurality of distributions relating to the plurality of distributions relating to past predetermined periods grouped by the grouping means. The entropy value of 2 and
The abnormal value calculation means calculates a probability distribution of the second entropy value, and calculates an upper rare rate or a lower rare rate with respect to the first entropy value based on the probability distribution. The log analyzer according to 1 . The distribution on the frequency axis belongs to a parameter in the log, and is a first distribution related to an event of a specific network, and a second distribution related to an event of a network other than the specific network belongs to the parameter in the log. And the distribution of
The entropy calculating means includes a first entropy value related to the first distribution grouped by the grouping means and a second entropy value related to the second distribution grouped by the grouping means. Calculate
The abnormal value calculation means calculates a probability distribution of the second entropy value, and calculates an upper rare rate or a lower rare rate with respect to the first entropy value based on the probability distribution. The log analyzer according to 1 . In a log analysis method that performs analysis processing based on logs collected from network devices,
Collecting logs output from the network device;
Obtaining a distribution on the time axis of events belonging to the parameters in the collected logs, and converting the distribution on the time axis to a distribution on the frequency axis;
There line grouping to divide the distribution on the frequency axis into a plurality of groups, for each frequency of the event in the distribution on the frequency axis, the event as the total amount of the events in each group approaches equally Grouping steps;
Calculating an entropy value based on the amount of events in each group in the grouped distribution on the frequency axis;
Based on the calculated entropy value, calculating a value related to the degree of network abnormality,
Log analysis program to make a computer execute. The distribution on the frequency axis belongs to the parameter in the log, and is a distribution related to events within a predetermined period,
In the step of calculating the value related to the degree of network abnormality based on the calculated entropy value, the calculated first entropy value related to the distribution in the specific predetermined period and the plurality of distributions related to the past predetermined period. The log analysis program according to claim 6 , wherein a ratio with the second entropy value is calculated. The distribution on the frequency axis belongs to a parameter in the log and relates to a first distribution relating to an event of a specific network, and relates to an event of a plurality of other networks belonging to the parameter in the log and excluding the specific network A second distribution,
In the step of calculating the value related to the degree of network abnormality based on the calculated entropy value, the calculated first entropy value related to the first distribution and the second entropy value related to the second distribution; The log analysis program according to claim 6 , wherein the ratio is calculated. The distribution on the frequency axis belongs to the parameter in the log, and is a distribution related to events within a predetermined period,
In the step of calculating an entropy value based on the amount of events in each group in the grouped distribution on the frequency axis, a first entropy value related to the distribution in a specific grouped period, and a group And calculating a second entropy value related to a plurality of the distributions related to the predetermined past period,
In the step of calculating a value related to the degree of network abnormality based on the calculated entropy value, a probability distribution of the second entropy value is obtained, and an upper rare rate or a lower rare rate related to the first entropy value is obtained. The log analysis program according to claim 6 , wherein: is calculated based on the probability distribution. The distribution on the frequency axis belongs to a parameter in the log and relates to a first distribution relating to an event of a specific network, and relates to an event of a plurality of other networks belonging to the parameter in the log and excluding the specific network A second distribution,
In the step of calculating the entropy value based on the amount of events in each group in the grouped distribution on the frequency axis, the first entropy value related to the grouped first distribution is grouped. Calculating a second entropy value for the second distribution;
In the step of calculating a value related to the degree of network abnormality based on the calculated entropy value, a probability distribution of the second entropy value is obtained, and an upper rare rate or a lower rare rate related to the first entropy value is obtained. The log analysis program according to claim 6 , wherein: is calculated based on the probability distribution. Any computer-readable recording medium which records a log analysis program according to the preceding claims 6 to claim 10.

Images