JP2005209115A - Log summarization device, log summarization program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a log summarization device and a log summarization program capable of visibly shaping an enormous amount of logs to provide useful information to a user; and to provide a recording medium. <P>SOLUTION: A summarization processing part 141 finds impurity (f) for each attribute data in a log by using an impurity function, and selects attribute data having the largest impurity (f) as attribute data of a summarization object. Then, the summarization processing part 141 replaces a plurality of item data with an identical numerical value or character string so as to lower the impurity of the attribute data based on the appearance frequency of the item data appearing in the selected attribute data. The summarization processing part 141 deletes a duplicated line created by the replacement. The summarization processing part 141 repeats the summarization process until the line number of the log is reduced to less than a predetermined line number. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a log summarization device, a log summarization program, and a recording medium that automatically summarize logs of network devices (communication devices, security devices) such as routers, firewalls, IDS (Intrusion Detection System).

  When devices such as routers, firewalls, packet filters, and IDSs are installed in the network, a huge amount of records (logs) are generated in these devices. It is difficult to confirm all of these enormous logs, and methods for summarizing or analyzing the logs have been proposed.

Patent Document 1 describes a control structure extraction method for extracting a control structure from a log such as an execution trace of a program and shaping the log using the information. Further, Patent Document 2 describes a method for reducing an access log that extracts an access log that is substantially meaningful from the access log. Patent Document 3 describes a log information analysis device that extracts log information in consideration of the appearance frequency of words or two-word concatenation phrases.
JP 2003-241998 A JP 2003-76814 A JP 2001-356939 A

  The log output in the network device is very large, and it is difficult for the user to confirm all of the logs. In addition, useful information is buried in other unimportant information, and it is difficult to confirm only useful information.

  The present invention has been made in view of the above-described problems, and provides a log summarization apparatus, a log summarization program, and a recording medium that can easily format a huge log and present useful information to a user. For the purpose.

  The present invention has been made to solve the above-described problems, and the invention according to claim 1 is a log summarizing apparatus that summarizes a log output in a network device, and includes attribute data recorded in the log. Based on the appearance frequency for each item data that appears, a calculation unit that calculates a value related to the uneven distribution degree of the appearance item of the attribute data, and selects attribute data to be aggregated based on a calculation result of the value related to the uneven distribution degree Produced by selection means, replacement means for replacing a plurality of item data with the same numerical value or character string based on the appearance frequency of item data appearing in the selected attribute data, and replacement of item data by the replacement means A log summarizing apparatus comprising: an aggregating means for aggregating overlapping lines into the same line.

  The attribute data represents a data type such as a file size, a file name, and an IP address recorded in the log. Item data represents specific contents (numerical values or character strings) of attribute data. The appearance frequency represents the frequency at which the same item data appears in the log, such as the number of appearances and the appearance probability. Information entropy, a gini coefficient, etc. are used in the calculation of the value regarding the degree of uneven distribution. The uneven distribution degree increases as the item data concentrates on specific data, and conversely, the uneven distribution degree decreases as the item data varies without concentrating on specific data.

  According to a second aspect of the present invention, in the log summarizing apparatus according to the first aspect, the attribute data is any one of continuous attribute data, discrete attribute data, and hierarchical structure attribute data, and the replacement means selects The plurality of item data is replaced with the same numerical value or character string so that the degree of uneven distribution of the attribute data is increased based on the appearance frequency of the item data appearing in the attribute data.

  The continuous attribute data has a magnitude relationship between item data, and the magnitude relationship is meaningful. In the discrete attribute data, there is no size relationship between item data, or even if there is a size relationship, there is no particular meaning in the size relationship. In the hierarchical structure attribute data, item data is represented by a hierarchical structure. An IP address, directory structure, directory path, URL (Uniform Resource Locator), URI (Uniform Resource Identifier), mail address, Xpath, and the like can be included in the hierarchical structure attribute data.

  According to a third aspect of the present invention, in the log summarizing apparatus according to the second aspect, when the attribute data is the continuous attribute data, the replacement means includes a difference between a plurality of item data appearing in the attribute data. A plurality of item data is replaced with the same numerical value or character string so that the degree of uneven distribution of the attribute data is increased based on the appearance frequency of the item data.

  According to a fourth aspect of the present invention, in the log summarizing apparatus according to the second aspect, when the attribute data is the discrete attribute data, the replacing means hums between a plurality of item data appearing in the attribute data. A plurality of item data is replaced with the same numerical value or character string so that the degree of uneven distribution of the attribute data is increased based on the distance and the appearance frequency of the item data.

  The Hamming distance is a value indicating how many characters at the same position of a plurality of character strings to be compared are different. If the character string lengths of the plurality of character strings to be compared are different, the difference between the character string lengths may be added to the Hamming distance.

  According to a fifth aspect of the present invention, in the log summarizing apparatus according to the second aspect, when the attribute data is the hierarchical structure attribute data, the replacement unit converts the item data appearing in the attribute data into a tree structure. Item data having a low appearance frequency is replaced with the same numerical value or character string so that the attribute data is allocated to nodes and the degree of uneven distribution of the attribute data is increased.

  According to a sixth aspect of the present invention, there is provided the log summarizing apparatus according to any one of the first to fifth aspects, wherein the log is divided into a plurality of logs, and the divided individual logs are summarized. It further comprises connecting means for connecting subsequent logs.

  The logs divided by the dividing unit are summarized by the processing of the calculating unit, selecting unit, replacing unit, and aggregating unit. The concatenation means concatenates the summarized individual logs after the division. In this case, the concatenation unit may concatenate individual logs every predetermined number to generate a plurality of logs. When a plurality of logs are generated, each log is summarized again by the processing of the calculation means, selection means, replacement means, and aggregation means. The connecting means connects the logs after summarization. The above processing may be repeated.

  The invention according to claim 7 is a log summarization program that causes a computer to execute a log summarization process of a log output in a network device, based on the appearance frequency for each item data appearing in the attribute data recorded in the log Calculating a value related to the uneven distribution degree of the appearance items of the attribute data, selecting attribute data to be aggregated based on a calculation result of the value related to the uneven distribution degree, and appearing in the selected attribute data A step of replacing a plurality of item data with the same numerical value or character string, and a step of consolidating duplicate rows generated by the replacement of the item data by the replacement means into the same row based on the appearance frequency of the item data to be performed This is a log summarization program to be executed.

  The invention according to claim 8 is the log summarization program according to claim 7, wherein the attribute data is any one of continuous attribute data, discrete attribute data, and hierarchical structure attribute data, and the selected attribute is selected. In the step of replacing the plurality of item data with the same numerical value or character string based on the appearance frequency of item data appearing in the data, based on the appearance frequency of the item data appearing in the selected attribute data, A plurality of item data is replaced with the same numerical value or character string so that the degree of uneven distribution of the attribute data is increased.

  The invention according to claim 9 is the log summarization program according to claim 8, wherein when the attribute data is the continuous attribute data, a difference between a plurality of item data appearing in the attribute data and the item data A plurality of item data is replaced with the same numerical value or character string so that the degree of uneven distribution of the attribute data is increased based on the appearance frequency.

  In the log summarizing program according to claim 8, when the attribute data is the discrete attribute data, the Hamming distance between a plurality of item data appearing in the attribute data and the item data The plurality of item data is replaced with the same numerical value or character string so that the degree of uneven distribution of the attribute data is increased based on the appearance frequency of.

  The invention according to claim 11 is the log summarization program according to claim 8, wherein when the attribute data is the hierarchical structure attribute data, item data appearing in the attribute data is assigned to a node of a tree structure, The item data with a low appearance frequency is replaced with the same numerical value or character string so that the degree of uneven distribution of the attribute data is increased.

  The invention according to claim 12 is the log summarization program according to any one of claims 7 to 11, wherein the log is divided into a plurality of logs before the step of calculating the uneven distribution degree. And a step of concatenating logs after summarizing the divided individual logs after the step of aggregating the duplicate lines.

  A thirteenth aspect of the present invention is a computer-readable recording medium on which the log summarizing program according to any one of the seventh to twelfth aspects is recorded.

  According to the present invention, since log summarization is performed in consideration of the appearance frequency of items for each attribute in the log, it is possible to easily format a huge log and present useful information to the user. An effect is obtained.

  The best mode for carrying out the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a log summarizing apparatus according to an embodiment of the present invention. The log summarization apparatus according to the present embodiment is connected to a network and summarizes logs output from various network devices. As the network device, a router, a firewall, an IDS, or the like is assumed.

  In FIG. 1, reference numeral 10 denotes a log collection unit, which collects logs from a network device via a network. Reference numeral 11 denotes an operation unit, which includes a keyboard and the like operated by the user. When the operation unit 11 is operated by the user, the operation unit 11 outputs a signal indicating an operation by the user to the control unit 15. The control unit 15 determines the content of the user operation based on this signal.

  Reference numeral 12 denotes a storage unit that stores logs collected by the log collection unit 10, operation programs for the control unit 15, setting information related to settings such as a log format, and the like. Reference numeral 13 denotes a display unit, which includes, for example, a liquid crystal display. The display unit 13 displays the result of the user operation, the log summary result, and the like based on the display data output by the control unit 15. Reference numeral 14 denotes a log summarization unit, which summarizes logs according to processing described later. The log summarization unit 14 includes a summarization processing unit 141 that performs summarization processing, and a processing area 142 such as a memory that temporarily stores processing data. The control unit 15 controls each unit.

  The log summarizing apparatus having the above configuration is realized by, for example, a general-purpose personal computer. The log summarization device may collect logs output from an external communication device via a network and summarize the collected logs. If the functions of various network devices are realized as software, A personal computer in which the software is installed may have the function of a log summarizing device.

  Next, the log in this embodiment will be described. In general, a log has a format defined in advance, and has a structure in which recording occurs in time series. The log format is defined for each network device, but a general-purpose format is also defined between logs with different devices. The log is composed of a plurality of lines, and the log data per line is composed of time data and a plurality of attribute data. The attributes in the present embodiment are attributes whose values change continuously (continuous attributes), attributes whose values change discretely (discrete attributes), and attributes whose values are represented by a hierarchical structure (hierarchical structure attributes) ) Is one of the attributes.

  FIG. 2 is a reference diagram showing the general format of the log in this embodiment. The log data in this embodiment is arranged in the order of time indicated by the first time data, and each log data has attribute data belonging to any of continuous attributes, discrete attributes, and hierarchical structure attributes. The attribute data belonging to the continuous attribute is represented by a numerical value, and the magnitude relationship for each item is meaningful. The attribute data belonging to the discrete attribute is represented by a character string or a numerical value, and does not have a magnitude relationship for each item, or if there is any, the magnitude relationship has no particular meaning. In the attribute data belonging to the hierarchical structure attribute, items are represented in a hierarchical structure. Note that attribute data in the present embodiment indicates a file size, a file name, an IP address, and the like. The attribute data is composed of a plurality of items, and the specific contents (numerical values and character strings) of the items are defined as item values (item data).

  FIG. 3 is a reference diagram illustrating an example of a log generated in an ftp (file transfer protocol) server. In the figure, each row constitutes an event. FIG. 4 is a reference diagram showing the format of this log. Hereinafter, the log format will be described using the data in the first line of the log in FIG. FIG. 4 shows that the log includes five attribute data of “current-time”, “transfer-time”, “remote-host”, “file-size”, and “filename”. “Mon Sep 1 04:48:02 2003” in FIG. 3 corresponds to “current-time” in FIG. The general format of “current-time” is expressed as “DDD MMM ddhh: mm: ss YYYY”, and the meaning of each variable is as described in FIG.

  “9” in FIG. 3 corresponds to “transfer-time” in FIG. 4 and represents the total transfer time in seconds. “192.168.80.19” in FIG. 3 corresponds to “remote-host” in FIG. 4, and is a remote host name represented by an IP address. “61687” in FIG. 3 corresponds to “file-size” in FIG. 4 and represents the size of the transferred file in bytes. “/Tm/packages/mictex-metafont-misc.cab” in FIG. 3 corresponds to “filename” in FIG. 4 and represents the transferred file name. This “filename” indicates the hierarchical structure of the transferred file.

  Among the above items excluding current-time which is the order of time-series data, “transfer-time” and “file-size” belong to continuous attributes, and “filename” and “remote-host” are hierarchical structure attributes. Belonging to. “Filename” and “remote-host” can be handled as discrete attributes, but in the present embodiment, they are handled as hierarchical structure attributes as described later.

  Next, the operation of the log summarizing apparatus according to the present embodiment will be described. FIG. 5 is a schematic reference diagram showing a state of log summarization in the present embodiment. In the present embodiment, the log summarization apparatus cuts out a log having an arbitrary length of N lines from a vast number of logs, and summarizes it into a log having a length less than a predetermined length of M lines. M and N are positive integers, and M ≦ N.

  The log collection unit 10 collects logs output from the network device and outputs them to the control unit 15. The control unit 15 stores the log output by the log collection unit 10 in the storage unit 12. When the operation unit 11 is operated by the user and the start of log summarization is instructed, the operation unit 11 outputs a signal indicating the start instruction by the user to the control unit 15. Based on this signal, the control unit 15 reads the log and setting information from the storage unit 12 and outputs the log and setting information to the log summarization unit 14. The log summarization unit 14 summarizes the log and outputs the summarized log to the control unit 15. The control unit 15 outputs display data for displaying this log to the display unit 13. The display unit 13 performs display based on the display data. A user such as a network administrator can confirm the log summary result by the display on the display unit 13.

  Details of the log summarization method performed by the log summarization unit 14 will be described below. In the setting information, information indicating the log format described above, the number of lines N indicating the length of log cutting, the number of lines M indicating the length of the log as a reference for the end of the summary process, and the like are recorded. These information can be arbitrarily changed by the user. The log and the setting information are stored in the processing area 142 by the summary processing unit 141 and are appropriately read out by the summary processing unit 141. Based on the setting information, the summary processing unit 141 converts the logs into a format that can be aggregated. That is, the attribute data that does not follow the format defined by the setting information is removed from the information recorded in the log, and attribute data called a counter is added to each event in the log. The value indicated by this counter indicates the number of events in which all the attribute data values in the event are the same value, and the initial value of the counter is 1.

  Subsequently, the summary processing unit 141 refers to the number N of rows indicating the log cut length in the setting information, and cuts out N rows of data (processing data) from a huge log. In the extraction of the processing data, N rows of data can be extracted within an arbitrary time range by referring to the time data recorded in the log.

Subsequently, the summary processing unit 141 selects attribute data to be aggregated. This selection is performed as follows. The summary processing unit 141 obtains the appearance rate (appearance probability) for each specific item value for each attribute data, and obtains the impurity using the obtained appearance rate and the impurity function. Then, the attribute data with the highest purity is selected and aggregated. When paying attention to one attribute (file size, file name, etc.), | S | is the total number of events of that attribute data (the same as the number of lines of processing data), and the item value (numerical value) that appears in the attribute data Or a character string) is C _i (i = 0, 1, 2,...), And the number of occurrences of C _{i in} the attribute data is | C _i |. The appearance rate p _i of C _i is expressed by the following [Equation 1].

  As the impurity function, an information amount (information entropy) or a gini coefficient is used. When the impurity function is an information amount, the impurity f is expressed by [Equation 2], and when the impurity function is a gini coefficient, the impurity f is expressed by [Equation 3].

  The impurity f is an index indicating the degree of uneven distribution of how much the item values are concentrated on a specific value for specific attribute data. For example, the larger the value, the larger the value of f (the degree of uneven distribution) becomes smaller, and the more the value concentrates on a specific value, the smaller the value of f (the degree of uneven distribution) becomes. An attribute with a small value of f is likely to contain important data for log analysis. For example, when an attack spreads rapidly on the network, the value of f becomes small because the item value tends to concentrate on a specific value, and analyzing such attributes is important for analyzing the attack It is. Therefore, an attribute having a large value of f often includes data that is not very important, and the summary processing unit 141 generates a log that is easy for the user to view by summarizing such data.

  The summarization processing unit 141 obtains the impurity f for each attribute data, and selects the attribute data having the largest f as the attribute data to be summarized. The operation of the summary processing unit 141 varies depending on the type of the selected attribute data. Hereinafter, the operation of the summary processing unit 141 will be described for each attribute data.

When the selected attribute data is a continuous attribute, the summary processing unit 141 stores the item value of the attribute data and the appearance frequency of the value in the processing area 142. Subsequently, the summary processing unit 141 obtains a value d ₁ represented by [Equation 4]. In [Expression 4], D _j is the j-th (j = 0, 1, 2,...) Item value in the attribute data, and | D _j | is the appearance rate (the appearance frequency of D _j is all appearing) Frequency divided by the number of log lines). | D _j | + | D _{j + 1} | is the sum of the frequency of appearance of adjacent item values, and | D _{j + 1} −D _j | represents the distance (difference) between adjacent item values.

D ₁ in [Expression 4] is a value corresponding to the density. As d ₁ is larger, the degree of concentration to a specific value is larger, and such data is important data for log analysis. Thus, summarization unit 141 to aggregate in order from the interval d ₁ is small. That is, the summary processing unit 141 stores in the processing area 142 replacement information that replaces adjacent item values D _j and D _{j + 1} that minimize d ₁ with the same value. The summary processing unit 141 repeats the above process a predetermined number of times. Alternatively, the summary processing unit 141 repeats the above-described processing until the number of types of item values reaches a predetermined number. This predetermined number of times or the predetermined number is recorded in advance in the setting information. As described above, by summarizing unimportant data, the summary processing unit 141 reduces the impurity f of the attribute data (increases the uneven distribution degree).

When the attribute data selected by the summary processing unit 141 is a discrete attribute, the summary processing unit 141 stores the item value of the attribute data and the appearance frequency of the value in the processing area 142. Subsequently, the summary processing unit 141 obtains a value d ₂ expressed by [Equation 5]. In [Expression 5], E _k is the k-th (k = 0, 1, 2,...) Item value in the attribute data, and | E _k | is the appearance rate (the appearance frequency of E _k is the total appearance). Divided by frequency). _{d hum (k, k + 1} ) is the Hamming distance (a value indicating whether the character is a number different from the same position of _{E k} and _{E k + 1)} of _{E k} and _{E k + 1.} When the character string lengths of E _k and E _{k + 1} to be compared are different, the difference between the character string lengths is added to the Hamming distance.

D ₂ in [Equation 5] is also a value corresponding to the density, similarly to d ₁ . Summary processor 141 to aggregate in order from the interval d ₂ is small. That is, the summary processing unit 141 stores in the processing area 142 replacement information that replaces adjacent item values E _j and E _{j + 1} that minimize d ₂ with the same value. The summary processing unit 141 repeats the above process a predetermined number of times or until the number of types of items reaches a predetermined number. As described above, by summarizing unimportant data, the summary processing unit 141 reduces the impurity f of the attribute data (increases the uneven distribution degree).

  When the attribute data selected by the summary processing unit 141 is a hierarchical structure attribute, the summary processing unit 141 stores the item value of the attribute data and the appearance frequency of the value in the processing area 142. Subsequently, the summary processing unit 141 assigns all item values of the attribute data to the nodes of the tree structure. FIG. 6 is a schematic reference diagram showing a method for assigning item values of attribute data to nodes of a tree structure when a directory structure is taken as an example. In the figure, reference numerals 601 to 604 denote nodes. The summary processing unit 141 reads the item value of the attribute data and recognizes the node of the tree structure by the appearance of the character “/”.

  For example, when the first read value is “/ usr /”, the summary processing unit 141 creates a node 601 associated with the leading “/”. Subsequently, the summary processing unit 141 creates a node 602 associated with “usr /” in a hierarchy below the node 601 and assigns this value to the node 602. When the next read value is “/ home / yamada /”, the summary processing unit 141 recognizes the node 601 by the leading “/”, and the node 603 associated with the subsequent “home /” is the node 601. Create in the hierarchy below. Subsequently, the summary processing unit 141 creates a node 604 associated with “yamada /” in a hierarchy below the node 603 and assigns the read value to the node 604. Similarly, a node 605 is created with the value “/ home / miyake /”, and this value is assigned to the node 605.

  7 and 8 are schematic reference diagrams showing how items in the hierarchical structure attribute are aggregated. FIG. 7A shows a tree structure before aggregation. Circles attached to the nodes of the tree structure indicate item values assigned to the nodes, and the size of the circles schematically indicates the number of occurrences of the item values. Assume that nodes 701 to 715 are assigned in advance to each hierarchical structure. The number of item values that appear at each node is as shown on the left side of FIG. For example, the number of occurrences of the item value (“/ usr / src / a /”) assigned to the node 708 is 1.

  The summary processing unit 141 assigns each item value to the node of the tree structure as described above, and calculates Th represented by the following [Equation 6]. In [Equation 6], S is the sum of the number of appearances of item values belonging to each node (total number of items), and is equal to the number of lines in the log. 7 to 9, S is 14 which is the sum of the numbers of appearances shown in FIG. O is the number of item values to be output, and is 4 in this example. S and O are recorded in advance in the setting information in the storage unit 12. In this example, it is assumed that the item values of the appearing items having the number of items of 14 are collected into four item values (14 items are assigned to four nodes).

The summary processing unit 141 calculates the above Th. Since S = 14 and O = 4, Th = 3.5. This Th indicates the average number of items per node after aggregation. The summary processing unit 141 performs the following processing in order from the item value with the smallest number of appearances.
(1) When aggregation is performed at a node in the next higher hierarchy, if the number of occurrences of item values at that node is equal to or less than Th, it is determined that aggregation is possible.
(2) In the case of (1), when it is possible to consolidate to the node of the hierarchy one level above, the item values belonging to the hierarchy below the node are consolidated to the node.
(3) In (1), when it is not possible to consolidate the nodes one level above, (1) is considered again for the nodes one level higher.

  The summary processing unit 141 repeats the above (1) to (3), and aggregates the item values until the number of item values to be output becomes O. If no assignable node is found and the search proceeds to the highest node, the node is assigned to the highest node. Hereinafter, a specific example will be described. In the state of FIG. 7A, the summary processing unit 141 selects an item value belonging to the node 708, which is the item value having the smallest number of appearances. In accordance with (1) above, the summary processing unit 141 tries to assign to the node 704 (see FIG. 7B). If an item value belonging to the node 708 is assigned to the node 704, the number of items belonging to the node 704 is 1, which is equal to or less than Th. Further, if the item values belonging to the node 709 belonging to the hierarchy below the node 704 are also assigned to the node 704, the total number of items belonging to the node 704 is 3, which is equal to or less than Th. Therefore, the item values belonging to the nodes 708 and 709 are assigned to the node 704 according to (2) above (see FIG. 7C).

  Subsequently, the summary processing unit 141 selects an item value belonging to the node 711, and tries to assign to the node 705 in accordance with the above (1) (see FIG. 7D). If the item value belonging to the node 711 is assigned to the node 711 and the item value belonging to the node 710 is also assigned to the node 711, the total number of items belonging to the node 711 is 4, which exceeds Th. Assignments are made. The summary processing unit 141 tries to assign the item value belonging to the node 711 to the node 702 in the higher hierarchy (see FIG. 7E). According to the above (2), when the item values belonging to the nodes 704 and 711 in the hierarchy below the node 702 are also assigned to the node 702, the total number of items exceeds Th, so that the summarization processing unit 141 has the highest node. Item values are assigned to 701 (FIG. 7 (f)).

  Subsequently, the summary processing unit 141 selects an item value belonging to the node 714, and tries to assign to the node 707 in accordance with the above (1) (see FIG. 8A). If all the item values belonging to the hierarchy below the node 707 are assigned to the node 707, the number of items is equal to or less than Th, and the item values belonging to the nodes 714 and 715 are assigned to the node 707 (see FIG. 8B). Subsequently, the summary processing unit 141 selects an item value belonging to the node 713 and tries to assign it to the node 705 (see FIG. 8C). When the item values belonging to the node 710 in the hierarchy below the node 705 are also assigned to the node 705 according to the above (2), the total number of items exceeds Th, so the summary processing unit 141 tries to assign to the node 703. (See FIG. 8D). Also in this case, according to (3), the summary processing unit 141 assigns the node 701 in the highest hierarchy (see FIG. 8E).

  Subsequently, the summary processing unit 141 selects an item value belonging to the node 707 (number of appearances 2) as the item value having the smallest number of appearances, and tries to assign it to the node 703 (see FIG. 8F). According to (3) above, the summary processing unit 141 assigns this item value to the node 701 in the highest hierarchy (see FIG. 8G). Up to this point, the number of item values has reached a predetermined value of 4, so the summary processing unit 141 ends the aggregation of items. Each item value after aggregation and the number of items are as shown on the right side of FIG. Based on the above aggregation, the summary processing unit 141 stores in the processing area 142 as replacement information what item value each item value is replaced with. As described above, the summary processing unit 141 reduces the impurity f of the attribute data (increases the uneven distribution degree) by collecting unimportant item values (item values with a small number of appearances) at the same node.

  Following the item aggregation for each attribute data as described above, the summary processing unit 141 replaces each item value in the log. That is, the summary processing unit 141 reads the replacement information from the processing area 142, and replaces the item value of the item having the same item value by aggregation with the same value based on the replacement information. For example, in the case of the above hierarchical structure attribute, the item values “/ usr / src / a /” (item value of node 708) and “/ usr / src / b” (item value of node 709) are “// It is replaced with “usr / src /” (the item value of the node 704).

  Subsequently, the summary processing unit 141 counts the number of duplicate lines (lines in which the item values of each attribute data all match) generated by the above replacement, stores the number in the counter, and leaves only one line to other duplicates. Delete the line. In this case, “current-time” is not involved in the above-described counting of duplicate rows. That is, the summary processing unit 141 determines whether or not there is duplication regarding attribute data excluding “current-time”. Alternatively, the summary processing unit 141 counts the duplicate lines after replacing “current-time” with the same value for all events.

  The summary processing unit 141 reads the setting information from the processing area 142 and refers to the setting information. If the number of remaining log lines is less than M as a result of the above-described deletion of duplicate lines, the summary processing unit 141 ends the summary process. The summary processing unit 141 selects attribute data based on the impurity f, and performs the item aggregation processing on the attribute data.

  In addition, regarding the item aggregation of hierarchical structure attributes, the directory structure has been described as an example, but the same applies to IP addresses. FIG. 10 is a schematic reference diagram showing a method for assigning IP addresses to a tree structure. As shown in the figure, the summary processing unit 141 converts each numerical value of the IP address divided by periods into a 32-bit binary number. Subsequently, the digest processing unit 141 inserts a delimiter between each bit, and assigns 1 or 0 delimited by the delimiter to the upper hierarchy from the left. For example, in the case of a binary number represented by “1100”, the numerical value of each bit is assigned to nodes 1102 to 1105 belonging to the lowermost node 1101 as shown in FIG. The summary processing unit 141 aggregates items for the tree structure constructed as described above.

  FIG. 11 is a flowchart showing the operation of the log summarization apparatus described above. When the operation unit 11 is operated by the user and the start of log summarization is instructed, the operation unit 11 outputs a signal indicating the start instruction by the user to the control unit 15. Based on this signal, the control unit 15 reads the log and setting information from the storage unit 12 and outputs the log and setting information to the log summarization unit 14. The log and setting information are input to the log summary unit 14 (step S1101). Subsequently, the summary processing unit 141 in the log summarizing unit 14 converts the logs into a format that can be aggregated based on the setting information (step S1102).

  Subsequently, the summary processing unit 141 refers to the number N of lines indicating the log cut length in the setting information, and cuts out N rows of processing data from the log (step S1103). The summary processing unit 141 obtains the impurity f for each attribute data using the impurity function, and selects the attribute data having the largest f as the attribute data to be summarized (step S1104). The summary processing unit 141 determines whether the selected attribute data belongs to a continuous attribute, a discrete attribute, or a hierarchical structure attribute (step S1105).

When selected attribute data is continuous attribute, abstract processor 141 to aggregate in order from d ₁ is small intervals indicated by [Equation 4] (Step S1106). Also, if the attribute data selected are discrete attributes, abstract processor 141 to aggregate in order from d ₂ is smaller intervals indicated by Equation 5 (step S1107). If the selected attribute data is a hierarchical structure attribute, the summary processing unit 141 assigns each item to a node of the tree structure, and aggregates the items in descending order of item values (step S1108).

  Subsequently, the summary processing unit 141 replaces the item values of the items that become the same by the aggregation (step S1109). The summary processing unit 141 counts the number of duplicate lines generated by the replacement, stores the number in a counter, and deletes the remaining duplicate lines while leaving only one line (step S1110). Subsequently, the summary processing unit 141 reads the setting information from the processing area 142 and refers to it, and determines whether the number of remaining log lines is less than M (step S1111). If the number of remaining log lines is less than M, the process ends. On the other hand, if the number of remaining log lines is M or more, the process returns to step S1104. The above processing is repeated until the number of log lines is less than M.

  When summarizing a large log, it may be repeated that the log is first divided into a plurality of logs and summarization is performed on the divided logs. For example, in FIG. The summary processing unit 141 divides the log 1000 into a plurality of logs 500, 510, 520, 530,... Having a predetermined length. Subsequently, the summary processing unit 141 summarizes the individual logs by the above-described summarization method, and generates logs 400, 410, 420, 430. The summary processing unit 141 concatenates each summarized log for each predetermined number, and generates logs 300 to 330. Subsequently, the summary processing unit 141 summarizes each connected log and generates logs 200 to 230. The summary processing unit 141 concatenates these again to generate the log 100, and summarizes this log to generate the log 20 having a number less than a predetermined number of lines.

  As described above, according to the present embodiment, an enormous log output in an IDS, firewall, router, Web server, ftp server, various syslogs, peer-to-peer systems, etc. is less than a predetermined number of lines (or below). Can be summarized. As a result, it is possible to easily format an enormous log and present useful information in the log to a network administrator or the like. Further, according to the present embodiment, summarization can be performed by specifying the scale (number of lines) of the log to be output. This makes it possible to reduce the size of the log to be output when you want to get an overview of the network, and to summarize by reducing the size of the log to be output when you want to know the details of the network. Summary can be made.

  Among the attributes output as logs, in particular, by assigning an IP address, directory structure, directory path, URL (Uniform Resource Locator), URI (Uniform Resource Identifier), mail address, Xpath, etc. to the tree structure, Useful summaries can be made. In the case of a hierarchical structure, each hierarchy has meaningful information, and the hierarchy of items to be aggregated has meaningful information. For example, in the case of a directory structure of “/ home / a /”, all of “/”, “/ home /”, and “/ home / a” that are candidates for aggregation are meaningful information. According to the hierarchical attribute summarization method according to the present embodiment, the hierarchical structure can be aggregated into meaningful information with a finer granularity. Many communication logs have hierarchical structure attributes such as an IP address, URI, and Xpath, and according to the present embodiment, these can be aggregated more finely.

  In addition, by repeating log summarization step by step, processing can be performed at a higher speed than generating a log with less than a predetermined number of lines in one summary. Furthermore, by storing the summary results, when a new log is output, only the new log needs to be summarized.

  As described above, the embodiments of the present invention have been described in detail with reference to the drawings, but the specific configuration is not limited to these embodiments, and includes design changes and the like within a scope not departing from the gist of the present invention. It is. For example, the log summarization apparatus in the above-described embodiment records a program for realizing the operation and function on a computer-readable recording medium, and causes the computer to read and execute the program recorded on the recording medium. May be realized.

  Here, the “computer” includes a homepage providing environment (or display environment) if the WWW system is used. The “computer-readable recording medium” refers to a storage device such as a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a hard disk built in the computer. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The log summarization program described above may be transmitted from a computer storing the program in a storage device or the like to another computer via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the log summarization program described above may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer, what is called a difference file (difference program) may be sufficient.

It is a block diagram which shows the structure of the log summarizing apparatus by one Embodiment of this invention. It is a reference figure for demonstrating the format of the log in this embodiment. It is a reference figure which shows the example of the log produced | generated in an ftp server. It is a reference figure which shows the format of the log produced | generated in an ftp server. It is a schematic reference figure which shows the mode of the log summarization by a log summarization apparatus. It is a schematic reference figure which shows the allocation method to the tree structure of the item value of a hierarchical structure attribute. It is a schematic reference figure which shows the mode of the item aggregation of a hierarchical structure attribute. It is a schematic reference figure which shows the mode of the item aggregation of a hierarchical structure attribute. It is a schematic reference drawing for demonstrating item aggregation of a hierarchical structure attribute. It is a schematic reference figure which shows the allocation method to the tree structure of an IP address. It is a flowchart which shows operation | movement of a log summarization apparatus. It is a schematic reference diagram for demonstrating the other method in the case of dividing | segmenting a big log.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Log collection part, 11 ... Operation part, 12 ... Memory | storage part, 13 ... Display part, 14 ... Log summary part, 15 ... Control part, 141 ... Summary process Part, 142... Processing area.

Claims (13)

In a log summarization device that summarizes logs output in network devices,
Calculation means for calculating a value related to the uneven distribution degree of the appearance items of the attribute data based on the appearance frequency for each item data appearing in the attribute data recorded in the log;
Selection means for selecting attribute data to be aggregated based on a calculation result of a value regarding the uneven distribution degree;
Replacement means for replacing a plurality of item data with the same numerical value or character string based on the appearance frequency of item data appearing in the selected attribute data;
Aggregating means for aggregating duplicate lines generated by replacement of item data by the replacing means into the same line;
A log summarizing apparatus comprising: The attribute data is any one of continuous attribute data, discrete attribute data, and hierarchical structure attribute data,
The replacement means replaces a plurality of item data with the same numerical value or character string so that the uneven distribution degree of the attribute data is increased based on the appearance frequency of the item data appearing in the selected attribute data. The log summarizing apparatus according to claim 1. When the attribute data is the continuous attribute data, the replacement unit increases the degree of uneven distribution of the attribute data based on the difference between the plurality of item data appearing in the attribute data and the appearance frequency of the item data. As described above, the log summarizing apparatus according to claim 2, wherein a plurality of item data is replaced with the same numerical value or character string. In the case where the attribute data is the discrete attribute data, the replacement means determines the degree of uneven distribution of the attribute data based on the Hamming distance between a plurality of item data appearing in the attribute data and the appearance frequency of the item data. The log summarizing apparatus according to claim 2, wherein a plurality of item data is replaced with the same numerical value or character string so as to increase. When the attribute data is the hierarchical structure attribute data, the replacement means assigns item data appearing in the attribute data to nodes of a tree structure, and an item with a low appearance frequency so that the degree of uneven distribution of the attribute data increases. The log summarizing apparatus according to claim 2, wherein the data is replaced with the same numerical value or character string. Dividing means for dividing the log into a plurality of logs;
A concatenation means for concatenating logs after the summarization of each divided log;
The log summarizing apparatus according to claim 1, further comprising: In a log summarization program that causes a computer to execute log summarization processing that is output in a network device,
Calculating a value related to the degree of uneven distribution of the appearance items of the attribute data based on the appearance frequency for each item data appearing in the attribute data recorded in the log;
Selecting attribute data to be aggregated based on a calculation result of a value regarding the uneven distribution degree;
Replacing a plurality of item data with the same numerical value or character string based on the appearance frequency of the item data appearing in the selected attribute data;
Aggregating duplicate lines generated by the replacement of item data by the replacement means into the same line;
Log summarization program to make the computer run. The attribute data is any one of continuous attribute data, discrete attribute data, and hierarchical structure attribute data,
In the step of replacing the plurality of item data with the same numerical value or character string based on the appearance frequency of the item data appearing in the selected attribute data, the item data appearing in the selected attribute data The log summarization program according to claim 7, wherein a plurality of item data is replaced with the same numerical value or character string so that the degree of uneven distribution of the attribute data is increased based on the appearance frequency. When the attribute data is the continuous attribute data, based on the difference between the plurality of item data appearing in the attribute data and the appearance frequency of the item data, a plurality of uneven distribution levels of the attribute data are increased. 9. The log summarization program according to claim 8, wherein item data is replaced with the same numerical value or character string. In a case where the attribute data is the discrete attribute data, a plurality of the attribute data is increased based on the Hamming distance between the item data appearing in the attribute data and the appearance frequency of the item data. The log summarization program according to claim 8, wherein the item data is replaced with the same numerical value or character string. When the attribute data is the hierarchical structure attribute data, item data that appears in the attribute data is assigned to nodes of a tree structure, and the item data with a low appearance frequency is assigned the same numerical value so that the degree of uneven distribution of the attribute data increases. The log summarization program according to claim 8, wherein the log summarization program is replaced with a character string. Dividing the log into a plurality of logs before calculating the uneven distribution;
Concatenating the logs after summarizing the individual divided logs after the step of aggregating the duplicate rows;
The log summarization program according to any one of claims 7 to 11, further comprising: A computer-readable recording medium on which the log summarizing program according to any one of claims 7 to 12 is recorded.

Images