JP6741217B2 - Log analysis system, method and program - Google Patents

Description

The present invention relates to a log analysis system, method and program for analyzing logs.

In a system executed on a computer, a log including the result of an event, a message, etc. is output. When performing log analysis for the purpose of pursuing the cause of system abnormality, the user (for example, operator) needs to refer to many logs output from the system. In order to reduce the burden on the user, it is required to provide information that assists the log-based analysis.

In Patent Document 1, an important part of a log is defined as a template, and a log related to the specific event is defined based on a change in the amount of logs corresponding to the template before and after a specific event such as a failure. Techniques for detecting are disclosed. Thereby, the occurrence of the specific event can be known based on the log.

JP, 2005-153077, A

However, since the technique of Patent Document 1 extracts a template log having a significant change in the amount of occurrence before and after the specific event that is the cause of the abnormality, the specific event in the past is known. Unless it is a situation, the log related to the abnormality cannot be extracted.

The present invention has been made in view of the above problems, and provides a log analysis system, method, and program capable of outputting information suggesting the cause of an abnormality even when the cause of the abnormality is unknown. With the goal.

A first aspect of the present invention is a log analysis system, wherein each log included in an analysis target log has a variable part that can change in the analysis target log and a constant part that does not change in the analysis target log. A format determination unit that determines which of a plurality of predetermined formats, including, a ratio calculation unit that calculates the respective appearance ratios of the plurality of formats in the analysis target log, and the appearance ratio And an output unit for outputting.
A second aspect of the present invention is a log analysis method, wherein each log included in the analysis target log has a variable part that can change in the analysis target log and a constant part that does not change in the analysis target log. Including a step of determining which of a plurality of predetermined formats matches, a step of calculating the appearance ratio of each of the plurality of formats in the analysis target log, and a step of outputting the appearance ratio , Is provided.
A third aspect of the present invention is a log analysis program, wherein each log included in an analysis target log is changed in a computer by a variable part that can be changed in the analysis target log and in the analysis target log. Including a constant part that does not match, a step of determining which of a plurality of predetermined formats is matched, a step of calculating an appearance ratio of each of the plurality of formats in the analysis target log, and outputting the appearance ratio And the step of performing.
A fourth aspect of the present invention is a log analysis system, wherein a log classification unit that divides each log included in the analysis target log into a plurality of classifications according to a predefined rule, and an appearance ratio of each of the plurality of classifications. A ratio calculation unit for calculating, the appearance ratio is divided for each time section, based on the appearance ratio in the time section, an abnormality determination unit for determining whether or not an abnormality has occurred in the time section, An output unit that outputs a determination result as to whether or not an abnormality has occurred in the time period.
A fifth aspect of the present invention is a log analysis method, which is a step of dividing each log included in the analysis target log into a plurality of classifications according to a predefined rule, and calculating an appearance ratio of each of the plurality of classifications. And a step of dividing the appearance ratio for each time section, determining whether an abnormality has occurred in the time section based on the appearance rate in the time section, and And a step of outputting a determination result of whether or not it has occurred.
A sixth aspect of the present invention is a log analysis program, comprising a step of causing a computer to divide each log included in an analysis target log into a plurality of classifications according to a predefined rule, and the appearance of each of the plurality of classifications. A step of calculating a ratio, a step of dividing the appearance ratio for each time section, and determining whether or not an abnormality has occurred in the time section based on the appearance ratio in the time section; And a step of outputting a determination result as to whether or not an abnormality has occurred.

According to the present invention, it is possible to calculate and output the appearance ratios of a plurality of formats or categories from the analysis target log even when the cause of the abnormality is unknown.

It is a block diagram of a log analysis system according to the first embodiment. FIG. 3 is a schematic diagram of an analysis target log according to the first embodiment. It is a schematic diagram of a format according to the first embodiment. It is a schematic diagram of the state definition information which concerns on 1st Embodiment. It is a schematic diagram of a state determination result according to the first embodiment. It is a schematic diagram of the screen which displays the state determination result which concerns on 1st Embodiment. It is a schematic diagram of the screen which displays the state determination result which concerns on 1st Embodiment. It is a schematic block diagram of the log analysis system which concerns on 1st Embodiment. It is a figure which shows the flowchart of the log analysis method which uses the log analysis system which concerns on 1st Embodiment. It is a schematic diagram of the state definition information which concerns on 2nd Embodiment. It is a schematic diagram of a state determination result according to the second embodiment. It is a schematic diagram of the screen which displays the state determination result which concerns on 2nd Embodiment. It is a block diagram of the log analysis system which concerns on 3rd Embodiment. It is a schematic diagram of the screen which displays the output content which concerns on 3rd Embodiment. It is a block diagram of the log analysis system which concerns on 4th Embodiment. It is a schematic diagram which shows the display screen of the exemplary abnormality determination result in 4th Embodiment. It is a block diagram of the log analysis system which concerns on 6th Embodiment. It is a block diagram of a log analysis system according to each embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the embodiments. In the drawings described below, components having the same function are designated by the same reference numeral, and repeated description thereof may be omitted.

(First embodiment)
FIG. 1 is a block diagram of a log analysis system 100 according to this embodiment. In FIG. 1, arrows indicate main data flows, and there may be data flows other than those shown in FIG. In FIG. 1, each block does not show a hardware (device) unit configuration but a functional unit configuration. As such, the blocks shown in FIG. 1 may be implemented in a single device or may be implemented separately in multiple devices. Data transfer between blocks may be performed via any means such as a data bus, a network, and a portable storage medium.

The log analysis system 100 includes a log input unit 110, a format determination unit 120, a log ratio calculation unit 130, a state determination unit 140, and an output unit 150 as processing units. The log analysis system 100 also includes a format storage unit 161 and a state storage unit 162 as storage units.

The log input unit 110 acquires the analysis target log 10 in the analysis target period and inputs it to the log analysis system 100. The analysis target log 10 may be acquired from outside the log analysis system 100, or may be acquired by reading a pre-recorded inside the log analysis system 100. The analysis target log 10 includes one or more logs output from one or more devices or programs. The analysis target log 10 is a log represented in an arbitrary data format (file format), and may be, for example, binary data or text data. The analysis target log 10 may be recorded as a database table or a text file.

FIG. 2A is a schematic diagram of an exemplary analysis target log 10. The analysis target log 10 in the present embodiment includes one log output from a device or a program as one unit, and includes an arbitrary number of one or more logs. One log may be a one-line character string or a plurality of line character strings. That is, the analysis target log 10 refers to the total of the logs included in the analysis target log 10, and the log refers to one log extracted from the analysis target log 10. Each log includes a time stamp, a message and the like. The log analysis system 100 is not limited to a specific type of log, and can analyze a wide variety of logs. For example, a log that records a message output from an operating system or an application such as a syslog or an event log can be used as the analysis target log 10. Further, a log of a security device on the network such as an IDS (Intrusion Detection System) or an IPS (Intrusion Prevention System) can also be used as the analysis target log 10.

The format determination unit 120 is a format determination unit, determines which of the formats recorded in the format storage unit 161 in advance is suitable for each log included in the analysis target log 10, and determines the compatible format. It is used to separate each log into a variable part and a constant part. The format is a known log format. The variable part is the part that can change in the format, and the constant part is the part that does not change in the log format. The value of the variable part in the input log (including numerical values, character strings and other data) is called the variable value.

FIG. 2B is a schematic diagram of an exemplary format recorded in the format storage unit 161. The format includes a string that represents the format associated with the unique ID. The format defines a variable part by describing a predetermined identifier in a variable part in the log, and defines a part other than the variable part in the log as a constant part. As an identifier of the variable part, for example, “<variable:timestamp>” indicates a variable part indicating a time stamp, “<variable:character string>” indicates a variable part indicating an arbitrary character string, and “<variable:numerical value”. >" indicates a variable part that represents an arbitrary numerical value, and "<variable:IP>" indicates a variable part that represents an arbitrary IP address. The identifier of the variable part is not limited to these, and may be defined by an arbitrary method such as a regular expression or a list of possible values. Also, the format may be composed of only the constant part without including the variable part, or may be composed of only the variable part without including the constant part.

For example, the format determination unit 120 determines that the log on the third line in FIG. 2A conforms to the format with the ID 223 in FIG. 2B. Then, the format determination unit 120 processes the log based on the determined format, and the time stamp is “2015/08/17 08:29:59”, the character string is “SV003”, and the IP address. "192.168.1.23" is determined as the variable value.

In FIG. 2B, the format is represented by a list of character strings for visibility, but it may be represented by any data format (file format), for example, binary data or text data. The format may be recorded as a text file in the format storage unit 161 or may be recorded as a database table in the format storage unit 161.

The log ratio calculation unit 130 calculates the time-series appearance frequency (appearance number) of each format for the analysis target log 10 whose format is determined by the format determination unit 120. Specifically, the log ratio calculation unit 130 counts the number of times each format appears in the analysis target log 10 for each predetermined time width (for example, 30 seconds). As a result, the log ratio calculation unit 130 acquires time-series information of the appearance frequency of the format of the analysis target log 10 in the entire period.

For example, the log ratio calculation unit 130 calculates the appearance frequency that the log with the format ID 039 appears once and the log with the format ID 223 appears twice in the analysis target log 10 for 30 seconds from a certain time. To do. In the present embodiment, the entire period of the analysis target log 10 is set as the analysis target, but a period designated by the user or a period preset in the system may be set as the analysis target.

Next, the log ratio calculation unit 130 divides the entire period of the analysis target log 10 into time intervals. The time section is a time period equal to or longer than the time width used for calculating the appearance frequency of the format. The time section may be set by a predetermined time (for example, 30 minutes).

Alternatively, the amount of change in the appearance ratio of the format when transitioning from one time (or a time zone divided by a predetermined time width) to the next time (or a time zone that has passed a predetermined time width from the previous time zone) The timing at which is greater than or equal to a predetermined threshold may be set as a break in the time interval. The amount of change in the appearance ratio of the format can be calculated using the distance between the feature vectors described later. For example, when the change amount of the appearance ratio of the format between the log from 8:00 to 8:5 and the log from 8:05 to 8:15 is equal to or more than a predetermined threshold value, the time from 8:00 to 8:00 The period of 5 minutes is set as the first time period, and the period of 8:5 to 8:15 is set as the second time period. In this specification, for simplification of the expression of time, the expression “first time to second time” includes the first time and does not include the second time (that is, the second time). It is before the time of 2). By setting the time interval by such a method, it is possible to determine the state for each period divided according to the change in the appearance ratio of the format.

Next, the log ratio calculation unit 130 calculates the appearance frequency of the format for each time section from the time series information of the appearance frequency of the format. Specifically, the appearance frequencies of the formats acquired for each predetermined time width (for example, 30 seconds) are totaled for each set time interval (for example, 30 minutes). Then, the log ratio calculation unit 130 calculates the appearance ratio of each format by dividing the appearance frequency of each format in each time interval by the total value of the appearance frequencies of all formats in the time interval.

The state determination unit 140 determines the state for each time section using the appearance ratio of the format calculated by the log ratio calculation unit 130. The state definition information is information that defines a state based on the appearance ratio of the format. Specifically, the state determination unit 140 determines, for each time interval, which state is recorded in advance in the state storage unit 162 based on the appearance ratio of the format calculated by the log ratio calculation unit 130. judge.

FIG. 3A is a schematic diagram of exemplary state definition information recorded in the state storage unit 162. The state definition information includes a list of state names and appearance frequencies associated with a state ID that is a unique identifier of a state. The state name is a character string that displays the name of each state to the user, and the same state name may be attached to a plurality of state IDs. The appearance frequency list is a list in which the format and the appearance frequency are associated and recorded. For example, {ID=2, V=20} indicates that a log whose format ID is 2 appears 20 times. The state definition information according to the present embodiment includes the appearance frequency, but may include a list of appearance ratios converted from the number of appearances.

In addition, the state definition information records the start time and end time of the time interval detected in the past as a past history for each state. As the past history, one time of the previous time may be recorded, or a predetermined number of times may be recorded retroactively from the previous time.

In FIG. 3A, the state definition information is represented as a character string for visibility, but may be represented in an arbitrary data format (file format), for example, binary data or text data. The state definition information is not limited to the specific definition shown here, and may be defined by any method. Also, the state definition information may be recorded separately in a plurality of files or tables.

The state determination unit 140 generates a feature vector (referred to as a target feature vector) in which the appearance ratios of the formats calculated by the log ratio calculation unit 130 are arranged for each time section. For example, when the appearance ratio of the format ID 1 is 0.7, the appearance ratio of the format ID 2 is 0.3, and no other format appears, (0.7, 0.3, 0 , 0,...) (the number of dimensions of the feature vector is equal to the number of all formats).

The state determination unit 140 also reads out the state definition information illustrated in FIG. 3A from the state storage unit 162. The state determination unit 140 calculates the appearance ratio of each format by dividing the number of appearances of each format in each state by the total value of the number of appearances of all formats in the state. The state determination unit 140 generates a feature vector (referred to as a reference feature vector) in which the appearance ratio of each format is arranged for each state.

Then, the state determination unit 140 calculates the distance between the target feature vector generated from each time interval of the analysis target log 10 and the reference feature vector generated from each state previously recorded in the state storage unit 162. It is possible to determine that the smaller the distance is, the more similar the time section of the analysis target log 10 is to the previously recorded state. Then, the state determination unit 140, when the distance calculated between the target feature vector of a certain time section and the reference feature vector of a certain state is equal to or less than a predetermined threshold, the time section corresponds to the state (similar). Then determine. The threshold value of the distance may be arbitrarily determined by experiment or simulation. The state determination unit 140 may determine that the time segment does not correspond to any of the recorded states when there is no state within a predetermined threshold for one time segment. The state determination unit 140 may determine that the time section corresponds to the state with the smallest distance when there are a plurality of states in which the distance is equal to or less than the predetermined threshold value for one time section. The state determination unit 140 repeats the state determination for all the time intervals and outputs it as the state determination result.

The state determination unit 140 may determine the state for each cluster after clustering the time intervals by a known clustering method. In this case, the state determination unit 140 determines a target feature vector representing a plurality of time intervals classified into the same cluster, and calculates the distance between the target vector and the reference feature vector of each state. By doing so, the state of the cluster is determined.
Further, the state determination unit 140 according to the present embodiment compares the appearance ratio of each format in the time interval with the appearance ratio of each format indicating the past state recorded in the state storage unit 162 without using the feature vector. However, it is also possible to judge the completely matched one as the state of the time.

The state determination unit 140 according to the present embodiment uses the appearance ratio of the log format for state determination, but may use other information related to the analysis target log 10. For example, the state determination unit 140 may determine the state based on the number of types of formats (that is, the number of different format IDs) for each time period. For example, a time section including only one type of format and a time section including 10 types of formats can be determined as different states. Further, for example, the state determination unit 140 may determine the state based on the amount of the analysis target log 10 for each time section. Further, for example, the state determination unit 140 may determine the state based on at least one of the time and the day of the week for each time section. For example, the state determination unit 140 may determine the state based on performance information (CPU usage rate or the like) of the system that outputs the analysis target log 10 for each time section. By using such information in combination with the appearance ratio of the format, it is possible to improve the accuracy of state determination. Alternatively, the state for each time section can be determined by using such information instead of the appearance ratio of the format.

FIG. 3B is a schematic diagram showing an exemplary state determination result by the state determination unit 140. The state determination result includes a range for each time section and a state ID and a state name determined for the time section. The state determination result shown in FIG. 3B is an example, and may be recorded in any format. In FIG. 3B, the state determination result is represented by a list of character strings for visibility, but may be represented in any data format (file format), for example, binary data or text data. Further, the status determination result may be recorded separately in a plurality of files or tables.

The output unit 150 outputs the state determination result by the state determination unit 140. In the present embodiment, the output unit 150 outputs the state determination result to the display device 20, and the display device 20 displays the state determination result as an image toward the user. The display device 20 includes a display unit such as a liquid crystal display for displaying an image or a CRT (Cathode Ray Tube) display.

4A and 4B are schematic diagrams showing exemplary display screens of the state determination results using the display device 20. The screen A shown in FIG. 4A displays a total graph A1 showing the total amount of log generation and an individual graph A2 showing the amount generated for each format. The horizontal axis of each of the graphs A1 and A2 represents time, and the vertical axis represents the appearance frequency (generation amount).

The total graph A1 shows the time change of the generation amount of the analysis target log 10. The individual graph A2 shows the time change of the appearance ratio for each format calculated by the log ratio calculation unit 130. The individual graph A2 displays graphs of values obtained by multiplying the generation amount of the analysis target log 10 by the appearance ratio for each format in a layered manner. The individual graph A2 may display only the formats having the appearance ratio equal to or higher than the predetermined threshold value, or only the predetermined number of formats in the descending order of the appearance ratio.

Different colors, patterns or combinations thereof are given to the individual graph A2 for each format, and the formats indicated by each individual graph A2 can be distinguished. A character string indicating the format ID of the format shown in the individual graph A2 may be displayed so as to overlap with or near the individual graph A2. Further, the screen A displays the state name A3 of the state for each time section determined by the state determination unit 140, overlapping the graphs A1 and A2 or in the vicinity thereof. This allows the user to easily identify the state of each time period.

When the user selects any one of the time intervals by performing an operation (that is, an operation from the outside) using an input device such as a mouse or a touch panel, the screen A displays a log corresponding to the time interval as shown in FIG. 4B. Display A4. The log analysis system 100 (output unit 150) not only displays the log of the time section selected by the user operation on the display device 20 but also outputs the log by an arbitrary method such as recording or printing. Good.

The screens shown in FIGS. 4A and 4B are merely examples, and any display method can be used as long as the information including the appearance ratio calculated by the log ratio calculation unit 130 and the state determination result by the state determination unit 140 can be visually displayed to the user. May be used. Further, the method of outputting information by the log analysis system 100 (output unit 150) is not limited to the image display for the user. For example, the output unit 150 may output information to be output as data, and the log analysis system 100 or another system may perform recording processing, printing processing, analysis processing, statistical processing, etc. on the data from the output unit 150. ..

By referring to the screens of FIGS. 4A and 4B, the user can know the time-series change in the appearance ratio of each format in the analysis target log 10 and the state determined based on the appearance ratio of the format. For example, even if the log generation amounts are the same, it can be determined that some kind of abnormality has occurred in the time intervals determined to be in different states because the logs of different types are included. Further, even if a log different from the normal one is generated in the time section determined to be in the test state, it can be determined that the log is not abnormal. The determination of the presence or absence of such an abnormality may be made by the user or automatically by the log analysis system 100.

FIG. 5 is a schematic configuration diagram showing an exemplary device configuration of the log analysis system 100 according to the present embodiment. The log analysis system 100 includes a CPU (Central Processing Unit) 101, a memory 102, a storage device 103, and a communication interface 104. The log analysis system 100 may be connected to the display device 20 via the communication interface 104, or may include the display device 20. The log analysis system 100 may be an independent device, or may be configured integrally with another device.

The communication interface 104 is a communication unit that transmits and receives data, and is configured to be able to execute at least one communication method of wired communication and wireless communication. The communication interface 104 includes a processor, an electric circuit, an antenna, a connection terminal and the like required for the communication method. The communication interface 104 is connected to a network using the communication method according to a signal from the CPU 101, and performs communication. The communication interface 104 receives, for example, the analysis target log 10 from the outside.

The storage device 103 stores a program executed by the log analysis system 100, data of processing results by the program, and the like. The storage device 103 includes a read-only ROM (Read Only Memory), a readable/writable hard disk drive, a flash memory, or the like. The storage device 103 may also include a computer-readable portable storage medium such as a CD-ROM. The memory 102 includes a RAM (Random Access Memory) for temporarily storing data being processed by the CPU 101, a program read from the storage device 103, and data.

The CPU 101 temporarily records temporary data used for processing in the memory 102, reads a program recorded in the storage device 103, and performs various calculations, control, determination, etc. on the temporary data according to the program. It is a processor as a processing unit that executes the processing operation of. Further, the CPU 101 records the processing result data in the storage device 103, and also transmits the processing result data to the outside via the communication interface 104.

In the present embodiment, the CPU 101 functions as the log input unit 110, the format determination unit 120, the log ratio calculation unit 130, the state determination unit 140, and the output unit 150 of FIG. 1 by executing the program recorded in the storage device 103. To do. Further, in the present embodiment, the storage device 103 functions as the format storage unit 161 and the state storage unit 162 of FIG.

The log analysis system 100 is not limited to the specific configuration shown in FIG. The log analysis system 100 is not limited to one device, and may be configured by connecting two or more physically separated devices by wire or wirelessly. Each unit included in the log analysis system 100 may be realized by an electric circuit configuration. Here, the electric circuit configuration is a wording that conceptually includes a single device, a plurality of devices, a chipset, or a cloud.

Further, at least a part of the log analysis system 100 may be provided in the SaaS (Software as a Service) format. That is, at least a part of the functions for realizing the log analysis system 100 may be executed by software executed via the network.

FIG. 6 is a diagram showing a flowchart of a log analysis method using the log analysis system 100 according to this embodiment. First, the log input unit 110 acquires the analysis target log 10 and inputs it to the log analysis system 100 (step S101). The format determination unit 120 determines whether or not one of the logs included in the analysis target log 10 input in step S101 is a determination target and conforms to any of the formats recorded in the format storage unit 161 (step). S102).

If the log of the determination target does not match any of the formats recorded in the format storage unit 161 in step S102 (NO in step S103), the next one log of the analysis target log 10 is set as the determination target in steps S102-. S103 is repeated.

If the log to be judged matches the format recorded in the format storage unit 161 in step S102 (YES in step S103), the format judging unit 120 uses the format to change the log to be judged to a variable. It is separated into a part and a constant part (step S104). The format determination unit 120 records the variable value in the determination target log.

When the analysis is not completed for all the logs in the analysis target log 10 (NO in step S105), steps S102 to S105 are repeated with the next one log of the analysis target log 10 as the determination target.

When the analysis is completed for all the logs in the analysis target log 10 (YES in step S105), the log ratio calculation unit 130 causes the analysis target log 10 to have a predetermined time width (for example, 30 seconds). The number of times each format has appeared (frequency of appearance) is counted (step S106). The time when the log is output is obtained from the variable value of the time stamp in the log. The log ratio calculation unit 130 calculates the time-series appearance frequency of the format for the entire period of the analysis target log 10.

Next, the log ratio calculation unit 130 divides the entire period of the analysis target log 10 into time sections (step S107). The time period may be set by dividing the entire period of the analysis target log 10 into each predetermined time (for example, 30 minutes), or the entire period of the analysis target log 10 may be changed by the amount of change in the appearance ratio of the format. It may be set by dividing every timing that is equal to or more than a predetermined threshold value.

The log ratio calculation unit 130 calculates the appearance ratio of each format for each time interval from the time-series appearance frequency of the format calculated in step S106 (step S108). The appearance ratio of the format is calculated by dividing the total value of the appearance frequencies in one time section of the format by the total value of the appearance frequencies of all the formats in the time section.

Next, the state determination unit 140 calculates the appearance ratio of the format for each state from the state definition information read from the state storage unit 162. The state determination unit 140 generates a target feature vector from the appearance ratio of each format for each time section, and generates a reference feature vector from the appearance ratio of each format for each state. The state determination unit 140 calculates the distance between the target feature vector of each time section and the reference feature vector of each state. Then, the state determination unit 140 determines that, for each time section, a state having a distance within a predetermined threshold corresponds to (similar to) the state of the time section (step S109). When there is no state within a predetermined threshold for one time section, the state determination unit 140 determines that the time section does not correspond to any recorded state. When there are a plurality of states in which the distance is within the predetermined threshold for one time section, the state determination unit 140 determines that the time section corresponds to the state in which the distance is the smallest.

Finally, the output unit 150 outputs the appearance ratio calculated in step S108 and the state determination result acquired in step S109 to the display device 20 and displays the result for the user (step S110).

A general system includes a plurality of devices and programs, and these devices and programs output many kinds of logs. For example, when an abnormality occurs, the generation amount of logs of one format has hardly changed, but the appearance ratio of logs of a plurality of formats may change significantly. However, since the technique of Patent Document 1 focuses only on the change in the generation amount of one template, such a change in the appearance ratio of logs in a plurality of formats cannot be detected. Further, it is possible that the amount of generated logs in many formats increases all at once when the system is started, but with the technique of Patent Document 1, such an increase in the amount of generated logs may be regarded as abnormal. There is a nature. On the other hand, the log analysis system 100 according to the present embodiment calculates the appearance ratio of logs in a plurality of formats in the analysis target log 10 and outputs the result of determining the state based on the appearance ratio. When an abnormality occurs in the system that outputs the analysis target log 10, the appearance ratio changes, so that the user can infer the occurrence of the abnormality and the cause thereof based on the information output from the log analysis system 100. The user can also determine the similarity between the time sections in the analysis target log 10 based on the output ratio of the output format and the state of the time section.

(Second embodiment)
In the first embodiment, the state is determined based on the appearance ratio of the format, but in the present embodiment, the state is determined based on the performance information of the device that outputs the log in addition to the appearance ratio of the format. This embodiment uses the same configuration as the log analysis system 100 of FIG.

FIG. 7A is a schematic diagram of exemplary state definition information recorded in the state storage unit 162 in this embodiment. The state definition information of FIG. 7A includes the range of the CPU usage rate as performance information in addition to the information of FIG. 3A. In this embodiment, different ranges of CPU usage rates are associated with the same appearance frequency list. Therefore, the state determination unit 140 determines a unique state ID based on the combination of the format appearance rate and the CPU usage rate.

The state determination unit 140 calculates the distance between the target feature vector generated from each time interval and the reference feature vector generated from each state previously recorded in the state storage unit 162, as in the first embodiment. To do. In addition, the state determination unit 140 acquires the average CPU usage rate in each time section from the performance information storage unit in which the performance information of the device that outputs the analysis target log 10 is recorded in advance. Then, the state determination unit 140 determines that the distance calculated between the target feature vector in a certain time section and the reference feature vector in a certain state is less than or equal to a predetermined threshold, and the average CPU usage rate in the time section is in the state. When it is included in the range of the CPU usage rate of, the time section is determined to correspond (similarly) to the state.

FIG. 7B is a schematic diagram showing an exemplary state determination result by the state determination unit 140 in the present embodiment. The state determination result of FIG. 7B includes an average CPU usage rate as performance information in addition to the information of FIG. 3B.

FIG. 8 is a schematic diagram showing an exemplary state determination result display screen in the present embodiment. The screen B shown in FIG. 8 displays the same name as the screen A shown in FIG. 4A, as well as the status name B1 of the status determined in the present embodiment. As can be seen from the screen B, in the present embodiment, different state names are assigned based on the difference in performance information even in time intervals having the same format appearance ratio. The screen B may display the average CPU usage rate for each time section.

As described above, in the present embodiment, the state to be determined changes depending on not only the appearance ratio of the format but also the performance state, and thus the state can be defined more finely. The performance information used for the state determination in the present embodiment is not limited to the CPU usage rate, and any information indicating the performance of the device that outputs the log, such as the memory usage rate and the disk read/write amount, may be used. Also, a plurality of performance information may be used in combination.

(Third Embodiment)
In the first embodiment, the state determined for each time interval is output together with the appearance ratio of the format, but the present embodiment outputs only the appearance ratio of the format.

FIG. 9 is a block diagram of the log analysis system 200 according to this embodiment. The log analysis system 200 includes a log input unit 110, a format determination unit 120, a log ratio calculation unit 130, and an output unit 150 as processing units. The log analysis system 200 also includes a format storage unit 161 as a storage unit. That is, the log analysis system 200 according to the present embodiment is obtained by omitting the state determination unit 140 and the state storage unit 162 from the log analysis system 100 according to the first embodiment.

FIG. 10 is a schematic diagram showing a display screen displaying exemplary output contents in the present embodiment. Similar to the screen A shown in FIG. 4A, the screen C shown in FIG. 10 displays a total graph C1 showing the total amount of log generation and an individual graph C2 showing the amount generated for each format. On the other hand, unlike the screen A, the screen C does not display the state name.

As described above, in this embodiment, only the appearance ratio of the format is output without determining the state. The user can infer that some kind of abnormality has occurred when the appearance ratios of the plurality of formats are different from the expected ones. Therefore, also in the present embodiment, it is possible to provide the user with information that suggests a log that may be the cause of the abnormality.

(Fourth Embodiment)
In the first embodiment, the state is determined for each time section based on the appearance ratio of the format, but in the present embodiment, the presence or absence of abnormality is determined based on the appearance ratio of the format.

FIG. 11 is a block diagram of the log analysis system 300 according to this embodiment. The log analysis system 300 includes a log input unit 110, a format determination unit 120, a log ratio calculation unit 130, an abnormality determination unit 340, and an output unit 150 as processing units. The log analysis system 300 also includes a format storage unit 161 and a reference storage unit 362 as storage units. That is, the log analysis system 300 according to the present embodiment includes an abnormality determination unit 340 and a reference storage unit 362 instead of the state determination unit 140 and the state storage unit 162 in the log analysis system 100 according to the first embodiment.

First, the log ratio calculation unit 130 divides the entire period of the analysis target log 10 into time intervals. In the present embodiment, the time period is set based on the format determined by the format determination unit 120. Specifically, the log ratio calculation unit 130 sets one continuous period in which a log in a specific format associated with the behavior of a component (server, virtual machine, etc.) that outputs the analysis target log 10 appears. Set as a time section. Then, the log ratio calculation unit 130 sets each continuous period that is not set by the format as one undefined time section. For example, immediately after the component is activated, the log corresponding to the format ID 039 in FIG. 2B is output. Therefore, the log ratio calculation unit 130 sets a continuous period in which a log corresponding to the format ID of 039 appears as a time section related to the system startup state, and sets other periods as undefined time sections. .. With such a configuration, in the present embodiment, it is possible to set the time interval based on the behavior of the constituent element and determine the abnormality.
Alternatively, the log ratio calculation unit may set consecutive time periods in which logs in a plurality of predetermined formats appear as one time section. For example, when the format ID 039 and the format ID 040 are defined as the predetermined format related to the system startup state, the continuous period including the log of the format is set as one time section related to the system startup state. ..

Similar to the first embodiment, the log ratio calculation unit 130 may set the time interval by a predetermined time (for example, 30 minutes), or set the time interval based on the change in the appearance ratio of the format. You may

The abnormality determination unit 340 determines the abnormality in the time-series information of the appearance frequency of the format acquired by the log ratio calculation unit 130, based on the reference information recorded in the reference storage unit 362. The reference information is information defining a reference for determining whether or not there is an abnormality based on the appearance ratio of the format. Similar to the state definition information shown in FIG. 3A, the reference information includes a list of appearance frequencies of a format that serves as a reference for abnormality determination.

The abnormality determination unit 340 generates a feature vector (target feature vector) in which the appearance ratios of the formats calculated by the log ratio calculation unit 130 are arranged. In addition, the abnormality determination unit 340 generates a feature vector (referred to as a reference feature vector) in which the appearance ratios of the respective formats that are pre-recorded in the reference storage unit 362 and are used as a reference for abnormality determination are arranged. The method of generating the target feature vector and the reference feature vector is the same as in the first embodiment.

Then, the abnormality determination unit 340 calculates the distance between the target feature vector generated from each time interval and the reference feature vector generated from the abnormality determination reference previously recorded in the reference storage unit 362. This distance is an index indicating how far from the reference. When the distance calculated between the target feature vector in a certain time section and the reference feature vector that is the reference for the abnormality determination is equal to or greater than a predetermined threshold, the abnormality determining unit 340 has an abnormality in the time section. Determine that. The threshold value of the distance may be arbitrarily determined by experiment or simulation. The abnormality determination unit 340 repeats the abnormality determination for all the time intervals and outputs it as the abnormality determination result.

FIG. 12 is a schematic diagram showing a display screen of an exemplary abnormality determination result in this embodiment. Similar to the screen A shown in FIG. 4A, the screen D shown in FIG. 12 displays a total graph D1 showing the total amount of log generation and an individual graph D2 showing the amount generated for each format. Further, the screen D displays a frame D3 indicating a time period in which the abnormality determination unit 340 has determined that an abnormality has occurred, and a message D4 indicating that the abnormality has occurred, so that the user is notified of an abnormality. To notify. The notification of the occurrence of the abnormality is not limited to these methods, and may be performed by changing the color or pattern of the graph in the time section.

As described above, in the present embodiment, instead of the determination of the state in the first embodiment, the occurrence of abnormality in each time section is determined and output based on the appearance ratio of the format. According to this embodiment, it is possible to directly provide the user with information indicating the occurrence of an abnormality.

The criteria for abnormality determination may not match the actual situation when time has passed from the set time or when the devices making up the system are changed. Therefore, the log analysis system 300 may change the abnormality determination reference based on the logs accumulated in the past. For example, the log analysis system 300 may use the average value of the appearance ratios of the respective formats in the logs accumulated within a predetermined period as a criterion for the abnormality determination after the change. Alternatively, the log analysis system 300 may use the value input by the user as a criterion for the abnormality determination after the change. As the update timing of the abnormality determination reference, the log analysis system 300 may set the changed abnormality determination reference when, for example, a predetermined time has elapsed from the day when the abnormality determination reference was set last time. In addition, the log analysis system 300 may set a criterion for the abnormality determination after the change when the change of the device that detects the change of the device that configures the system that outputs the analysis target log 10 is replaced. Further, the log analysis system 300 may set the changed abnormality determination standard triggered by the user's instruction.

(Fifth Embodiment)
In each of the above-described embodiments, the log appearance ratio is calculated based on the format that separates the variable part and the constant part, but the appearance ratio calculation method is not limited to this. Any method can be used as long as the analysis target log 10 is divided into a plurality of classifications according to a predefined rule and the appearance ratio of each of the plurality of classifications for each time section can be calculated.

For example, clustering can be performed based on the similarity between logs, and each cluster can be classified into one classification. In this case, the appearance frequency and the appearance ratio in each time segment of the cluster as the classification may be calculated.

Specifically, in the log analysis system 300 of FIG. 11, the format determination unit 120 is a log classification unit and divides the analysis target log 10 into a plurality of classifications according to a predetermined rule. The log ratio calculation unit 130 calculates the appearance ratio based on the classification by the format determination unit 120 as the log classification unit. Then, the abnormality determination unit 340 determines whether or not an abnormality has occurred for each time section based on the appearance ratio of the classification for each time section.

As described above, each embodiment of the present invention is not limited to the definition of a specific format, the logs are classified according to a predefined rule, and the appearance ratio of each time section of the classification is calculated to calculate the appearance ratio. Output, state determination, abnormality determination, etc. can be performed.

(Sixth Embodiment)
In this embodiment, a learning unit for learning the format and the state is provided. FIG. 13 is a block diagram of the log analysis system 400 according to this embodiment. The log analysis system 400 includes a format learning unit 471 and a state learning unit 472 in addition to the configuration of FIG.

When the format determination unit 120 determines the format, the format learning unit 471 creates a new format and formats it if the log to be determined does not match any of the formats recorded in the format storage unit 161. It is recorded in the storage unit 161.

As a first method for the format learning unit 471 to learn the format, the format learning unit 471 accumulates a plurality of logs of unknown formats, and a variable part that statistically changes with respect to them and a constant that does not change. It can be defined as a new format by separating it from the part. As a second method for the format learning unit 471 to learn the format, the format learning unit 471 reads a list of known variable values and matches or is similar to known variable values in a log whose format is unknown. A new format can be defined by determining the part to be changed as the variable part and the other part as the constant part. The value itself may be used as the known variable value, or a pattern such as a regular expression may be used. The learning method of the format is not limited to these, and an arbitrary learning algorithm capable of defining a new format for the input log may be used.

When the state determination unit 140 determines the state, the state learning unit 472 determines whether the time interval to be determined does not correspond (similarly) to any of the states recorded in the state storage unit 162. Such information is recorded in the state storage unit 162 as a new state. Specifically, the state learning unit 472 records, in the state storage unit 162, a list of the frequency of formats included in the time period that does not correspond to any state, together with the newly generated state ID and state name. The state ID and the state name may be automatically generated based on a predetermined rule (date and time, serial number, etc.), or may be input by the user via an input device such as a keyboard.

Since the log analysis system 400 according to this embodiment includes the format and state learning unit, it is possible to newly generate and record a format or state from a log of an unknown format or state.

(Other embodiments)
FIG. 14 is a schematic configuration diagram of the log analysis system 100, 200, 300, 400 according to each of the above-described embodiments. FIG. 14 shows a configuration example in which the log analysis system 100, 200, 300, 400 functions as a device that outputs the appearance ratio of the format in the log. In the log analysis systems 100, 200, 300, and 400, each log included in the analysis target log includes a variable part that can change in the analysis target log and a constant part that does not change in the analysis target log. A format determination unit 120 as a format determination unit that determines which of a plurality of predetermined formats matches, and a log ratio as a ratio calculation unit that calculates the appearance ratio of each of the plurality of formats in the analysis target log. A calculation unit 130 and an output unit 150 that outputs the appearance ratio are provided.

The present invention is not limited to the above-described embodiments, and can be appropriately modified without departing from the spirit of the present invention.

A program for operating the configuration of the embodiment so as to realize the functions of the above-described embodiment (more specifically, a program for causing a computer to execute the process illustrated in FIG. 6) is recorded on the recording medium, and the recording medium is recorded on the recording medium. A processing method of reading the recorded program as a code and executing it in a computer is also included in the category of each embodiment. That is, a computer-readable recording medium is also included in the scope of each embodiment. Further, not only the recording medium in which the above program is recorded but also the program itself is included in each embodiment.

As the recording medium, for example, a floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, CD-ROM, magnetic tape, non-volatile memory card, ROM can be used. Further, the embodiment is not limited to the one executing the process by the program recorded in the recording medium, and the one executing the process by operating on the OS in cooperation with other software and the function of the expansion board. It is included in the category of.

The whole or part of the exemplary embodiments disclosed above can be described as, but not limited to, the following supplementary notes.

(Appendix 1)
Each log included in the analysis target log matches any of a plurality of predetermined formats including a variable part that can change in the analysis target log and a constant part that does not change in the analysis target log. A format determination unit for determining whether
A ratio calculation unit that calculates the respective appearance ratios of the plurality of formats in the analysis target log;
An output unit for outputting the appearance ratio,
A log analysis system equipped with.

(Appendix 2)
The appearance determination unit further comprising a state determination unit that divides the appearance ratio for each time section and determines which state the time section corresponds to based on the appearance ratio in the time section. Log analysis system.

(Appendix 3)
The log analysis system according to Note 2, wherein the time period is determined based on a predetermined time.

(Appendix 4)
The log analysis system according to Note 2, wherein the time period is determined based on a change in the appearance ratio.

(Appendix 5)
Any one of the states in which the time interval is predetermined based on whether or not the appearance ratio calculated from the analysis target log is similar to the appearance ratio associated with the state. 5. The log analysis system according to any one of appendices 2 to 4, which determines whether or not the above applies.

(Appendix 6)
The state determination unit calculates a feature vector indicating the appearance ratio calculated from the analysis target log and a feature vector indicating the appearance ratio associated with the state, and based on the distance between the feature vectors. The log analysis system according to appendix 5, wherein the log analysis system determines which of the predetermined states the time interval corresponds to.

(Appendix 7)
The log analysis according to Appendix 1, further comprising: an abnormality determination unit that divides the appearance ratio for each time section and determines whether or not an abnormality has occurred in the time section based on the appearance ratio in the time section. system.

(Appendix 8)
The log analysis system according to appendix 7, wherein the time period is determined based on a period in which a predetermined format among the plurality of formats appears.

(Appendix 9)
9. The log analysis system according to attachment 8, wherein the predetermined format is a format that is associated in advance with the behavior of a component that outputs the analysis target log.

(Appendix 10)
Note that the abnormality determination unit determines whether or not an abnormality has occurred in the time period based on the appearance ratio calculated from the analysis target log and the appearance ratio associated with a predetermined criterion. The log analysis system according to any one of 7 to 9.

(Appendix 11)
The abnormality determination unit calculates a feature vector indicating the appearance ratio calculated from the analysis target log and a feature vector indicating the appearance ratio associated with the predetermined criterion, and a distance between the feature vectors. 11. The log analysis system according to appendix 10, which determines whether or not an abnormality has occurred in the time section based on.

(Appendix 12)
12. The log analysis system according to any one of appendices 1 to 11, wherein the output unit outputs a log corresponding to the appearance ratio selected by an external operation.

(Appendix 13)
Each log included in the analysis target log matches any of a plurality of predetermined formats including a variable part that can change in the analysis target log and a constant part that does not change in the analysis target log. The process of determining whether
Calculating the respective appearance ratios of the plurality of formats in the analysis target log;
Outputting the appearance ratio,
A log analysis method comprising.

(Appendix 14)
On the computer,
Each log included in the analysis target log matches any of a plurality of predetermined formats including a variable part that can change in the analysis target log and a constant part that does not change in the analysis target log. The process of determining whether
Calculating the respective appearance ratios of the plurality of formats in the analysis target log;
Outputting the appearance ratio,
A log analysis program that executes.

(Appendix 15)
A log classification unit that divides each log included in the analysis target log into a plurality of classifications according to a predefined rule,
A ratio calculation unit for calculating the respective appearance ratios of the plurality of classifications,
The appearance ratio is divided for each time section, based on the appearance ratio in the time section, an abnormality determination unit that determines whether or not an abnormality has occurred in the time section,
An output unit that outputs a determination result of whether or not an abnormality has occurred in the time period,
A log analysis system equipped with.

(Appendix 16)
Note that the abnormality determination unit determines whether or not an abnormality has occurred in the time period based on the appearance ratio calculated from the analysis target log and the appearance ratio associated with a predetermined criterion. 15. The log analysis system according to 15.

(Appendix 17)
The abnormality determination unit calculates a feature vector indicating the appearance ratio calculated from the analysis target log and a feature vector indicating the appearance ratio associated with the predetermined criterion, and a distance between the feature vectors. 17. The log analysis system according to appendix 16, which determines whether or not an abnormality has occurred in the time section based on.

(Appendix 18)
A step of dividing each log included in the analysis target log into a plurality of classifications according to a predefined rule,
Calculating the respective appearance ratios of the plurality of classifications,
Dividing the appearance ratio for each time section, and determining whether or not an abnormality has occurred in the time section based on the appearance ratio in the time section,
Outputting a determination result of whether or not an abnormality has occurred in the time period,
A log analysis method comprising.

(Appendix 19)
On the computer,
A step of dividing each log included in the analysis target log into a plurality of classifications according to a predefined rule,
Calculating the respective appearance ratios of the plurality of classifications,
Dividing the appearance ratio for each time section, and determining whether or not an abnormality has occurred in the time section based on the appearance ratio in the time section,
Outputting a determination result of whether or not an abnormality has occurred in the time period,
A log analysis program that executes.

(Appendix 20)
18. The log analysis system according to any one of appendices 10, 11, 16 and 17, which changes the predetermined criterion at a predetermined timing.

(Appendix 21)
A log classification unit that divides each log included in the analysis target log into a plurality of classifications according to a predefined rule,
A ratio calculation unit for calculating the respective appearance ratios of the plurality of classifications,
The appearance ratio is divided for each time section, and based on the appearance ratio in the time section, a state determination unit that determines which of the predetermined states the time section corresponds to,
An output unit for outputting the state,
A log analysis system equipped with.

This application claims the priority on the basis of Japanese application Japanese Patent Application No. 2005-237235 for which it applied on December 4, 2015, and takes in those the indications of all here.

Claims (10)

Each log included in the analysis target log matches any of a plurality of predetermined formats including a variable part that can change in the analysis target log and a constant part that does not change in the analysis target log. A format determination unit for determining whether
A ratio calculation unit that calculates the respective appearance ratios of the plurality of formats in the analysis target log;
An output unit for outputting the appearance ratio,
A log analysis system equipped with. The state determination unit that further divides the appearance ratio for each time section, and determines which of the predetermined states the time section corresponds to based on the appearance ratio in the time section. Log analysis system. Any one of the states in which the time interval is predetermined based on whether the appearance ratio calculated from the analysis target log is similar to the appearance ratio associated with the state. The log analysis system according to claim 2, wherein it is determined whether or not the above applies. The log according to claim 1, further comprising: an abnormality determination unit that divides the appearance ratio for each time section and determines whether or not an abnormality has occurred in the time section based on the appearance ratio in the time section. Analysis system. The abnormality determination unit determines whether or not an abnormality has occurred in the time period based on the appearance ratio calculated from the analysis target log and the appearance ratio associated with a predetermined reference. Item 4. The log analysis system according to item 4. A log analysis method executed by a computer,
Each log included in the analysis target log matches any of a plurality of predetermined formats including a variable part that can change in the analysis target log and a constant part that does not change in the analysis target log. The process of determining whether
Calculating the respective appearance ratios of the plurality of formats in the analysis target log;
Outputting the appearance ratio,
A log analysis method comprising. On the computer,
Each log included in the analysis target log matches any of a plurality of predetermined formats including a variable part that can change in the analysis target log and a constant part that does not change in the analysis target log. The process of determining whether
Calculating the respective appearance ratios of the plurality of formats in the analysis target log;
Outputting the appearance ratio,
A log analysis program that executes. A log classification unit that divides each log included in the analysis target log into a plurality of classifications according to a predefined rule,
A ratio calculation unit for calculating the respective appearance ratios of the plurality of classifications,
The appearance ratio is divided for each time section, based on the appearance ratio in the time section, an abnormality determination unit that determines whether or not an abnormality has occurred in the time section,
An output unit that outputs a determination result of whether or not an abnormality has occurred in the time period,
A log analysis system equipped with. A log analysis method executed by a computer,
A step of dividing each log included in the analysis target log into a plurality of classifications according to a predefined rule,
Calculating the respective appearance ratios of the plurality of classifications,
Dividing the appearance ratio for each time section, and determining whether or not an abnormality has occurred in the time section based on the appearance ratio in the time section,
Outputting a determination result of whether or not an abnormality has occurred in the time period,
A log analysis method comprising. On the computer,
A step of dividing each log included in the analysis target log into a plurality of classifications according to a predefined rule,
Calculating the respective appearance ratios of the plurality of classifications,
Dividing the appearance ratio for each time section, and determining whether or not an abnormality has occurred in the time section based on the appearance ratio in the time section,
Outputting a determination result of whether or not an abnormality has occurred in the time period,
A log analysis program that executes.

Images